Thanks to people who told us that sometimes the game crashes and sent us their crash logs. We are fixing this ASAP.
*Update*
It seems like this problem exists just for players using xbox controllers. Should work fine if you are using vive controllers. We are fixing this for xbox controller users ASAP.