Context-Aware Recommendation Systems has gained lots of attention in both industry and academic research. Factorization Machines (FM) based recommendation has been successfully used in sparse industrial datasets for user personalized video recommendations. FM is a collaborative filtering technique for predicting a target such as user rating, given observations of interaction between some users and items. The model can incorporate any available auxiliary information about the user, the item, or the interaction which serves as context or features of the data. In this paper, we propose a framework to automatically select features on FM-based recommender systems to improve the prediction quality. FM requires the input data as a one-hot encoded feature vector in a binary space domain. We use the values of the FM parameters in the binary space to determine the importance of the context. We consider the density of the important features in the binary space to rank and select the relevant features in the original data. Experiments on multiple datasets have been conducted to validate the efficiency and robustness of our method.