Challenges in building recommendation systems
A recommendation engine can be gem whereas same engine can be nightmare if it can be easily fooled by the people on the system can be manipulated easily.
Here are few basic situations and their solution to prevent this from happening. If you are new to recommendation you should definitely check this post and continue to rest of the article.
Collaborative filtering is the technique of recommending products to users with the help of other similar users. The core idea is that people with similar preferences will like similar type of products. Collaborative filtering is a general term and there are many algorithms that use this concept to recommend products. Latent collaborative filtering is one of the most used collaborative filtering algorithm that performs matrix factorization to recommend most relevant product to users. There is also deep learning approach for collaborative filtering that outperforms most of other traditional techniques. While implementing these algorithms in our application we come up with few problems that are described below:
Cold Start Problem:
How do you deal with new users and products that don’t have any history?
Solution:
Use content-boosted filtering approach. It is combination of content-based filtering and collaborative filtering. You can use product description and attributes as well as user demographic to recommend products to users.
Data Sparsity:
UserItem rating matrix is very sparse( many null items ) because stores have many products and all those products will not be rated by many users. Actually very few people rate products. Think about how many times have you rated products after you bought products online? This sparsity makes training computationally inefficient.
Solution:
Use dimensionality reduction. Remove unnecessary users and products from where we are not learning much and reduce sparsity of user-item rating matrix.
Grey-Sheep Problem:
Now here comes a weirdo person in our app. From the name grey sheep we understand his behavior are unpredictable. He may say Game Of Thrones 1 is good and Game Of Thrones 2 is worst. Basically how do we deal with these weird people whose opinion are inconsistent.
Solution:
Pure collaborative filtering does not work. So use content-boosted filtering like in cold start problem.
Synonymy:
How will you deal with the products that are practically same but different. For example, different editions of a book or pdf or physical copy of book. Since you don’t use product description for collaborative filtering you can miss the information about synonymy. Since online stores have different codes for these items, finding synonymy can be problem.
Solution:
Latent collaborative filtering is the type of algorithm that can identify hidden factors from the data. This algorithm works really well for synonymy as well. So if we have lot of items with synonymy this is the way to go.
Shilling Attacks:
How do you deal with people who are trying to game the recommendation system? For example our system has a weird author who gave ton of ratings for his books and tons of negative ratings for other people’s book.
Solution:
Take precautions and monitor user behavior.
I will discuss more about recommendation systems in my coming articles. If you like this post, don’t forget to clap the post and follow me on medium and on twitter.
