Build a recommendation engine from scratch for your university project

found in internet

Almost every CS student need to complete a final year project. There is a lot of confusion in what language to choose, what frameworks to pick, what type of project to build, should it be AI based or simple CRUD application, and many more. I was at the same stage before. Then I realized building a recommendation system would be a great choice. Some of the reason why I got interested in building a recommendation system are:

1. I have been reading and watching a lot of AI resources earlier but never bothered myself building one because I thought I was not ready to tackle complex algorithms and implement them in jumbo Python libraries and I thought that I needed to learn much more. So I decided this time I won’t make any excuses and get my hands dirty.
2. An ai-based system that can learn from data is given high priority for final year projects by the universities.
3. Recommendation systems are one of the widely used AI application so learning how to build one and wire up the components of recommendation system would be a good programming skill to have and to keep in the resume for a future internship in AI-based companies.

Technologies I planned to use:

To build a fully functional recommendation system, we need 3 components working together. They are:

1. A user interacting with an application through the browser,
2. A web application that can manage movies, users, ratings
3. Recommendation engine

To render recommendations to the browser:

If you were like me then you must think about things like I need to implement an algorithm in Python or R but I should build a web-based recommendation system. I know how to build a web-based application using nodejs/ROR/Golang or any other technologies but not in Python because Django’s configuration is a pain in the ass. For me, Rails is the best choice for building web application because you can have all your app’s models and their relationships can be set up by firing few commands in the terminal. I don’t even have to touch code to get my app running. I can scaffold most part of the application. Just use the programming language and framework of your choice.

To implement the recommendation engine:

For this part, we are going to use Python. Because it is very easy to implement recommendation system algorithm in Python using libraries like NumPy and SciPy. We are going to build a python API that will take user’s information as input and predict the recommendation for that user and respond back with the recommendations.

System architecture:

We will have clients interacting with the web application and the web application fetches recommendations for that particular user from the recommendation engine.

Why tornado as API server for our recommendation engine?

1. It is an async web application framework that won’t block new incoming requests when previous requests are still being processed.
2. Synchronous web framework won’t scale in the production environment because we need to asynchronously process many requests at the same time.

What to recommend?

This is one of the other important questions we need to know before we start building a system. For my case, I love watching movies and I used to search for movies in imdb to find the movie I want to watch. I choose to recommend movies because I know more about movies than other things. Then I started searching for the dataset that is publicly available and that will get my job done. Then I landed in the movie lens dataset and downloaded the 100k dataset.
After I downloaded the dataset, I created a nice looking web application that shows lists of movies, has an ability to login/signup, manage profile and be able to track user’s activity. The dataset I downloaded, had data about the user(age, gender, address, occupation, email), the movie(title, imdburl, genres), ratings(userid, movieid, rating, timestamp). I then seeded all these information into my database. But before I seeded the database I created all the database schema and required relationships in the web application.

Now I at this point, I have a fully functional web application and all the data in the database. The application can log in the user and show personalized and non-personalized recommendations.

Non-personalized recommendations:

1. Trending movies => Sort in descending order of number of views
2. Popular movies => Movies that are rated highly.
3. You can also show movies that are highly rated for each genre etc.

Personalized recommendations:

1. ‘Just for you’ section => This is a highly personalized section of recommendation. To show recommendations we provide user’s information to recommendation engine and it will give us recommendations that we are going to display.

non-personalized recommendations

Personalized recommendations

The continue watching section tracks what movies the user viewed in the past and shows movie information based on that.

I have the show movie page as:

The dataset I downloaded did not have any image references so I downloaded few movie images from Google and displayed them randomly. Also, I created a dynamic movie description myself. The user should be able to manage their profile too.

Track the movies that the user was previously watching.

Now we are almost done for the web application part. We have not yet fetched the recommendation from the API. In my next post I will discuss how to build a recommendation engine.

If you like this post, don’t forget to clap the post and follow me on medium and on twitter.