Implementing Regression algorithms from Scratch: Experience— #P3 of 100MLProjects

Hey there! People!
I’ve just completed Project3 or Shortly P3 of #100MLProjects, where I attempted to build Machine Learning Regression algorithms from scratch.

This is not a tutorial or walkthrough blog, but I'm detailing my experience doing this project. This project really turned out to be a really good learning experience for me, I had to face some challenges, but at the same time, I understood the areas that I need to work on.

I picked two regression algorithms for this project, to build from scratch with a goal of getting a deep understanding of the math behind it. I can now strongly say, I have made a significant leap in that section.

Math Math Math! Math everywhere!

Most of the math involved in deriving these algorithms are taught in Indian high schools and in engineering colleges. As I come from an engineering background, I did learn most of these concepts too, but one of my biggest problems is I didn't know the practical applications for those concepts, it was just brute memorization for scoring good grades(Truth Bomb). This time when I made a decision to strengthen my math foundation to advance in ML/DL, I learned (and learning) math out of curiosity and not for grades or evaluation purposes. This simple shift in mindset made me realize how beautiful math is, and how naive I have been.

Math is now becoming my new fav subject. I still have a long way to go, but I’m enjoying the process. Thanks to Khanacademy and 3Blue1Brown for teaching me Math the right and exciting way. If you are reading this post and you are trying to learn machine learning, I would highly suggest you to check out 3Blue1Brown YouTube channel, it’s awesome!

Challenges Faced:

Coming back, I faced a few challenges during this project.

1. Comprehending the Math Notations
   I had trouble understanding the mathematical notation that the technical articles and literature review papers used. They were greek and Latin to me. It was really hard to comprehend, and it took me hours to understand the common notations statisticians and data scientists used. How did I overcome it? I just dived straight in, hoping I would catch on as I keep reading. Machine Learning by Sebastian Raschka's book really helped me overcome this, a really good book and I would highly recommend it.
2. Math itself, SalKhan came to the rescue!
   My math knowledge was wide but not deep. I knew the concepts, but I couldn't clearly understand why a particular approach was used. So I went back to Khanacademy, and SalKhan is serving as a light and showing me the way through the dark ML Math Cave. I took courses on probability, statistics, linear algebra, bivariate regression, and a few others.
3. The temptation to take the shorter route
   I could have copy-pasted other implementations of the same algorithms that are available in GitHub, but the ‘Why’ power kept me on track. The temptation was high, but remembering why I actually started doing this project prevented me from deviating.

Those who have a ‘why’ to live, can bear with almost any ‘how’.
-Viktor Frankl, Man’s search for Meaning

Experience and Lessons:

I have improved my reading research papers now! and I might start building models based on research papers!

Math and theory knowledge behind an algorithm is very essential to build efficient models and improve its performance, and also comes in very handy while we are trying to tune the hyperparameters.

Learning to build the algorithm from scratch not only gives us an intuitive knowledge but also empowers us to know which model can be used where. previous to this project, I only knew there were several regression models — linear, logistic, Huber, ridge, lasso, etc. For example, Linear regression cannot perform efficiently for data with too many outliers, the ordinary least squares method will deviate from the optimal solution while trying to accommodate the outliers. Here’s where algorithms such as Huber, Lasso can perform better. These regression algorithms reduce the impact of outliers while deducing the regression model.

(Note: Initially I had the plan to build Huber regression too, but dropped the idea as I have decided to add that to a future project — Advanced Regression algorithms from scratch).

I have completed the following implementations:

1. Linear Regression using Gradient Descent Optimization
2. Linear Regression using Ordinary least squares
3. Logistic regression — Binary Classification
4. Logistic regression — Multiclass Classification — One vs All method\

My source code for this project is available in GitHub: Project3–100MLProjects

Check out my LinkedIn Profile:

Lakshmanan Meiyappan - Chennai, Tamil Nadu, India | Professional Profile | LinkedIn

Check out my GitHub Repository:

laxmena/100MLProjects