What Exactly a Bias Term is?

What Exactly a Bias Term is?

Hey folks

In this post, I’ll talk about the Intuition behind the bias term in linear regression.

As you already know a linear regression is all about finding the best line which fits our data, and those who don’t know about it don’t worry, let’s take it another way.

You probably learned about the equation of a line in your High school i.e y=m*x+c

So here c is a constant term while in Machine Learning it’s called the Bias Term.

Alright, let’s take an example to make it intuitive but please don’t get irritated as I’ll going to take the most common examples in the history of Machine Learning and that is Housing Price Prediction.

So for simplicity let’s only consider 2 features.

Let’s consider our hypothesis function or the prediction function for this problem.

H(theta)= theta0+ x1*theta1+ x2*theta2

Where x1 and x2 are the features, while theta0, theta1, and theta2 are the weights or the learnable parameters. So here theta0 is a bias term, but exactly what it conveys?

So before moving forward just look at the small example. Suppose you have a list of heights of your batchmates and you have a missing value so what you will be going to do. So the most common approach is to fill that missing height value with the average of other heights. Okay then, let’s keep this example in your mind.

Here comes the intuitive part.

Just think about this way, consider a scenario when you don’t any information about the house like you know nothing about it, so in that case, what will you do? (Hint: Remember the missing height example)

You’ll consider the price as the average of all house prices for that house.

Mathematically put x1 and x2 as 0 and it gives us the price value as theta0.

So by the above result theta0 should be the average of all the house prices, that’s the significance of Bias term, as it gives the average of output when we don’t have prior knowledge of inputs.

You can prove it mathematically also.

Consider the equation of a line for a single variable i.e y=m*x+c

So my hypothesis is c= average(y)

Let’s calculate the average of y for -a to a (where, i!=0)

Let’s expand this above equation.

So it’s mathematically proven that the constant term is nothing but the average of the other output terms.

I hope now the bias term makes sense to you and it’s not only useful in Linear Regression, but it’s also a part of deep learning and other fields of engineering like Instrumentation and Control. As in the PID controller, there’s also a Bias term and that’s what its significance is.