Linear Regression Foundations

1. What supervised regression does

1. Linear Regression is a supervised learning method used for predicting a numerical target from one or more input features.
2. A training dataset contains examples where both the inputs and the target are known.
3. The model learns coefficients from these examples, then uses the fitted equation to make predictions for new inputs.

• Feature: an input variable used to make a prediction.
• Target: the numerical value the model is trying to predict.
• Prediction: the output calculated by the fitted model.
• Parameter: a value learned by the model, such as a slope or intercept.

2. Simple Linear Regression equation

For one feature, the model is

$$\hat y = b_0 + b_1x.$$

• $x$ is the input feature.
• $\hat y$ is the predicted target.
• $b_0$ is the intercept: the prediction when $x=0$.
• $b_1$ is the slope: the change in the prediction for a one-unit increase in $x$.

The hat on $\hat y$ indicates a prediction, not the observed target $y$.

Example

fitted model is

$$\hat y = 1 + 2x.$$

For $x=3$,

$$\hat y = 1 + 2(3) = 1+6 = 7.$$

The model predicts 7. This calculation alone does not tell us whether 7 is close to the real target. We need the observed value to calculate an error.

import numpy as np

x = np.array([1, 2, 3], dtype=float)
y = np.array([3, 5, 7], dtype=float)

# Start with a deliberately imperfect line so errors are visible.
b0 = 0.0
b1 = 1.5
y_pred = b0 + b1 * x

print("Predictions:", y_pred)
print("Observed targets:", y)

Predictions: [1.5 3.  4.5]
Observed targets: [3. 5. 7.]

3. Residuals: observed minus predicted

A residual is the difference between an observed target and its prediction:

$$e_i = y_i - \hat y_i.$$

The sign matters:

• Positive residual: the model predicted too low.
• Negative residual: the model predicted too high.
• Zero residual: the prediction equals the observed value.

Some gradient formulas use $\hat y-y$, the opposite sign. That is not a contradiction: the residual definition and the derivative's error term serve different roles. State the convention before calculating.

residuals = y - y_pred
print("Residuals:", residuals)

Residuals: [1.5 2.  2.5]

4. Squared error, SSE and MSE

Squaring residuals prevents positive and negative errors from cancelling and gives larger errors more influence.

Sum of Squared Errors (SSE):

$$SSE = \sum_{i=1}^{n}(y_i-\hat y_i)^2.$$

Mean Squared Error (MSE):

$$MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i-\hat y_i)^2.$$

SSE is the total squared error. MSE is the average squared error per observation.

MSE is expressed in squared target units.

Hand calculation

example: let $y=[3,5,7]$ and $\hat y=[2,5,8]$

$y_i$	$\hat y_i$	Residual $y_i-\hat y_i$	Squared residual
3	2	1	1
5	5	0	0
7	8	-1	1

Therefore,

$SSE=1+0+1=2,$

and with $n=3$,

$MSE=\frac{2}{3}\approx0.667.$

y_example = np.array([3, 5, 7], dtype=float)
pred_example = np.array([2, 5, 8], dtype=float)

sse = np.sum((y_example - pred_example) ** 2)
mse = np.mean((y_example - pred_example) ** 2)

print("SSE:", sse)
print("MSE:", mse)

SSE: 2.0
MSE: 0.6666666666666666

5. Cost function: a quantity to minimize

Training often means choosing parameters that minimize an objective. In this learning path, one commonly used cost is half the MSE:

$$J(b_0,b_1)=\frac{1}{2n}\sum_{i=1}^{n}(\hat y_i-y_i)^2.$$

Since squaring removes the sign, using $\hat y-y$ instead of $y-\hat y$ gives the same squared error. The factor $1/2$ is included because it cancels the factor 2 that appears when differentiating a square.

Be careful: this cost is not exactly MSE. It equals $MSE/2$. Always label the quantity you calculate.

cost_half_mse = np.sum((y_pred - y) ** 2) / (2 * len(y))
mse = np.mean((y_pred - y) ** 2)

print("Half-MSE cost:", cost_half_mse)
print("MSE:", mse)

Half-MSE cost: 2.0833333333333335
MSE: 4.166666666666667

6. Practice checkpoints

1. If $\hat y=4+3x$, calculate the prediction for $x=2$.
2. If $y=12$ and $\hat y=15$, calculate the residual using $e=y-\hat y$. Is the model overpredicting or underpredicting?
3. For residuals $[2,-1,0]$, calculate SSE and MSE by hand.
4. Explain why the half-MSE cost and MSE have different numerical values.

# 1. y_hat = 4 + 3x, prediction at x = 2
x= 2
y_hat = 4 + 3 * x
print("Prediction for x=2:", y_hat)

Prediction for x=2: 10

# Q2: y = 12, y_hat = 15, residual e = y - y_hat
y_obs = 12
y_hat_q2 = 15
residual = y_obs - y_hat_q2
print("Residual:", residual)

if residual > 0:
    print("The model is underpredicting.")
elif residual < 0:
    print("The model is overpredicting.")
else:
    print("The model is neither overpredicting nor underpredicting.")

Residual: -3
The model is overpredicting.

# Q3: Find SSE and MSE for residuals [2, -1, 0]

residuals = np.array([2, -1, 0], dtype=float)

SSE= np.sum(residuals**2)
MSE= np.mean(residuals**2)

print("SSE:", SSE)
print("MSE:", MSE)

SSE: 5.0
MSE: 1.6666666666666667

4. Why do half-MSE cost and MSE have different values?

Because the half-MSE cost divides the sum of squared errors by 2n (n = number of observations), while MSE divides by n. So the half-MSE cost is half the value of MSE. This is useful in gradient descent because the 1/2 cancels the 2 that comes from differentiating the square.

Takeaways

Simple Linear Regression predicts a numerical target using an intercept and a slope. Residuals describe prediction errors; SSE, MSE, and half-MSE cost measure error in related but distinct ways.

Notes

The original toy data $x=[1,2,3]$, $y=[3,5,7]$ lie exactly on $y=1+2x$. That is useful for checking the fitted coefficients, but it hides what nonzero errors look like. This notebook therefore uses an intentionally imperfect line in the first prediction example.

A point to keep clear: the residual convention used here is $y-\hat y$. The Gradient Descent derivative later uses $\hat y-y$. These differ by a sign, so write down which expression you are using before calculating.