The purpose of this page is to provide supplementary materials for the ordinary least squares article, reducing the load of the main article with mathematics and improving its accessibility, while at the same time retaining the completeness of exposition.
Derivation of the normal equations
Define the th residual to be
Then the objective can be rewritten
Given that S is convex, it is minimized when its gradient vector is zero (This follows by definition: if the gradient vector is not zero, there is a direction in which we can move to minimize it further – see maxima and minima.) The elements of the gradient vector are the partial derivatives of S with respect to the parameters:
The derivatives are
Substitution of the expressions for the residuals and the derivatives into the gradient equations gives
Thus if minimizes S, we have
Upon rearrangement, we obtain the normal equations:
The normal equations are written in matrix notation as
(where XT is the matrix transpose of X).
The solution of the normal equations yields the vector of the optimal parameter values.
Derivation directly in terms of matrices
The normal equations can be derived directly from a matrix representation of the problem as follows. The objective is to minimize
Here has the dimension 1x1 (the number of columns of ), so it is a scalar and equal to its own transpose, hence and the quantity to minimize becomes
Differentiating this with respect to and equating to zero to satisfy the first-order conditions gives
which is equivalent to the above-given normal equations. A sufficient condition for satisfaction of the second-order conditions for a minimum is that have full column rank, in which case is positive definite.
Derivation without calculus
When is positive definite, the formula for the minimizing value of can be derived without the use of derivatives. The quantity
can be written as
where depends only on and , and is the inner product defined by
It follows that is equal to
and therefore minimized exactly when
Generalization for complex equations
In general, the coefficients of the matrices and can be complex. By using a Hermitian transpose instead of a simple transpose, it is possible to find a vector which minimizes , just as for the real matrix case. In order to get the normal equations we follow a similar path as in previous derivations:
where stands for Hermitian transpose.
We should now take derivatives of with respect to each of the coefficients , but first we separate real and imaginary parts to deal with the conjugate factors in above expression. For the we have
and the derivatives change into
After rewriting in the summation form and writing explicitly, we can calculate both partial derivatives with result:
which, after adding it together and comparing to zero (minimization condition for ) yields
In matrix form:
Least squares estimator for β
Using matrix notation, the sum of squared residuals is given by
Since this is a quadratic expression, the vector which gives the global minimum may be found via matrix calculus by differentiating with respect to the vector (using denominator layout) and setting equal to zero:
By assumption matrix X has full column rank, and therefore XTX is invertible and the least squares estimator for β is given by
Unbiasedness and variance of
Plug y = Xβ + ε into the formula for and then use the law of total expectation:
where E[ε|X] = 0 by assumptions of the model. Since the expected value of equals the parameter it estimates, , it is an unbiased estimator of .
For the variance, let the covariance matrix of be
(where is the identity matrix), and let X be a known constant. Then,
where we used the fact that is just an affine transformation of by the matrix .
For a simple linear regression model, where ( is the y-intercept and is the slope), one obtains
Expected value and biasedness of
First we will plug in the expression for y into the estimator, and use the fact that X'M = MX = 0 (matrix M projects onto the space orthogonal to X):
Now we can recognize ε′Mε as a 1×1 matrix, such matrix is equal to its own trace. This is useful because by properties of trace operator, tr(AB) = tr(BA), and we can use this to separate disturbance ε from matrix M which is a function of regressors X: