Projection matrix on arvau

Wikipedia / Projection matrix

In statistics, the projection matrix, sometimes also called the influence matrix or hat matrix, maps the vector of response values (dependent variable values) to the vector of fitted values (or predicted values). It describes the influence each response value has on each fitted value. The diagonal elements of the projection matrix are the leveragess, which describe the influence each response value has on the fitted value for that same observation.

Definition

If the vector of response values,

is also named hat matrix as it "puts a hat".

Application for residuals

The formula for the vector of residuals can also be expressed compactly using the projection matrix:

is the identity matrix is sometimes referred to as the residual maker matrix or the annihilator matrix.

The covariance matrix, by error propagation, equals

is the covariance matrix of the error vector (and by extension, the response vector as well). For the case of linear models with independent and identically distributed, this reduces to:

Intuition

. A vector that is orthogonal to the column space of a matrix is in the nullspace of the matrix transpose, so

From there, one rearranges, so

Linear model

Suppose that we wish to estimate a linear model using linear least squares. The model can be written as

is a matrix of explanatory variables (the design matrix), β is a vector of unknown parameters to be estimated, and ε is the error vector.

Many types of models and techniques are subject to this formulation. A few examples are linear least squares, smoothing splines, regression splines, local regression, kernel regression, and linear filtering.

Ordinary least squares

When the weights for each observation are identical and the errors are uncorrelated, the estimated parameters are

so the fitted values are

Therefore, the projection matrix (and hat matrix) is given by

Weighted and generalized least squares

The above may be generalized to the cases where the weights are not identical and/or the errors are correlated. Suppose that the covariance matrix of the errors is Σ. Then since

the hat matrix is thus

, though now it is no longer symmetric.

Properties

The projection matrix has a number of useful algebraic properties. In the language of linear algebra, the projection matrix is the orthogonal projection onto the column space. is the pseudoinverse of X.) Some facts of the projection matrix in this setting are summarized as follows:

.
.
is an
The eigenvalues consist of r ones and consist of ones and r zeros.
.
is unique for certain subspaces.

The projection matrix corresponding to a linear model is symmetric and idempotent. However, this is not always the case; in locally weighted scatterplot smoothing (LOESS), for example, the hat matrix is in general neither symmetric nor idempotent.

For linear models, the trace of the projection matrix is equal to the rank, which is the number of independent parameters of the linear model., the projection matrix can be used to define the effective degrees of freedom of the model.

Practical applications of the projection matrix in regression analysis include leverage and Cook's distance, which are concerned with identifying influential observations, i.e. observations which have a large effect on the results of a regression.

Blockwise formula

. Then the projection matrix can be decomposed as follows:

is a column of all ones, which allows one to analyze the effects of adding an intercept term to a regression. Another use is in the fixed effects model is a large sparse matrix, which might be too large to fit into computer memory.

History

The hat matrix was introduced by John Wilder in 1972. An article by Hoaglin, D.C. and Welsch, R.E. (1978) gives the properties of the matrix and also many examples of its application.