Smoothing spline on arvau

Wikipedia / Smoothing spline

Smoothing splines, in order to balance a measure of goodness of fit is a vector quantity.

Cubic spline definition

is defined to be the unique minimizer, in the Sobolev space on a compact interval, of

Remarks:

is a smoothing parameter, controlling the trade-off between fidelity to the data and roughness of the function estimate. This is often estimated by generalized cross-validation, or by restricted marginal likelihood (REML)).
.
(no smoothing), the smoothing spline converges to the interpolating spline.
(infinite smoothing), the roughness penalty becomes paramount and the estimate converges to a linear least squares estimate.
The roughness penalty based on the second derivative is the most common in modern statistics literature, although the method can easily be adapted to penalties based on other derivatives.
, second or third-order differences were used in the penalty, rather than derivatives. See also Whittaker–Henderson smoothing.
The penalized sum of squares smoothing objective can be replaced by a penalized likelihood objective in which the sum of squares terms is replaced by another log-likelihood based measure of fidelity to the data..

Derivation of the cubic smoothing spline

It is useful to think of fitting a smoothing spline in two steps:

.
for all x.

Now, treat the second step first.

, and the minimizer is a natural cubic spline. This interpolating spline is a linear operator, and can be written in the form

are a set of spline basis functions. As a result, the roughness penalty has the form

matrix of second differences with elements:

symmetric tri-diagonal matrix with elements:

, the distances between successive knots (or x values).

Now back to the first step. The penalized sum-of-squares can be written as

De Boor's approach

De Boor's approach exploits the same idea, of finding a balance between having a smooth curve and being close to the given data.

). In practice, since cubic splines was proposed by Christian Reinsch in 1967. converges to the "natural" spline interpolant to the given data. was introduced for convenience. meets the following condition:

until the condition is met. is an estimation of the standard deviation means the solution is the "natural" spline interpolant. means we obtain a smoother curve by getting farther from the given data.

Multidimensional splines

we might use the Thin plate spline minimizing

The thin plate spline approach can be generalized to smoothing with respect to more than two dimensions and to other orders of differentiation in the penalty. As the dimension increases there are some restrictions on the smallest order of differential that can be used, but actually Duchon's original paper, gives slightly more complicated penalties that can avoid this restriction.

co-ordinate system the estimate will not change, but also that we are assuming that the same level of smoothing is appropriate in all directions. This is often considered reasonable when smoothing with respect to spatial location, but in many other cases isotropy is not an appropriate assumption and can lead to sensitivity to apparently arbitrary choices of measurement units. For example, if smoothing with respect to distance and time an isotropic smoother will give different results if distance is measure in metres and time in seconds, to what will occur if we change the units to centimetres and hours.

The second class of generalizations to multi-dimensional smoothing deals directly with this scale invariance issue using tensor product spline constructions. Such splines have smoothing penalties with multiple smoothing parameters, which is the price that must be paid for not assuming that the same degree of smoothness is appropriate in all directions.

Related methods

Smoothing splines are related to, but distinct from:

Regression splines. In this method, the data is fitted to a set of spline basis functions with a reduced set of knots, typically by least squares. No roughness penalty is used. (See also multivariate adaptive regression splines.)
Penalized splines. This combines the reduced knots of regression splines, with the roughness penalty of smoothing splines.
Thin plate splines and Elastic maps method for manifold learning. This method combines the least squares penalty for approximation error with the bending and stretching penalty of the approximating manifold and uses the coarse discretization of the optimization problem.

Source code

Source code for spline smoothing can be found in the examples from Carl de Boor's book A Practical Guide to Splines. The examples are in the Fortran programming language. The updated sources are available also on Carl de Boor's official site http://pages.cs.wisc.edu/~deboor/.