↑ arvau

Wikipedia / Preconditioner

In mathematics, preconditioning is the application of a transformation, called the preconditioner, that conditions a given problem into a form that is more suitable for numerical solving methods. Preconditioning is typically related to reducing a condition number of the problem. The preconditioned problem is then usually solved by an iterative method.

Preconditioning for linear systems

In linear algebra and numerical analysis, a preconditioner has a smaller condition number, i.e., multiplication of a column vector, is commonly performed in a matrix-free fashion) are explicitly available in a matrix form.

Preconditioners are useful in iterative methods since the rate of convergence for most iterative linear solvers increases because the condition number of a matrix decreases as a result of preconditioning. Preconditioned iterative solvers typically outperform direct solvers, e.g., Gaussian elimination, for large, especially for sparse, matrices. Iterative solvers can be used as matrix-free methods is not stored explicitly, but is accessed by evaluating matrix-vector products.

Description

, one may consider the right.

Alternatively, one may solve the left

is nonsingular. The left preconditioning is more traditional.

The two-sided is also symmetric. The two-sided preconditioning is common for diagonal scaling, e.g., in order to decrease the dynamic range of entries of the matrix.

The goal of preconditioning is reducing the condition number. Small condition numbers benefit fast convergence of iterative solvers and improve stability of the solution with respect to perturbations in the system matrix and the right-hand side, e.g., allowing for more aggressive quantization of the matrix entries using lower computer precision.

to a given vector may need to be computed.

which has optimal condition number as simple as possible. Some examples of typical preconditioning approaches are detailed below.

Preconditioned iterative methods

For example, the standard Richardson iteration

Examples of popular preconditioned iterative methods for linear systems include the preconditioned conjugate gradient method, the biconjugate gradient method, and generalized minimal residual method

Matrix splitting

A stationary iterative method. Assuming that

the condition number

Geometric interpretation

For a symmetric positive definite-based scalar product. In this case, the desired effect in applying a preconditioner is to make the quadratic form-based scalar product to be nearly spherical.

Variable and non-linear preconditioning

may not be linear. Typical examples involve using non-linear iterative methods, e.g., the conjugate gradient method, as a part of the preconditioner construction. Such preconditioners may be practically very efficient, however, their behavior is hard to predict theoretically.

Random preconditioning

One interesting particular case of variable preconditioning is random preconditioning, e.g., multigrid preconditioning on random coarse grids. If used in gradient descent methods, random preconditioning can be viewed as an implementation of stochastic gradient descent and can lead to faster convergence, compared to fixed preconditioning, since it breaks the asymptotic "zig-zag" pattern of the gradient descent.

Spectrally equivalent preconditioning

The most common use of preconditioning is for iterative solution of linear systems resulting from approximations of partial differential equations to be bounded from above by a constant independent of the matrix size, which is called spectrally equivalent preconditioning by D'yakonov by a vector.

Examples

Jacobi (or diagonal) preconditioner

The Jacobi preconditioner It is efficient for diagonally dominant matrices. It is used in analysis software for beam problems or 1-D problems (EX:- STAAD.Pro)

SPAI

The Sparse Approximate Inverse is the Frobenius norm is from some suitably constrained set of sparse matrices. The method was introduced by M.J. Grote and T. Huckle together with an approach to selecting sparsity patterns.

Other preconditioners

Successive over-relaxation

Preconditioning for eigenvalue problems

Eigenvalue problems can be framed in several alternative ways, each leading to its own preconditioning. The traditional preconditioning is based on the so-called spectral transformations. Knowing (approximately) the targeted eigenvalue, one can compute the corresponding eigenvector by solving the related homogeneous linear system, thus allowing to use preconditioning for linear system. Finally, formulating the eigenvalue problem as optimization of the Rayleigh quotient brings preconditioned optimization techniques to the scene.

Spectral transformations

By analogy with linear systems, for an eigenvalue. However, this makes sense only if the seeking eigenvectors are the same. This is the case for spectral transformations.

The most popular spectral transformation is the so-called shift-and-invert, called the shift. The eigenvectors are preserved, and one can solve the shift-and-invert problem by an iterative solver, e.g., the power iteration. This gives the Inverse iteration. The Rayleigh quotient iteration is a shift-and-invert method with a variable shift.

Spectral transformations are specific for eigenvalue problems and have no analogs for linear systems. They require accurate numerical calculation of the transformation involved, which becomes the main bottleneck for large problems.

General preconditioning

is the preconditioner, which we can try to solve using the Richardson iteration

The ideal preconditioning

The Moore–Penrose pseudoinverse is the preconditioner, which makes the Richardson iteration. Second, the exact Moore–Penrose pseudoinverse requires the knowledge of the eigenvector, which we are trying to find. This can be somewhat circumvented by the use of the Jacobi–Davidson preconditioner, which becomes as expensive for large problems as the shift-and-invert method above. If the solution is not accurate enough, step two may be redundant.

Practical preconditioning

in the Richardson iteration

using the Rayleigh quotient allows one to easily utilize for eigenvalue problems the vast variety of preconditioners developed for linear systems.

, a comprehensive theoretical convergence analysis is much more difficult, compared to the linear systems case, even for the simplest methods, such as the Richardson iteration.

Preconditioning in optimization

In optimization, preconditioning is typically used to accelerate first-order optimization algorithms.

Description

For example, to find a local minimum using gradient descent, one takes steps proportional to the negative of the gradient

Preconditioning here can be viewed as changing the geometry of the vector space with the goal to make the level sets look like circles. In this case the preconditioned gradient aims closer to the point of the extrema as on the figure, which speeds up the convergence.

Connection to linear systems

is a real symmetric positive-definite matrix, the preconditioned gradient descent

This is the preconditioned Richardson iteration for solving a system of linear equations.

Connection to eigenvalue problems

The minimum of the Rayleigh quotient is a real symmetric positive-definite matrix, is the smallest eigenvalue, while the minimizer is the corresponding eigenvector, the preconditioned gradient descent

This is an analog of preconditioned Richardson iteration for solving eigenvalue problems.

Variable preconditioning

In many cases, it may be beneficial to change the preconditioner at some or even every step of an iterative algorithm

, a BFGS approximation of the inverse hessian matrix, this method is referred to as a Quasi-Newton method.

References

Sources