In calculus, Newton's method (also called Newton–Raphson) is an iterative method for finding the roots of a differentiable function, which are solutions to the equation. We can therefore use Newton's method on its derivative, also known as the critical points. These solutions may be minima, maxima, or saddle points; see section "Several variables" in Critical point (mathematics) and also section "Geometric interpretation" in this article. This is relevant in optimization.

Newton's method

The central problem of optimization is minimization of functions. Let us first consider the case of univariate functions, i.e., functions of a single real variable. We will later consider the more general and more practically useful multivariate case.

, we seek to solve the optimization problem

Newton's method attempts to solve this problem by constructing a sequence around the iterates. The second-order Taylor expansion of is

, and its minimum can be found by setting the derivative to zero. Since

the minimum is achieved for

Putting everything together, Newton's method performs the iteration

Geometric interpretation

The geometric interpretation of Newton's method is that at each iteration, it amounts to the fitting of a parabola to the graph, having the same slope and curvature as the graph at that point, and then proceeding to the maximum or minimum of that parabola (in higher dimensions, this may also be a saddle point happens to a quadratic function, then the exact extremum is found in one step.

Higher dimensions

The above iterative scheme dimensions by replacing the derivative with the gradient), and the reciprocal of the second derivative with the inverse of the Hessian matrix). One thus obtains the iterative scheme

Often Newton's method is modified to include a small step size:

This is often done to ensure that the Wolfe conditions, or much simpler and efficient Armijo's condition, are satisfied at each step of the method. For step sizes other than 1, the method is often referred to as the relaxed or damped Newton's method.

Convergence

If quadratically fast. That is,

Computing the Newton direction

as the solution to the system of linear equations

which may be solved by various factorizations or approximately (but to great accuracy) using iterative methods. Many of these methods are only applicable to certain types of equations, for example the Cholesky factorization and conjugate gradient is not positive definite, then the iterations are converging to a saddle point and not a minimum.

On the other hand, if a constrained optimization is done (for example, with Lagrange multipliers variant of Cholesky factorization or the conjugate residual method.

There also exist various quasi-Newton methods, where an approximation for the Hessian (or its inverse directly) is built up from changes in the gradient.

If the Hessian is close to a non-invertible matrix.

An approach exploited in the Levenberg–Marquardt algorithm and small Hessian, the iterations will behave like gradient descent. This results in slower but more reliable convergence where the Hessian doesn't provide useful information.

Some caveats

Newton's method, in its original version, has several caveats:

1. It does not work if the Hessian is not invertible. This is clear from the very definition of Newton's method, which requires taking the inverse of the Hessian.

2. It may not converge at all, but can enter a cycle having more than 1 point. See § Computing the Newtonian direction.

3. It can converge to a saddle point instead of to a local minimum, as per § Geometric interpretation.

The popular modifications of Newton's method, such as quasi-Newton methods or Levenberg-Marquardt algorithm mentioned above, also have caveats:

For example, it is usually required that the cost function is (strongly) convex and the Hessian is globally bounded or Lipschitz continuous, for example this is mentioned in the section "Convergence" in this article. If one looks at the papers by Levenberg and Marquardt in the reference for Levenberg–Marquardt algorithm, which are the original sources for the mentioned method, one can see that there is basically no theoretical analysis in the paper by Levenberg, while the paper by Marquardt only analyses a local situation and does not prove a global convergence result. One can compare with a backtracking line search on gradient descent, which has good theoretical guarantee under more general assumptions, and can be implemented and works well in practical large scale problems such as deep neural networks.

See also

• Quasi-Newton method

• Gradient descent

• Gauss–Newton algorithm

• Levenberg–Marquardt algorithm

• Trust region

• Optimization

• Nelder–Mead method

• Self-concordant function - a function for which Newton's method has very good global convergence rate.

Notes

References

External links

fr:Méthode de Newton