In statistics, the delta method is a method of deriving the asymptotic distribution of a random variable. It is applicable when the random variable being considered can be defined as a differentiable function of a random variable which is asymptotically Gaussian. More generally, the delta method applies to Hadamard directionally differentiable functionals of stochastic processes that converge to a limiting process.

History

The delta method was derived from propagation of error, and the idea behind was known in the early 20th century. Its statistical application can be traced as far back as 1928 by T. L. Kelley. A formal description of the method was presented by J. L. Doob in 1935. Robert Dorfman also described a version of it in 1938.

Univariate delta method

While the delta method generalizes easily to a multivariate setting, careful motivation of the technique is more easily demonstrated in univariate terms. Roughly, if there is a sequence of random variables satisfying

where θ and σ2 denotes convergence in distribution, then

for any function g exists and is non-zero valued.

The intuition of the delta method is that any such g function, in a "small enough" range of the function, can be approximated via a first order Taylor polynomial, i.e., a linear function. If the random variable is roughly normal then a linear transformation of it is also normal. A small range can be ensured when approximating the function around the concentrating point of the sequence of random variables, so long as the variance is "small enough". When g is applied to a random variable such as the mean, the delta method would tend to work better as the sample size increases, since it would help reduce the variance, and thus the Taylor approximation would be applied to a smaller range of the function g at the point of interest.

Proof in the univariate case

. To begin, we use the mean value theorem (i.e.: the first order approximation of a Taylor series using Taylor's theorem):

lies between and θ and since is continuous, applying the continuous mapping theorem yields

denotes convergence in probability tends to infinity.

gives

Since

by assumption, it follows from Slutsky's theorem that

This concludes the proof.

Proof with an explicit order of approximation

Alternatively, one can add one more step at the end, to obtain the order of approximation:

This demonstrates that the error in the approximation converges to 0 in probability.

Multivariate delta method

By definition, a consistent estimator B converges in probability to its true value β, and often a central limit theorem can be applied to obtain asymptotic normality:

where n is the number of observations and Σ is a (symmetric positive semi-definite) covariance matrix. Suppose we want to estimate the variance of a scalar-valued function h of the estimator B. Keeping only the first two terms of the Taylor series, and using vector notation for the gradient, we can estimate h(B) as

which implies the variance of h(B) is approximately

One can use the mean value theorem (for real-valued functions of many variables) to see that this does not rely on taking first order approximation.

The delta method therefore implies that

or in univariate terms,

Example: the binomial proportion

Suppose Xn is binomial and n. Since

we can apply the Delta method with to see

Hence, even though for any finite n does not actually exist (since Xn does exist and is equal to

Note that since p>0 is finite for large n.

are estimates of different group rates from independent samples of sizes n and m respectively, then the logarithm of the estimated relative risk has asymptotic variance equal to

This is useful to construct a hypothesis test or to make a confidence interval for the relative risk.

Alternative form

The delta method is often used in a form that is essentially identical to that above, but without the assumption that or B is asymptotically normal. Often the only context is that the variance is "small". The results then just give approximations to the means and covariances of the transformed quantities. For example, the formulae presented in Klein (1953, p.258) are:

where is the rth element of h(B) and Bi is the ith element of B.

Second-order delta method

When the delta method cannot be applied. However, if .

Note that when using the second-order method, the expectation of is no longer simply . The intuition is that if the function has significant curvature, even if the input variable is Gaussian (and has symmetric tails), the expected value of the function is no longer equal to the function of the mean, .

can be approximated as the weighted sum of a standard normal and a chi-square with 1 degree of freedom.

Nonparametric delta method

A version of the delta method exists in nonparametric statistics be an independent and identically distributed with an empirical distribution function is Hadamard differentiable with respect to the Chebyshev metric, then

denoting the empirical influence function is therefore given by

-quantile of the standard normal.

Examples

is obtained by replacing the unknown population quantities by suitable sample estimates.

Mean

For the mean functional

gives

Therefore, the influence function is

, its empirical version is

Consequently,

and the standard error obtained from the general formula above is

Correlation coefficient

be the Pearson correlation coefficient

Writing

the influence function of the correlation functional is

Let

be the sample correlation coefficient. The empirical influence values are then

Thus, the variance factor in the general standard-error formula may be estimated by

Quantile

Let

. The influence function of the quantile functional is

is an estimate of the density at that quantile, the influence function may be estimated by

It follows that

At the population level,

so the asymptotic variance of the sample quantile is

See also

• Taylor expansions for the moments of functions of random variables

• Variance-stabilizing transformation

References

Further reading

External links