Rejection sampling on arvau

Wikipedia / Rejection sampling

In numerical analysis and computational statistics, rejection sampling is a basic technique used to generate observations from a distribution. It is also commonly called the acceptance-rejection method with a density.

Rejection sampling is based on the observation that to sample a random variable in one dimension, one can perform a uniformly random sampling of the two-dimensional Cartesian graph, and keep the samples in the region under the graph of its density function. Note that this property can be extended to N-dimension functions.

Algorithmic Definition

The algorithm, which was used by John von Neumann and dates back to Buffon and his needle as follows:

Rejection Sampling

Algorithm

rejections to obtain a sample.

Detailed Description

To visualize the motivation behind rejection sampling, imagine graphing the probability density function‑positions of these darts will be distributed according to the random variable's density. This is because there is the most room for the darts to land where the curve is highest and thus the probability density is greatest.

) that we know how to sample from (for example, using inversion sampling). Its shape must be at least as high at every point as the distribution we want to sample from, so that the former completely encloses the latter. Otherwise, there would be parts of the curved area we want to sample from that could never be reached.

Rejection sampling works as follows:

‑axis from the proposal distribution.
‑position, up to the y-value of the probability density function of the proposal distribution.
‑value is a sample from the desired distribution.

This algorithm can be used to sample from the area under any curve, regardless of whether the function integrates to 1. In fact, scaling a function by a constant has no effect on the sampled . Thus, the algorithm can be used to sample from a distribution whose normalizing constant is unknown, which is common in computational statistics.

Theory

. The rejection sampling method generates sampling values from a target distribution with probability density function over the support.

. There are a number of extensions to this algorithm, such as the Metropolis algorithm.

This method relates to the general field of Monte Carlo techniques, including Markov chain Monte Carlo. It forms the basis for algorithms such as the Metropolis algorithm.

of the proposal distribution.

to obtain an accepted value thus follows a geometric distribution is the expected number of the iterations that are needed, as a measure of the computational complexity of the algorithm.

, i.e. that the target and proposal distributions are actually the same distribution.

times the cost of these operations—which is the expected cost of obtaining a sample with rejection sampling—is lower than the cost of obtaining a sample using the other method.

Advantages over sampling using naive methods

Rejection sampling can be far more efficient compared with the naive methods in some situations can be easily simulated, using the naive methods (e.g. by inverse transform sampling):

(see also truncation (statistics))

is large and the rejection rate is high, the algorithm can be very inefficient. The Natural Exponential Family and speed up the computations (see examples: working with Natural Exponential Families).

Rejection sampling using exponential tilting

. Choose the proposal as

, is from a natural exponential family. Moreover, the likelihood ratio is

implies that it is indeed a cumulant-generating function, that is,

It is easy to derive the cumulant-generation function of the proposal and therefore the proposal's cumulants.

. The analysis goes as follows:

, with cumulant-generating function as
1. ,
which further implies it is a normal distribution.
is to set
1. ,
.
Explicitly write out the target, the proposal and the likelihood ratio
, therefore
, if

until acceptance.

, which is far more inefficient.

In general, exponential tilting, among the class of simple distributions, the trick is to use natural exponential family, which helps to gain some control over the complexity and considerably speed up the computation. Indeed, there are deep mathematical reasons for using natural exponential family.

Drawbacks

Rejection sampling requires knowing the target distribution (specifically, ability to evaluate target PDF at any point).

Rejection sampling can lead to a lot of unwanted samples being taken if the function being sampled is highly concentrated in a certain region, for example a function that has a spike at some location. For many distributions, this problem can be solved using an adaptive extension (see adaptive rejection sampling), or with an appropriate change of variables with the method of the ratio of uniforms. In addition, as the dimensions of the problem get larger, the ratio of the embedded volume to the "corners" of the embedding volume tends towards zero, thus a lot of rejections can take place before a useful sample is generated, thus making the algorithm inefficient and impractical. See curse of dimensionality. In high dimensions, it is necessary to use a different approach, typically a Markov chain Monte Carlo method such as Metropolis sampling or Gibbs sampling. (However, Gibbs sampling, which breaks down a multi-dimensional sampling problem into a series of low-dimensional samples, may use rejection sampling as one of its steps.)

Regenerative Rejection Sampling

, as follows.

Regenerative Rejection Sampling

Input:
.
Algorithm:

.
.
.
Output:
.

converges in distribution.

is the empirical supremum rejection sampling method.

Adaptive rejection sampling

For many distributions, finding a proposal distribution that includes the given distribution without a lot of wasted space is difficult. An extension of rejection sampling that can be used to overcome this difficulty and efficiently sample from a wide variety of distributions (provided that they have log-concave density functions, which is in fact the case for most of the common distributions—even those whose density functions are not concave themselves) is known as adaptive rejection sampling (ARS).

There are three basic ideas to this technique as ultimately introduced by Gilks in 1992:

directly.

may be easier to work with or, at least, closer to piecewise linear).

Instead of a single uniform envelope density function, use a piecewise linear density function as your envelope instead.

. This therefore reduces the chance that your next attempt will be rejected. Asymptotically, the probability of needing to reject your sample should converge to zero, and in practice, often very rapidly.
As proposed, any time we choose a point that is rejected, we tighten the envelope with another line segment that is tangent to the curve at the point with the same x-coordinate as the chosen point.
A piecewise linear model of the proposal log distribution results in a set of piecewise exponential distributions (i.e. segments of one or more exponential distributions, attached end to end). Exponential distributions are well behaved and well understood. The logarithm of an exponential distribution is a straight line, and hence this method essentially involves enclosing the logarithm of the density in a series of line segments. This is the source of the log-concave restriction: if a distribution is log-concave, then its logarithm is concave (shaped like an upside-down U), meaning that a line segment tangent to the curve will always pass over the curve.
If not working in log space, a piecewise linear density function can also be sampled via triangle distributions

when your sample is accepted.

that we had to evaluate in the current chain of rejections, we can also construct a piecewise linear lower bound (the "squeezing" function) using these values as well.
to see if your sample will be accepted, we may already know in this case) squeezing function that have available.
This squeezing step is optional, even when suggested by Gilks. At best it saves you from only one extra evaluation of your (messy and/or expensive) target density. However, presumably for particularly expensive density functions (and assuming the rapid convergence of the rejection rate toward zero) this can make a sizable difference in ultimate runtime.

The method essentially involves successively determining an envelope of straight-line segments that approximates the logarithm better and better while still remaining above the curve, starting with a fixed number of segments (possibly just a single tangent line). Sampling from a truncated exponential random variable is straightforward. Just take the log of a uniform random variable (with appropriate interval and corresponding truncation).

Unfortunately, ARS can only be applied for sampling from log-concave target densities. For this reason, several extensions of ARS have been proposed in literature for tackling non-log-concave target distributions. Furthermore, different combinations of ARS and the Metropolis-Hastings method have been designed in order to obtain a universal sampler that builds a self-tuning proposal densities (i.e., a proposal automatically constructed and adapted to the target). This class of methods are often called as Adaptive Rejection Metropolis Sampling (ARMS) algorithms. The resulting adaptive techniques can be always applied but the generated samples are correlated in this case (although the correlation vanishes quickly to zero as the number of iterations grows).