Wikipedia / F-divergence
Wikipedia / F-divergence
In probability theory-divergence that measures the difference between two probability distributions. Many common divergences, such as KL-divergence, Hellinger distance, and total variation distance-divergence.
History
These divergences were introduced by Alfréd Rényi in the same paper where he introduced the well-known Rényi entropy. He proved that these divergences decrease in Markov processeses. f-divergences were studied further independently by , and -divergences, Csiszár–Morimoto divergences, or Ali–Silvey distances.
Definition
Non-singular case
is absolutely continuous). Then, for a convex function is defined as
.
, the reference distribution is the Lebesgue measure, then we can use Radon–Nikodym theorem to take their probability densities, giving
, and proceed as above. This is a useful technique in more abstract proofs.
Extension to singular measuressingular measures
is no longer satisfied (Definition 7.1 of ).
.
.
Properties
Basic relations between f-divergences
.
.
Basic properties of f-divergences
In particular, the monotonicity implies that if a Markov process is a solution of the Kolmogorov forward equations (or Master equation), used to describe the time evolution of the probability distribution in the Markov process. This means that all f are the Lyapunov functions, for some convex function f. For example, Bregman divergences in general do not have such property and can increase in Markov processes.
Analytic properties
The f-divergences can be expressed using Taylor series and rewritten using a weighted sum of chi-type distances ().
Basic variational representation
be the convex conjugate be the effective domain, which we describe below.
Under the above setup,
This is Theorem 7.24 in.
Example applications
varies is affine-invariant, we can use up the affine-invariance to obtain a leaner expression.
which is just a few steps away from the Hammersley–Chapman–Robbins bound and the Cramér–Rao bound (Theorem 29.1 and its corollary in ).
.
-divergence obtained above.
.
varies is merely scale invariant This is strictly less efficient than the Donsker–Varadhan representation This defect is fixed by the next theorem.
Improved variational representation
Assume the setup in the beginning of this section ("Variational representations").
This is Theorem 7.25 in.
Example applications
Applying this theorem to KL-divergence yields the Donsker–Varadhan representation.
does not yield a closed-form solution.
Common examples of f-divergences
-divergences.
, its generating function. This freedom is not only convenient, but actually necessary.. In particular, this shows that the squared Hellinger distance and Jensen-Shannon divergence are symmetric.
-divergences are sometimes parametrized as
.
Relations to other statistical divergences
Here, we compare f-divergences with other statistical divergences.
Rényi divergence
The Rényi divergences is a family of divergences defined by
by taking the limit.
-divergence defined above.
Bregman divergence
The only f-divergence that is also a Bregman divergence is the KL divergence.
Integral probability metrics
The only f-divergence that is also an integral probability metric is the total variation.
Financial interpretation
A pair of probability distributions can be viewed as a game of chance in which one of the distributions defines the official odds and the other contains the actual probabilities. Knowledge of the actual probabilities allows a player to profit from the game. For a large class of rational players the expected profit rate has the same general form as the ƒ-divergence.