Wikipedia / Conditional expectation
Wikipedia / Conditional expectation
In probability theory, the conditional expectation, conditional expected value, or conditional mean of a random variable is its expected value evaluated with respect to the conditional probability distribution. If the random variable can take on only a finite number of values, the "conditions" are that the variable can only take on a subset of those values. More formally, in the case when the random variable is defined over a discrete probability space, the "conditions" are a partition of this probability space.
analogously to conditional probability.
Examples
Example 1: Dice rolling
Consider the roll of a fair dice and let A = 1 if the number is even (i.e., 2, 4, or 6) and A = 0 otherwise. Furthermore, let B = 1 if the number is prime (i.e., 2, 3, or 5) and B = 0 otherwise., but the expectation of A conditional on B, and the expectation of A conditional on B, and the expectation of B conditional on A.
Example 2: Rainfall data
Suppose we have daily rainfall data (mm of rain each day) collected by a weather station on every day of the ten-year (3652-day) period from January 1, 1990, to December 31, 1999. The unconditional expectation of rainfall for an unspecified day is the average of the rainfall amounts for those 3652 days. The conditional expectation of rainfall for an otherwise unspecified day known to be (conditional on being) in the month of March, is the average of daily rainfall over all 310 days of the ten–year period that fall in March. Similarly, the conditional expectation of rainfall conditional on days dated March 2 is the average of the rainfall amounts that occurred on the ten days with that specific date.
History
The related concept of conditional probability dates back at least to Laplace, who calculated conditional distributions. It was Andrey Kolmogorov who, in 1933, formalized it using the Radon–Nikodym theorem. In works of Paul Halmos and Joseph L. Doob from 1953, conditional expectation was generalized to its modern definition using sub-σ-algebras.
Definitions
Conditioning on an event
If with nonzero probability, and is a discrete random variable, the conditional expectation of given is
where the sum is taken over all possible outcomes of .
, the conditional expectation is undefined due to the division by zero.
Discrete random variables
If and are discrete random variables, the conditional expectation of given is
is the joint probability mass function of and . The sum is taken over all possible outcomes of .
.
Conditioning on a discrete random variable is the same as conditioning on the corresponding event:
where .
Continuous random variables
be continuous random variables is
When the denominator is zero, the expression is undefined.
as it was in the discrete case. For a discussion, see Conditioning on an event of probability zero. Not respecting this distinction can lead to contradictory conclusions as illustrated by the Borel-Kolmogorov paradox.
L2 random variables
, that is square integrable. In its full generality, conditional expectation is developed without this assumption, see below under Conditional expectation with respect to a sub-σ-algebra theory is, however, considered more intuitive and admits important generalizations random variables, conditional expectation is also called regression.
and variance minimizes the mean squared error:
The conditional expectation of be a random vector is a measurable function such that
is not generally unique: there may be multiple minimizers of the mean squared error.
Uniqueness
Example 1: Consider the case where is the constant random variable that is always 1. Then the mean squared error is minimized by any function of the form
Example 2: Consider the case where . Then clearly
or infinitely many other ways. In the context of linear regression, this lack of uniqueness is called multicollinearity.
. The measure used is the pushforward measure induced by .
In the first example, the pushforward measure is a Dirac distribution, so that any set not intersecting it has measure 0.
Existence
is non-trivial. It can be shown that
. By the Hilbert projection theorem, the necessary and sufficient condition in we have
In words, this equation says that the residual is orthogonal to the space of all functions of . This orthogonality condition, applied to the indicator functions, is used below to extend conditional expectation to the case that and .
Connections to regression
The conditional expectation is often approximated in applied mathematics and statistics due to the difficulties in analytically calculating it, and for interpolation.
The Hilbert subspace
defined above is replaced with subsets thereof by restricting the functional form of , rather than allowing any measurable function. Examples of this are decision tree regression when is required to be a simple function, linear regression when is required to be affine, etc.
These generalizations of conditional expectation come at the cost of many of its properties no longer holding. For example, let be the space of all linear functions of does not contain the constant functions, the tower property will not hold.
An important special case is when and are jointly normally distributed. In this case it can be shown that the conditional expectation is equivalent to linear regression:
described in Multivariate normal distribution#Conditional distributions.
Conditional expectation with respect to a sub-σ-algebra
Consider the following:
is a probability space.
is a random variable on that probability space with finite expectation.
is a sub-σ-algebra.
with the help of the conditional expectation.
A conditional expectation of X-measurable function which satisfies:
.
discussion, this condition is equivalent to saying that the residual:
Existence
is a finite measure that is absolutely continuous is the natural injection, because the condition
implies
Thus, we have
where the derivatives are Radon–Nikodym derivatives of measures.
Conditional expectation with respect to a random variable
Consider, in addition to the above,
A measurable space, and
.
The conditional expectation of given is defined by applying the above construction on the σ-algebra generated by :
By the Doob–Dynkin lemma such that
Discussion
This is not a constructive definition; we are merely given the required property that a conditional expectation must satisfy.
.
Uniqueness can be shown to be almost sure: that is, versions of the same conditional expectation will only differ on a set of probability zero.
may have a different null set. Because countable unions of null sets are null sets, for a countable set with a countable set of them. This directly corresponds to the conditions for creating a regular probability measure, which are separability and completeness.
The σ over a finer (larger) σ retains information about the probabilities of a larger class of events. A conditional expectation over a coarser (smaller) σ-algebra averages over more events.
Conditional probability
For a Borel subset , one can consider the collection of random variables
It can be shown that they form a Markov kernel is a probability measure.
The Law of the unconscious statistician is then
This shows that conditional expectations are, like their unconditional counterparts, integrations, against a conditional measure.
General Definition
In full generality, consider:
.
A Banach space.
A sub-σ.
The conditional expectation satisfying
.
.
Basic properties
All the following formulas are to be understood in an almost sure sense.
Pulling out independent factors:
is independent.
, so we get that
.
.
Stability:
.
, or equivalently
-measurable (the former property holds by definition; the latter property is key here), from this one can show
In particular, for sub-σ. (Note this is different from the tower property below.)
If Z.
Pulling out known factors:
.
.
almost everywhere.
.
.
Combining the special case proved for simple functions, the definition of conditional expectation, and deploying the monotone convergence theorem:
If Z.
Tower property:
For sub-σ.
.
A special case is when Z.
.
.
.
.
.
Conditional variance: Using the conditional expectation we can define, by analogy with the definition of the variance as the mean square deviation from the average, the conditional variance
Martingale convergence is an increasing series of sub-σ is a decreasing series of sub-σ.
are in the Hilbert space of square-integrable real random variables (real random variables with finite second moment) then
is in the sense of the L2(P) scalar product the orthogonal projection to the linear subspace-measurable functions. (This allows to define and prove the existence of the conditional expectation based on the Hilbert projection theorem.)
is self-adjoint
Conditioning is a contractive projection of Lp for any p≥1.
Doob's conditional independence property: are conditionally independent).
See also
Probability laws
Law of total cumulance (generalizes the other three)
Notes
References
William Feller, An Introduction to Probability Theory and its Applications, vol 1, 1950, page 223
Paul A. Meyer, Probability and Potentials, Blaisdell Publishing Co., 1966, page 28
, pages 67–69
External links