Cross-entropy on arvau

Wikipedia / Cross-entropy

In information theory, the cross-entropy between two probability distributions, over the same underlying set of events, measures the average number of bits.

Definition

over a given set is defined as follows:

is the expected value.

The definition may be formulated using the Kullback–Leibler divergence (also known as the relative entropy).

is the entropy.

For discrete with the same support, this means

The situation for continuous are absolutely continuous with respect to some reference measure is a Lebesgue measure on a Borel σ-algebra be probability density functions. Then

and therefore

is also used for a different concept, the joint entropy.

Motivation

In information theory, the Kraft–McMillan theorem is

Estimation

is unknown. An example is language modeling is the distribution of words as predicted by the model. Since the true distribution is unknown, cross-entropy cannot be directly calculated. In these cases, an estimate of cross-entropy is calculated using the following formula:

words of the test. This is a Monte Carlo estimate.

Relation to maximum likelihood

The cross entropy arises in classification problems when introducing a logarithm in the guise of the log-likelihood function.

from a training set, obtained from conditionally independent, by the law of large numbers.

where we have used the calculation rules for the logarithm Since the logarithm is a monotonically increasing function. So we observe that the likelihood maximization amounts to minimization of the cross-entropy.

Cross-entropy minimization

, cross-entropy and KL divergence is fixed): According to the Gibbs' inequality for cross-entropy. In the engineering literature, the principle of minimizing KL divergence (Kullback's "Principle of Minimum Discrimination Information") is often called the Principle of Minimum Cross-Entropy (MCE), or Minxent.

However, as discussed in the article Kullback–Leibler divergence as possible, subject to some constraint. In this case the two minimizations are not. In fact, cross-entropy is another name for relative entropy; see Cover and Thomas and Good. does not agree with the literature and can be misleading.

Cross-entropy loss function and logistic regression

Cross-entropy can be used to define a loss function in machine learning and optimization. Mao, Mohri, and Zhong (2023) give an extensive analysis of the properties of the family of cross-entropy loss functions in machine learning, including theoretical learning guarantees and extensions to adversarial learning. is the predicted value of the current model. This is also known as the log loss (or logarithmic loss or logistic loss); the terms "log loss" and "cross-entropy loss" are used interchangeably.

More specifically, consider a binary regression, can be interpreted as a probability, which serves as the basis for classifying the observation. In logistic regression, the probability is modeled using the logistic function is optimized through some appropriate algorithm such as gradient descent

Logistic regression typically optimizes the log loss for all the observations on which it is trained, which is the same as optimizing the average cross-entropy in the sample. Other loss functions that penalize errors differently can be also used for training, resulting in models with different final test accuracy.. The average of the loss function is then given by

as the logistic function as before.

Relation to linear regression

The gradient of the cross-entropy loss for logistic regression is equal to the gradient of the squared-error loss for linear regression (up to a constant factor). To see this, define

Then we have the result

Proof: we have

Amended cross-entropy

It may be beneficial to train an ensemble of models that have diversity, such that when they are combined, their predictive accuracy is augmented. we would like the classifier to be as diverse as possible.