In information theory, the cross entropy between two probability distributions measures the average number of bits needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution q, rather than the "true" distribution p.

The cross entropy for two distributions p and q over the same probability space is thus defined as follows:

\mathrm{H}(p, q) = \mathrm{E}_p[-\log q] = \mathrm{H}(p) + D_{\mathrm{KL}}(p \| q),\!

where \mathrm{H}(p) is the entropy of p, and D_{\mathrm{KL}}(p || q) is the Kullback-Leibler divergence of q from p (also known as the relative entropy).

For discrete p and q this means

\mathrm{H}(p, q) = -\sum_x p(x)\, \log q(x). \!

The situation for continuous distributions is analogous:

-\int_X p(x)\, \log q(x)\, dx. \!

NB: The notation \mathrm{H}(p,q) is sometimes used for both the cross entropy as well as the joint entropy of p and q.


In information theory, the Kraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value xi out of a set of possibilities X can be seen as representing an implicit probability distribution q(xi) = 2li over X, where li is the length of the code for xi in bits. Therefore, Cross entropy can be interpreted as the expected message-length per datum when a wrong distribution Q is assumed however the data actually follows a distribution P - that's why the expectation is taken over the probability distribution P and not Q.

\mathrm{H}(p, q) = \mathrm{E}_p[l_i] = \mathrm{E}_p[\log \frac{1}{q(x_i)}]
\mathrm{H}(p, q) = \sum_{x_i} p(x_i)\, \log \frac{1}{q(x_i)} \!
\mathrm{H}(p, q) = -\sum_x p(x)\, \log q(x). \!


There are many situations where cross-entropy needs to be measured but the distribution of p is unknown. An example is language modeling, where a model is created based on a training set T, and then its cross-entropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example, p is the true distribution of words in any corpus, and q is the distribution of words as predicted by the model. Since the true distribution is unknown, cross-entropy cannot be directly calculated. In these cases, an estimate of cross-entropy is calculated using the following formula:

H(T,q) = -\sum_{i=1}^N \frac{1}{N} \log_2 q(x_i)

where N is the size of the test set, and q(x) is the probability of event x estimated from the training set. The sum is calculated over N. This is a Monte Carlo estimate of the true cross entropy, where the training set is treated as samples from p(x).

Cross-entropy minimization

Cross-entropy minimization is frequently used in optimization and rare-event probability estimation; see the cross-entropy method.

When comparing a distribution q against a fixed reference distribution p, cross entropy and KL divergence are identical up to an additive constant (since p is fixed): both take on their minimal values when p = q, which is 0 for KL divergence, and \mathrm{H}(p) for cross entropy. In the engineering literature, the principle of minimising KL Divergence (Kullback's "Principle of Minimum Discrimination Information") is often called the Principle of Minimum Cross-Entropy (MCE), or Minxent.

However, as discussed in the article Kullback-Leibler divergence, sometimes the distribution q is the fixed prior reference distribution, and the distribution p is optimised to be as close to q as possible, subject to some constraint. In this case the two minimisations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be DKL(p||q), rather than H(p,q).

See also

External links

  • What is cross-entropy, and why use it?
This article was sourced from Creative Commons Attribution-ShareAlike License; additional terms may apply. World Heritage Encyclopedia content is assembled from numerous content providers, Open Access Publishing, and in compliance with The Fair Access to Science and Technology Research Act (FASTR), Wikimedia Foundation, Inc., Public Library of Science, The Encyclopedia of Life, Open Book Publishers (OBP), PubMed, U.S. National Library of Medicine, National Center for Biotechnology Information, U.S. National Library of Medicine, National Institutes of Health (NIH), U.S. Department of Health & Human Services, and USA.gov, which sources content from all federal, state, local, tribal, and territorial government publication portals (.gov, .mil, .edu). Funding for USA.gov and content contributors is made possible from the U.S. Congress, E-Government Act of 2002.
Crowd sourced content that is contributed to World Heritage Encyclopedia is peer reviewed and edited by our editorial staff to ensure quality scholarly research articles.
By using this site, you agree to the Terms of Use and Privacy Policy. World Heritage Encyclopedia™ is a registered trademark of the World Public Library Association, a non-profit organization.