World Library  
Flag as Inappropriate
Email this Article

Differential entropy

 

Differential entropy

Differential entropy (also referred to as continuous entropy) is a concept in information theory that extends the idea of (Shannon) entropy, a measure of average surprisal of a random variable, to continuous probability distributions.

Contents

  • Definition 1
  • Properties of differential entropy 2
  • Maximization in the normal distribution 3
  • Example: Exponential distribution 4
  • Differential entropies for various distributions 5
  • Variants 6
  • See also 7
  • References 8
  • External links 9

Definition

Let X be a random variable with a probability density function f whose support is a set \mathbb X. The differential entropy h(X) or h(f) is defined as

h(X) = -\int_\mathbb{X} f(x)\log f(x)\,dx.

For probability distributions which don't have an explicit density function expression, but have an explicit quantile function expression, Q(p), then h(Q) can be defined in terms of the derivative of Q(p) i.e. the quantile density function Q'(p) as [1]

h(Q) = \int_0^1 \log Q'(p)\,dp.

As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2 (i.e., the units are bits). See logarithmic units for logarithms taken in different bases. Related concepts such as joint, conditional differential entropy, and relative entropy are defined in a similar fashion. Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure X.[2] For example, the differential entropy of a quantity measured in millimeters will be log(1000) more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of log(1000) more than the same quantity divided by 1000.

One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, Uniform(0,1/2) has negative differential entropy

\int_0^\frac{1}{2} -2\log(2)\,dx=-\log(2)\,.

Thus, differential entropy does not share all properties of discrete entropy.

Note that the continuous mutual information I(X;Y) has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of partitions of X and Y as these partitions become finer and finer. Thus it is invariant under non-linear homeomorphisms (continuous and uniquely invertible maps) ,[3] including linear [4] transformations of X and Y, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.

Properties of differential entropy

h(X_1, \ldots, X_n) = \sum_{i=1}^{n} h(X_i|X_1, \ldots, X_{i-1}) \leq \sum h(X_i).
  • Differential entropy is translation invariant, i.e., h(X + c) = h(X) for a constant c.
  • Differential entropy is in general not invariant under arbitrary invertible maps. In particular, for a constant a, h(aX) = h(X) + log|a|. For a vector valued random variable X and a matrix A, h(A X) = h(X) + log|det(A)|.
  • In general, for a transformation from a random vector to another random vector with same dimension Y = m(X), the corresponding entropies are related via
h(\mathbf{Y}) \leq h(\mathbf{X}) + \int f(x) \log \left\vert \frac{\partial m}{\partial x} \right\vert dx
where \left\vert \frac{\partial m}{\partial x} \right\vert is the Jacobian of the transformation m. The above inequality becomes an equality if the transform is a bijection. Furthermore, when m is a rigid rotation, translation, or combination thereof, the Jacobian determinant is always 1, and h(Y) = h(X).
  • If a random vector X in Rn has mean zero and covariance matrix K, h(\mathbf{X}) \leq \frac{1}{2} \log[(2\pi e)^n \det{K}] with equality if and only if X is jointly gaussian (see below).

However, differential entropy does not have other desirable properties:

  • It is not invariant under change of variables, and is therefore most useful with dimensionless variables.
  • It can be negative.

A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points).

Maximization in the normal distribution

With a normal distribution, differential entropy is maximized for a given variance. The following is a proof that a Gaussian variable has the largest entropy amongst all random variables of equal variance, or, alternatively, that the maximum entropy distribution under constraints of mean and variance is the Gaussian.

Let g(x) be a Gaussian PDF with mean μ and variance σ2 and f(x) an arbitrary PDF with the same variance. Since differential entropy is translation invariant we can assume that f(x) has the same mean of μ as g(x).

Consider the Kullback–Leibler divergence between the two distributions

0 \leq D_{KL}(f || g) = \int_{-\infty}^\infty f(x) \log \left( \frac{f(x)}{g(x)} \right) dx = -h(f) - \int_{-\infty}^\infty f(x)\log(g(x)) dx.

Now note that

\begin{align} \int_{-\infty}^\infty f(x)\log(g(x)) dx &= \int_{-\infty}^\infty f(x)\log\left( \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\right) dx \\ &= \int_{-\infty}^\infty f(x) \log\frac{1}{\sqrt{2\pi\sigma^2}} dx + \log(e)\int_{-\infty}^\infty f(x)\left( -\frac{(x-\mu)^2}{2\sigma^2}\right) dx \\ &= -\tfrac{1}{2}\log(2\pi\sigma^2) - \log(e)\frac{\sigma^2}{2\sigma^2} \\ &= -\tfrac{1}{2}\left(\log(2\pi\sigma^2) + \log(e)\right) \\ &= -\tfrac{1}{2}\log(2\pi e \sigma^2) \\ &= -h(g) \end{align}

because the result does not depend on f(x) other than through the variance. Combining the two results yields

h(g) - h(f) \geq 0 \!

with equality when g(x) = f(x) following from the properties of Kullback–Leibler divergence.

This result may also be demonstrated using the variational calculus. A Lagrangian function with two Lagrangian multipliers may be defined as:

L=\int_{-\infty}^\infty g(x)\ln(g(x))\,dx-\lambda_0\left(1-\int_{-\infty}^\infty g(x)\,dx\right)-\lambda\left(\sigma^2-\int_{-\infty}^\infty g(x)(x-\mu)^2\,dx\right)

where g(x) is some function with mean μ. When the entropy of g(x) is at a maximum and the constraint equations, which consist of the normalization condition \left(1=\int_{-\infty}^\infty g(x)\,dx\right) and the requirement of fixed variance \left(\sigma^2=\int_{-\infty}^\infty g(x)(x-\mu)^2\,dx\right), are both satisfied, then a small variation δg(x) about g(x) will produce a variation δL about L which is equal to zero:

0=\delta L=\int_{-\infty}^\infty \delta g(x)\left (\ln(g(x))+1+\lambda_0+\lambda(x-\mu)^2\right )\,dx

Since this must hold for any small δg(x), the term in brackets must be zero, and solving for g(x) yields:

g(x)=e^{-\lambda_0-1-\lambda(x-\mu)^2}

Using the constraint equations to solve for λ0 and λ yields the normal distribution:

g(x)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}

Example: Exponential distribution

Let X be an exponentially distributed random variable with parameter λ, that is, with probability density function

f(x) = \lambda e^{-\lambda x} \mbox{ for } x \geq 0.

Its differential entropy is then

h_e(X)\, =-\int_0^\infty \lambda e^{-\lambda x} \log (\lambda e^{-\lambda x})\,dx
= -\left(\int_0^\infty (\log \lambda)\lambda e^{-\lambda x}\,dx + \int_0^\infty (-\lambda x) \lambda e^{-\lambda x}\,dx\right)
= -\log \lambda \int_0^\infty f(x)\,dx + \lambda E[X]
= -\log\lambda + 1\,.

Here, h_e(X) was used rather than h(X) to make it explicit that the logarithm was taken to base e, to simplify the calculation.

Differential entropies for various distributions

In the table below \Gamma(x) = \int_0^{\infty} e^{-t} t^{x-1} dt is the gamma function, \psi(x) = \frac{d}{dx} \ln\Gamma(x)=\frac{\Gamma'(x)}{\Gamma(x)} is the digamma function, B(p,q) = \frac{\Gamma(p)\Gamma(q)}{\Gamma(p+q)} is the beta function, and γE is Euler's constant.[5]

Table of differential entropies
Distribution Name Probability density function (pdf) Entropy in nats Support
Uniform f(x) = \frac{1}{b-a} \ln(b - a) \, [a,b]\,
Normal f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) \ln\left(\sigma\sqrt{2\,\pi\,e}\right) (-\infty,\infty)\,
Exponential f(x) = \lambda \exp\left(-\lambda x\right) 1 - \ln \lambda \, [0,\infty)\,
Rayleigh f(x) = \frac{x}{\sigma^2} \exp\left(-\frac{x^2}{2\sigma^2}\right) 1 + \ln \frac{\sigma}{\sqrt{2}} + \frac{\gamma_E}{2} [0,\infty)\,
Beta f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)} for 0 \leq x \leq 1 \ln B(\alpha,\beta) - (\alpha-1)[\psi(\alpha) - \psi(\alpha +\beta)]\,
- (\beta-1)[\psi(\beta) - \psi(\alpha + \beta)] \,
[0,1]\,
Cauchy f(x) = \frac{\gamma}{\pi} \frac{1}{\gamma^2 + x^2} \ln(4\pi\gamma) \, (-\infty,\infty)\,
Chi f(x) = \frac{2}{2^{k/2} \Gamma(k/2)} x^{k-1} \exp\left(-\frac{x^2}{2}\right) \ln{\frac{\Gamma(k/2)}{\sqrt{2}}} - \frac{k-1}{2} \psi\left(\frac{k}{2}\right) + \frac{k}{2} [0,\infty)\,
Chi-squared f(x) = \frac{1}{2^{k/2} \Gamma(k/2)} x^{\frac{k}{2}\!-\!1} \exp\left(-\frac{x}{2}\right) \ln 2\Gamma\left(\frac{k}{2}\right) - \left(1 - \frac{k}{2}\right)\psi\left(\frac{k}{2}\right) + \frac{k}{2} [0,\infty)\,
Erlang f(x) = \frac{\lambda^k}{(k-1)!} x^{k-1} \exp(-\lambda x) (1-k)\psi(k) + \ln \frac{\Gamma(k)}{\lambda} + k [0,\infty)\,
F f(x) = \frac{n_1^{\frac{n_1}{2}} n_2^{\frac{n_2}{2}}}{B(\frac{n_1}{2},\frac{n_2}{2})} \frac{x^{\frac{n_1}{2} - 1}}{(n_2 + n_1 x)^{\frac{n_1 + n2}{2}}} \ln \frac{n_1}{n_2} B\left(\frac{n_1}{2},\frac{n_2}{2}\right) + \left(1 - \frac{n_1}{2}\right) \psi\left(\frac{n_1}{2}\right) -
\left(1 + \frac{n_2}{2}\right)\psi\left(\frac{n_2}{2}\right) + \frac{n_1 + n_2}{2} \psi\left(\frac{n_1\!+\!n_2}{2}\right)
[0,\infty)\,
Gamma f(x) = \frac{x^{k - 1} \exp(-\frac{x}{\theta})}{\theta^k \Gamma(k)} \ln(\theta \Gamma(k)) + (1 - k)\psi(k) + k \, [0,\infty)\,
Laplace f(x) = \frac{1}{2b} \exp\left(-\frac{|x - \mu|}{b}\right) 1 + \ln(2b) \, (-\infty,\infty)\,
Logistic f(x) = \frac{e^{-x}}{(1 + e^{-x})^2} 2 \, (-\infty,\infty)\,
Lognormal f(x) = \frac{1}{\sigma x \sqrt{2\pi}} \exp\left(-\frac{(\ln x - \mu)^2}{2\sigma^2}\right) \mu + \frac{1}{2} \ln(2\pi e \sigma^2) [0,\infty)\,
Maxwell–Boltzmann f(x) = \frac{1}{a^3}\sqrt{\frac{2}{\pi}}\,x^{2}\exp\left(-\frac{x^2}{2a^2}\right) \ln(a\sqrt{2\pi})+\gamma_E-\frac{1}{2} [0,\infty)\,
Generalized normal f(x) = \frac{2 \beta^{\frac{\alpha}{2}}}{\Gamma(\frac{\alpha}{2})} x^{\alpha - 1} \exp(-\beta x^2) \ln{\frac{\Gamma(\alpha/2)}{2\beta^{\frac{1}{2}}}} - \frac{\alpha - 1}{2} \psi\left(\frac{\alpha}{2}\right) + \frac{\alpha}{2} (-\infty,\infty)\,
Pareto f(x) = \frac{\alpha x_m^\alpha}{x^{\alpha+1}} \ln \frac{x_m}{\alpha} + 1 + \frac{1}{\alpha} [x_m,\infty)\,
Student's t f(x) = \frac{(1 + x^2/\nu)^{-\frac{\nu+1}{2}}}{\sqrt{\nu}B(\frac{1}{2},\frac{\nu}{2})} \frac{\nu\!+\!1}{2}\left(\psi\left(\frac{\nu\!+\!1}{2}\right)\!-\!\psi\left(\frac{\nu}{2}\right)\right)\!+\!\ln \sqrt{\nu} B\left(\frac{1}{2},\frac{\nu}{2}\right) (-\infty,\infty)\,
Triangular f(x) = \begin{cases} \frac{2(x-a)}{(b-a)(c-a)} & \mathrm{for\ } a \le x \leq c, \\[4pt] \frac{2(b-x)}{(b-a)(b-c)} & \mathrm{for\ } c < x \le b, \\[4pt] \end{cases} \frac{1}{2} + \ln \frac{b-a}{2} [0,1]\,
Weibull f(x) = \frac{k}{\lambda^k} x^{k-1} \exp\left(-\frac{x^k}{\lambda^k}\right) \frac{(k-1)\gamma_E}{k} + \ln \frac{\lambda}{k} + 1 [0,\infty)\,
Multivariate normal f_X(\vec{x}) =
\frac{\exp \left( -\frac{1}{2} ( \vec{x} - \vec{\mu})^\top \Sigma^{-1}\cdot(\vec{x} - \vec{\mu}) \right)} {(2\pi)^{N/2} \left|\Sigma\right|^{1/2}}
\frac{1}{2}\ln\{(2\pi e)^{N} \det(\Sigma)\} (-\vec{\infty},\vec{\infty})\,

(Many of the differential entropies are from.[6]

Variants

As described above, differential entropy does not share all properties of discrete entropy. For example, the differential entropy can be negative; also it is not invariant under continuous coordinate transformations. Edwin Thompson Jaynes showed in fact that the expression above is not the correct limit of the expression for a finite set of probabilities.[7]

A modification of differential entropy adds an invariant measure factor to correct this, (see limiting density of discrete points). If m(x) is further constrained to be a probability density, the resulting notion is called relative entropy in information theory:

D(p||m) = \int p(x)\log\frac{p(x)}{m(x)}\,dx.

The definition of differential entropy above can be obtained by partitioning the range of X into bins of length h with associated sample points ih within the bins, for X Riemann integrable. This gives a quantized version of X, defined by Xh = ih if ihX ≤ (i+1)h. Then the entropy of Xh is

H_h=-\sum_i hf(ih)\log (f(ih)) - \sum hf(ih)\log(h).

The first term on the right approximates the differential entropy, while the second term is approximately −log(h). Note that this procedure suggests that the entropy in the discrete sense of a continuous random variable should be ∞.

See also

References

  1. ^ Vasicek, Oldrich (1976), "A Test for Normality Based on Sample Entropy",  
  2. ^ Pages 183-184,  
  3. ^ Kraskov, Alexander; Stögbauer, Grassberger (2004). "Estimating mutual information". Phys. Rev. E 60: 066138.  
  4. ^ Fazlollah M. Reza (1994) [1961]. An Introduction to Information Theory. Dover Publications, Inc., New York.  
  5. ^ Park, Sung Y.; Bera, Anil K. (2009). "Maximum entropy autoregressive conditional heteroskedasticity model" (PDF). Journal of Econometrics (Elsevier): 219–230. Retrieved 2011-06-02. 
  6. ^ Lazo, A. and P. Rathie (1978). "On the entropy of continuous probability distributions". Information Theory, IEEE Transactions on. 24(1): 120–122.  
  7. ^  
  • Thomas M. Cover, Joy A. Thomas. Elements of Information Theory New York: Wiley, 1991. ISBN 0-471-06259-6

External links

  • Hazewinkel, Michiel, ed. (2001), "Differential entropy",  
  • Differential entropy at PlanetMath.org.
This article was sourced from Creative Commons Attribution-ShareAlike License; additional terms may apply. World Heritage Encyclopedia content is assembled from numerous content providers, Open Access Publishing, and in compliance with The Fair Access to Science and Technology Research Act (FASTR), Wikimedia Foundation, Inc., Public Library of Science, The Encyclopedia of Life, Open Book Publishers (OBP), PubMed, U.S. National Library of Medicine, National Center for Biotechnology Information, U.S. National Library of Medicine, National Institutes of Health (NIH), U.S. Department of Health & Human Services, and USA.gov, which sources content from all federal, state, local, tribal, and territorial government publication portals (.gov, .mil, .edu). Funding for USA.gov and content contributors is made possible from the U.S. Congress, E-Government Act of 2002.
 
Crowd sourced content that is contributed to World Heritage Encyclopedia is peer reviewed and edited by our editorial staff to ensure quality scholarly research articles.
 
By using this site, you agree to the Terms of Use and Privacy Policy. World Heritage Encyclopedia™ is a registered trademark of the World Public Library Association, a non-profit organization.
 


Copyright © World Library Foundation. All rights reserved. eBooks from Project Gutenberg are sponsored by the World Library Foundation,
a 501c(4) Member's Support Non-Profit Organization, and is NOT affiliated with any governmental agency or department.