World Library  
Flag as Inappropriate
Email this Article

Fisher information

Article Id: WHEBN0000598971
Reproduction Date:

Title: Fisher information  
Author: World Heritage Encyclopedia
Language: English
Subject: Beta distribution, Optimal design, Maximum likelihood, Cramér–Rao bound, Founders of statistics
Collection: Design of Experiments, Estimation Theory, Information Theory
Publisher: World Heritage Encyclopedia
Publication
Date:
 

Fisher information

Ronald Fisher
Francis Ysidro Edgeworth

In mathematical statistics, the Fisher information (sometimes simply called information[1]) is a way of measuring the amount of information that an observable random variable X carries about an unknown parameter θ of a distribution that models X. Formally, it is the variance of the score, or the expected value of the observed information. In Bayesian statistics, the asymptotic distribution of the posterior mode depends on the Fisher information and not on the prior (according to the Bernstein–von Mises theorem, which was anticipated by Laplace for exponential families).[2] The role of the Fisher information in the asymptotic theory of maximum-likelihood estimation was emphasized by the statistician Ronald Fisher (following some initial results by Francis Ysidro Edgeworth). The Fisher information is also used in the calculation of the Jeffreys prior, which is used in Bayesian statistics.

The Fisher-information matrix is used to calculate the covariance matrices associated with maximum-likelihood estimates. It can also be used in the formulation of test statistics, such as the Wald test.

Statistical systems of a scientific nature (physical, biological, etc.) whose likelihood functions obey shift invariance have been shown to obey maximum Fisher information.[3] The level of the maximum depends upon the nature of the system constraints.

Contents

  • Definition 1
    • Informal derivation of the Cramér–Rao bound 1.1
    • Single-parameter Bernoulli experiment 1.2
  • Matrix form 2
    • Orthogonal parameters 2.1
    • Multivariate normal distribution 2.2
  • Properties 3
    • Reparametrization 3.1
  • Applications 4
    • Optimal design of experiments 4.1
    • Jeffreys prior in Bayesian statistics 4.2
    • Computational neuroscience 4.3
  • Relation to relative entropy 5
  • History 6
  • See also 7
  • Notes 8
  • References 9

Definition

The Fisher information is a way of measuring the amount of information that an observable random variable X carries about an unknown parameter θ upon which the probability of X depends. The probability function for X, which is also the likelihood function for θ, is a function f(X; θ); it is the probability mass (or probability density) of the random variable X conditional on the value of θ. The partial derivative with respect to θ of the natural logarithm of the likelihood function is called the score.

Under certain regularity conditions,[4] it can be shown that the first moment of the score (that is, its expected value) is 0:

\operatorname{E} \left[\left. \frac{\partial}{\partial\theta} \log f(X;\theta)\right|\theta \right] = \operatorname{E} \left[\left. \frac{\frac{\partial}{\partial\theta} f(X;\theta)}{f(X; \theta)}\right|\theta \right] = \int \frac{\frac{\partial}{\partial\theta} f(x;\theta)}{f(x; \theta)} f(x;\theta)\; \mathrm{d}x =
= \int \frac{\partial}{\partial\theta} f(x;\theta)\; \mathrm{d}x = \frac{\partial}{\partial\theta} \int f(x; \theta)\; \mathrm{d}x = \frac{\partial}{\partial\theta} \; 1 = 0.

The second moment is called the Fisher information:

\mathcal{I}(\theta)=\operatorname{E} \left[\left. \left(\frac{\partial}{\partial\theta} \log f(X;\theta)\right)^2\right|\theta \right] = \int \left(\frac{\partial}{\partial\theta} \log f(x;\theta)\right)^2 f(x; \theta)\; \mathrm{d}x\,,

where, for any given value of θ, the expression E[...|θ] denotes the conditional expectation over values for X with respect to the probability function f(x; θ) given θ. Note that 0 \leq \mathcal{I}(\theta) < \infty. A random variable carrying high Fisher information implies that the absolute value of the score is often high. The Fisher information is not a function of a particular observation, as the random variable X has been averaged out.

Since the expectation of the score is zero, the Fisher information is also the variance of the score.

If log f(x; θ) is twice differentiable with respect to θ, and under certain regularity conditions, then the Fisher information may also be written as[5]

\mathcal{I}(\theta) = - \operatorname{E} \left[\left. \frac{\partial^2}{\partial\theta^2} \log f(X;\theta)\right|\theta \right]\,,

since

\frac{\partial^2}{\partial\theta^2} \log f(X;\theta) = \frac{\frac{\partial^2}{\partial\theta^2} f(X;\theta)}{f(X; \theta)} \;-\; \left( \frac{\frac{\partial}{\partial\theta} f(X;\theta)}{f(X; \theta)} \right)^2 = \frac{\frac{\partial^2}{\partial\theta^2} f(X;\theta)}{f(X; \theta)} \;-\; \left( \frac{\partial}{\partial\theta} \log f(X;\theta)\right)^2

and

\operatorname{E} \left[\left. \frac{\frac{\partial^2}{\partial\theta^2} f(X;\theta)}{f(X; \theta)}\right|\theta \right] = \cdots = \frac{\partial^2}{\partial\theta^2} \int f(x; \theta)\; \mathrm{d}x = \frac{\partial^2}{\partial\theta^2} \; 1 = 0.

Thus, the Fisher information is the negative of the expectation of the second derivative with respect to θ of the natural logarithm of f. Information may be seen to be a measure of the "curvature" of the support curve near the maximum likelihood estimate of θ. A "blunt" support curve (one with a shallow maximum) would have a low negative expected second derivative, and thus low information; while a sharp one would have a high negative expected second derivative and thus high information.

Information is additive, in that the information yielded by two independent experiments is the sum of the information from each experiment separately:

\mathcal{I}_{X,Y}(\theta) = \mathcal{I}_X(\theta) + \mathcal{I}_Y(\theta).

This result follows from the elementary fact that if random variables are independent, the variance of their sum is the sum of their variances. In particular, the information in a random sample of size n is n times that in a sample of size 1, when observations are independent and identically distributed.

The information provided by a sufficient statistic is the same as that of the sample X. This may be seen by using Neyman's factorization criterion for a sufficient statistic. If T(X) is sufficient for θ, then

f(X;\theta) = g(T(X), \theta) h(X) \!

for some functions g and h. See sufficient statistic for a more detailed explanation. The equality of information then follows from the following fact:

\frac{\partial}{\partial\theta} \log \left[f(X ;\theta)\right] = \frac{\partial}{\partial\theta} \log \left[g(T(X);\theta)\right]

which follows from the definition of Fisher information, and the independence of h(X) from θ. More generally, if T = t(X) is a statistic, then

\mathcal{I}_T(\theta) \leq \mathcal{I}_X(\theta)

with equality if and only if T is a sufficient statistic.

Informal derivation of the Cramér–Rao bound

The Cramér–Rao bound states that the inverse of the Fisher information is a lower bound on the variance of any unbiased estimator of θ. H.L. Van Trees (1968) and B. Roy Frieden (2004) provide the following method of deriving the Cramér–Rao bound, a result which describes use of the Fisher information, informally:

Consider an unbiased estimator \hat\theta(X). Mathematically, we write

\operatorname{E}\left[ \left. \hat\theta(X) - \theta \right| \theta \right] = \int \left[ \hat\theta(x) - \theta \right] \cdot f(x ;\theta) \, \mathrm{d}x = 0.

The likelihood function f(X; θ) describes the probability that we observe a given sample x given a known value of θ. If f is sharply peaked with respect to changes in θ, it is easy to intuit the "correct" value of θ given the data, and hence the data contains a lot of information about the parameter. If the likelihood f is flat and spread-out, then it would take many, many samples of X to estimate the actual "true" value of θ. Therefore, we would intuit that the data contain much less information about the parameter.

Now, we use the product rule to differentiate the unbiased-ness condition above to get

\frac{\partial}{\partial\theta} \int \left[ \hat\theta(x) - \theta \right] \cdot f(x ;\theta) \, \mathrm{d}x = \int \left(\hat\theta(x)-\theta\right) \frac{\partial f}{\partial\theta} \, \mathrm{d}x - \int f \, \mathrm{d}x = 0.

We now make use of two facts. The first is that the likelihood f is just the probability of the data given the parameter. Since it is a probability, it must be normalized, implying that

\int f \, \mathrm{d}x = 1.

Second, we know from basic calculus that

\frac{\partial f}{\partial\theta} = f \, \frac{\partial \log f}{\partial\theta}.

Using these two facts in the above let us write

\int \left(\hat\theta-\theta\right) f \, \frac{\partial \log f}{\partial\theta} \, \mathrm{d}x = 1.

Factoring the integrand gives

\int \left(\left(\hat\theta-\theta\right) \sqrt{f} \right) \left( \sqrt{f} \, \frac{\partial \log f}{\partial\theta} \right) \, \mathrm{d}x = 1.

If we square the equation, the Cauchy–Schwarz inequality lets us write

\left[ \int \left(\hat\theta - \theta\right)^2 f \, \mathrm{d}x \right] \cdot \left[ \int \left( \frac{\partial \log f}{\partial\theta} \right)^2 f \, \mathrm{d}x \right] \geq 1.

The right-most factor is defined to be the Fisher Information

\mathcal{I}\left(\theta\right) = \int \left( \frac{\partial \log f}{\partial\theta} \right)^2 f \, \mathrm{d}x.

The left-most factor is the expected mean-squared error of the estimator θ^, since

\operatorname{E}\left[ \left. \left( \hat\theta\left(X\right) - \theta \right)^2 \right| \theta \right] = \int \left(\hat\theta - \theta\right)^2 f \, \mathrm{d}x.

Notice that the inequality tells us that, fundamentally,

\operatorname{Var}\left(\hat\theta\right) \, \geq \, \frac{1}{\mathcal{I}\left(\theta\right)}.

In other words, the precision to which we can estimate θ is fundamentally limited by the Fisher Information of the likelihood function.

Single-parameter Bernoulli experiment

A Bernoulli trial is a random variable with two possible outcomes, "success" and "failure", with success having a probability of θ. The outcome can be thought of as determined by a coin toss, with the probability of heads being θ and the probability of tails being 1 − θ.

The Fisher information contained in n independent Bernoulli trials may be calculated as follows. In the following, A represents the number of successes, B the number of failures, and n = A + B is the total number of trials.

\begin{align} \mathcal{I}(\theta) & = -\operatorname{E} \left[ \left. \frac{\partial^2}{\partial\theta^2} \log(f(A;\theta)) \right| \theta \right] \qquad (1) \\ & = -\operatorname{E} \left[ \left. \frac{\partial^2}{\partial\theta^2} \log \left( \theta^A(1-\theta)^B\frac{(A+B)!}{A!B!} \right) \right| \theta \right] \qquad (2) \\ & = -\operatorname{E} \left[ \left. \frac{\partial^2}{\partial\theta^2} \left( A \log (\theta) + B \log(1-\theta) \right) \right| \theta \right] \qquad (3) \\ & = -\operatorname{E} \left[ \left. \frac{\partial}{\partial\theta} \left( \frac{A}{\theta} - \frac{B}{1-\theta} \right) \right| \theta \right] \qquad (4) \\ & = +\operatorname{E} \left[ \left. \frac{A}{\theta^2} + \frac{B}{(1-\theta)^2} \right| \theta \right] \qquad (5) \\ & = \frac{n\theta}{\theta^2} + \frac{n(1-\theta)}{(1-\theta)^2} \qquad (6) \\ & \text{since the expected value of }A\text{ given }\theta\text{ is }n\theta,\text{ etc.} \\ & = \frac{n}{\theta(1-\theta)} \qquad (7) \end{align}

(1) defines Fisher information. (2) invokes the fact that the information in a sufficient statistic is the same as that of the sample itself. (3) expands the natural logarithm term and drops a constant. (4) and (5) differentiate with respect to θ. (6) replaces A and B with their expectations. (7) is algebra.

The end result, namely,

\mathcal{I}(\theta) = \frac{n}{\theta(1-\theta)},

is the reciprocal of the variance of the mean number of successes in n Bernoulli trials, as expected (see last sentence of the preceding section).

Matrix form

When there are N parameters, so that θ is a N × 1 vector \theta = \begin{bmatrix} \theta_{1}, \theta_{2}, \dots , \theta_{N} \end{bmatrix}^{\mathrm T}, then the Fisher information takes the form of an N × N matrix, the Fisher Information Matrix (FIM), with typical element

{\left(\mathcal{I} \left(\theta \right) \right)}_{i, j} = \operatorname{E} \left[\left. \left(\frac{\partial}{\partial\theta_i} \log f(X;\theta)\right) \left(\frac{\partial}{\partial\theta_j} \log f(X;\theta)\right) \right|\theta\right].

The FIM is a N × N positive semidefinite symmetric matrix, defining a Riemannian metric on the N-dimensional parameter space, thus connecting Fisher information to differential geometry. In that context, this metric is known as the Fisher information metric, and the topic is called information geometry.

Under certain regularity conditions, the Fisher Information Matrix may also be written as

{\left(\mathcal{I} \left(\theta \right) \right)}_{i, j} = - \operatorname{E} \left[\left. \frac{\partial^2}{\partial\theta_i \, \partial\theta_j} \log f(X;\theta) \right|\theta\right]\,.

The metric is interesting in several ways; it can be derived as the Hessian of the relative entropy; it can be understood as a metric induced from the Euclidean metric, after appropriate change of variable; in its complex-valued form, it is the Fubini–Study metric.

Orthogonal parameters

We say that two parameters θi and θj are orthogonal if the element of the ith row and jth column of the Fisher information matrix is zero. Orthogonal parameters are easy to deal with in the sense that their maximum likelihood estimates are independent and can be calculated separately. When dealing with research problems, it is very common for the researcher to invest some time searching for an orthogonal parametrization of the densities involved in the problem.

Multivariate normal distribution

The FIM for a N-variate multivariate normal distribution has a special form. Let \mu(\theta) = \begin{bmatrix} \mu_{1}(\theta), \mu_{2}(\theta), \dots , \mu_{N}(\theta) \end{bmatrix}^\mathrm{T}, and let Σ(θ) be the covariance matrix. Then the typical element \mathcal{I}_{m,n}, 0 ≤ m, n < K, of the FIM for XN(μ(θ), Σ(θ)) is:

\mathcal{I}_{m,n} = \frac{\partial \mu^\mathrm{T}}{\partial \theta_m} \Sigma^{-1} \frac{\partial \mu}{\partial \theta_n} + \frac{1}{2} \operatorname{tr} \left( \Sigma^{-1} \frac{\partial \Sigma}{\partial \theta_m} \Sigma^{-1} \frac{\partial \Sigma}{\partial \theta_n} \right),

where (..)^\mathrm{T} denotes the transpose of a vector, tr(..) denotes the trace of a square matrix, and:

  • \frac{\partial \mu}{\partial \theta_m} = \begin{bmatrix} \frac{\partial \mu_1}{\partial \theta_m} & \frac{\partial \mu_2}{\partial \theta_m} & \cdots & \frac{\partial \mu_N}{\partial \theta_m} \end{bmatrix}^\mathrm{T};
  • \frac{\partial \Sigma}{\partial \theta_m} = \begin{bmatrix} \frac{\partial \Sigma_{1,1}}{\partial \theta_m} & \frac{\partial \Sigma_{1,2}}{\partial \theta_m} & \cdots & \frac{\partial \Sigma_{1,N}}{\partial \theta_m} \\ \\ \frac{\partial \Sigma_{2,1}}{\partial \theta_m} & \frac{\partial \Sigma_{2,2}}{\partial \theta_m} & \cdots & \frac{\partial \Sigma_{2,N}}{\partial \theta_m} \\ \\ \vdots & \vdots & \ddots & \vdots \\ \\ \frac{\partial \Sigma_{N,1}}{\partial \theta_m} & \frac{\partial \Sigma_{N,2}}{\partial \theta_m} & \cdots & \frac{\partial \Sigma_{N,N}}{\partial \theta_m} \end{bmatrix}.

Note that a special, but very common, case is the one where Σ(θ) = Σ, a constant. Then

\mathcal{I}_{m,n} = \frac{\partial \mu^\mathrm{T}}{\partial \theta_m} \Sigma^{-1} \frac{\partial \mu}{\partial \theta_n}.\

In this case the Fisher information matrix may be identified with the coefficient matrix of the normal equations of least squares estimation theory.

Another special case is that the mean and covariance depends on two different vector parameters, say, β and θ. This is especially popular in the analysis of spatial data, which uses a linear model with correlated residuals. We have

\mathcal{I}\left( \beta ,\theta \right)=\text{diag}\left( \mathcal{I}\left( \beta \right),\mathcal{I}\left( \theta \right) \right)

where

\mathcal{I}=\frac{\partial }}{\partial }\frac{\partial \mu }{\partial },

\mathcal{I}=\frac{1}{2}\operatorname{tr}\left( \frac{\partial \Sigma }{\partial }\frac{\partial \Sigma }{\partial } \right)

The proof of this special case is given in literature.[6] Using the same technique in this paper, it's not difficult to prove the original result.

Properties

Reparametrization

The Fisher information depends on the parametrization of the problem. If θ and η are two scalar parametrizations of an estimation problem, and θ is a continuously differentiable function of η, then

{\mathcal I}_\eta(\eta) = {\mathcal I}_\theta(\theta(\eta)) \left( \frac_{\text{Fisher info.}}(\theta'-\theta)+\cdots

Thus the Fisher information represents the curvature of the relative entropy.

Schervish (1995: §2.3) says the following.

One advantage Kullback-Leibler information has over Fisher information is that it is not affected by changes in parameterization. Another advantage is that Kullback-Leibler information can be used even if the distributions under consideration are not all members of a parametric family. ...
Another advantage to Kullback-Leibler information is that no smoothness conditions on the densities … are needed.

History

The Fisher information was discussed by several early statisticians, notably F. Y. Edgeworth.[14] For example, Savage[15] says: "In it [Fisher information], he [Fisher] was to some extent anticipated (Edgeworth 1908–9 esp. 502, 507–8, 662, 677–8, 82–5 and references he [Edgeworth] cites including Pearson and Filon 1898 [. . .])." There are a number of early historical sources[16] and a number of reviews of this early work.[17][18][19]

See also

Other measures employed in information theory:

Notes

  1. ^ Lehmann & Casella, p. 115
  2. ^ Lucien Le Cam (1986) Asymptotic Methods in Statistical Decision Theory: Pages 336 and 618–621 (von Mises and Bernstein).
  3. ^ Frieden & Gatenby (2013)
  4. ^ Suba Rao. "Lectures on statistical inference" (PDF). 
  5. ^ Lehmann & Casella, eq. (2.5.16).
  6. ^ Mardia, K. V.; Marshall, R. J. (1984). "Maximum likelihood estimation of models for residual covariance in spatial regression".  
  7. ^ Lehmann & Casella, eq. (2.5.11).
  8. ^ Lehmann & Casella, eq. (2.6.16)
  9. ^ Janke, W.; Johnston, D. A.; Kenna, R. (2004). "Information Geometry and Phase Transitions". Physica A 336 (1–2): 181.  
  10. ^ Prokopenko, M.; Lizier, Joseph T.; Lizier, J. T.; Obst, O.; Wang, X. R. (2011). "Relating Fisher information to order parameters". Physical Review E 84 (4): 041116.  
  11. ^ Pukelsheim, Friedrick (1993). Optimal Design of Experiments. New York: Wiley.  
  12. ^ Bernardo, Jose M.; Smith, Adrian F. M. (1994). Bayesian Theory. New York: John Wiley & Sons.  
  13. ^ Gourieroux & Montfort (1995), page 87
  14. ^ Savage (1976)
  15. ^ Savage(1976), page 156
  16. ^ Edgeworth (September 1908, December 1908)
  17. ^ Pratt (1976)
  18. ^ Stigler (1978, 1986, 1999)
  19. ^ Hald (1998, 1999)

References

  •  
  •  
  •  
  • Frieden, B. R. (2004) Science from Fisher Information: A Unification. Cambridge Univ. Press. ISBN 0-521-00911-1.
  • Frieden, B. Roy; Gatenby, Robert A. (2013). "Principle of maximum Fisher information from Hardy's axioms applied to statistical systems". Physical Review E 88 (4).  
  • Hald, A. (May 1999). "On the History of Maximum Likelihood in Relation to Inverse Probability and Least Squares".  
  • Hald, A. (1998). A History of Mathematical Statistics from 1750 to 1930. New York: Wiley.  
  •  
  •  
  • Pratt, John W. (May 1976). "F. Y. Edgeworth and R. A. Fisher on the Efficiency of Maximum Likelihood Estimation".  
  •  
  • Schervish, Mark J. (1995). Theory of Statistics. New York: Springer.  
  •  
  •  
  •  
  • Van Trees, H. L. (1968). Detection, Estimation, and Modulation Theory, Part I. New York: Wiley.  
This article was sourced from Creative Commons Attribution-ShareAlike License; additional terms may apply. World Heritage Encyclopedia content is assembled from numerous content providers, Open Access Publishing, and in compliance with The Fair Access to Science and Technology Research Act (FASTR), Wikimedia Foundation, Inc., Public Library of Science, The Encyclopedia of Life, Open Book Publishers (OBP), PubMed, U.S. National Library of Medicine, National Center for Biotechnology Information, U.S. National Library of Medicine, National Institutes of Health (NIH), U.S. Department of Health & Human Services, and USA.gov, which sources content from all federal, state, local, tribal, and territorial government publication portals (.gov, .mil, .edu). Funding for USA.gov and content contributors is made possible from the U.S. Congress, E-Government Act of 2002.
 
Crowd sourced content that is contributed to World Heritage Encyclopedia is peer reviewed and edited by our editorial staff to ensure quality scholarly research articles.
 
By using this site, you agree to the Terms of Use and Privacy Policy. World Heritage Encyclopedia™ is a registered trademark of the World Public Library Association, a non-profit organization.
 


Copyright © World Library Foundation. All rights reserved. eBooks from Project Gutenberg are sponsored by the World Library Foundation,
a 501c(4) Member's Support Non-Profit Organization, and is NOT affiliated with any governmental agency or department.