Information Geometry - Information Geometry

Information geometry studies probability models using geometric objects that describe changes in the distributions themselves. A parameter vector gives coordinates for a model. The score describes a first-order change in its log density, and Fisher information measures the size of that change.

Let $p_\theta$ be a density with respect to a fixed measure, with $\theta$ in an open subset of $\mathbb R^d$ . Let $X$ have law $P_\theta$ , and write $s_\theta(x)=\nabla_\theta\log p_\theta(x)$ for the score vector. We work in a regular interior region with common positive support, square-integrable scores, and smooth positive definite Fisher information. The Fisher note states the differentiation assumptions used below.

Fix a parameter value $\theta$ . A smooth perturbation $\theta(\varepsilon)$ through $\theta(0)=\theta$ has velocity $v=\dot\theta(0)$ : its first-order parameter displacement is $\varepsilon v$ . This velocity is a tangent vector at $\theta$ , not an additional model parameter. Let $w$ be another such velocity at the same point. The Fisher inner product per observation is

g_\theta(v,w) =\mathbb E_\theta\!\left[ (v^\mathsf{T}s_\theta(X))(w^\mathsf{T}s_\theta(X)) \right].

(1)

To interpret (1) statistically, hold an observation $x$ fixed and apply the chain rule:

\left.\frac{d}{d\varepsilon}\log p_{\theta(\varepsilon)}(x) \right|_{\varepsilon=0} =v^\mathsf{T}s_\theta(x).

(2)

Thus $v^\mathsf{T}s_\theta(x)$ is the directional score: the rate at which the log density changes when we perturb the parameter along $v$ . The Fisher speed $\sqrt{g_\theta(v,v)}$ is its root mean square under the current model $P_\theta$ . A larger speed means that the same small increment in $\varepsilon$ produces a larger typical first-order change in log density. Here $\varepsilon$ indexes a chosen perturbation; it need not represent time. Since regular-model scores have mean zero, $g_\theta(v,w)$ is also the covariance of the two directional scores.

The first note works this out for $N(\mu,\sigma^2)$ , with mean $\mu$ and standard deviation $\sigma>0$ . In coordinates $\theta=(\mu,\sigma)^\mathsf{T}$ , a velocity has two components: the rate of change of the mean and the rate of change of the standard deviation. The notes then explain coordinate invariance and the two affine structures of exponential families.

Reading path¶

The prerequisites are multivariable calculus, elementary linear algebra, and probability densities and expectation. No previous course in differential geometry is assumed. The first four notes supply the background; the last two develop the first geometric results.

Note	Main question
1. From Euclidean space to a manifold	What survives a change of coordinates, and what extra structure measures lengths?
2. Exponential families and sufficient statistics	Why does one log-partition function determine means, covariances, and likelihood equations?
3. Latent variables and ordinary EM	How does posterior completion produce a likelihood-increasing update?
4. Conditional expectation as projection	In which space is conditional expectation orthogonal projection?
5. Fisher geometry and the meaning of L2	Why does Euclidean distance between parameter vectors miss the model’s statistical sensitivity?
6. Dual coordinates and KL projections	How do orthogonality, Pythagoras, and two notions of straightness fit together?

For a geometry-first route, read 1, 2, 5, and 6, then return to EM and conditional expectation. For a likelihood-first route, begin with 2 and 3, then read 4 before the geometry notes. KL is introduced where EM needs it; a separate information-theory course is not required.

Conventions¶

Symbols follow the site-wide shared notation.

Densities are taken with respect to a stated common measure, which may be counting measure. Logarithms are natural. In exponential families, $\theta$ denotes natural coordinates, $\psi$ the log-partition function, and $\eta=\nabla\psi$ expectation coordinates. A latent model has joint density $P_\theta(x,z)$ , observable marginal $p_\theta(x)$ , and posterior $r_\theta(z\mid x)$ .

Smoothness, common support, integrability, and nonsingularity are stated where they are used. Most geometric calculations concern interior regular models; boundary distributions and redundant parameters require separate treatment. The worked examples are exact calculations. No numerical approximation is used as a proof.

Where this leads¶

The next questions concern what happens when a latent variable is hidden: how marginalization changes scores and divergences, how lost information affects EM, and when dual flatness is lost. GH/Normix specialization and curvature calculations belong to that later sequence.

The conditioning example gives a first normal-mixture calculation. The Normix theory track links to the independently maintained package. Return to the mathematical notes index for the other tracks.