Fisher geometry and the meaning of L2

Euclidean distance between parameter vectors measures changes in the chosen coordinates. Fisher geometry instead measures first-order changes in the log density, averaged under the model. Its coordinate matrix changes when the parameters change, while the length assigned to the same statistical motion stays fixed.

A regular model and its scores¶

Let $p_\theta$ be densities with respect to a fixed measure $\nu$, with $\theta$ in an open subset of $\mathbb R^d$. Assume a common positive support, twice continuously differentiable densities, and local integrable bounds that justify differentiating the normalizing integral twice. Assume the scores are square integrable. For the KL expansion below, also require that the expected log density has a second-order Taylor expansion with an integrable remainder.

Write $\ell_\theta(x)=\log p_\theta(x)$ and define the column score vector

Along a curve with velocity $v$, the log density changes at rate $v^\mathsf{T}s_\theta(x)$. Differentiating $\int p_\theta\,d\nu=1$ gives

The Fisher information matrix per observation is

Thus $g_\theta(v,v)$ is the mean square directional change in log density. The matrix is always positive semidefinite. It gives a Riemannian metric on a regular region where it is positive definite and smooth. A null direction means a zero first-order change in the density almost everywhere; global identifiability alone does not exclude such singular parameterizations.

Local KL divergence gives the same quadratic form¶

Differentiating the normalizer a second time, using $\partial_i\partial_jp=p(\partial_i\partial_j\ell+ \partial_i\ell\,\partial_j\ell)$, gives

Now hold the first distribution fixed in

Taylor expansion of the second log density has a linear term with expectation zero and a quadratic term involving $-I_\theta$. Therefore

This is a local expansion as $\delta\to0$ at a fixed interior parameter. Finite KL is generally asymmetric and is not squared Riemannian distance. For $n$ independent observations, scores add and their cross-covariances vanish, so the information matrix is $nI_\theta$.

Reparameterization changes the matrix, not the length¶

Use a smooth invertible chart $\theta=\theta(\phi)$ and let $J=\partial\theta/\partial\phi$. By the chain rule,

Since $d\theta=J\,d\phi$, the two quadratic expressions agree:

This is exactly the metric transformation law. A Euclidean metric can also be transformed correctly. The problem arises if one declares the identity matrix to be the metric anew in every nonlinear chart: that changes the geometry.

The normal running example¶

For $X\sim N(\mu,\sigma^2)$ with $\mu\in\mathbb R$ and $\sigma>0$, use $\theta=(\mu,\sigma)^\mathsf{T}$. The introductory calculation gives the Fisher matrix $I(\theta)=\operatorname{diag}(\sigma^{-2},2\sigma^{-2})$. Its transformation to mean and variance coordinates is worked out there as well.

We can check the local KL expansion directly for this family. Let $\widetilde\theta=(\widetilde\mu,\widetilde\sigma)^\mathsf{T}$, with $\widetilde\sigma>0$. Taking the expected log density ratio and using $\mathbb E_\theta[(X-\widetilde\mu)^2]=\sigma^2+(\mu-\widetilde\mu)^2$ gives the exact formula

For a fixed velocity $v=(v^\mu,v^\sigma)^\mathsf{T}$, set $\widetilde\theta=\theta+\varepsilon v$. Expanding the logarithm and the reciprocal square at $\varepsilon=0$ cancels the linear terms and yields

Thus the squared Fisher speed determines the leading KL change along either parameter direction, or any combination of them. The expansion is local at a fixed $\sigma>0$, with $\sigma+\varepsilon v^\sigma>0$. If $\sigma$ is known and only $\mu$ varies, the restricted metric is $ds^2=d\mu^2/\sigma^2$, so a scaled Euclidean metric is appropriate for that subfamily.

Bernoulli probability and log odds¶

For $X\sim\operatorname{Bernoulli}(q)$ with $0<q<1$, differentiation gives

Because $\operatorname{Var}(X)=q(1-q)$,

Equal small changes in probability have different local statistical sizes: the coefficient is 4 at $q=1/2$ and $10000/99$ at $q=1/100$. This statement uses the infinitesimal metric, not a finite-step equality for KL.

Now take the natural parameter $\theta=\log(q/(1-q))$. Since $dq/d\theta=q(1-q)$, the same metric is

This agrees with $I_\theta=\psi''(\theta)$ from the exponential-family calculation.

Three different uses of L2¶

The Euclidean norm on a finite parameter vector is more precisely an $\ell^2$ norm. It should be distinguished from two function-space constructions.

Raw density $L^2$ is invariant under parameter relabeling because the densities do not change. It generally depends on the measurement coordinates. For Lebesgue densities and $Y=aX$ with $a>0$, the transformed densities satisfy $p_Y(y)=p_X(y/a)/a$, so

A fixed invertible transformation of the observation contributes a parameter-independent Jacobian to the log density, so it leaves scores and Fisher information unchanged. This explains a distinction between Fisher geometry and raw density $L^2$ that parameter invariance alone cannot explain.

Square-root densities recover Fisher geometry¶

There is a useful $L^2(\nu)$ representation. Map a density to $u_\theta=2\sqrt{p_\theta}$, which has norm 2. Assume this map is differentiable in $L^2(\nu)$, with derivative along $v$ given by

Then its squared $L^2(\nu)$ norm is precisely $v^\mathsf{T}I_\theta v$. This is an isometric realization of the tangent metric by square-root densities. It is not the raw density-difference construction above.

With the convention

the same differentiability gives

The factors $1/2$ for KL and $1/8$ for this squared Hellinger convention describe the same local Fisher metric. Finite Hellinger distance measures a chord between square-root densities; Fisher–Rao distance minimizes path length within the specified statistical model.

For background on the metric and these representations, see §§3.9–3.12 of Frank Nielsen’s An elementary introduction to information geometry. The conditional-expectation note explains projection in the other Hilbert space used here, $L^2(P_\theta)$.

Continue with dual coordinates and KL projections, or return to the Information Geometry hub.