Exponential families and sufficient statistics

An exponential family separates the data-dependent quantities from their parameter-dependent coefficients in the log density. This structure makes sufficiency, likelihood, and the local geometry accessible through one function.

Begin with a sufficient statistic¶

Let $X_1,\ldots,X_n$ be independent Bernoulli variables with success probability $q\in(0,1)$. Their joint mass function is

The sample enters the parameter-dependent part only through the count $S_n=\sum_i x_i$. Conditional on $S_n=s$, all binary sequences with $s$ successes have equal probability, independent of $q$. This directly shows that $S_n$ is sufficient: the conditional distribution of the full data given the statistic contains no further dependence on the parameter.

Set $\theta=\log(q/(1-q))$. For a single observation,

The subtraction of $\psi$ normalizes the two masses. This calculation motivates the general form; it does not derive every exponential family from an arbitrary sufficient statistic.

The general form and its domain¶

Fix a measure $\nu$, a nonnegative carrier $h$, and a vector-valued statistic $T:\mathcal X\to\mathbb R^d$. Define

The natural parameter domain consists of parameters where the integral is positive and finite. We work on its nonempty open interior $\Theta$ and assume the support $\{h>0\}$ is fixed. On this interior the integral admits local differentiation of all orders; exponential integrability in a neighborhood controls the moments of $T$. This is the regular setting used below.

The domain is convex: Hölder’s inequality bounds the normalizing integral at a convex combination of two parameters by the corresponding geometric mean of their integrals. In particular, $\psi$ is convex.

For independent observations from this family, the density factors as

The factorization criterion for dominated models therefore makes $\sum_i T(X_i)$ sufficient. Its dimension stays fixed as $n$ grows. Sufficiency alone is a broader concept: the full data are always sufficient, and converse characterizations of exponential families require additional hypotheses.

Differentiating the normalizer¶

Write $Z(\theta)=e^{\psi(\theta)}$. Differentiation under the integral yields

Define the expectation parameter $\eta(\theta)=\mathbb E_\theta[T(X)]$. Since $\partial_jp_\theta=p_\theta(T_j-\eta_j)$, a second derivative gives

Thus $\nabla^2\psi=\operatorname{Cov}_\theta(T)$ is positive semidefinite. For any vector $a$,

The representation is minimal if no nonzero $a$ makes $a^\mathsf{T}T$ constant almost everywhere on the support. Under minimality the variance above is positive for every $a\ne0$, so $\psi$ is strictly convex and its gradient has an invertible derivative. Natural and expectation parameters are then smooth local coordinates for the same family. Strict convexity also makes $\nabla\psi$ injective on $\Theta$; its image must still be distinguished from boundary moments or the entire set of conceivable moment vectors.

The score, the derivative of the log density, is

Its covariance is the Fisher information matrix in natural coordinates: $I_\theta=\nabla^2\psi$. The Fisher note derives this metric for general regular models.

Two examples¶

For Bernoulli observations, direct differentiation gives

The expectation coordinate is the success probability. Natural coordinates range over $\mathbb R$, while expectation coordinates range over $(0,1)$.

For the normal running example, with mean $\mu\in\mathbb R$ and standard deviation $\sigma>0$, we now use $\theta$ for natural coordinates in place of the earlier location and scale coordinates. With Lebesgue measure and $h(x)=1$, the representation is

Completing the square in the integral verifies this normalizer. Differentiating recovers $\eta=(\mu,\mu^2+\sigma^2)^\mathsf{T}$. The expectation domain is $\eta_2>\eta_1^2$; equality would describe zero variance, outside the family.

This is a full two-dimensional family: the natural coordinates vary over the open half-plane $\theta_2<0$. Restricting, for example, to $\sigma^2=1+\mu^2$ produces a one-dimensional curved subfamily in that natural coordinate space. The ambient potential still exists, but the restricted model does not automatically inherit the full family’s affine coordinates.

Likelihood becomes moment matching¶

For observed data, let $\overline T=n^{-1}\sum_iT(x_i)$. Up to a parameter-independent term, the average log likelihood is

An interior optimum of a full family must satisfy

Its Hessian is $-\nabla^2\psi$, so in a minimal family an interior solution is the unique maximum. Existence is a separate question. If all Bernoulli observations are successes, then $\overline T=1$ lies outside $(0,1)$ and the likelihood supremum is approached as $\theta\to+\infty$; no finite natural parameter attains it.

For a constrained parameterization $\theta=\theta(\phi)$ with Jacobian $J$, the stationary equation is instead $J^\mathsf{T}(\overline T-\eta)=0$. Matching every ambient sufficient-statistic moment is generally too strong. This distinction will matter when an EM M-step fits a constrained joint model.

For further reading, Michael I. Jordan’s The Exponential Family: Basics, §§8.3–8.7, treats moment derivatives, sufficiency, and likelihood. That reference uses different symbols for natural and expectation parameters.

Continue with latent variables and EM, or take the geometry route to Fisher information. The hub lists both routes.