Mixture architecture — joint and marginal laws

A normal variance–mean mixture has two related statistical descriptions: the joint law of an observation and its mixing variable, and the marginal law of the observation alone. Keeping these distinct explains the sufficient statistics and expectation-to-parameter map used by EM.

Conditional, joint, and marginal descriptions¶

Let $Y>0$ be independent of $Z\sim\mathcal N_d(0,I_d)$ , and let $LL^\top=\Sigma\succ0$ . Define

X=\mu+\gamma Y+\sqrt Y\,LZ,\qquad X\mid Y=y\sim\mathcal N_d(\mu+\gamma y,y\Sigma).

(1)

For $d=1$ , this is conditioning a mixture with $Y=W$ , $\gamma=\beta$ , and $\Sigma=\sigma^2$ . For a mixing density $g$ ,

f(x,y)=f(x\mid y)g(y),\qquad f_X(x)=\int_0^\infty f(x,y)\,dy,\qquad f(y\mid x)=\frac{f(x,y)}{f_X(x)}.

(2)

Law	Variables available	Statistical role
Conditional normal	$X$ at fixed $Y=y$	Specifies the sampling mechanism
Joint	$(X,Y)$	Complete-data likelihood, sufficient statistics, joint divergences
Marginal	$X$	Observed likelihood, moments, observable probabilities
Posterior	$Y$ given $X=x$	Converts observations into expected complete-data statistics

Drawing both variables and discarding $Y$ produces a marginal draw. Integrating out $Y$ is an operation on a probability model, not an assertion that $Y$ was observed.

When the joint is an exponential family¶

For GIG mixing, expansion of $\log f(x,y)$ gives the six statistic blocks

t(x,y)=(\log y,y^{-1},y,x,x/y,xx^\top/y).

(3)

The scalar coefficients in the first three slots are

p-1-d/2,\qquad -\tfrac12(b+\mu^\top\Sigma^{-1}\mu),\qquad -\tfrac12(a+\gamma^\top\Sigma^{-1}\gamma).

(4)

The factors of $1/2$ follow from the GIG kernel $y^{p-1}e^{-(ay+b/y)/2}$ . The remaining coefficients and the log-partition are derived in GH. Matrix blocks use the trace inner product on symmetric matrices.

The first three expectations are always ordered as $(\mathbb E[\log Y],\mathbb E[Y^{-1}],\mathbb E[Y])$ in this batch. Upstream implementation storage may use another order; descriptive moments, rather than slot numbers, identify the mathematical correspondence.

The joint GIG-normal model is an exponential family in these statistics. An arbitrary mixing density does not automatically give this particular finite-dimensional representation. Likewise, marginalization does not generally preserve exponential-family structure. For the full observable GH model, Bessel terms couple parameters to the observation inside the log density. The mere presence of a Bessel function or integral is not a proof that a model is outside the exponential family; GIG itself is a counterexample to that reasoning. Special restricted marginal families can behave differently.

Recovering a model from expected statistics¶

Let $\eta_i=\mathbb E[t_i(X,Y)]$ in the order (3). Conditional Gaussian moments imply

\eta_4=\mu+\gamma\eta_3,\qquad \eta_5=\mu\eta_2+\gamma.

(5)

If $\eta_2\eta_3\ne1$ , this linear system gives

\mu=\frac{\eta_4-\eta_3\eta_5}{1-\eta_2\eta_3},\qquad \gamma=\frac{\eta_5-\eta_2\eta_4}{1-\eta_2\eta_3}.

(6)

The sixth moment gives

\Sigma=\eta_6-\eta_5\mu^\top-\mu\eta_5^\top +\eta_2\mu\mu^\top-\eta_3\gamma\gamma^\top.

(7)

The mixing law is recovered from its own moment equations. This is the joint family’s inverse expectation map, with the domain conditions stated in the exponential-family core. An arbitrary tuple of six arrays need not be a feasible expectation vector. In particular, the recovered covariance must be positive definite and the mixing moments must admit the claimed distribution.

In EM, replace each $\eta_i$ by an average of conditional expectations given the observations. This uses the same mathematical map, but the expectations now come from the current posterior. See latent variables and ordinary EM for why this improves the observed likelihood under the stated assumptions.

Conditional parameters and observable moments¶

The normal parameters enter the marginal moments through

\mathbb E[X]=\mu+\gamma\mathbb E[Y],\qquad \operatorname{Cov}(X)=\mathbb E[Y]\Sigma +\operatorname{Var}(Y)\gamma\gamma^\top.

(8)

Thus $\mu$ and $\Sigma$ describe the conditional mechanism, and cannot simply be read as the mean and covariance of the observed distribution. The normal-mixtures note works through this distinction. Moreover, a latent rescaling can change the joint representation while preserving the marginal law; see GH identifiability.

This note covers the full positive-definite covariance model. Factor-analysis constructions introduce additional latent variables and different complete-data statistics; the factor-analysis note derives their posterior moments and constrained M-step.

Class naming, storage, constructor recipes, and fitter contracts remain in the upstream mixture design, EM framework, and package API.

Source and adaptation¶

Rewritten as a mathematical note from xshi19/normix, docs/design/mixtures.md, at revision 763bb3608920661a012cf089888d349fbf680aad (2026-09-13 import). Copyright (c) 2020 xshi19. Licensed under MIT. The pinned source records the original version. Notation, mathematical qualifications, and links were adapted for this site; API recipes, executable package cells, and notebook plots are omitted. No upstream benchmark execution or formal-proof verification is claimed for this adaptation.