Mean Excess Function

Statement¶

For a random variable $X$ and a threshold $u$ with $\mathbb P(X>u)>0$ and finite conditional expectation above $u$, the mean excess function is

It measures the expected overshoot beyond $u$ after the threshold has already been crossed. For a Pareto Type I tail with exponent $\alpha>1$ and $u\ge x_m$,

For a generalized Pareto variable with location zero, shape $\xi$, scale $\beta>0$, and threshold $u$ in its support with $\beta+\xi u>0$, the mean excess function is linear when $\xi<1$:

The exponential distribution is the boundary case $\xi=0$, where $e(u)$ is constant. Here $u$ is the threshold, $\xi$ the GPD shape, and $\beta$ its scale at threshold zero.

Diagnostic intuition¶

The mean excess function asks what remains after an event is already large. Thin-tail intuition often expects the remaining excess to be tame once a high threshold has been crossed. A Pareto tail says the opposite: the expected additional excess grows in proportion to the threshold itself.

The exponential, positive-shape GPD with $\xi<1$, and negative-shape GPD models have constant, increasing linear, and decreasing linear population mean excess functions, respectively. Curvature departs from an exact GPD model over that threshold range. An empirical plot can also bend because of sampling noise, especially when few observations exceed the threshold; its shape alone does not identify a model failure or a body-to-tail transition.

Pareto derivation and GPD sketch¶

We derive the Pareto mean-excess formula from the conditional survival function. The generalized Pareto linear formula follows from standard GPD threshold stability and the GPD mean formula. The GPD page derives the stability identity; its moment formulas are cited rather than reproved here.

For Pareto Type I,

For $u\ge x_m$ and $\alpha>1$,

Taking the ratio $\bar F(u+y)/\bar F(u)$ for $y\ge0$ gives

Therefore

For a generalized Pareto distribution, the threshold-stability property gives another generalized Pareto distribution above threshold $u$ with updated scale $\beta+\xi u$, provided $u$ is in the support and $\beta+\xi u>0$. Its mean exists only for $\xi<1$, and equals $(\beta+\xi u)/(1-\xi)$.

Mean-excess diagnostic¶

Exact theoretical shapes provide a baseline for the empirical diagnostic. Always report how many observations exceed each threshold.

The theoretical shapes can be compared directly:

For observations $x_1,\ldots,x_n$, let $k(u)=\sum_i\mathbf1_{\{x_i>u\}}$. The empirical diagnostic is

For the fixed observations $(1,2,2,4)$ at $u=2$, only 4 exceeds the threshold, so $k(2)=1$ and $\widehat e_n(2)=2$. At $u=4$ the mean excess estimate is undefined because the exceedance count is zero. The counting example uses the same strict inequality convention.

Pareto mean excess rises with the threshold, exponential mean excess is flat, and the displayed uniform mean excess decreases toward its endpoint. For any distribution with finite right endpoint $x_F=\sup\{x:F(x)<1\}$ and $u<x_F$, conditioning on $X>u$ gives $0<X-u\le x_F-u$ almost surely. Taking conditional expectations proves

This bound does not imply monotonicity. For example, if $\mathbb P(X=1)=\mathbb P(X=3)=1/2$, both atoms exceed 0.9, so $e(0.9)=[(1-0.9)+(3-0.9)]/2=1.1$. At 1.1 only the atom at 3 exceeds the threshold, so $e(1.1)=3-1.1=1.9$. The mean excess has increased even though $x_F=3$.

Empirical values are diagnostics; high thresholds aim to reduce tail-model bias but leave fewer exceedances and more noise.

Population curves and sample estimates¶

Use mean_excess; masking retains the counts and reports an unavailable estimate as nan:

from xmath.incerto.estimators import mean_excess

result = mean_excess([1, 2, 2, 4], [1, 2, 4], min_exceedances=2)
for u, count, mean in zip(result["threshold"], result["exceedances"], result["mean_excess"]):
    print(f"u={u:.4g}: count={int(count):d}, mean_excess={mean:.4g}")

Output

u=1: count=3, mean_excess=1.667
u=2: count=1, mean_excess=nan
u=4: count=0, mean_excess=nan

Regenerate the figure with uv run python demos/incerto/tail_estimation.py. The threshold comparison adds the shared mixture sample, whose body changes the population curve below 3.

Caveats¶

• The mean excess function is itself a mean. If the fitted tail has $\xi\ge1$, the theoretical mean excess is infinite.

• Empirical mean excess plots are unstable at high thresholds. Always inspect exceedance counts.

• Linear-looking behavior is suggestive, not decisive. Mixtures and finite upper truncation can create misleading curvature.

• For two-sided returns, apply the function to a one-sided loss variable or to absolute returns after stating the modeling choice.

References¶

• Davison and Smith, “Models for Exceedances over High Thresholds” 1990.

• Coles, An Introduction to Statistical Modeling of Extreme Values 2001.

• Embrechts, Klueppelberg, and Mikosch, Modelling Extremal Events 1997.

Backlinks¶

• Depends on: Pickands-Balkema-de Haan Theorem, Generalized Pareto Distribution, Pareto Distribution, and threshold notation in Shared notation.

• Used by: S&P 500 Tail Diagnostics and Tail Threshold Selection.