Tail Threshold Selection

Statement¶

Tail threshold selection is the modeling step that chooses where the body of a sample ends and where a tail estimator or peaks-over-threshold model begins. For a threshold $u$, the exceedance sample is

For a Hill estimator, the equivalent tuning parameter is the number $k$ of upper order statistics, with empirical threshold $x_{n-k:n}$. Hill is designed for positive, regularly varying right tails with extreme-value index $\xi>0$; for a Pareto-type tail, $\alpha=1/\xi$. A threshold selection report should include at least:

• the candidate thresholds or $k$ values;

• the number of exceedances retained;

• Hill or EVI stability over the candidate range;

• mean-excess or GPD-shape stability where relevant;

• the reason the selected range was kept or rejected.

The goal is not to find a universally correct threshold. It is to make the bias-variance tradeoff visible.

Here $x_{1:n}\le\cdots\le x_{n:n}$ are the sorted observations, $k$ counts the upper values used by Hill, and $\xi$ is the extreme-value index. Ties at the threshold can make the strict exceedance count differ from $k$ and should be reported explicitly.

Bias-variance intuition¶

Every tail model asks the same awkward question: how far out is “tail”? Set the threshold too low and the estimator is biased by body observations. Set it too high and the estimator is dominated by too few extremes. A good threshold analysis therefore looks for a region where several diagnostics stop moving violently while enough exceedances remain to estimate anything at all.

This page is distinct from the body-shoulder-tail diagnostic. Body/shoulder geometry describes how a distribution’s density responds to variance mixing. Threshold selection is an empirical modeling decision for tail estimation.

Diagnostics¶

For a one-sided positive sample:

1. Choose a grid of candidate thresholds, often empirical quantiles.

2. Record exceedance counts at each threshold.

3. Plot Hill estimates over $k$ or threshold values.

4. Plot the mean-excess function for the same threshold range.

5. If using a GPD model, fit shape and scale over multiple thresholds.

6. Prefer a range where estimates are reasonably stable and exceedance counts are not too small.

The accepted range should be reported, not hidden. Downstream quantities such as moment existence, return levels, or expected shortfall can be highly sensitive to $\xi$. Under an exact GPD model, finite variance requires $\xi<1/2$, while finite mean and expected shortfall require $\xi<1$. For a general regularly varying tail, moments at the boundary also depend on the slowly varying factor. High return-level estimates are sensitive to $\xi$ but do not have mathematical singularities specifically at those two values.

Static body-plus-tail example¶

Consider the population model from the source example: with probability 0.82, $X$ is uniform on $[1,3]$; with probability 0.18, it is Pareto with cutoff 3 and exponent $\alpha=1.7=17/10$. Thus

Above 3, the conditional tail is exactly Pareto. Its shape is $\xi=10/17$, its excess scale is $\beta(u)=10u/17$, and its mean excess is $e(u)=u/(\alpha-1)=10u/7$. For an iid sample of size $n=30{,}000$, the expected number of exceedances is $n\bar F(u)$.

These counts are expectations under a specified population, not realized sample counts or fitted estimates. At $u=2$, the exact mean excess instead is

where 0.205 is the body contribution to $\mathbb E[(X-2)_+]$ and $51/7$ is the conditional tail mean. Extending the Pareto line below its cutoff would incorrectly give $20/7$.

At thresholds at least 3, raising $u$ introduces no tail-model bias in this exact construction but still loses observations. In empirical data, the cutoff is unknown and there may be no exact Pareto region at any finite threshold. The table therefore does not select an empirical threshold or report an uncertainty interval.

For a general GPD tail, threshold stability is $\beta(u')=\beta(u)+\xi(u'-u)$. Hence shape and modified scale $\beta(u)-\xi u$ should be inspected together. The simpler relation $\beta(u)/u=\xi$ is specific to the exact Pareto tail. Selected-threshold uncertainty, dependence, and tail approximation error are additional to the sampling uncertainty of a fit at a fixed threshold.

Coordinated diagnostics on one sample¶

The GPD fits hold excess location at zero and check numerical convergence, positive scale and support for every excess. We plot both fitted shape and modified scale $\widehat\beta(u)-\widehat\xi(u)u$. Success of these checks means the numerical fit is usable for the display; it does not establish a global likelihood optimum or validate the model.

Use pareto_type1 and gpd_threshold_fit. This reproduces the three fixed-threshold fits without writing figures:

import numpy as np
from xmath.incerto.distributions import pareto_type1
from xmath.incerto.estimators import gpd_threshold_fit

rng = np.random.default_rng(20261021)
tail = rng.random(30000) < 0.18
x = rng.uniform(1, 3, 30000)
x[tail] = pareto_type1.rvs(1.7, scale=3, size=tail.sum(), random_state=rng)
for u in (2, 3, 12):
    fit = gpd_threshold_fit(x, u, min_exceedances=50)
    print(f"u={u:d}: count={fit['count']:d}, status={fit['status']}")
    print(f"  xi={fit['xi']:.4g}, beta={fit['beta']:.4g}, modified_scale={fit['modified_scale']:.4g}")

u=2: count=17596, status=ok
  xi=0.5065, beta=0.8523, modified_scale=-0.1608
u=3: count=5344, status=ok
  xi=0.5929, beta=1.744, modified_scale=-0.03471
u=12: count=501, status=ok
  xi=0.5553, beta=7.346, modified_scale=0.683

Regenerate both figures with uv run python demos/incerto/tail_estimation.py. The runner also exports the Hill and mean-excess comparisons; its fixed sample sizes and fit-iteration caps bound the work.

Caveats¶

• Threshold selection is a modeling judgment, not a theorem. Different diagnostics can disagree.

• The Hill estimator is a right-tail regular-variation tool for $\xi>0$, not a general estimator for Gumbel-type or bounded-tail domains.

• The ratio $\hat\beta(u)/u$ is an exact-Pareto diagnostic. For a general GPD threshold model, threshold stability is $\beta(u')=\beta(u)+\xi(u'-u)$, so $\beta(u)-\xi u$ is the stable quantity.

• Very high thresholds may reduce approximation bias but can leave too few exceedances for stable estimation.

• Dependence, volatility clustering, rounding, reporting limits, and truncation can all make threshold diagnostics misleading.

• A selected threshold for one sample window or horizon should not be reused mechanically for a different dataset.

• Thresholds for right-tail positive data do not automatically apply to two-sided returns or losses without an explicit transformation.

References¶

• Hill, “A Simple General Approach to Inference About the Tail of a Distribution” 1975.

• Davison and Smith, “Models for Exceedances over High Thresholds” 1990.

• Coles, An Introduction to Statistical Modeling of Extreme Values 2001.

• Embrechts, Klueppelberg, and Mikosch, Modelling Extremal Events 1997.

Backlinks¶

• Depends on: Extreme-Value Index Estimation, Hill Estimator, Mean Excess Function, and Generalized Pareto Distribution.

• Related: Body, Shoulders, and Tails is a density geometry diagnostic, not a threshold-selection rule.

• Used by: S&P 500 Tail Diagnostics.