coverage

probly.metrics.coverage = <flextype.singledispatch.Flexdispatch object>

Compute the empirical coverage of a predicted set against true labels.

Coverage is the fraction of samples whose true value lies inside the predicted set. Higher is better; a well-calibrated conformal predictor targeting confidence 1 - alpha should yield coverage close to 1 - alpha.

Per-credal-set semantics:

  • Singleton: degenerates to top-1 accuracy on the single distribution.

  • Discrete: covered iff some vertex distribution puts its argmax on the true class. Discrete coverage is always less than or equal to Convex coverage on the same vertices: a true class with probability strictly between two vertex argmaxes is included by the convex hull’s interval-dominance set but missed by the vertex-only rule. Use Discrete when the modeled object is the finite vertex set (e.g. an ensemble); use Convex when the convex hull is the modeled object.

  • Convex / DistanceBased / ProbabilityIntervals / DirichletLevelSet: interval-dominance prediction set built from lower() / upper() envelopes. Note that the DistanceBased envelope (clip(nominal +- radius, 0, 1)) ignores the simplex constraint and is therefore loose; the resulting coverage is consistent with the declared envelope but over-conservative relative to the tight L1-ball envelope.

Parameters:
  • y_pred – A predicted-set representation (conformal set or credal set). See module docstring for the registered types.

  • y_true – Ground-truth values aligned with y_pred. Integer class labels for classification, or scalar targets for regression.

Returns:

The empirical coverage as a float in [0, 1].

Raises:

NotImplementedError – If no implementation is registered for the type of y_pred.

Examples using probly.metrics.coverage

Classification Conformal Prediction — sklearn

Classification Conformal Prediction — sklearn

Classification Conformal Prediction — PyTorch

Classification Conformal Prediction — PyTorch

How Conformal Sets Change with the Score and with Alpha

How Conformal Sets Change with the Score and with Alpha

A Gentle Introduction to Conformal Prediction

A Gentle Introduction to Conformal Prediction

Adaptive Conformal Intervals on Heteroscedastic Data

Adaptive Conformal Intervals on Heteroscedastic Data

Quantile Regression Conformal Prediction — sklearn

Quantile Regression Conformal Prediction — sklearn

Quantile Regression Conformal Prediction — PyTorch

Quantile Regression Conformal Prediction — PyTorch

Regression Conformal Prediction — sklearn

Regression Conformal Prediction — sklearn

Regression Conformal Prediction — PyTorch

Regression Conformal Prediction — PyTorch

Mixing torch-uncertainty and probly

Mixing torch-uncertainty and probly

Cyclone forecast uncertainty with WeatherNext

Cyclone forecast uncertainty with WeatherNext