coverage¶
- probly.metrics.coverage = <flextype.singledispatch.Flexdispatch object>¶
Compute the empirical coverage of a predicted set against true labels.
Coverage is the fraction of samples whose true value lies inside the predicted set. Higher is better; a well-calibrated conformal predictor targeting confidence
1 - alphashould yield coverage close to1 - alpha.Per-credal-set semantics:
Singleton: degenerates to top-1 accuracy on the single distribution.
Discrete: covered iff some vertex distribution puts its argmax on the true class. Discrete coverage is always less than or equal to Convex coverage on the same vertices: a true class with probability strictly between two vertex argmaxes is included by the convex hull’s interval-dominance set but missed by the vertex-only rule. Use Discrete when the modeled object is the finite vertex set (e.g. an ensemble); use Convex when the convex hull is the modeled object.
Convex / DistanceBased / ProbabilityIntervals / DirichletLevelSet: interval-dominance prediction set built from
lower()/upper()envelopes. Note that the DistanceBased envelope (clip(nominal +- radius, 0, 1)) ignores the simplex constraint and is therefore loose; the resulting coverage is consistent with the declared envelope but over-conservative relative to the tight L1-ball envelope.
- Parameters:
y_pred – A predicted-set representation (conformal set or credal set). See module docstring for the registered types.
y_true – Ground-truth values aligned with
y_pred. Integer class labels for classification, or scalar targets for regression.
- Returns:
The empirical coverage as a float in
[0, 1].- Raises:
NotImplementedError – If no implementation is registered for the type of
y_pred.
Examples using probly.metrics.coverage¶
How Conformal Sets Change with the Score and with Alpha
Adaptive Conformal Intervals on Heteroscedastic Data
Quantile Regression Conformal Prediction — sklearn
Quantile Regression Conformal Prediction — PyTorch