selective_prediction¶
- probly.evaluation.selective_prediction.selective_prediction = <flextype.singledispatch.Flexdispatch object>¶
Selective prediction downstream task for evaluation.
Perform selective prediction based on criterion and losses. The criterion is used to sort the losses. In line with uncertainty literature the sorting is done in descending order, i.e. the losses with the largest criterion are rejected first. Ties are broken by input position: among equal criterion values, earlier instances are rejected first, so results are deterministic across backends.
- Parameters:
criterion – Criterion values of shape (n_instances,).
losses – Loss values of shape (n_instances,).
n_bins – Number of bins.
- Returns:
aurc: Area under the risk / loss curve.
bin_losses: Loss per bin of shape (n_bins,).
- Return type:
A tuple containing
- Raises:
NotImplementedError – If no implementation is registered for the type of
criterion.