probly.evaluation.tasks.selective_prediction¶

probly.evaluation.tasks.selective_prediction(criterion: ndarray, losses: ndarray, n_bins: int = 50) → tuple[float, ndarray][source]¶

Selective prediction downstream task for evaluation.

Perform selective prediction based on criterion and losses. The criterion is used the sort the losses. In line with uncertainty literature the sorting is done in descending order, i.e. the losses with the largest criterion are rejected first.

Based on [LPB17a].

Parameters:

criterion – Criterion values of shape (n_instances,).
losses – Loss values of shape (n_instances,).
n_bins – Number of bins.

Returns:

auroc: Area under the loss curve.
bin_losses: Loss per bin of shape (n_bins,).

Return type:

A tuple containing