selective_prediction¶

probly.evaluation.tasks.selective_prediction(criterion: ndarray, losses: ndarray, n_bins: int = 50) → tuple[float, ndarray][source]¶

Selective prediction downstream task for evaluation.

Perform selective prediction based on criterion and losses. The criterion is used the sort the losses. In line with uncertainty literature the sorting is done in descending order, i.e. the losses with the largest criterion are rejected first.

Parameters:

criterion – Criterion values of shape (n_instances,).
losses – Loss values of shape (n_instances,).
n_bins – Number of bins.

Returns:

aurc: Area under the risk / loss curve.
bin_losses: Loss per bin of shape (n_bins,).

Return type:

A tuple containing