probly.evaluation.tasks

Collection of downstream tasks to evaluate the performance of uncertainty pipelines.

Functions

selective_prediction(criterion, losses[, n_bins])

Selective prediction downstream task for evaluation based on :cite: lakshminarayananSimpleScalable2017.

probly.evaluation.tasks.selective_prediction(criterion, losses, n_bins=50)

Selective prediction downstream task for evaluation based on :cite: lakshminarayananSimpleScalable2017.

Perform selective prediction based on criterion and losses. The criterion is used the sort the losses. In line with uncertainty literature the sorting is done in descending order, i.e. the losses with the largest criterion are rejected first.

Parameters:

• criterion (ndarray) – numpy.ndarray shape (n_instances,), criterion values

• losses (ndarray) – numpy.ndarray shape (n_instances,), loss values

• n_bins (int) – int, number of bins

Returns:

float, area under the loss curve bin_losses: numpy.ndarray shape (n_bins,), loss per bin

Return type:

auroc