classwise_ece

probly.metrics.classwise_ece = <flextype.singledispatch.Flexdispatch object>

Compute the classwise expected calibration error (classwise-ECE) [KPerelloNKangsepp+19].

For every class j, the predicted probabilities p_j are grouped into num_bins equal-width bins; the per-class error is the bin-size-weighted mean absolute difference between the empirical frequency of class j and its mean predicted probability within each bin, and the classwise-ECE averages these errors over all classes. In contrast to the confidence-based expected_calibration_error(), it is also sensitive to miscalibration on non-maximal class probabilities.

Parameters:
  • y_prob – Predicted class probabilities of shape (n, k), or a categorical distribution.

  • y_true – Integer ground-truth class labels of shape (n,).

  • num_bins – Number of equal-width probability bins per class.

Returns:

The classwise expected calibration error in [0, 1].

Raises:

NotImplementedError – If no implementation is registered for the type of y_prob.

Examples using probly.metrics.classwise_ece

Dirichlet Calibration on MNIST

Dirichlet Calibration on MNIST