expected_calibration_error

probly.metrics.expected_calibration_error = <flextype.singledispatch.Flexdispatch object>

Compute the confidence expected calibration error (ECE) [GPSW17].

Predictions are grouped into num_bins equal-width bins by confidence, i.e. the maximum predicted probability, and the ECE is the frequency-weighted mean absolute gap between accuracy and mean confidence across bins. In contrast to classwise_ece(), only the winning class probability enters.

Parameters:
  • y_prob – Predicted class probabilities of shape (n, k), or a categorical distribution.

  • y_true – Integer ground-truth class labels of shape (n,).

  • num_bins – Number of equal-width confidence bins.

Returns:

The expected calibration error in [0, 1].

Raises:

NotImplementedError – If no implementation is registered for the type of y_prob.

Examples using probly.metrics.expected_calibration_error

Isotonic Regression on MNIST (Odd vs. Even)

Isotonic Regression on MNIST (Odd vs. Even)

Platt Scaling on MNIST (Odd vs. Even)

Platt Scaling on MNIST (Odd vs. Even)

Temperature Scaling on MNIST

Temperature Scaling on MNIST

Vector Scaling on MNIST

Vector Scaling on MNIST