3. Uncertainty Methods¶
As argued in Representing Uncertainty, choosing an uncertainty method is to a
large extent choosing a representation. The methods probly provides are
therefore grouped by the representation they produce rather than by the
mechanism that produces it.
Roughly speaking, the four families answer four different questions about a prediction. A second-order distribution asks how much would the predictive distribution change had the model been trained differently? A credal set asks which distributions remain compatible with what the model has learned?, and declines to rank them. Conformal prediction asks which outcomes must be retained so that the truth is covered with a prescribed probability?, and answers with a set of labels or an interval instead of a distribution. Calibration, finally, asks can the predicted probabilities be taken at face value?, and corrects their scale where they cannot.
Within each family, the methods range from post-hoc wrappers around an already-trained network to methods that change the architecture, the loss, or both. In practice, this is often the more decisive distinction, since it determines whether a method is applicable to a model that cannot be retrained. Every entry states the idea behind the method, the representation it returns, its main advantages and disadvantages, and the paper it originates from, and links to a worked example in the gallery.
- 3.1. Second-Order Distributions
- 3.1.1.
dropout - 3.1.2.
dropconnect - 3.1.3.
bayesian - 3.1.4.
laplace - 3.1.5.
ensemble - 3.1.6.
batchensemble - 3.1.7.
subensemble - 3.1.8.
dare - 3.1.9.
duq - 3.1.10.
ddu - 3.1.11.
deup - 3.1.12.
sngp - 3.1.13.
evidential_classification - 3.1.14.
posterior_network - 3.1.15.
mahalanobis - 3.1.16.
natural_posterior_network - 3.1.17.
prior_network - 3.1.18.
evidential_regression - 3.1.19.
het_net - 3.1.20. Choosing Among Them
- 3.1.21. Full API
- 3.1.1.
- 3.2. Credal Sets
- 3.3. Conformal Prediction
- 3.4. Calibration