3.1. Second-Order Distributions

The methods on this page turn a point predictor into one that outputs a distribution over distributions. This additional order is what makes the aleatoric/epistemic decomposition computable, at least in its most common form: the spread between the predicted first-order distributions is attributed to epistemic uncertainty, and the average spread within them to aleatoric uncertainty.

The methods differ in many respects, but one distinction cuts deeper than the others, since it determines where the computational cost is paid. Sampled methods represent the second-order distribution implicitly, by a finite set of first-order distributions obtained from T stochastic forward passes, N ensemble members, or S posterior weight samples. The resolution of this representation is paid for at every prediction.

Parameterized methods, on the other hand, output the second-order distribution in closed form, typically as a Dirichlet distribution over the simplex, from a single forward pass. The cost does not disappear, however; it moves to training, where the architecture, the loss, or both have to change.

A third and smaller group is distance-aware. Instead of placing a posterior over parameters, these methods measure where an input falls relative to the training data in feature space. They tend to separate in-distribution from out-of-distribution inputs well, but the distance itself carries no information about aleatoric uncertainty, which has to come from elsewhere, if at all. A few methods fall outside all three groups, most notably deup, which learns to predict the model’s error, and het_net, which models aleatoric uncertainty only.

3.1.1. dropout

Dropout layers are kept active at inference, and each stochastic forward pass is treated as a draw from an approximate posterior. For a network that was trained with dropout, no retraining is required at all, which makes this the cheapest retrofit available. The transform inserts a dropout layer in front of every linear layer; the torch backend additionally offers shared_mask, which draws a single mask per forward pass, shared across the batch, instead of an independent mask per sample.

Idea:

Monte Carlo sampling of sub-networks by keeping dropout active at test time.

Representation:

Sampled second order, i.e., a Sample of categorical distributions.

Advantages:

No retraining if the model was trained with dropout; the cheapest retrofit available.

Disadvantages:

T forward passes per prediction; the quality depends on where the dropout layers sit, and a model trained without dropout may have to be retrained once they are inserted.

Reference:

[GG16]

Uncertainty for Hugging Face transformers

Uncertainty for Hugging Face transformers

Uncertainty for Hugging Face vision transformers

Uncertainty for Hugging Face vision transformers

MC-Dropout uncertainty on a 2-D stream

MC-Dropout uncertainty on a 2-D stream

MC Dropout on Two Moons

MC Dropout on Two Moons

MC Dropout on MNIST

MC Dropout on MNIST

3.1.2. dropconnect

The same Monte Carlo argument, applied one level lower: DropConnect sets individual weights rather than entire activations to zero, so that every forward pass samples a different sparse weight matrix. The space of sub-networks is thus larger than for dropout, which tends to yield more diverse samples at the same drop rate.

Idea:

Monte Carlo sampling over randomly dropped weights.

Representation:

Sampled second order.

Advantages:

A larger sub-network space than dropout, and hence more diverse samples at the same drop rate; as with dropout, retraining is not strictly required.

Disadvantages:

T forward passes per prediction; masking individual weights costs more per pass than masking activations.

Reference:

[MYM+21]

DropConnect on Two Moons

DropConnect on Two Moons

DropConnect on MNIST

DropConnect on MNIST

3.1.3. bayesian

A genuine variational Bayesian neural network: every weight is replaced by a Gaussian with learned mean and standard deviation, fitted by Bayes by Backprop against a prior. Unlike with dropout, the posterior is explicit and is itself the object of training, which is also why it cannot simply be added to a model that has already been trained.

Idea:

Mean-field Gaussian variational posterior over the weights.

Representation:

Sampled second order, obtained by sampling weights.

Advantages:

An explicit posterior that is trained for, rather than an approximation added afterwards.

Disadvantages:

Requires training with the ELBO and roughly doubles the parameter count; cannot be applied to a trained model; S forward passes per prediction.

Reference:

[BCKW15]

Bayesian Neural Network on Two Moons

Bayesian Neural Network on Two Moons

Bayesian Neural Network on MNIST

Bayesian Neural Network on MNIST

3.1.4. laplace

A post-hoc Bayesian treatment: the network is trained as usual, and a Gaussian is then placed around the resulting MAP weights, with the inverse Hessian of the loss, or in practice an approximation of it, as its covariance. probly integrates the laplace-torch package and exposes its predictive_samples through the standard representer interface; only classification is currently supported.

Idea:

Second-order Taylor expansion of the loss around the MAP estimate.

Representation:

Sampled second order.

Advantages:

No change to training; a post-hoc treatment of an already trained network.

Disadvantages:

A Hessian approximation after training, then S forward passes per prediction; classification only.

Reference:

Wraps the external laplace-torch package.

Laplace Approximation on Two Moons

Laplace Approximation on Two Moons

Laplace on MNIST

Laplace on MNIST

3.1.5. ensemble

N copies of the same architecture are trained from different initializations, and their predictions are treated as samples. Deep ensembles have proven hard to beat, and what explains their success appears to be the diversity of the loss basins in which the members settle rather than any Bayesian argument.

Idea:

Independent retraining; the disagreement between members is the epistemic signal.

Representation:

Sampled second order with N members.

Advantages:

Consistently among the strongest baselines, without relying on Bayesian assumptions.

Disadvantages:

N times the training cost, N times the memory, and N forward passes per prediction.

Reference:

[LPB17]

Quantile Regression Conformal Prediction — PyTorch

Quantile Regression Conformal Prediction — PyTorch

Depth estimation uncertainty with transformers

Depth estimation uncertainty with transformers

Forecast uncertainty with transformers

Forecast uncertainty with transformers

LoRA adapter ensembles with peft

LoRA adapter ensembles with peft

Uncertainty for Hugging Face transformers

Uncertainty for Hugging Face transformers

Ensemble Ordinal Classification Uncertainty

Ensemble Ordinal Classification Uncertainty

Ensemble Regression Uncertainty

Ensemble Regression Uncertainty

Deep Ensemble on Two Moons

Deep Ensemble on Two Moons

Deep Ensemble on MNIST

Deep Ensemble on MNIST

3.1.6. batchensemble

An ensemble that fits into roughly the memory of a single model. Each member owns only a pair of rank-one factors (r, s) that modulate a shared “slow” weight matrix, so N members cost N additional pairs of vectors per layer rather than N additional matrices. The members are trained jointly, in a single pass over inputs that are tiled N times.

Idea:

Rank-one, per-member perturbations of a shared weight matrix.

Representation:

Sampled second order with N members.

Advantages:

Close to a single model in memory, with a single training run, yet recovers much of the benefit of a full ensemble.

Disadvantages:

Inputs are tiled N times per batch, and diversity is lower than for fully independent members.

Reference:

[WTB20]

BatchEnsemble on Two Moons

BatchEnsemble on Two Moons

BatchEnsemble on MNIST

BatchEnsemble on MNIST

3.1.7. subensemble

The expensive backbone is shared, and only the head is ensembled. The split is either taken from an existing model, in which case the last head_layer layers form the head, or specified explicitly. Since all members see the same features, diversity is lower than for a full ensemble, but the cost is close to that of a single model.

Idea:

One shared feature extractor and N independently initialized heads.

Representation:

Sampled second order with N heads.

Advantages:

Close to a single model in cost: one backbone and one training run.

Disadvantages:

Lower diversity than a full ensemble, since all members share the same features.

Reference:

[VT19]

Sub-Ensemble on Two Moons

Sub-Ensemble on Two Moons

Sub-Ensemble on MNIST

Sub-Ensemble on MNIST

3.1.8. dare

An ensemble whose members are actively pushed apart. The anti-regularization term rewards large weights instead of penalizing them, but is only active as long as the training loss stays below a threshold, so that the fit to the training data is maintained. Members with large weights tend to disagree away from the training data, which is precisely where conventional ensembles tend to collapse into agreement; as a result, the epistemic estimates under distribution shift improve.

Idea:

Deep anti-regularized ensembles: increase member diversity by rewarding, rather than penalizing, large weights.

Representation:

Sampled second order with N members.

Advantages:

Wider and more informative epistemic estimates under distribution shift, where ordinary ensembles tend to collapse into agreement.

Disadvantages:

N times the training cost, plus the anti-regularization term and the loss threshold it requires.

Reference:

[dMDMV23]

DARE on Two Moons

DARE on Two Moons

DARE on MNIST

DARE on MNIST

3.1.9. duq

Replaces the linear classification head by a set of learned per-class centroids in feature space and scores an input by its RBF-kernel distance to them. A single deterministic forward pass thus yields both the prediction and the uncertainty. During training, the centroids are updated by an exponential moving average, and a gradient penalty prevents feature collapse, i.e., out-of-distribution inputs being mapped close to the centroids.

Idea:

Distance to learned class centroids in feature space, in place of a softmax head.

Representation:

RBF kernel scores, presented as a categorical distribution; total uncertainty only.

Advantages:

A single deterministic forward pass yields both the prediction and the uncertainty.

Disadvantages:

Requires training with a gradient penalty and a modified head; does not separate aleatoric from epistemic uncertainty.

Reference:

[vASTG20]

DUQ on Two Moons

DUQ on Two Moons

DUQ on MNIST

DUQ on MNIST

3.1.10. ddu

Also distance-aware, but with a division of labor: spectral normalization keeps the feature extractor sensitive to changes in the input, so that distances in feature space remain meaningful, and a Gaussian mixture density fitted post-hoc on these features provides the epistemic score. The softmax entropy is retained as a measure of aleatoric uncertainty.

Idea:

Spectral normalization for a well-behaved feature space, plus a Gaussian density estimate over it.

Representation:

The feature-space density as epistemic score, and the softmax as first-order predictive distribution.

Advantages:

One forward pass, with the density fitted after training; strong out-of-distribution detection.

Disadvantages:

Spectral normalization must be applied during training, so the method is not a pure retrofit.

Reference:

[MKvA+23]

DDU on Two Moons

DDU on Two Moons

DDU on MNIST

DDU on MNIST

3.1.11. deup

Rather than deriving uncertainty from a posterior, DEUP predicts the model’s own error. A second head is trained on held-out data to regress the per-sample loss of the frozen main model, and at inference time this predicted loss serves as the epistemic score. Training therefore proceeds in two phases: first the classifier, then the error head, on data the classifier has not seen.

Idea:

Learn a direct regressor of the main model’s generalization error.

Representation:

A scalar error score, which the decomposition assigns entirely to the epistemic term.

Advantages:

One forward pass at inference, and no assumptions about a posterior.

Disadvantages:

Requires an additional head and a second training phase on a held-out split.

Reference:

[LJN+23]

DEUP on Two Moons

DEUP on Two Moons

DEUP on MNIST

DEUP on MNIST

3.1.12. sngp

Combines the two ingredients that make a single deterministic network distance-aware: spectral normalization of the hidden layers, which makes the representation approximately distance-preserving, and a Gaussian process output layer, approximated by random Fourier features with a Laplace-approximated posterior, whose uncertainty grows away from the training data.

Idea:

Spectrally normalized backbone with a Gaussian process output layer approximated by random Fourier features.

Representation:

A Gaussian over the logits, which yields a second-order predictive distribution in closed form.

Advantages:

Distance-aware uncertainty with a closed-form predictive distribution from a single forward pass.

Disadvantages:

A modified head, spectral normalization, and an additional covariance update pass at the end of training.

Reference:

[LLP+20]

SNGP Distance Awareness on 2D Toys

SNGP Distance Awareness on 2D Toys

SNGP on MNIST

SNGP on MNIST

3.1.13. evidential_classification

Here, the second order is carried by evidence rather than by sampling: the network directly outputs the concentration parameters of a Dirichlet distribution, which are interpreted as accumulated evidence for each class. The entire second-order distribution thus results from a single forward pass, and the epistemic signal lies in the total amount of evidence rather than in the spread across samples; low total evidence means, roughly, “I have not seen anything like this”. Note, however, that nothing in the training objective forces the evidence to decrease away from the training data.

Idea:

Predict Dirichlet concentrations as per-class evidence.

Representation:

Parameterized second order, i.e., a DirichletDistribution.

Advantages:

The entire second-order distribution from a single forward pass; low total evidence flags unfamiliar inputs.

Disadvantages:

Requires an evidential loss and a positive activation; nothing forces the evidence to decrease away from the data.

Reference:

[SKK18]

Evidential on Two Moons

Evidential on Two Moons

Evidential on MNIST

Evidential on MNIST

3.1.14. posterior_network

Addresses the main weakness of purely evidential losses, namely that nothing forces the evidence to decrease away from the data, by making the concentration parameters proportional to an estimated density. This density is provided by a normalizing flow over a low-dimensional latent space, so that regions with little training data receive little evidence by construction.

Idea:

Density-based Dirichlet concentrations via a normalizing flow in latent space.

Representation:

Parameterized second order, i.e., a DirichletDistribution.

Advantages:

Evidence decreases by construction where training data is sparse, which removes the main weakness of evidential losses; one forward pass.

Disadvantages:

Requires an encoder, one normalizing flow per class, and a Bayesian training loss; the per-class flows scale poorly in the number of classes.

Reference:

[CZugnerGunnemann20]

Posterior Network on Two Moons

Posterior Network on Two Moons

Posterior Network on MNIST

Posterior Network on MNIST

3.1.15. mahalanobis

Strictly speaking, a post-hoc out-of-distribution detector rather than a method for predictive uncertainty. Class-conditional Gaussians with a shared covariance matrix are fitted to the penultimate, and optionally intermediate, features of a trained network. The score is the Mahalanobis distance to the nearest class mean, optionally sharpened by a small FGSM-style perturbation of the input and combined across layers by logistic regression.

Idea:

Class-conditional Gaussians in feature space; the distance to the nearest one is the out-of-distribution score.

Representation:

A scalar out-of-distribution score alongside the base first-order prediction.

Advantages:

No retraining; a single fitting pass over the training features.

Disadvantages:

An out-of-distribution detector rather than a predictive-uncertainty method; the multi-layer combination requires both in- and out-of-distribution data.

Reference:

[LLLS18]

Mahalanobis OOD on Two Moons

Mahalanobis OOD on Two Moons

Mahalanobis OOD on MNIST

Mahalanobis OOD on MNIST

3.1.16. natural_posterior_network

Generalizes posterior_network to targets from the exponential family and is the natural first choice among the density-based methods. A single shared flow over the latent space provides log p(z), a linear classifier provides the class log-probabilities, and the Dirichlet parameters follow the Bayesian update alpha = alpha_prior + n(x) * chi(x), where n(x) is a pseudo-count derived from the density and scaled by a certainty budget. Sharing one flow across classes is what allows the method to scale to many classes.

Idea:

A Bayesian posterior update per input, with a density-derived pseudo-count as the evidence.

Representation:

Parameterized second order, i.e., a DirichletDistribution.

Advantages:

A single shared flow scales to many classes, unlike the per-class flows it replaces; one forward pass.

Disadvantages:

Requires an encoder, a shared normalizing flow, and a modified loss.

Reference:

[CBZugner+22]

Natural Posterior Network on Two Moons

Natural Posterior Network on Two Moons

Natural Posterior Network on MNIST

Natural Posterior Network on MNIST

3.1.17. prior_network

The simplest way of obtaining a Dirichlet distribution from an existing classifier: the logits are exponentiated and interpreted as concentration parameters. In its original form, the network is trained to output a flat Dirichlet on out-of-distribution data and a sharp one in-distribution, which requires out-of-distribution examples during training. Without them, the method amounts to little more than a reparameterization of the logits.

Idea:

An exponential activation on the logits turns them into Dirichlet concentrations.

Representation:

Parameterized second order, i.e., a DirichletDistribution.

Advantages:

One forward pass and no architectural change; the simplest route to a Dirichlet distribution.

Disadvantages:

The intended training scheme requires out-of-distribution data; without it, the method is merely a reparameterization of the logits.

Reference:

[MG18]

Prior Network on Two Moons

Prior Network on Two Moons

Prior Network on MNIST

Prior Network on MNIST

3.1.18. evidential_regression

The regression counterpart of evidential classification. The final linear layer is replaced by a head that outputs the four parameters of a Normal-Inverse-Gamma distribution, i.e., a distribution over both the mean and the variance of the target. A single forward pass therefore yields an aleatoric estimate, the expected variance, as well as an epistemic one, the variance of the mean.

Idea:

Predict a Normal-Inverse-Gamma prior over the mean and variance of the target.

Representation:

Parameterized second order over a real-valued target.

Advantages:

A single forward pass yields both an aleatoric (expected variance) and an epistemic (variance of the mean) estimate.

Disadvantages:

A modified head and an evidential regression loss.

Reference:

[ASSR20]

Evidential Regression on a 1D Example

Evidential Regression on a 1D Example

3.1.19. het_net

Models aleatoric uncertainty only, and is included because no other method on this page captures correlated label noise. Instead of a single logit vector, the network predicts a full input-dependent covariance over the logits, parameterized in low rank to keep it affordable. This is useful where labels are genuinely ambiguous, but it says nothing about whether the model has seen similar data before.

Idea:

Input-dependent, low-rank correlated noise over the logits.

Representation:

A Gaussian over the logits; aleatoric uncertainty only.

Advantages:

Models correlated, input-dependent label noise at the cost of one forward pass and a few additional output channels.

Disadvantages:

Says nothing about epistemic uncertainty; relies on a sampled softmax during training.

Reference:

[CMK+21]

Het-Net on Two Moons

Het-Net on Two Moons

Het-Net on MNIST

Het-Net on MNIST

3.1.20. Choosing Among Them

The first question is usually whether the model can be retrained at all. If it cannot, the options reduce to laplace, mahalanobis, and, provided the network was trained with dropout, dropout. If N training runs are affordable, ensemble is the baseline against which everything else should be measured, and batchensemble or subensemble recover much of its benefit for a fraction of the memory. If inference latency is the binding constraint, a parameterized method is the natural choice, such as natural_posterior_network for classification or evidential_regression for regression. If the goal is specifically out-of-distribution detection, the distance-aware methods ddu, sngp, and duq are preferable.

Whichever method is chosen, whether it actually works can only be decided empirically, with the tools described in Evaluating Uncertainty.

3.1.21. Full API

dropout

Create a Dropout predictor from a base predictor based on [GG16].

dropconnect

Create a DropConnect predictor from a base predictor based on [MYM+21].

bayesian

Create a Bayesian predictor from a base predictor based on [BCKW15].

laplace

Laplace approximation method.

ensemble

Create an ensemble predictor from a base predictor based on [LPB17].

batchensemble

Create a BatchEnsemble predictor from a base predictor based on [WTB20].

subensemble

Create a subensemble predictor from a base model and optional head based on [VT19].

dare

Create a DARE predictor from a base predictor based on [dMDMV23].

duq

Replace the classifier head with an RBF centroid head based on [vASTG20].

ddu

Add spectral normalization and a Gaussian density head based on [MKvA+23].

deup

Transform a model for Direct Epistemic Uncertainty Prediction [LJN+23].

sngp

Wrap a model with SNGP (Spectral-normalized Neural Gaussian Process) based on [LLP+20].

evidential_classification

Create an Evidential Deep Learning predictor based on [SKK18].

posterior_network

Create a Posterior Network predictor from an encoder based on [CZugnerGunnemann20].

mahalanobis

Turn a classifier into a Mahalanobis-distance OOD detector based on [LLLS18].

natural_posterior_network

Create a Natural Posterior Network predictor based on [CBZugner+22].

prior_network

Create a Prior Network predictor from a base predictor based on [MG18].

evidential_regression

Replace the final linear layer with a Normal-Inverse-Gamma head based on [ASSR20].

het_net

Create a HetNets predictor from a base predictor base on [CMK+21].