Uncertainty Quantification for Machine Learning

Turn any model into one that knows what it doesn’t know.

probly is a library-agnostic toolkit for uncertainty representation and quantification in machine learning. Make any PyTorch, Flax/JAX, scikit-learn, River, or Hugging Face model uncertainty-aware in a single line, then represent, quantify, and decompose its predictive uncertainty into aleatoric and epistemic components. It ships 40+ methods, from Bayesian nets and deep ensembles to evidential, credal, and conformal prediction, all behind the same unified API.

Python 3.12+ · MIT license · Published at NeurIPS 2026 (oral presentation)

Uncertainty in a few lines

Transform a model, represent its predictions, quantify the uncertainty, and evaluate it on a downstream task:

from probly.method import dropout
from probly.representer import representer
from probly.quantification import quantify
from probly.evaluation.ood import evaluate_ood

# transform: keep dropout active at inference (MC dropout)
model = dropout(net, p=0.25, predictor_type="logit_classifier")
train(model)  # train as usual

# represent: turn stochastic forward passes into a predictive distribution
rep = representer(model, num_samples=50)
out_id, out_ood = rep.represent(data_id), rep.represent(data_ood)

# quantify epistemic (model) uncertainty
eu_id = quantify(out_id).epistemic.detach().numpy()
eu_ood = quantify(out_ood).epistemic.detach().numpy()

# evaluate: does uncertainty separate in-distribution from out-of-distribution?
print(evaluate_ood(eu_id, eu_ood))  # {'auroc': 0.94}

Swap dropout for ensemble, laplace, sngp, or any other method and the rest of the pipeline stays the same:

See it in action

Catch LLM hallucinations with semantic entropy

Sample several answers, cluster them by meaning, and decompose the resulting semantic entropy. A model that says the same thing five ways is confident; one that says five different things is likely hallucinating.

Animated demo: a factual question collapses to one meaning with zero uncertainty, while a trick question scatters into seven meanings with high uncertainty Animated demo: a factual question collapses to one meaning with zero uncertainty, while a trick question scatters into seven meanings with high uncertainty
https://github.com/pwhofman/probly/blob/main/examples/llm/semantic_entropy.py
Regression: a band that widens off the data

A deep ensemble turns member disagreement into an epistemic band: tight near the training data, wide in the gap and beyond.

Deep-ensemble regression band, tight over the data and wide in the gap Deep-ensemble regression band, tight over the data and wide in the gap
Ensemble Regression Uncertainty
Out-of-distribution detection at ImageNet scale

A credal Bayesian network scores in-distribution ImageNet near zero and out-of-distribution iNaturalist much higher.

Histogram of uncertainty scores for in-distribution ImageNet and OoD iNaturalist Histogram of uncertainty scores for in-distribution ImageNet and OoD iNaturalist
Evaluating Uncertainty
Credal sets

Represent ignorance as a set of distributions on the simplex.

Convex credal set
Conformal prediction

Intervals with finite-sample coverage guarantees for any regressor.

Regression Conformal Prediction — sklearn
Streaming data

Track aleatoric and epistemic uncertainty online and spot concept drift.

Streaming uncertainty with ARFClassifier
Hugging Face vision

Per-pixel uncertainty for a pretrained depth-estimation transformer.

Depth estimation uncertainty with transformers
Second-order distributions

A Dirichlet over the simplex: concentrated with evidence, flat without.

Second order: a parameterized Dirichlet
Active learning

Query the samples the model is most uncertain about and learn faster.

Active Learning with sklearn - Margin vs Random

One pipeline, 20+ methods

Because every method shares the same interface, comparing them is a matter of changing one line. Out-of-distribution detection on ImageNet with a ResNet50, far-OoD on the left and near-OoD on the right, mean over three runs:

Bar charts of OoD detection AUROC for 20+ uncertainty methods on ImageNet Bar charts of OoD detection AUROC for 20+ uncertainty methods on ImageNet

Philosophy

Library-agnostic

Works with the framework you already use: PyTorch, Flax / JAX, scikit-learn, River, Hugging Face, and more.

Model-agnostic

From linear models to CNNs, graph networks, and transformer-based LLMs: probly fits into the models and pipelines you already have.

Ante-hoc and post-hoc

Bring your own model and let probly transform it, or build uncertainty-native models from the start with the built-in layers and methods.