Uncertainty Quantification for Machine Learning¶
Turn any model into one that knows what it doesn’t know.
probly is a library-agnostic toolkit for uncertainty representation and quantification
in machine learning. Make any PyTorch, Flax/JAX, scikit-learn, River, or Hugging Face model
uncertainty-aware in a single line, then represent, quantify, and decompose its
predictive uncertainty into aleatoric and epistemic components. It ships 40+ methods,
from Bayesian nets and deep ensembles to evidential, credal, and conformal prediction, all behind
the same unified API.
Uncertainty in a few lines¶
Transform a model, represent its predictions, quantify the uncertainty, and evaluate it on a downstream task:
from probly.method import dropout
from probly.representer import representer
from probly.quantification import quantify
from probly.evaluation.ood import evaluate_ood
# transform: keep dropout active at inference (MC dropout)
model = dropout(net, p=0.25, predictor_type="logit_classifier")
train(model) # train as usual
# represent: turn stochastic forward passes into a predictive distribution
rep = representer(model, num_samples=50)
out_id, out_ood = rep.represent(data_id), rep.represent(data_ood)
# quantify epistemic (model) uncertainty
eu_id = quantify(out_id).epistemic.detach().numpy()
eu_ood = quantify(out_ood).epistemic.detach().numpy()
# evaluate: does uncertainty separate in-distribution from out-of-distribution?
print(evaluate_ood(eu_id, eu_ood)) # {'auroc': 0.94}
Swap dropout for ensemble, laplace, sngp, or any other method and the rest of the
pipeline stays the same:
See it in action¶
Sample several answers, cluster them by meaning, and decompose the resulting semantic entropy. A model that says the same thing five ways is confident; one that says five different things is likely hallucinating.
A deep ensemble turns member disagreement into an epistemic band: tight near the training data, wide in the gap and beyond.
A credal Bayesian network scores in-distribution ImageNet near zero and out-of-distribution iNaturalist much higher.
Represent ignorance as a set of distributions on the simplex.
Intervals with finite-sample coverage guarantees for any regressor.
Track aleatoric and epistemic uncertainty online and spot concept drift.
Per-pixel uncertainty for a pretrained depth-estimation transformer.
A Dirichlet over the simplex: concentrated with evidence, flat without.
Query the samples the model is most uncertain about and learn faster.
One pipeline, 20+ methods¶
Because every method shares the same interface, comparing them is a matter of changing one line. Out-of-distribution detection on ImageNet with a ResNet50, far-OoD on the left and near-OoD on the right, mean over three runs:
Philosophy¶
Works with the framework you already use: PyTorch, Flax / JAX, scikit-learn, River, Hugging Face, and more.
From linear models to CNNs, graph networks, and transformer-based LLMs: probly fits into
the models and pipelines you already have.
Bring your own model and let probly transform it, or build uncertainty-native models
from the start with the built-in layers and methods.









