References and Further Reading¶
This section provides the theoretical background for the models implemented in probly as well
as references to relevant literature and acknowledgments.
Related Research Papers¶
The core functionality of this package is built upon the following foundational research papers. We recommend reading the papers to gain a deeper understanding of the algorithms and methodologies used:
Moloud et al. Abdar. A review of uncertainty quantification in deep learning. Information Fusion, 2021.
Joaqu\'ın Abellán, George J. Klir, and Seraf\'ın Moral. Disaggregated total uncertainty measure for credal sets. Int. J. Gen. Syst., 35(1):29–44, 2006. URL: https://doi.org/10.1080/03081070500473490, doi:10.1080/03081070500473490.
Joaqu\'ın Abellán and Seraf\'ın Moral. A non-specificity measure for convex sets of probability distributions. Int. J. Uncertain. Fuzziness Knowl. Based Syst., 8(3):357–368, 2000. URL: https://doi.org/10.1142/S0218488500000253, doi:10.1142/S0218488500000253.
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. 2020. URL: https://proceedings.neurips.cc/paper/2020/hash/aab085461de182608ee9f607f3f7d18f-Abstract.html.
Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. CoRR, 2021. URL: https://arxiv.org/abs/2107.07511, arXiv:2107.07511.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. ICML, 2017.
Thomas Augustin, Frank Coolen, Gert de Cooman, and Matthias Troffaes. Introduction to Imprecise Probabilities. Wiley, 2014.
Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? CoRR, 2020. URL: https://arxiv.org/abs/2006.07159, arXiv:2006.07159.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, New York, NY, USA, 2006. ISBN 978-0-387-31073-2. URL: https://link.springer.com/book/10.1007/978-0-387-45528-0, doi:10.1007/978-0-387-45528-0.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, 1613–1622. JMLR.org, 2015. URL: http://proceedings.mlr.press/v37/blundell15.html.
Michele Caprio, Souradeep Dutta, Kuk Jin Jang, Vivian Lin, Radoslav Ivanov, Oleg Sokolsky, and Insup Lee. Credal bayesian deep learning. arXiv preprint arXiv:2302.09656, 2023.
Bob Carpenter, Andrew Gelman, Matthew D. Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A probabilistic programming language. Journal of Statistical Software, 76(1):1–32, 2017. URL: https://www.jstatsoft.org/article/view/v076i01, doi:10.18637/jss.v076.i01.
Bertrand Charpentier, Oliver Borchert, Daniel Zügner, Simon Geisler, and Stephan Günnemann. Natural posterior network: deep bayesian predictive uncertainty for exponential family distributions. In International Conference on Learning Representations. 2022.
Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. Posterior network: uncertainty estimation without OOD samples via density-based pseudo-counts. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. 2020. URL: https://proceedings.neurips.cc/paper/2020/hash/0eac690d7059a8de4b48e90f14510391-Abstract.html.
Mark Collier, Basil Mustafa, Efi Kokipoulou, Rodolphe Jenatton, and Jesse Berent. Correlated input-dependent label noise in large-scale image classification. arXiv preprint arXiv:2105.10305, 2021.
John P. Cunningham and others. Ensemble subsampling for efficient uncertainty estimation. In ICLR Workshop on Uncertainty and Robustness in Deep Learning. 2020.
Fabio Cuzzolin. The intersection probability: betting with probability intervals. CoRR, 2022. URL: https://arxiv.org/abs/2201.01729, arXiv:2201.01729.
Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, 1192–1201. PMLR, 2018. URL: http://proceedings.mlr.press/v80/depeweg18a.html.
Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does it matter? Structural Safety, 2009.
Thomas G. Dietterich. Ensemble methods in machine learning. In International Workshop on Multiple Classifier Systems, 1–15. Springer, 2000.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: a loss landscape perspective. arXiv preprint arXiv:1912.02757, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: representing model uncertainty in deep learning. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, 1050–1059. JMLR.org, 2016. URL: http://proceedings.mlr.press/v48/gal16.html.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: representing model uncertainty in deep learning. ICML, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML), 1050–1059. 2016.
Yotam Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In NeurIPS. 2017.
Andrew Gelman and Jennifer Hill. Data Analysis Using Regression and Multilevel/Hierarchical Models. Analytical Methods for Social Research. Cambridge University Press, New York, NY, 2007. ISBN 978-0-521-86706-1. URL: https://doi.org/10.1017/CBO9780511790942, doi:10.1017/CBO9780511790942.
Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. JASA, 2007.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Andreas Griewank and Andrea Walther. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. Volume 105 of Other Titles in Applied Mathematics. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2 edition, 2008. ISBN 978-0-89871-659-7. URL: https://doi.org/10.1137/1.9780898717761, doi:10.1137/1.9780898717761.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. ICML, 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, 1321–1330. PMLR, 2017. URL: http://proceedings.mlr.press/v70/guo17a.html.
Marton Havasi and others. Training independent subnetworks for robust prediction. Proceedings of the National Academy of Sciences, 2021.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ICLR, 2017.
Paul Hofman, Timo Löhr, Maximilian Muschalik, Yusuf Sale, and Eyke Hüllermeier. Efficient credal prediction through decalibration. In The Fourteenth International Conference on Learning Representations. 2026.
Robin J. Hyndman and George Athanasopoulos. Forecasting: Principles and Practice. OTexts, Australia, 2nd edition, 2018. URL: https://otexts.com/fpp2/.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning, 2021. arXiv:1910.09457.
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87, 1991.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 2017.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR 2014). 2014. arXiv:1312.6114. URL: https://arxiv.org/abs/1312.6114.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 6402–6413. 2017. URL: https://proceedings.neurips.cc/paper/2017/hash/9ef2ed4b7fd2c810847ffa5fa85bce38-Abstract.html.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NeurIPS, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 6402–6413. 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
Julian Lienen and Eyke Hüllermeier. From label smoothing to label relaxation. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, 8583–8591. AAAI Press, 2021. URL: https://doi.org/10.1609/aaai.v35i10.17041, doi:10.1609/AAAI.V35I10.17041.
Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, 2999–3007. IEEE Computer Society, 2017. URL: https://doi.org/10.1109/ICCV.2017.324, doi:10.1109/ICCV.2017.324.
Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax-Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. arXiv preprint arXiv:2006.10108, 2020.
Timo Löhr, Paul Hofman, Felix Mohr, and Eyke Hüllermeier. Credal prediction based on relative likelihood. arXiv preprint arXiv:2505.22332, 2025.
Andrey Malinin, Sergey Chervontsev, Ivan Provilkov, and Mark Gales. Regression prior networks. arXiv preprint arXiv:2006.11590, 2020.
Andrey Malinin and Mark J. F. Gales. Predictive uncertainty estimation via prior networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, 7047–7058. 2018. URL: https://proceedings.neurips.cc/paper/2018/hash/3ea2db50e62ceefceaf70a9d9a56a6f4-Abstract.html.
Charlie Marsh. Astral docs. https://docs.astral.sh/uv/, 2024. Accessed: 2024-01-15.
Aryan Mobiny, Hien Van Nguyen, Supratik Moulik, Naveen Garg, and Carol C. Wu. Dropconnect is effective in modeling uncertainty of bayesian deep networks. CoRR, 2019. URL: http://arxiv.org/abs/1906.04569, arXiv:1906.04569.
Aryan Mobiny, Pengyu Yuan, Supratik K. Moulik, Naveen Garg, Carol C. Wu, and Hien Van Nguyen. Dropconnect is effective in modeling uncertainty of bayesian deep networks. Scientific Reports, 11:5458, 2021. doi:10.1038/s41598-021-84854-x.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip H.S. Torr, and Yarin Gal. Deep deterministic uncertainty: a new simple baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 24384–24394. 2023.
n/a. Google python style guide. https://google.github.io/styleguide/pyguide.html#docstrings, 2026. Accessed: 2025-01-15.
Vu-Linh Nguyen, Haifei Zhang, and Sébastien Destercke. Credal ensembling in multi-class classification. Mach. Learn., 114(1):19, 2025. URL: https://doi.org/10.1007/s10994-024-06703-y, doi:10.1007/S10994-024-06703-Y.
David Opitz and Richard Maclin. Popular ensemble methods: an empirical study. Journal of Artificial Intelligence Research, 11:169–198, 1999.
Yaniv Ovadia and others. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 2019.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1–64, 2021. URL: https://www.jmlr.org/papers/volume22/19-1028/19-1028.pdf.
Suvendu K. Pati. Best practices for organizing and coding data science projects — part 1. 2025. The Deep Hub (Medium), posted March 27, 2025. URL: https://medium.com/thedeephub/best-practices-for-organizing-and-coding-data-science-projects-part-1-72539e14a7a0.
Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, 9616–9625. IEEE, 2019. URL: https://doi.org/10.1109/ICCV.2019.00971, doi:10.1109/ICCV.2019.00971.
Danilo J. Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, 1530–1538. PMLR, 2015. URL: https://proceedings.mlr.press/v37/rezende15.html.
Yusuf Sale, Viktor Bengs, Michele Caprio, and Eyke Hüllermeier. Second-order uncertainty quantification: A distance-based approach. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL: https://openreview.net/forum?id=VJjjNrUi8j.
Lars Schmarje, Vasco Grossmann, Claudius Zelenka, Sabine Dippel, Rainer Kiko, Mariusz Oszust, Matti Pastell, Jenny Stracke, Anna Valros, Nina Volkmann, and Reinhard Koch. Is one annotation enough? - A data-centric image classification benchmark for noisy and ambiguous label estimation. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. 2022. URL: http://papers.nips.cc/paper_files/paper/2022/hash/d6c03035b8bc551f474f040fe8607cab-Abstract-Datasets_and_Benchmarks.html.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. arXiv preprint arXiv:1910.09457, 2019.
Murat Sensoy, Lance M. Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, 3183–3193. 2018. URL: https://proceedings.neurips.cc/paper/2018/hash/a981f2b708044d6fb4a71a1463242520-Abstract.html.
Claude Shannon. A mathematical theory of communication. Bell System Technical Journal, 1948.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, and others. Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations. 2017.
Dustin et al. Tran. Bayesian layers: a module for neural network uncertainty. In NeurIPS. 2019.
Theodoros Tsiligkaridis. Information robust dirichlet networks for predictive uncertainty estimation. arXiv preprint arXiv:1910.04819, 2019.
Xiaoguang Tu, Zhi He, Yi Huang, Zhi-Hao Zhang, Ming Yang, and Jian Zhao. An overview of large AI models and their applications. Visual Intelligence, 2(1):34, 2024. URL: https://doi.org/10.1007/s44267-024-00065-8, doi:10.1007/s44267-024-00065-8.
Ankush Jitendrakumar Tyagi. Scaling deep learning models: challenges and solutions for large-scale deployments. World Journal of Advanced Engineering Technology and Sciences, 16(2):10–20, 2025. URL: https://doi.org/10.30574/wjaets.2025.16.2.1252, doi:10.30574/wjaets.2025.16.2.1252.
Matías Valdenegro-Toro. Deep sub-ensembles for fast uncertainty estimation in image classification. Machine Learning, 2019. arXiv:1910.08168.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Proceedings of Machine Learning Research, 9690–9700. PMLR, 2020. URL: http://proceedings.mlr.press/v119/van-amersfoort20a.html.
vitkl. Softplus transform as a more numerically stable way to enforce positive constraint. GitHub issue #855 on the \emph NumPyro repository, December 2020. Issue opened December 31, 2020. URL: https://github.com/pyro-ppl/numpyro/issues/855.
Kaizheng Wang, Fabio Cuzzolin, Shireen K Manchingal, Keivan Shariatmadar, David Moens, and Hans Hallez. Credal deep ensembles for uncertainty quantification. Advances in Neural Information Processing Systems, 37:79540–79572, 2024.
Kaizheng Wang, Fabio Cuzzolin, Keivan Shariatmadar, David Moens, and Hans Hallez. Credal wrapper of model averaging for uncertainty estimation in classification. arXiv preprint arXiv:2405.15047, 2024.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL: https://openreview.net/forum?id=Sklf1yrYDr.
Yiyou et al. Yang. Generalized ood detection via bayesian uncertainty estimation. CVPR, 2021.
Martin Zinkevich. Rules of machine learning: best practices for ML engineering. Google Developers, n.d. Google Developers guide, accessed 19 November 2025. URL: https://developers.google.com/machine-learning/guides/rules-of-ml.
Contributors to the Pyro Project. NumPyro transforms module. Documentation, 2019. NumPyro documentation, transforms module, version 0.4.1. URL: https://num.pyro.ai/en/0.4.1/_modules/numpyro/distributions/transforms.html.
Python Software Foundation. Python Software Foundation Code of Conduct – Python Software Foundation Policies. https://policies.python.org/python.org/code-of-conduct/, 2018. Accessed: 2025-12-07.
Python Software Foundation. The python profilers. n.d. In Python Documentation. URL: https://docs.python.org/3/library/profile.html.
Stan Development Team. Constraint transforms. 2025. In Stan Reference Manual, Version 2.37. URL: https://mc-stan.org/docs/reference-manual/transforms.html.
Stan Development Team. 10.2 lower bounded scalar. n.d. In Stan Reference Manual, Version 2.22. URL: https://mc-stan.org/docs/2_22/reference-manual/lower-bound-transform-section.html.
TensorFlow Probability. Tfp.bijectors.bijector; tfp.bijectors.softplus. 2023. In TensorFlow Probability API documentation. URL: https://www.tensorflow.org/probability/api_docs/python/tfp/bijectors/Bijector.
TensorFlow Probability. Module: tfp.bijectors. n.d. In TensorFlow Probability API documentation. URL: https://www.tensorflow.org/probability/api_docs/python/tfp/bijectors.
The JAX Authors. Pseudorandom numbers. 2024. In JAX documentation. URL: https://docs.jax.dev/en/latest/random-numbers.html.