Note
Go to the end to download the full example code.
Bayesian Neural Network on Two Moons¶
Replace point-estimate weights with distributions and train them with the ELBO loss. Every forward pass samples new weights, so predictions are inherently stochastic.
from __future__ import annotations
from sklearn.datasets import make_moons
import torch
from probly.representer import representer
from probly.transformation import bayesian
from probly.losses.torch import elbo_loss
from probly.transformation.bayesian import collect_kl_divergence
from examples.utils.model import MLPClassifier
from examples.utils.plotting import plot_example_uncertainty
Setup¶
X, y = make_moons(n_samples=500, noise=0.05, random_state=0)
X_tensor = torch.from_numpy(X).float()
y_tensor = torch.from_numpy(y).long()
Model¶
base_model = MLPClassifier()
bayesian_model = bayesian(
base_model,
use_base_weights=False, # initialize posterior means randomly rather than from base_model
posterior_std=0.05, # initial posterior std; small = near-deterministic start
prior_mean=0.0,
prior_std=1.0, # smaller = stronger regularization toward zero
predictor_type="logit_classifier",
)
Training¶
elbo_loss(out, y, kl, kl_penalty=beta) computes: cross_entropy(out, y) + beta * kl. beta = 1/N scales the KL so its magnitude is independent of dataset size. collect_kl_divergence walks the model and sums the KL from every BayesianLinear layer, which must be called after each forward pass because each forward pass draws new weight samples.
Uncertainty Evaluation¶
bayesian_model.eval()
rep = representer(bayesian_model, num_samples=200)
plot = plot_example_uncertainty(X, y, rep, title="Bayesian Predictive Uncertainty", notion="total")
plot.show()

Total running time of the script: (0 minutes 5.466 seconds)