probly.conformal_prediction.scores.common

Common structures for conformal prediction scores.

Classes

ClassificationScore(model, score_func[, ...])

Generic implementation for classification scores.

ClassificationScoreProtocol(*args, **kwargs)

Nonconformity scores for classification tasks.

RegressionScore(model, score_func[, ...])

Generic implementation for regression scores.

RegressionScoreProtocol(*args, **kwargs)

Nonconformity scores for regression (e.g.. Residuals).

Score(*args, **kwargs)

Interface for nonconformity scores used in split conformal prediction.
class probly.conformal_prediction.scores.common.ClassificationScore(model, score_func, randomize=False, random_state=None)[source]

Bases: ClassificationScoreProtocol

Generic implementation for classification scores.

Handles APS, LAC, RAPS, SAPS by delegating to a score_func. Randomization must be built into the score_func if needed.

Parameters:

  • model (Predictor)

  • score_func (Callable[[Any], Any])

  • randomize (bool)

  • random_state (int | None)

calibration_nonconformity(x_cal, y_cal, probs=None)[source]

Compute calibration scores (vectorized, backend-agnostic).

Parameters:

  • x_cal (Sequence[Any])

  • y_cal (Sequence[Any])

  • probs (Any | None)

Return type:

npt.NDArray[np.floating]

predict_nonconformity(x_test, probs=None)[source]

Compute scores for all labels (stays on original device).

Parameters:

  • x_test (Sequence[Any])

  • probs (Any | None)

Return type:

Any

class probly.conformal_prediction.scores.common.ClassificationScoreProtocol(*args, **kwargs)[source]

Bases: Score, Protocol

Nonconformity scores for classification tasks.

calibration_nonconformity: 1D scores from Score. predict_nonconformity: 2D scores (n_instances, n_labels).

calibration_nonconformity(x_cal, y_cal)

Return 1D array of nonconformity scores for calibration instances.

Parameters:

  • x_cal (Sequence[Any])

  • y_cal (Sequence[Any])

Return type:

npt.NDArray[np.floating]

predict_nonconformity(x_test, probs=None)[source]

Return 2D score matrix of shape (n_instances, n_labels).

Parameters:

  • x_test (Sequence[Any])

  • probs (Any | None)

Return type:

Any

class probly.conformal_prediction.scores.common.RegressionScore(model, score_func, interval_func=None)[source]

Bases: RegressionScoreProtocol

Generic implementation for regression scores.

Handles AbsoluteError, CQR, etc.

Parameters:

  • model (Predictor)

  • score_func (Callable[[Any, Any], Any])

  • interval_func (Callable[[Any, float], Any] | None)

calibration_nonconformity(x_cal, y_cal, y_pred=None)[source]

Compute calibration scores.

Parameters:

  • x_cal (Sequence[Any])

  • y_cal (Sequence[Any])

  • y_pred (Any | None)

Return type:

npt.NDArray[np.floating]

construct_intervals(y_hat, threshold)[source]

Construct prediction intervals (Preserves backend).

Parameters:
Return type:

Any

predict_nonconformity(x_test)[source]

For regression, return predictions for interval construction.

Parameters:

x_test (Sequence[Any])

Return type:

Any

class probly.conformal_prediction.scores.common.RegressionScoreProtocol(*args, **kwargs)[source]

Bases: Score, Protocol

Nonconformity scores for regression (e.g.. Residuals).

calibration_nonconformity: 1D scores (|y - y_hat|, standardized Residuals, …). predict_nonconformity: 1D scores or local scales (n_instances,).

calibration_nonconformity(x_cal, y_cal)

Return 1D array of nonconformity scores for calibration instances.

Parameters:

  • x_cal (Sequence[Any])

  • y_cal (Sequence[Any])

Return type:

npt.NDArray[np.floating]

construct_intervals(y_hat, threshold)[source]

Construct prediction intervals based on model output and threshold.

Parameters:

  • y_hat (Any) – Model output (n_samples, ) or (n_samples, 2) etc.

  • threshold (float) – Calibrated q-hat.

Returns:

Intervals as (n_samples, 2) matrix [lower, upper].

Return type:

Any

predict_nonconformity(x_test)[source]

Return 1D scores or local scales of shape (n_instances,).

Parameters:

x_test (Sequence[Any])

Return type:

Any

class probly.conformal_prediction.scores.common.Score(*args, **kwargs)[source]

Bases: Protocol

Interface for nonconformity scores used in split conformal prediction.

Each score (APS, LAC, RAPS, …) must implement: - calibration_nonconformity: used on calibration data. - predict_nonconformity: used on test data, must return a score matrix

of shape according to the specific score type (classification or regression).

calibration_nonconformity(x_cal, y_cal)[source]

Return 1D array of nonconformity scores for calibration instances.

Parameters:

  • x_cal (Sequence[Any])

  • y_cal (Sequence[Any])

Return type:

npt.NDArray[np.floating]