Reference / API reference / dorothy / dorothy.inference / dorothy.inference.evaluation_utils

dorothy.inference.evaluation_utils¶

Evaluate a trained model with the training validation pipeline.

evaluate_model() runs a trained model over a set of rows using the same batch construction, label normalization and predicted-scatter definition as the trainer, and scores every output head (label source) in both normalized space (directly comparable to the training validation metrics, unless calibration factors are given) and physical units. It also returns the physical-unit predictions for plotting.

The per-layout helpers (evaluate_on_test_set, evaluate_on_dense_data, evaluate_on_single_survey_data and the normalize_* and build_batch_from_sparse utilities) wrap it and report the primary head.

Functions

`build_batch_from_sparse`(data, global_indices)	Build one dense batch from sparse storage (same as training).
`evaluate_model`(model, normalizer, config, ...)	Evaluate a trained model on the given rows of a dataset.
`evaluate_on_dense_data`(model, X_dict, ...[, ...])	Evaluate on dense multi-survey dict data (primary head).
`evaluate_on_single_survey_data`(model, X, y, ...)	Evaluate on single-survey arrays (primary head).
`evaluate_on_test_set`(model, data, ...[, ...])	Evaluate on sparse multi-survey data (primary head).
`normalize_dense_data`(X_dict, y_dict, ...)	Normalize dense dict labels as in training.
`normalize_single_survey_data`(X, y, normalizer)	Normalize single-survey labels as in training.
`normalize_sparse_data`(data, normalizer)	Normalize the labels of sparse data as in training.

Classes

HeadEvaluation(label_source, normalized, ...)

Evaluation of one output head (label source) on a set of stars.

class dorothy.inference.evaluation_utils.HeadEvaluation(label_source, normalized, physical, parameter_names, y_pred, pred_scatter, y_true, y_err, mask, z_label_errors, calibration, uncalibrated)[source]¶

Bases: object

Evaluation of one output head (label source) on a set of stars.

label_source¶: Label source name of the head.

normalized¶: Metrics in normalized space (comparable to training validation), or in raw model units without a normalizer.

physical¶: Metrics in physical units (None without a normalizer).

parameter_names¶: Parameter names, in column order.

y_pred, pred_scatter, y_true, y_err, mask: Physical-unit arrays of shape (n_stars, n_params) (model units without a normalizer).

z_label_errors¶: The label errors that the z-scores add to the predicted scatter: y_err for a model with an AdditiveOutput, else None.

calibration¶: Parameter name -> the calibration factor that scales the predicted scatter of the metrics and of pred_scatter, or None for the raw scatter.

uncalibrated¶: Metrics in normalized units with the raw scatter, when calibration is set; else None.

to_dict()[source]¶

Metrics, and the calibration factors if any, as a JSON-serializable dict.

Returns:: Dict with normalized_space, physical_space (if a normalizer was used) and, when calibration is set, calibration_factors and uncalibrated_normalized_space.
Return type:: dict

dorothy.inference.evaluation_utils.evaluate_model(model, normalizer, config, data, rows, batch_size=1024, device='auto', calibration=None)[source]¶

Evaluate a trained model on the given rows of a dataset.

The model reads no label errors, so it predicts with its missing vector. The z-scores divide the residuals by s for a model with label-aware uncertainty, and by sqrt(sigma_y^2 + s^2) for a model with an AdditiveOutput.

Without calibration factors, the metrics on the validation stars equal the ones recorded during training. A head with factors scales its predicted scatter as prediction does, so its metrics and pred_scatter describe the calibrated uncertainties; it also reports the normalized-unit metrics of the raw scatter.

Parameters:

model (nn.Module) – Trained MLP or MultiHeadMLP (moved to device, eval mode).
normalizer (LabelNormalizer | None) – Label normalizer from training (None if not used).
config (ExperimentConfig) – The checkpoint’s experiment config.
data (TrainingData) – Dataset loaded with load_training_data().
rows (np.ndarray) – Row indices to evaluate (e.g. the test split).
batch_size (int) – Batch size for inference.
device (str | torch.device) – Device for inference.
calibration (dict[str, dict[str, float]] | None) – Label source -> parameter name -> calibration factor, as in LoadedCheckpoint.calibration. None evaluates the raw scatter.

Returns:

Label source name -> HeadEvaluation, in the model’s head order. A single-head model’s head is named after the configured label source.

Return type:

dict[str, HeadEvaluation]

dorothy.inference.evaluation_utils.evaluate_on_test_set(model, data, test_indices, normalizer, config, batch_size=1024, device='auto')[source]¶

Evaluate on sparse multi-survey data (primary head).

Parameters:

model (nn.Module) – Trained MLP or MultiHeadMLP.
data (SparseMergedData) – Sparse merged data with physical labels.
test_indices (np.ndarray) – Row indices to evaluate.
normalizer (LabelNormalizer | None) – Label normalizer from training (None if not used).
config (ExperimentConfig) – The checkpoint’s experiment config.
batch_size (int) – Batch size for inference.
device (str | torch.device) – Device for inference.

Returns:

Tuple of (normalized-space result, physical-space result) of the primary head; the second is None without a normalizer.

Return type:

tuple[EvaluationResult, EvaluationResult | None]

dorothy.inference.evaluation_utils.evaluate_on_dense_data(model, X_dict, y_dict, has_data_dict, test_indices, normalizer, config, batch_size=1024, device='auto')[source]¶

Evaluate on dense multi-survey dict data (primary head).

Parameters:

model (nn.Module) – Trained MLP or MultiHeadMLP.
X_dict (dict[str, np.ndarray]) – Survey name -> spectra, shape (N, 3, n_wavelengths).
y_dict (dict[str, np.ndarray]) – Label source -> physical labels, shape (N, 3, n_params).
has_data_dict (dict[str, np.ndarray]) – Survey name -> boolean array of shape (N,), True where the star has a spectrum in that survey.
test_indices (np.ndarray) – Row indices to evaluate.
normalizer (LabelNormalizer | None) – Label normalizer from training (None if not used).
config (ExperimentConfig) – The checkpoint’s experiment config.
batch_size (int) – Batch size for inference.
device (str | torch.device) – Device for inference.

Returns:

Tuple of (normalized-space result, physical-space result) of the primary head; the second is None without a normalizer.

Return type:

tuple[EvaluationResult, EvaluationResult | None]

dorothy.inference.evaluation_utils.evaluate_on_single_survey_data(model, X, y, test_indices, normalizer, config, batch_size=1024, device='auto')[source]¶

Evaluate on single-survey arrays (primary head).

Parameters:

model (nn.Module) – Trained MLP or MultiHeadMLP.
X (np.ndarray) – Spectra, shape (N, 3, n_wavelengths).
y (np.ndarray) – Physical labels, shape (N, 3, n_params).
test_indices (np.ndarray) – Row indices to evaluate.
normalizer (LabelNormalizer | None) – Label normalizer from training (None if not used).
config (ExperimentConfig) – The checkpoint’s experiment config.
batch_size (int) – Batch size for inference.
device (str | torch.device) – Device for inference.

Returns:

Tuple of (normalized-space result, physical-space result) of the primary head; the second is None without a normalizer.

Return type:

tuple[EvaluationResult, EvaluationResult | None]

dorothy.inference.evaluation_utils.build_batch_from_sparse(data, global_indices, device='cpu')[source]¶

Build one dense batch from sparse storage (same as training).

Parameters:

data (SparseMergedData) – Sparse merged data.
global_indices (ndarray) – Row indices of the stars in the batch.
device (device | str) – Device of the batch tensors.

Returns:

(X_batch, has_data_batch, y_batch); y_batch is a dict per label source when the data has several, else the primary labels tensor.

Return type:

tuple[dict[str, Tensor], dict[str, Tensor], Tensor | dict[str, Tensor]]

dorothy.inference.evaluation_utils.normalize_sparse_data(data, normalizer)[source]¶

Normalize the labels of sparse data as in training.

Parameters:

data (SparseMergedData) – Sparse merged data with physical labels.
normalizer (LabelNormalizer) – Label normalizer from training.

Returns:

A copy of data with normalized labels and labels_dict.

Return type:

SparseMergedData

dorothy.inference.evaluation_utils.normalize_dense_data(X_dict, y_dict, has_data_dict, normalizer)[source]¶

Normalize dense dict labels as in training.

Parameters:

X_dict (dict[str, np.ndarray]) – Survey name -> spectra.
y_dict (dict[str, np.ndarray]) – Label source -> physical labels, shape (N, 3, n_params).
has_data_dict (dict[str, np.ndarray]) – Survey name -> boolean presence array.
normalizer (LabelNormalizer) – Label normalizer from training.

Returns:

(X_dict, normalized y_dict, has_data_dict); X_dict and has_data_dict are returned unchanged.

Return type:

tuple[dict[str, np.ndarray], dict[str, np.ndarray], dict[str, np.ndarray]]

dorothy.inference.evaluation_utils.normalize_single_survey_data(X, y, normalizer)[source]¶

Normalize single-survey labels as in training.

Parameters:

X (np.ndarray) – Spectra, shape (N, 3, n_wavelengths).
y (np.ndarray) – Physical labels, shape (N, 3, n_params).
normalizer (LabelNormalizer) – Label normalizer from training.

Returns:

(X unchanged, normalized y).

Return type:

tuple[np.ndarray, np.ndarray]