Reference / API reference / dorothy / dorothy.inference / dorothy.inference.evaluator

dorothy.inference.evaluator¶

Per-parameter evaluation metrics for DOROTHY stellar parameter predictions.

Metrics computed:

Mean-based (outlier-sensitive):

RMSE: Root Mean Square Error
Bias: Mean (predicted - true)
SD: Standard deviation of residuals

Quantile-based (robust):

MAE: Mean Absolute Error
Median offset: Median (predicted - true)
Robust scatter: (P84 - P16) / 2 of residuals

Z-score metrics (calibration check, when uncertainties provided):

z_median: Median of z-scores
z_robust_scatter: (P84 - P16) / 2 of z-scores (should be ~1 if calibrated)

Predicted uncertainty statistics:

pred_unc_p16, pred_unc_p50, pred_unc_p84: Percentiles of predicted scatter

Examples

>>> evaluator = Evaluator(parameter_names=["teff", "logg", "fe_h"])
>>> result = evaluator.evaluate(y_pred, y_true, pred_scatter, label_errors)
>>> print(result.summary())

Functions

`evaluate_predictions`(y_pred, y_true[, ...])	Evaluate predictions with a default `Evaluator`.
`evaluate_predictions_by_survey`(y_pred, y_true)	Evaluate predictions per survey with a default `Evaluator`.

Classes

`EvaluationResult`([metrics, parameter_names, ...])	Container for evaluation results across all parameters.
`Evaluator`([parameter_names, teff_in_log, ...])	Evaluator for stellar parameter predictions.
`ParameterMetrics`(name, n_samples, rmse, ...)	Metrics for a single stellar parameter.
`SurveyEvaluationResult`(overall[, by_survey, ...])	Container for evaluation results broken down by survey.

class dorothy.inference.evaluator.ParameterMetrics(name, n_samples, rmse, bias, sd, mae, median_offset, robust_scatter, z_median=None, z_robust_scatter=None, pred_unc_p16=None, pred_unc_p50=None, pred_unc_p84=None)[source]¶

Bases: object

Metrics for a single stellar parameter.

Mean-based metrics are sensitive to outliers; quantile-based metrics are robust to them. z-scores check the calibration of the uncertainties.

name¶: Parameter name.

n_samples¶: Number of samples evaluated.

rmse¶: Root mean square error.

bias¶: Mean of (predicted - true).

sd¶: Standard deviation of the residuals.

mae¶: Mean absolute error.

median_offset¶: Median of (predicted - true).

robust_scatter¶: (P84 - P16) / 2 of the residuals.

z_median¶: Median of the z-scores (None without uncertainties).

z_robust_scatter¶: (P84 - P16) / 2 of the z-scores (None without uncertainties).

pred_unc_p16¶: 16th percentile of the predicted scatter (None without uncertainties).

pred_unc_p50¶: 50th percentile of the predicted scatter (None without uncertainties).

pred_unc_p84¶: 84th percentile of the predicted scatter (None without uncertainties).

to_dict()[source]¶

Convert to a dict with one entry per field.

Returns:: Field name -> value; unset optional fields are None.
Return type:: dict[str, Any]

class dorothy.inference.evaluator.EvaluationResult(metrics=<factory>, parameter_names=<factory>, survey_name=None)[source]¶

Bases: object

Container for evaluation results across all parameters.

metrics¶: Dictionary mapping parameter names to their metrics.

parameter_names¶: List of parameter names in order.

survey_name¶: Optional name of the survey (for per-survey results).

summary(format='text', include_zscore=True)[source]¶

Generate a summary of evaluation results.

Parameters:

format (str) – Output format (“text” or “markdown”).
include_zscore (bool) – Whether to include z-score metrics if available.

Returns:

Formatted summary string.

Return type:

str

to_dict()[source]¶

Convert all metrics to a dictionary.

Returns:: Parameter name -> ParameterMetrics.to_dict().
Return type:: dict[str, dict[str, Any]]

class dorothy.inference.evaluator.SurveyEvaluationResult(overall, by_survey=<factory>, survey_names=<factory>, survey_sample_counts=<factory>)[source]¶

Bases: object

Container for evaluation results broken down by survey.

Holds both overall metrics and per-survey metrics for multi-survey training scenarios.

overall¶: Overall metrics across all surveys.

by_survey¶: Dictionary mapping survey names to their EvaluationResult.

survey_names¶: List of survey names in order.

survey_sample_counts¶: Number of samples per survey.

summary(format='text', include_zscore=True, compact=False)[source]¶

Generate a summary of per-survey evaluation results.

Parameters:

format (str) – Output format (“text” or “markdown”).
include_zscore (bool) – Whether to include z-score metrics if available.
compact (bool) – If True, show a per-survey table of the RMSE of the first four parameters (text) or omit the per-survey details (markdown), instead of full details.

Returns:

Formatted summary string.

Return type:

str

to_dict()[source]¶

Convert all metrics to a dictionary.

Returns:: Dict with overall, by_survey, survey_names and survey_sample_counts.
Return type:: dict[str, Any]

class dorothy.inference.evaluator.Evaluator(parameter_names=None, teff_in_log=True, scatter_floor=0.01)[source]¶

Bases: object

Evaluator for stellar parameter predictions.

Computes per-parameter metrics of predictions against ground truth. With teff_in_log, a parameter named “teff” also gets a “log_teff” entry.

parameter_names¶: Names of parameters to evaluate.

teff_in_log¶: Whether to also compute Teff metrics in log10 space.

scatter_floor¶: Scatter floor used by the model (pred_scatter already includes it).

Examples

>>> evaluator = Evaluator(["teff", "logg", "fe_h"])
>>> result = evaluator.evaluate(predictions, ground_truth, pred_scatter, label_errors)
>>> print(result.summary())

evaluate(y_pred, y_true, pred_scatter=None, label_errors=None, mask=None)[source]¶

Evaluate predictions against ground truth.

Parameters:

y_pred (NDArray[float32]) – Predicted values of shape (n_samples, n_parameters).
y_true (NDArray[float32]) – Ground truth values of shape (n_samples, n_parameters).
pred_scatter (NDArray[float32] | None) – Predicted scatter s, shape (n_samples, n_parameters). None skips the z-score and scatter statistics.
label_errors (NDArray[float32] | None) – Label errors sigma_label, shape (n_samples, n_parameters). With pred_scatter, z-scores divide by sqrt(sigma_label^2 + s^2) instead of s.
mask (NDArray[float32] | None) – Optional mask of shape (n_samples, n_parameters); labels are valid where > 0. Only valid samples contribute to the metrics of each parameter.

Returns:

EvaluationResult with one entry per parameter, plus “log_teff” if teff_in_log is set and a “teff” column (any case) has valid samples with positive predicted and true values.

Raises:

ValueError – If y_pred and y_true differ in shape, y_pred has other than len(parameter_names) columns, or an optional array differs in shape from y_pred.

Return type:

EvaluationResult

evaluate_by_survey(y_pred, y_true, survey_ids=None, survey_names=None, pred_scatter=None, label_errors=None, mask=None, has_data=None)[source]¶

Evaluate predictions with breakdown by survey.

Computes overall metrics and per-survey metrics for multi-survey training scenarios. Supports two modes:

Exclusive mode (survey_ids): Each sample belongs to exactly one survey. Use this when each spectrum comes from a single survey.
Non-exclusive mode (has_data): Samples can belong to multiple surveys. Use this for multi-survey training where stars may have spectra from multiple surveys (e.g., both BOSS and DESI). Sample counts may sum to more than the total number of samples.

Parameters:

y_pred (NDArray[float32]) – Predicted values of shape (n_samples, n_parameters).
y_true (NDArray[float32]) – Ground truth values of shape (n_samples, n_parameters).
survey_ids (NDArray[int32] | NDArray[str_] | list[str] | None) – Survey membership for each sample (exclusive mode). Can be: - Integer array of survey indices (0, 1, 2, …) - String array of survey names - List of survey names (one per sample) Ignored if has_data is provided.
survey_names (list[str] | None) – List of survey names corresponding to survey indices. If None, string ids give their sorted unique values and integer ids give “survey_<id>”. String ids not in survey_names and integer ids out of range are excluded from the per-survey results, with a warning. Ignored if has_data is provided.
pred_scatter (NDArray[float32] | None) – Predicted scatter s, shape (n_samples, n_parameters).
label_errors (NDArray[float32] | None) – Label errors sigma_label, shape (n_samples, n_parameters).
mask (NDArray[float32] | None) – Optional mask of shape (n_samples, n_parameters); see evaluate().
has_data (dict[str, NDArray[bool]] | None) – Survey name -> boolean array of shape (n_samples,), True where the sample has data from that survey (non-exclusive mode). Takes precedence over survey_ids.

Returns:

SurveyEvaluationResult with overall and per-survey metrics.

Raises:

ValueError – If neither survey_ids nor has_data is provided, or if the shapes of the arrays do not match, as in evaluate().

Return type:

SurveyEvaluationResult

Example (exclusive mode):

>>> evaluator = Evaluator(["teff", "logg", "fe_h"])
>>> survey_ids = np.array(["boss", "lamost", "boss", "lamost"])
>>> result = evaluator.evaluate_by_survey(
...     y_pred, y_true, survey_ids,
...     pred_scatter=pred_scatter
... )
>>> print(result.summary(compact=True))

Example (non-exclusive mode):

>>> has_data = {
...     "boss": np.array([True, True, False, True]),
...     "desi": np.array([False, True, True, True]),
... }
>>> result = evaluator.evaluate_by_survey(
...     y_pred, y_true, has_data=has_data,
...     pred_scatter=pred_scatter
... )

dorothy.inference.evaluator.evaluate_predictions(y_pred, y_true, pred_scatter=None, label_errors=None, parameter_names=None, mask=None)[source]¶

Evaluate predictions with a default Evaluator.

Parameters:

y_pred (NDArray[float32]) – Predicted values, shape (n_samples, n_parameters).
y_true (NDArray[float32]) – Ground truth values, same shape.
pred_scatter (NDArray[float32] | None) – Predicted scatter s, same shape.
label_errors (NDArray[float32] | None) – Label errors sigma_label, same shape.
parameter_names (list[str] | None) – Parameter names, one per column. None uses [“teff”, “logg”, “fe_h”].
mask (NDArray[float32] | None) – Optional mask, same shape; labels are valid where > 0.

Returns:

EvaluationResult with per-parameter metrics, plus “log_teff” as in Evaluator.evaluate().

Raises:

ValueError – As Evaluator.evaluate().

Return type:

EvaluationResult

dorothy.inference.evaluator.evaluate_predictions_by_survey(y_pred, y_true, survey_ids=None, survey_names=None, pred_scatter=None, label_errors=None, parameter_names=None, mask=None, has_data=None)[source]¶

Evaluate predictions per survey with a default Evaluator.

Parameters:

y_pred (NDArray[float32]) – Predicted values, shape (n_samples, n_parameters).
y_true (NDArray[float32]) – Ground truth values, same shape.
survey_ids (NDArray[int32] | NDArray[str_] | list[str] | None) – Survey of each sample, shape (n_samples,), integer indices or names (exclusive mode). Ignored if has_data is given.
survey_names (list[str] | None) – Survey names; see Evaluator.evaluate_by_survey().
pred_scatter (NDArray[float32] | None) – Predicted scatter s, same shape as y_pred.
label_errors (NDArray[float32] | None) – Label errors sigma_label, same shape as y_pred.
parameter_names (list[str] | None) – Parameter names, one per column. None uses [“teff”, “logg”, “fe_h”].
mask (NDArray[float32] | None) – Optional mask, same shape as y_pred.
has_data (dict[str, NDArray[bool]] | None) – Survey name -> boolean array of shape (n_samples,) (non-exclusive mode). Takes precedence over survey_ids.

Returns:

SurveyEvaluationResult with overall and per-survey metrics.

Raises:

ValueError – As Evaluator.evaluate_by_survey().

Return type:

SurveyEvaluationResult