DOROTHY home

How it works / Overview

Overview

This page follows the data from the catalogue to a prediction. Each step names the code that runs it and links to the page that explains it.

The DOROTHY pipeline: the super-catalogue gives the stars; load_training_data selects and splits them; spectra and labels are preprocessed; the Trainer fits the model and its calibration factors; the checkpoint feeds dorothy evaluate, which scores the test stars, and dorothy predict, which gives labels with uncertainties for new spectra. Evaluation and prediction reuse the same selection, split and preprocessing. Catalogue HDF5, one row per star CatalogueLoader Select and split stars with a spectrum and a valid label load_training_data Preprocess flux, ρ, mask; normalized labels encode_precision Train fit μ and ln s, then calibrate s Trainer Checkpoint weights, normalizer, factors, split save_checkpoint dorothy evaluate metrics on the test stars dorothy predict new spectra in, labels μ ± c s out same selection, split and preprocessing
The DOROTHY pipeline: the super-catalogue gives the stars; load_training_data selects and splits them; spectra and labels are preprocessed; the Trainer fits the model and its calibration factors; the checkpoint feeds dorothy evaluate, which scores the test stars, and dorothy predict, which gives labels with uncertainties for new spectra. Evaluation and prediction reuse the same selection, split and preprocessing. Catalogue HDF5, one row per star CatalogueLoader Select and split stars with a spectrum and a valid label load_training_data Preprocess flux, ρ, mask; normalized labels encode_precision Train fit μ and ln s, then calibrate s Trainer Checkpoint weights, normalizer, factors, split save_checkpoint dorothy evaluate metrics on the test stars dorothy predict new spectra in, labels μ ± c s out Evaluation and prediction reuse the same selection, split and preprocessing.

The pipeline: training, evaluation and prediction share one implementation of each step.

The steps

  1. Select and split. load_training_data() keeps the stars that have a spectrum from a configured survey and a valid label from a configured label source. It splits them into training, validation and test stars (Training).

  2. Preprocess. Each spectrum becomes three channels: flux, the precision encoding ρ and a mask. Each label is scaled by the median and interquartile range of the training labels (Preprocessing).

  3. Predict. The model maps the spectra of a star to a prediction μ and a log predicted scatter ln s for every label, one head per label source (The model).

  4. Train. The Trainer minimizes the negative log-likelihood of the labels and keeps the weights of the epoch with the lowest validation loss (Uncertainties, Training).

  5. Calibrate. After training, the trainer fits one calibration factor per head and parameter, so that the validation z-scores have a robust spread of 1 (Uncertainties).

  6. Save. The checkpoint holds the weights, the configuration, the label normalization, the calibration factors and the split (Training).

  7. Evaluate. dorothy evaluate scores the test stars in normalized and physical units (Evaluation).

  8. Predict new stars. dorothy predict reads spectra from a FITS file and writes each label with its uncertainty (Commands).

One definition per step

Training, evaluation and prediction call the same functions for each step. These cover the data selection, the preprocessing, the label normalization, the predicted scatter, its calibration and the forward pass. Evaluation on the validation stars therefore reproduces the metrics recorded during training. Shared definitions lists the functions.