Contribute / Code map

Code map

This page shows where each step of the pipeline lives in the code, so that you can find the file to read or change.

The pipeline: the builder writes the catalogue, training writes a checkpoint, and evaluation and prediction read it.

Packages

Package	What it does	Start reading at
dorothy.config	Validates a configuration file	ExperimentConfig
dorothy.data	Reads the catalogue, selects and splits stars, preprocesses spectra and labels, masks inputs and labels during training	load_training_data(), CatalogueLoader
dorothy.models	The networks and their output layers	MLP, MultiHeadMLP, LabelAwareOutput
dorothy.losses	The loss and the predicted scatter	HeteroscedasticLoss, predicted_scatter()
dorothy.training	The training loop, batches and checkpoints	Trainer, forward_label_sources()
dorothy.inference	Loads checkpoints, predicts, evaluates, calibrates	load_checkpoint(), Predictor, evaluate_model()
dorothy.analysis	Saliency maps, anomaly detection, comparison of label sources	SaliencyAnalyzer, AnomalyDetector, compare_label_sources()
dorothy.visualization	Training and evaluation plots	generate_training_report()
dorothy.cli	The dorothy command	main in dorothy/cli/main.py

Outside the package:

Path	What it holds
scripts/build_catalogue_v2.py	The catalogue builder
scripts/legacy/	The deprecated v1 pipeline
examples/	Training configurations and saliency scripts
tests/	The test suite; tests/synthetic.py writes small catalogues with all four surveys and both label sources
docs/	These docs (Build the docs)

Training

dorothy train runs cmd_train in dorothy/cli/main.py, which calls these functions in order.

1. complete_config_from_catalogue() fills in the settings that come from the catalogue, and ExperimentConfig validates the result.

2. load_training_data() selects the stars and loads them in one of three layouts (below).

3. split() divides the rows into training, validation and test sets with the configured seed.

4. fit() passes the data to the Trainer.fit* method of its layout. All of these methods share one training loop.

5. save_checkpoint() and save() write the checkpoint directory (File formats).

Layout	Used for	Storage
single_survey	One survey and one label source	Dense arrays
dense_multi_label	Several label sources with data.use_dense_loading: true	Dense arrays per survey and label source
sparse_multi_survey	Every other configuration	SparseMergedData

The training loop takes batches from TensorBatchSource (dense arrays) or SparseBatchSource (sparse storage). A Batch holds a dict of spectra per survey, a dict of has_data masks and a dict of labels per label source.

Evaluation and prediction

dorothy evaluate loads the checkpoint with load_checkpoint(), reloads the data with the same load_training_data, and scores the test rows with evaluate_model(). That function runs the model through forward_label_sources, as training validation does.

dorothy predict runs predict_from_fits(): it reads the FITS file with FITSLoader, preprocesses the spectra and predicts with Predictor.

Where to make a change

To change	Edit
A configuration setting	Its Field in dorothy/config/schema.py; the description there is its documentation
Which stars are used, or the split	dorothy/data/training_data.py
Spectrum or label preprocessing	dorothy/data/fits_loader.py, dorothy/data/normalizer.py
The network	dorothy/models/
The loss	dorothy/losses/heteroscedastic.py
The training loop	dorothy/training/trainer.py
The files of a checkpoint	Trainer.save_checkpoint and dorothy/inference/checkpoint.py; then Old checkpoints
A command or its options	dorothy/cli/main.py; then Commands
The catalogue layout	scripts/build_catalogue_v2.py, dorothy/data/catalogue_loader.py and tests/synthetic.py

Several of these steps have one shared implementation; Shared definitions lists them. To add a survey, follow Add a survey.