Reference / API reference / dorothy / dorothy.data / dorothy.data.fits_loader

dorothy.data.fits_loader¶

Load stellar spectra and labels from FITS files, and preprocess spectra.

FITSLoader reads FITS files with DESI, BOSS or LAMOST spectra cross-matched with APOGEE stellar parameters. The preprocessing functions (zero_invalid_pixels(), normalize_spectra(), encode_precision()) are shared with the catalogue loaders.

Expected FITS file structure:

APOGEE extension: Table with stellar parameters and quality flags
SPECTRA extension: Table with flux arrays for each star
SPEC_IVAR extension: Table with inverse variance arrays
WAVELENGTH extension: Table with wavelength grid

The loader normalizes spectra using median/IQR normalization per-star and creates the 3-channel input format (flux, sigma, mask) of SpectralData.get_model_input_3channel().

Functions

`apply_quality_filter`(labels, errors, ...)	Create a quality mask based on physical bounds and valid values.
`encode_precision`(spectra[, inplace])	Replace the sigma channel of model inputs with the precision encoding.
`ivar_to_error_and_mask`(ivar)	Convert inverse variance to error (sigma) and mask.
`normalize_spectra`(flux, ivar[, chunk_size, ...])	Per-spectrum median/IQR normalization over valid pixels.
`normalize_spectrum`(flux, ivar)	Normalize a single spectrum using median/IQR scaling.
`zero_invalid_pixels`(flux, ivar[, inplace])	Set the flux and ivar of invalid pixels to 0.

Classes

`FITSLoader`(fits_path[, parameters])	Loader for FITS files containing stellar spectra and parameters.
`SpectralData`(flux, ivar, wavelength, labels, ...)	Container for loaded spectral data.

class dorothy.data.fits_loader.SpectralData(flux, ivar, wavelength, labels, errors, ids, quality_mask, parameter_names=<factory>, spectral_mask=None, spectral_error=None)[source]¶

Bases: object

Container for loaded spectral data.

flux¶: Flux of shape (n_samples, n_wavelengths), normalized if requested, and 0 at invalid pixels (zero_invalid_pixels()).

ivar¶: Inverse variance with the same shape, scaled with the flux, and 0 at invalid pixels.

wavelength¶: Wavelength grid of shape (n_wavelengths,).

labels¶: Stellar parameter labels of shape (n_samples, n_parameters).

errors¶: Label uncertainties of shape (n_samples, n_parameters).

ids¶: Star identifiers (e.g., APOGEE_ID) of length n_samples.

quality_mask¶: Boolean mask where True = good quality (for samples).

parameter_names¶: List of parameter names in order.

spectral_mask¶: Binary mask where 1=valid, 0=masked (per wavelength). Shape is (n_samples, n_wavelengths). Derived from ivar > 0.

spectral_error¶: Error array (sigma) converted from ivar. Shape is (n_samples, n_wavelengths). Computed as 1/sqrt(ivar).

property n_samples: int¶: Number of spectra in the dataset.

property n_wavelengths: int¶: Number of wavelength bins.

property n_parameters: int¶: Number of stellar parameters.

get_model_input_3channel(apply_quality_mask=True)[source]¶

Get combined flux/error/mask array for model input (3-channel format).

This format is consistent with label format [values | errors] and includes an explicit mask channel for handling missing data and dynamic block masking augmentation.

Parameters:

apply_quality_mask (bool) – Whether to filter by sample quality mask.

Returns:

Array of shape (n_samples, 3, n_wavelengths) containing –

Channel 0: Normalized flux (masked wavelengths → 0)
Channel 1: Error (sigma, converted from ivar, masked → 0)
Channel 2: Mask (1=valid, 0=masked)

Raises:

ValueError – If spectral_mask or spectral_error are not available.

Return type:

NDArray[float32]

get_labels(apply_quality_mask=True)[source]¶

Get labels, optionally filtered by quality mask.

Parameters:: apply_quality_mask (bool) – Whether to keep only quality_mask stars.
Returns:: Array of shape (n_samples, n_parameters).
Return type:: NDArray[float32]

get_errors(apply_quality_mask=True)[source]¶

Get label errors, optionally filtered by quality mask.

Parameters:: apply_quality_mask (bool) – Whether to keep only quality_mask stars.
Returns:: Array of shape (n_samples, n_parameters).
Return type:: NDArray[float32]

dorothy.data.fits_loader.zero_invalid_pixels(flux, ivar, inplace=False)[source]¶

Set the flux and ivar of invalid pixels to 0.

A pixel is valid when its flux is finite and its ivar is finite and positive. Every loading path applies this rule before it builds the model inputs (normalize_spectra() applies it first), so the mask ivar > 0 marks exactly the valid pixels and a masked pixel enters the model as zeros.

Parameters:

flux (NDArray) – Flux array of shape (n_spectra, n_wavelengths) or (n_wavelengths,).
ivar (NDArray) – Inverse variance array with the same shape as flux.
inplace (bool) – Write the results into flux and ivar (must be float32 arrays) instead of allocating new ones.

Returns:

Tuple of (flux, ivar), float32, same shape as input.

Raises:

ValueError – If flux and ivar differ in shape, or inplace is set and they are not float32.

Return type:

tuple[NDArray[float32], NDArray[float32]]

dorothy.data.fits_loader.normalize_spectra(flux, ivar, chunk_size=4096, inplace=False)[source]¶

Per-spectrum median/IQR normalization over valid pixels.

For each spectrum (row):

normalized_flux = (flux - median) / IQR
scaled_ivar = IQR^2 * ivar

where the median and IQR are computed over valid pixels only (finite flux, finite and positive ivar). Invalid pixels are returned with flux = 0 and ivar = 0 (zero_invalid_pixels()), so downstream code that derives the pixel mask from ivar > 0 treats them as masked. So are pixels whose normalized flux or scaled ivar overflows to infinity. A spectrum with no valid pixels uses median = 0 and IQR = 1; one with zero IQR uses IQR = 1.

Parameters:

flux (NDArray) – Flux array of shape (n_spectra, n_wavelengths) or (n_wavelengths,).
ivar (NDArray) – Inverse variance array with the same shape as flux.
chunk_size (int) – Number of spectra processed at once (bounds peak memory).
inplace (bool) – Write the results into flux and ivar (must be float32 arrays) instead of allocating new ones.

Returns:

Tuple of (normalized_flux, scaled_ivar), float32, same shape as input.

Raises:

ValueError – If flux and ivar differ in shape, or inplace is set and they are not float32.

Return type:

tuple[NDArray[float32], NDArray[float32]]

dorothy.data.fits_loader.encode_precision(spectra, inplace=False)[source]¶

Replace the sigma channel of model inputs with the precision encoding.

For each pixel with uncertainty sigma and mask m:

rho = log(1 + 1/sigma) if m > 0, else 0

Flux and mask are unchanged. rho falls to 0 as sigma grows, so a pixel without information reads like a masked one. Apply it after zero_invalid_pixels() and normalize_spectra(), which rescales sigma. A valid pixel needs sigma > 0; a masked pixel may hold any value, including NaN.

Parameters:

spectra (NDArray) – Model inputs of shape (N, 3, n_wavelengths) = [flux, sigma, mask].
inplace (bool) – Write the result into spectra (must be float32) instead of a new array.

Returns:

Float32 array of shape (N, 3, n_wavelengths) = [flux, rho, mask].

Raises:

ValueError – If spectra does not have shape (N, 3, n_wavelengths), has sigma <= 0 or NaN at a valid pixel, or inplace is set and it is not float32.

Return type:

NDArray[float32]

dorothy.data.fits_loader.normalize_spectrum(flux, ivar)[source]¶

Normalize a single spectrum using median/IQR scaling.

Calls normalize_spectra() on one spectrum.

Parameters:

flux (NDArray) – Flux array of shape (n_wavelengths,).
ivar (NDArray) – Inverse variance array of shape (n_wavelengths,).

Returns:

Tuple of (normalized_flux, scaled_ivar), float32.

Raises:

ValueError – If flux and ivar differ in shape.

Return type:

tuple[NDArray, NDArray]

dorothy.data.fits_loader.ivar_to_error_and_mask(ivar)[source]¶

Convert inverse variance to error (sigma) and mask.

Parameters:

ivar (NDArray[float32]) – Inverse variance array of shape (n_wavelengths,) or (n_samples, n_wavelengths). Can contain NaN, Inf, or <= 0 values indicating invalid data.

Returns:

Tuple of (error, mask) where –

error: Standard deviation (sigma = 1/sqrt(ivar)) where valid, else 0
mask: Binary mask (1=valid, 0=masked)

Return type:

tuple[NDArray[float32], NDArray[float32]]

dorothy.data.fits_loader.apply_quality_filter(labels, errors, parameter_names)[source]¶

Create a quality mask based on physical bounds and valid values.

A star passes if, for every parameter, the value and error are finite, the error is positive, the value lies within PARAMETER_BOUNDS (if the parameter has bounds) and the value is nonzero (except fe_h).

Parameters:

labels (NDArray) – Array of shape (n_samples, n_parameters).
errors (NDArray) – Array of shape (n_samples, n_parameters).
parameter_names (list[str]) – Parameter name of each column.

Returns:

Boolean mask of shape (n_samples,) where True = good quality.

Return type:

NDArray[bool]

class dorothy.data.fits_loader.FITSLoader(fits_path, parameters=None)[source]¶

Bases: object

Loader for FITS files containing stellar spectra and parameters.

Reads FITS files in the DOROTHY format, normalizes spectra and computes a quality mask from the label values and the APOGEE ASPCAPFLAG bits.

fits_path¶: Path to the FITS file.

parameters¶: List of stellar parameter names to load.

Examples

>>> loader = FITSLoader(Path("data/loa_trainingset.fits"))
>>> data = loader.load()
>>> X = data.get_model_input_3channel()  # (n_good_stars, 3, n_wavelengths)
>>> y = data.get_labels()  # (n_good_stars, n_parameters)

load(apply_normalization=True, apply_quality_filter=True, compute_spectral_mask=False, require_labels=True)[source]¶

Load spectral data from the FITS file.

Parameters:

apply_normalization (bool) – Whether to normalize spectra (recommended). Either way, invalid pixels get flux 0 and ivar 0.
apply_quality_filter (bool) – Whether to compute quality mask.
compute_spectral_mask (bool) – Whether to compute spectral_mask and spectral_error from ivar. Required for 3-channel format.
require_labels (bool) – Require an APOGEE extension with all label columns. When False (e.g. for prediction), missing labels are returned as NaN and the quality mask keeps every spectrum.

Returns:

SpectralData object containing all loaded data. ids come from the APOGEE_ID or TARGET_ID column; without an APOGEE extension, from the first column of a GAIA_ID extension; else they are the row numbers. wavelength is arange(n_wavelengths) without a WAVELENGTH extension.

Raises:

FileNotFoundError – If FITS file doesn’t exist.
ValueError – If a required extension is missing, or, with require_labels, the APOGEE extension or a parameter’s columns are missing.

Return type:

SpectralData