Changelog

The changes of each release since the 2022 package. The build copies this page from CHANGELOG.md.

Unreleased

Changed

• README.md, the home page and the Install page give pip install pygspice as the install command.

2.0.0 - 2026-10-29

Added

• Golden tests in tests/golden/: the outputs of every public function of the 2022 package (commit 1fbab9f) on synthetic inputs, and tests that compare an implementation with them.
• Continuous integration on Linux with Python 3.11 to 3.15, on macOS and Windows with Python 3.11 and 3.15, and on the oldest supported numpy and scipy.
• A CI job that builds the wheel and the sdist, checks that the sdist holds the sources and tests and that the wheel holds py.typed and no tests, and runs a smoke test of the wheel in a clean environment.
• The package is on PyPI as pygspice, because PyPI does not allow the name gspice: pip install pygspice. The import name is gspice. Before you install it, run pip uninstall gspice if you installed GSPICE from GitHub, because both packages write the same files.
• A workflow that, for each tag vX.Y.Z, runs CI, uploads the wheel and the sdist to PyPI once the lead approves, and publishes a GitHub release with them and the CHANGELOG section of X.Y.Z. It checks that the tag names the version of the built package.
• pyproject.toml: pip install . installs the package. It requires Python 3.11 or later, numpy 1.26 or later and scipy 1.11 or later.
• gspice.__version__, which is 2.0.0.
• CITATION.cff, the citation metadata for the software and the paper, which the sdist holds.
• The license is declared as GPL-3.0-or-later, the GPL v3 of the 2022 code in its “or later” form.
• Type checks with mypy in strict mode for the code outside gspice.legacy.
• A test coverage floor of 90% for the code outside gspice.legacy.
• pre-commit hooks that run ruff.
• Log records on the gspice logger: DEBUG lines on the time and size of the work of the main functions, INFO lines on the clipping passes of fit, and a WARNING from fit for each clipping pass whose threshold exceeds the in-sample ceiling of |z|. The package adds a logging.NullHandler to the logger, so the application decides where the records go.
• benchmarks/run.py, which times pixelwise, infill, fit_blocks, repair, fit, the noise models, the 2022 gp_interp and the 2022 Schur route against solve_pattern, and prints the timings and the setup as JSON.
• gspice.scale, gspice.unscale and gspice.snr. scale interpolates each spectrum over its bad pixels and scales it to unit mean noise, unscale returns scaled values and variances to flux units, and snr gives the median signal-to-noise ratio of each spectrum.
• gspice.Model, which holds the mean, the covariance and the training count of each S/N bin, and gspice.save and gspice.load, which write and read a model as one .npz file.
• gspice.pixelwise, which predicts each pixel of each spectrum from the pixels farther than nguard from it, and its result gspice.Prediction. A model keeps the precision matrix and the guard weights of each bin after their first use.
• gspice.simulate, which draws synthetic continuum-normalized spectra for tests and examples, and its result gspice.Simulation. The spectra have absorption lines whose depths follow two latent parameters in two populations, Gaussian noise, and optional cosmic rays, bad columns and shifted blocks. With gaussian=True and no artifacts they are exactly Gaussian.
• gspice.infill, which replaces chosen pixels of each spectrum by their Gaussian conditional mean given the other pixels, and its result gspice.Repaired. It reports the replaced and ignored pixels and χ², with covariance=True the conditional covariance of the replaced pixels of each spectrum, and with a numpy.random.Generator as rng a draw from the conditional distribution in Repaired.draws. Without rng, spectra of one S/N bin with the same replaced pixels share one factorization of the precision matrix at the pixels that all of them exclude, also where each spectrum has a few bad pixels of its own; repair and calibration take the same route where their patterns allow it.
• gspice.fit, which fits a model to training spectra with the iterative outlier clipping of paper Alg. 4, and returns the model and the final training mask.
• gspice.finite_sample_factor, which gives f_s(N_s, N_k, N_), the mean χ² per predicted pixel of a held-out Gaussian spectrum when a model fitted to N_s spectra predicts N_ pixels from N_k others.
• The option inflate of gspice.pixelwise and gspice.infill, which multiplies the variances and covariances of a mean-centered, unweighted model by f_s(N_s, N_k, 1). The field Repaired.fs holds the factor of each spectrum by which chi2 / dof is divided, so that chi2 / (dof * fs) has the mean 1 for held-out Gaussian spectra of such a model, with or without inflate.
• gspice/py.typed, the PEP 561 marker, so that type checkers read the annotations of an installed gspice. The functions of gspice.legacy keep their 2022 signatures without annotations.
• gspice.fit_blocks, which computes from spectra without artifacts a χ² threshold for each block of consecutive pixels and each S/N bin (paper section 3.2), and returns the model with them. gspice.Blocks holds the blocks and their thresholds in the new field Model.blocks, which save and load keep.
• gspice.noise_audit, which compares the residuals of gspice.pixelwise with the reported noise at each pixel. It returns rho, the root mean of (f − f̂)² ivar over the spectra that are good at the pixel; on spectra not used in the fit and away from bad runs longer than nguard, rho² has a mean above 1 even where ivar is right.
• gspice.calibration, which predicts a run or a random set of n good pixels of each held-out spectrum from its other good pixels, for each of several sizes n, with masks drawn independently of the data, and its result gspice.Calibration. Calibration holds χ², n and the finite-sample factor f_s of each prediction, and also takes the results of infill and pixelwise. Its methods give the mean of χ²/(n·f_s) and its standard error, the fraction of χ²/f_s above a quantile of the χ² distribution with n degrees of freedom, and the normal scores of a QQ plot, for all spectra or a selection of them. An error of a prediction names the row of its spectrum in flux.
• gspice.repair, which detects and replaces bad pixels with the procedure that the paper validates on simulated spectra: two pixelwise passes, a block pass with the thresholds of fit_blocks, an exclusion of the pixels around extended artifacts, and a final infill. The second pixelwise pass tests again each pixel that the first one flagged. The field Repaired.blocks holds the blocks that the block pass flagged in each spectrum.
• The option rank of gspice.fit, which fits a low-rank model: the covariance C = V diag(values) Vᵀ + diag(d) holds the K leading eigenvectors V and eigenvalues of the sample covariance, and d keeps its diagonal. Where K ≤ npix/80, Lanczos finds them, with a check that falls back to the eigendecomposition of the K pairs and logs it at INFO. The clipping passes use the same rank, so they also run with fewer spectra than pixels. gspice.LowRank holds V, the values and d, and save and load keep it. A model computes the precision matrix of a low-rank covariance by the Woodbury identity, without forming C, and raises numpy.linalg.LinAlgError where its condition bound κ_W exceeds 1e13. inflate needs a dense model, and Repaired.fs and Calibration.fs are 1 for a low-rank one.
• The option snr_bins of gspice.fit, which fits a model of several S/N bins: either their number n, with edges at the quantiles that give each bin the same count of the spectra that fit uses before clipping, or the interior edges. Each bin runs the clipping passes and the final fit on its own spectra, so it gets its own mask, mean, covariance and training count, as fit gives for those spectra alone.
• The option noise of gspice.fit, which fits a noise model in flux units. Its covariance is that of the noise-free flux, from an estimator that is unbiased with per-pixel masks, weights and the ivar of each spectrum, with its negative eigenvalues set to 0; with rank=K, its K leading eigenpairs plus the diagonal of the other eigenpairs, with their negative eigenvalues set to 0. The clipping passes run as without noise. The new field Model.noise marks such a model, and save and load keep it. pixelwise, infill, repair, fit_blocks, calibration and noise_audit condition each spectrum on that covariance plus its own noise variances, 1/ivar at good pixels and 1/a_s² at bad pixels, with a_s² the mean ivar of the good pixels, and form its precision matrix once per call, or, for a dense noise model in a call that forms no precision matrix for other steps, such as infill and calibration, only its Cholesky factor. pixelwise of a noise model computes each mean from its precision matrix times the spectrum, so an outlier within nguard of a pixel cancels in its mean only to rounding, linearly in the outlier: 9.3e-10 in z at 1e6σ in the review of PR #103. Their infills of a noise model with rank=K use the K eigenpairs and the diagonal plus the noise variances instead, at about npix K² flops per pattern, without the precision matrix; they raise where κ_K = 1 + λ_max(Λ^½ Vᵀ D_s⁻¹ V Λ^½) exceeds 1e13, which accepts every C_s that κ_W accepts. noise takes no center or snr_bins, inflate needs a model without noise, and Repaired.fs and Calibration.fs are 1 for a noise model.

Changed

• The documentation site is built with Astro Starlight instead of Sphinx. Its API pages come from the docstrings, as before. Page addresses end in / instead of .html, for example https://joshspeagle.github.io/GSPICE/start/install/, and the anchors of the API and glossary entries are unchanged.
• The documentation site has the visual design of docs/brand.md: dark and light themes on a navy header, the Archivo and IBM Plex Mono fonts, square corners, a new logo and favicon drawn in pixel cells, output blocks joined to their code, and framed figures.
• The documentation site opens with a Getting started group: Is GSPICE right for my data?, Install, Quickstart, Prepare your spectra, and What you get back. The home page has an interactive figure that predicts the pixels under a block that you drag over a spectrum. The installation and quickstart pages moved to start/install/ and start/quickstart/, and their old addresses redirect there.
• The documentation site has a Guides group of seven task pages on simulated spectra, which need no download: Fill known bad pixels, Find outliers, Find and repair artifacts, Choose a model, Check your model, Save and load, and Troubleshooting. They replace the tutorials on pixelwise prediction, infilling, repair, calibration, the low-rank and noise-aware models, and saving, whose old addresses redirect to them. The glossary adds artifact, clean spectrum, flagged pixel, outlier, scaled model and shifted segment.
• The docstrings, and so the API pages, use the terms of the glossary and show their equations as rendered maths. README.md has a figure of the quickstart, an example, badges and links to the pages of the site.
• The documentation site has a Concepts group of eight pages in place of the six Background pages: The model, Fitting, Prediction, Held-out uncertainty, Detection, Diagnostics, Numerics and Glossary. They hold the ideas, the equations and their limits, with new figures drawn from gspice.simulate by docs/scripts/concepts.py, and no code to run. Held-out uncertainty and Numerics are new, Numerics with benchmark times. The glossary adds a symbol table and terms such as conditional mean, covariance, ivar and S/N, and keeps its anchors. The old Background addresses redirect to the new pages.
• The documentation site has interactive explainers: on The model, a covariance explorer, a two-pixel explainer and the full mask explainer; in Find outliers, the guard window; and in Find and repair artifacts, the steps of repair. A link to a glossary entry shows the entry in a card on hover or keyboard focus.
• The documentation site replaces its two tutorials with four pages. A survey end to end runs GSPICE on the 10000 simulated LAMOST-like spectra of docs/scripts/spectra.py, with figures in the style of the site. Migrate from the 2022 package gives in one table the new function, the outputs and the units for each 2022 function, then the deprecation timeline. Cite GSPICE gives the paper, CITATION.cff and the data of the worked example, and the Changelog page shows this file. The old tutorial addresses redirect to the new pages.

Deprecated

• An access to one of the 11 functions of the 2022 package through gspice, as gspice.<name> or from gspice import <name>, emits a DeprecationWarning. It names the replacement and the removal of the name in 3.0.0. The access returns gspice.legacy.<name>, which keeps the 2022 results, except for the behavior changes below, and does not warn. Type checkers report gspice.<name> as an unknown attribute.
• gspice.standard_scale: use gspice.scale.
• gspice.covar and gspice.covar_iter_mask: use gspice.fit.
• gspice.get_chimask and gspice.gp_interp: use gspice.pixelwise.
• gspice.gaussian_estimate: use gspice.infill.
• gspice.cholesky_inv, gspice.submatrix_inv, gspice.submatrix_inv_mult, gspice.maskinterp1 and gspice.maskinterp: use gspice.legacy.<name>.

Removed

• The IDL code in pro/ and the Julia code in julia/.
• The notebooks and notes of 2021 and 2022 in notebooks/ and notes/, and the 2021 draft of the paper in paper/.
• setup.py, which pyproject.toml replaces.
• The egg in dist/ and the 2022 tests in gspice/tests/, which could not run and which the golden tests replace.
• The import of astropy, which the 2022 code did not use.
• The 10 prints of gspice.legacy, which reported progress, timings, shapes and notices such as “imask does not remove any rows or columns.” covar(checkmean=True) still prints its check of the column means.
• The LAPACK message that cholesky_inv printed for a 0 × 0 matrix.

Behavior changes

• The second pixelwise pass of repair tests each pixel again against its prediction with the replaced pixels infilled. A pixel above nsigma_replace in the first pass stays flagged only while it exceeds nsigma_ignore there, and the exclusion of step 4 counts the new sets. The paper and r26 gp_repair keep every such pixel, which replaced artifact-free pixels whose first-pass prediction a strong artifact had disturbed; design decision A30 gives the measurements.
• Step 5 of repair on a dense noise model agrees with infill to rounding, not bit for bit, where steps 1 to 4 form the precision matrix of the spectrum: infill then uses its Cholesky factor. In the test of the steps, flux differed by 1.4e-14σ.
• The block χ² of fit_blocks and repair reads the guard pixels of each block, which the earlier unreleased code never read. An outlier in a guard cancels only to rounding, so χ²_A changes linearly with its size: by up to 6e-8 at κ 1e8 and 1e4σ, and by 8.6 at κ 1e10 and 1e10σ (measured in PR #99).
• The golden tests compare 11 outputs at a relative tolerance looser than 1e-10: 1e-9 for 7 outputs and 1e-8 for 4. These outputs come from ill-conditioned inputs, on which the 2022 results differ from a direct Cholesky solve by up to 7.6e-10.
• The 2022 modules gspice.specUtils, gspice.matrixUtils and gspice.djs_maskinterp moved to gspice.legacy.specUtils, gspice.legacy.matrixUtils and gspice.legacy.djs_maskinterp, and the unused gspice.gspiceMain is deleted. Imports of the old module paths fail, and pickles that name the old paths no longer load. To migrate, insert legacy. into the module path, for example from gspice.legacy.specUtils import standard_scale.
• gspice.__all__ holds the new public names and legacy, so from gspice import * binds them. It no longer binds the 11 functions and the 3 modules of the 2022 package. It can shadow names such as load and save of an earlier from numpy import *. To migrate, use from gspice.legacy import *, which binds the same 14 names as the 2022 star import, without a warning.
• The 2022 asserts of gspice.legacy raise _LegacyError, a subclass of AssertionError and ValueError, with a message that names the argument. An except ValueError clause now catches them, and the checks also run under python -O.
• gp_interp accepts irange as a list, a tuple or an array of at least two integers of any integer type, and raises unless 0 <= irange[0] <= irange[1] <= npix - 1. The 2022 code rejected a 2-element irange and a uint64 array, and for a negative start with irange[1] = npix - 1 it returned leading columns of zeros. A numpy bool in irange[0] or irange[1] raises TypeError, and spectra without pixels raise.
• gp_interp raises for a negative nguard, where the 2022 code predicted each pixel from itself. Under numpy 2, fixed-width numpy integers now work for nguard, where the 2022 code gave wrong windows or raised OverflowError or TypeError. A numpy bool or an array of one or more dimensions raises TypeError for nguard.
• When bruteforce is False and imask keeps every row, submatrix_inv and submatrix_inv_mult return a copy of Minv and MinvY. The 2022 code returned the argument itself, so a later change to the result changed the caller’s array.
• gp_interp with bruteforce=True returns pred and predvar in the layout of the default path, where the 2022 code raised ValueError. The golden tests compare them with the 2022 outputs of the default path at the tolerances of those outputs, which are 1e-8 for 4 of them.
• In an array with one good entry, maskinterp1 sets a non-finite value at a bad entry to the good value, and so do maskinterp and djs_maskinterp.average_bilinear, which call it. The 2022 code gave NaN there. standard_scale spread that NaN to a whole column of Dvec, and get_chimask and covar_iter_mask then raised ValueError.
• standard_scale ignores a non-finite ivar at a bad pixel. The 2022 code made the refscale of that spectrum NaN, and with it every value of Dvec and refmean, so get_chimask and covar_iter_mask raised ValueError.
• standard_scale and get_chimask raise _LegacyError for a spectrum without a good pixel, and name its row. covar_iter_mask raises it for a kept spectrum that its mask or the clipping masks completely, and names its row in flux. The 2022 code returned NaN or raised another error.
• covar_iter_mask accepts mask=None, which marks the pixels where ivar == 0, also for maxbadpix. The 2022 code raised numpy’s AxisError.
• scale raises ValueError for a non-finite flux at a good pixel, a non-finite or negative ivar at a pixel not in mask, and zero spectra without refmean. For such inputs, the 2022 standard_scale returned NaN or a wrong refscale.
• scale counts a pixel with ivar = 0 as bad also where mask is given. It interpolates the flux of such a pixel and leaves the pixel out of refscale. The 2022 standard_scale with a mask used that flux and counted the 0 in the mean ivar.
• scale casts flux to float64 before it interpolates. The 2022 standard_scale interpolated in the dtype of the flux, so it truncated the interpolated values of integer flux and rounded those of float32 flux to float32.
• scale checks and computes a long double flux or ivar as float64. It raises ValueError for a long double beyond the float64 range at a good pixel, and an ivar that rounds to 0 in float64 marks a bad pixel. The 2022 standard_scale computed in long double and returned a finite result for such values.
• scale and snr raise ValueError for a spectrum without a good pixel, and name its row. For such a spectrum, the 2022 standard_scale returned NaN in every value of Dvec and refmean.
• pixelwise computes the conditional variances through the precision matrix, where the 2022 gp_interp used the Schur updates of a precomputed inverse. On the golden data set D3, whose covariance has the condition number 8.6e4, the 2022 predvar differs from a 60-digit reference by up to 7.6e-10 at nguard 5 and 2.7e-10 at nguard 20, relative. pixelwise differs by at most 1.7e-13 and 8.7e-14.
• infill computes the conditional covariance through the precision matrix, where the 2022 gaussian_estimate used the Schur updates of a precomputed inverse. On D3, the 2022 predcovar differs from a 60-digit reference by up to 3.0e-10 relative on the diagonal, and infill by at most 8.8e-14.
• fit masks the flags of every clipping pass, as paper Alg. 4 does. The 2022 covar_iter_mask masked only the flags of the last two passes; fit(cumulative=False) keeps that rule. On the golden data sets D1 to D4, the final mask of the default fit differs from the 2022 mask in 157, 152, 106 and 8 entries.
• fit flags a pixel whose |z| exceeds the threshold. The 2022 get_chimask also flagged a pixel whose |z| equals it.
• fit drops a spectrum that has no good pixel, or that a clipping mask covers completely. For such a spectrum, the 2022 covar_iter_mask returned a covariance of NaN or raised an error.
• fit needs at least npix + 1 spectra for a mean-centered model, and a good pixel in one of them at every pixel, and raises ValueError otherwise. The 2022 covar_iter_mask accepted npix spectra, whose covariance is singular, and then raised LinAlgError or flagged most pixels. It filled a pixel that is bad in every spectrum by interpolation, which also makes the covariance singular; on D1 it then raised LinAlgError.
• fit counts a pixel with ivar = 0 as bad, also against maxbadpix. The 2022 covar_iter_mask counted only the pixels in its mask.