Reference / binference.PopulationModel

binference.PopulationModel

class binference.PopulationModel(likelihoods, prior=None, *, selection=None, selection_mode=None, bins=None)

Likelihoods + prior (+ selection): the object every inference engine works on.

Parameters:

likelihoodsLikelihoods or array_like (N, K)

Per-object bin-averaged likelihoods. A raw array is wrapped with Likelihoods(array, bins).

priorPopulationPrior, optional

Population prior; defaults to Dirichlet(1.0). Use GammaRates to infer rates instead of fractions.

selectionarray_like (K,), callable or Selection, optional

Per-bin detection probability (fractions) or exposure (rates), or a function of the true value averaged over each bin. Overrides any selection attached to likelihoods.

selection_mode{“on_measurement”, “on_truth”}, optional

Required with an array or callable selection. "on_measurement": objects are selected on their measured data (or the likelihoods already include the detection probability); only the normalisation term is added. "on_truth": objects are detected with probability S_k depending on their true bin; likelihood rows are multiplied by S_k as well. Selection.combined() gives both at once.

binsBins, optional

Bins for a raw array of likelihoods; ignored otherwise.

Notes

likelihoods must be ordinary likelihoods \(p(d_i \mid z)\) evaluated at the observed data, not renormalised over the detectable data space (see Selection and rates).

__init__(likelihoods, prior=None, *, selection=None, selection_mode=None, bins=None)

Methods

__init__(likelihoods[, prior, selection, ...])
draw_total(value, rng[, S])	Draws of the total \(\Lambda \mid \rho \sim \mathrm{Gamma}(W, S\cdot\rho)\).
effective_likelihoods(likelihoods)	Likelihood matrix of (new) objects with this model's selection applied.
expected_counts(value)	Expected latent counts \(\hat n_k = \sum_i w_i r_{ik}\).
from_value(value)	An unconstrained point corresponding to fractions (or rates) value (and to the selection's values, or the response's posterior means, when they are inferred).
kkt_gap(value[, include_prior])	Violation of the first-order optimality conditions at value.
log_density_unconstrained(x)	Log posterior density in the unconstrained coordinates x (with the Jacobian).
log_density_unconstrained_and_grad(x)	Log posterior density and its gradient in the unconstrained coordinates x.
log_likelihood(value[, absolute, S, response])	Marginal log-likelihood of the population (including the selection term).
log_posterior(value)	Unnormalised log posterior density of fractions (on the simplex) or rates.
log_prior(value)	Prior log-density of fractions (on the simplex) or rates, where defined.
missing_information(value[, max_objects, ...])	Largest fraction of missing information at value (0 to 1).
response_at(X)	The response \(R\) (..., J, K) at unconstrained points X (..., dim), or None when it is fixed.
responsibilities(value[, idx])	Per-object posteriors over bins given the population: \(r_{ik}\).
sample_prior(n[, rng])	Draws of fractions (or rates) from the prior.
selection_at(X)	The selection \(S\) at unconstrained points X (..., dim), or None when it is fixed.
to_rho(x)	Population fractions at unconstrained x.
to_value(x)	Fractions (or rates, for rate priors) at unconstrained x.
transform(X)	Map many unconstrained draws (n, dim) to values (n, K).

Attributes

K
N
bins
dim	Dimension of the unconstrained parameter vector: the prior's, plus K for \(\mathrm{logit}\,S\) when the selection is inferred, or the log-ratios of the response's columns (J - 1 or J each, J with a lost share) when the response is inferred.
joint	Whether a selection from injections or a tallied response is inferred with the population, as extra unconstrained coordinates (DECISIONS #80, #81).
kind	"fractions" or "rates" (set by the prior).
selection_uncertain	Whether the selection comes from injections and is inferred with the population.
total_exposure	\(S\) of the Poisson law of the number of objects, \(W \sim \mathrm{Poisson}(\Lambda\,S\cdot\rho)\): detection probabilities or exposures (ones without a selection); None for a relative selection, which leaves the total unidentified (DECISIONS #46).
response	a response tallied from events, inferred with the population (or None)
selection	per-bin selection values (or None)
selection_row	factor on the likelihood columns (selection on the truth), or None

property dim

Dimension of the unconstrained parameter vector: the prior’s, plus K for \(\mathrm{logit}\,S\) when the selection is inferred, or the log-ratios of the response’s columns (J - 1 or J each, J with a lost share) when the response is inferred.

draw_total(value, rng, S=None)

Draws of the total \(\Lambda \mid \rho \sim \mathrm{Gamma}(W, S\cdot\rho)\).

One draw per row of fractions value (..., K): its exact posterior under \(p(\Lambda) \propto 1/\Lambda\). Returns None without an exposure.

Parameters:

valuendarray (…, K)

Fractions.

rngGenerator

Random number generator.

Sndarray (…, K), optional

Selection per row (draws of an uncertain one); None uses total_exposure.

effective_likelihoods(likelihoods)

Likelihood matrix of (new) objects with this model’s selection applied.

Parameters:

likelihoodsLikelihoods

Likelihoods of the objects, on this model’s bins.

expected_counts(value)

Expected latent counts \(\hat n_k = \sum_i w_i r_{ik}\).

Parameters:

valuendarray (K,)

Fractions or rates.

from_value(value)

An unconstrained point corresponding to fractions (or rates) value (and to the selection’s values, or the response’s posterior means, when they are inferred).

Parameters:

valuendarray (K,)

Fractions or rates.

property joint

Whether a selection from injections or a tallied response is inferred with the population, as extra unconstrained coordinates (DECISIONS #80, #81).

property kind

"fractions" or "rates" (set by the prior).

kkt_gap(value, include_prior=True)

Violation of the first-order optimality conditions at value.

Zero (to numerical precision) at the maximum-likelihood / MAP estimate on the simplex (or the positive orthant for rates), and infinite where an object has zero likelihood. For fractions it is normalised by the total weight; for rates it is the larger of \(\max_k G_k/S_k\) and \(|\lambda\cdot G|/(S\cdot\lambda)\) for the gradient \(G\) in \(\lambda\), so it does not depend on the exposure’s units (DECISIONS #56, #71). Self-consistency of the EM fixed point is necessary but not sufficient for optimality; this certificate checks the rest.

Parameters:

valuendarray (K,)

Fractions or rates.

include_priorbool, optional

Include the prior’s pseudo-counts in the optimality conditions.

log_density_unconstrained(x)

Log posterior density in the unconstrained coordinates x (with the Jacobian).

See log_density_unconstrained_and_grad().

Parameters:

xndarray (dim,)

Unconstrained point.

log_density_unconstrained_and_grad(x)

Log posterior density and its gradient in the unconstrained coordinates x.

This is the density of x (the prior’s isometric log-ratios, whitened fields or log-rates), so it includes the Jacobian of the map from x to the fractions or rates (DECISIONS #15); its mode differs from that of log_posterior(). With an inferred selection x ends with \(\mathrm{logit}\,S\), and with an inferred response with the log-ratios of its columns; their densities given the injections or tallies are included. NUTS, Laplace and L-BFGS use it. Up to a constant.

Parameters:

xndarray (dim,)

Unconstrained point.

log_likelihood(value, absolute=True, S=None, response=None)

Marginal log-likelihood of the population (including the selection term).

Parameters:

valuendarray (K,)

Fractions or rates.

absolutebool, optional

Include the per-object log-scales of the likelihoods, so values compare across binnings and models.

Sndarray (K,), optional

Value of an inferred selection; None uses its values.

responsendarray (J, K), optional

Value of an inferred response; None uses the rows of the likelihoods (its posterior means).

log_posterior(value)

Unnormalised log posterior density of fractions (on the simplex) or rates.

A density in the fractions or rates themselves, without the Jacobian of the unconstrained coordinates, so its mode is the MAP estimate on the simplex (MAP(space="simplex")); compare log_density_unconstrained() (DECISIONS #15).

Parameters:

valuendarray (K,)

Fractions or rates.

log_prior(value)

Prior log-density of fractions (on the simplex) or rates, where defined.

A density in the fractions (or rates) themselves, without the Jacobian of any reparametrisation (DECISIONS #15). On the boundary it is -inf where the density is zero, +inf where it is unbounded and NaN where it is undefined (a Polya tree); NaN for priors without a closed-form density (logistic priors).

Parameters:

valuendarray (K,)

Fractions or rates.

missing_information(value, max_objects=20000, rng=0, include_prior=False, chunk=4096)

Largest fraction of missing information at value (0 to 1).

Close to 0 when per-object PDFs are narrow compared with the bins and close to 1 when they are broad. It predicts the convergence rate of EM and the lag-one autocorrelation of the slowest mode of the Gibbs sampler (integrated autocorrelation time roughly (1 + f) / (1 - f)). With a selection function the missed objects count as missing data, as EM and Gibbs impute them; include_prior adds a Dirichlet prior’s term, as in the MAP EM (audit M-2; other priors add none; NaN where a prior with concentrations below one outweighs the data). Computed on the support of value and on a random subset of at most max_objects objects, chunk objects at a time.

Parameters:

valuendarray (K,)

Fractions or rates.

max_objectsint, optional

Largest random subset of objects used.

rngint, Generator or None, optional

Random number generator for the subset; the default 0 makes it reproducible.

include_priorbool, optional

Add a Dirichlet prior’s term, as the MAP EM does.

chunkint, optional

Objects processed per block, which bounds memory.

response

a response tallied from events, inferred with the population (or None)

response_at(X)

The response \(R\) (..., J, K) at unconstrained points X (..., dim), or None when it is fixed.

Parameters:

Xndarray (…, dim)

Unconstrained points.

responsibilities(value, idx=None)

Per-object posteriors over bins given the population: \(r_{ik}\).

Parameters:

valuendarray (K,)

Fractions or rates.

idxindex, optional

Objects to use; None uses all of them.

sample_prior(n, rng=None)

Draws of fractions (or rates) from the prior.

Parameters:

nint

Number of draws.

rngint, numpy.random.Generator, or None

Random number generator.

selection

per-bin selection values (or None)

selection_at(X)

The selection \(S\) at unconstrained points X (..., dim), or None when it is fixed.

Parameters:

Xndarray (…, dim)

Unconstrained points.

selection_row

factor on the likelihood columns (selection on the truth), or None

property selection_uncertain

Whether the selection comes from injections and is inferred with the population.

Methods that take a population value (log_likelihood, kkt_gap, …) then use its values.

to_rho(x)

Population fractions at unconstrained x.

Parameters:

xndarray (dim,)

Unconstrained point.

to_value(x)

Fractions (or rates, for rate priors) at unconstrained x.

Parameters:

xndarray (dim,)

Unconstrained point.

property total_exposure

\(S\) of the Poisson law of the number of objects, \(W \sim \mathrm{Poisson}(\Lambda\,S\cdot\rho)\): detection probabilities or exposures (ones without a selection); None for a relative selection, which leaves the total unidentified (DECISIONS #46).

transform(X)

Map many unconstrained draws (n, dim) to values (n, K).

Parameters:

Xndarray (n, dim)

Unconstrained draws.