Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Writing a quality function

A quality function receives a positional Boolean coverage mask and a fitted target. It returns a number, with larger meaning more interesting. This chapter derives a small linear cost measure directly, then adds a valid bound and batching without changing the score.

Define the objective first

Suppose each covered positive has benefit one and each covered negative has cost c≥0c\ge0. Let tt and ff be their counts. Define

q(G)=t−cfN.q(G)=\frac{t-cf}{N}.

This differs from WRAcc: its trade-off is an explicit negative-row cost, not the population target rate. It is useful when that cost interpretation matches the application.

Score: 0.375

target_type declares compatibility. _mask checks that the target is fitted and coverage has the correct Boolean shape. Keep configuration in the quality function and per-row arrays in the target.

This class works with ordinary exhaustive searches, which enumerate without a quality bound, and with heuristics. It can also be used as an additional measure in result tables and as a statistic in randomization tests.

Derive an optimistic estimate

For any refinement HH of GG, tH≤tGt_H\le t_G and fH≥0f_H\ge0. Since c≥0c\ge0,

q(H)=tH−cfHN≤tGN.q(H)=\frac{t_H-cf_H}{N}\le\frac{t_G}{N}.

Thus tG/Nt_G/N is a valid bound, attained by the arbitrary subset containing only the parent’s positives when there are any. Such a subset need not be expressible by a legal description, which is why the estimate remains optimistic.

Bound: 0.5

Check the claim independently on every subset

Testing only the score of the parent cannot establish a refinement bound. For this tiny example, enumerate every nonempty subset of its rows:

Best arbitrary subset: 0.5

For a production measure, include boundary cases, ties, missing-target policies, all-negative parents, and both scalar and batched paths. A proof defines why pruning is correct; tests catch implementation mistakes in that proof’s use.

Add batching from the same sufficient statistics

All scores here depend only on child sizes and positive counts. The batched protocol supplies these without assembling every child mask in Python:

Loading...

The corresponding statistics interface also provides sums and maxima of target arrays. Batch methods should return one NumPy value per child and reproduce the scalar definition, including undefined cases. A misleading batchable=True declaration can break searches that automatically choose this path.

Integrate with the rest of the workflow

Illustrative resampling p-values: [0.53 1.  ]

This last computation illustrates API compatibility on a tiny dataset. It is not holdout validation: the same rows were used to discover these results. Use the separate-data procedure for a validation analysis.

Experiment. Add a reward for negatives instead of a penalty. Why does the bound derived above fail when cost is negative? Derive a replacement before allowing that parameter range.