Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Validation after searching many subgroups

A discovery algorithm deliberately selects descriptions with high observed quality. Even noise can contain an extreme-looking winner if enough descriptions are tried. A holdout procedure separates choosing descriptions from testing their statistics Webb, 2007Siegl et al., 2025.

First split, then choose numeric cuts

For a strict discovery/validation separation, generate data-dependent candidate thresholds on discovery rows only. The validation rows should not help select cuts, descriptions, weights, or reporting thresholds.

Discovery rows: 75 validation rows: 75

refit creates a new target with the same fitting specification. Reusing the discovery target on validation rows would mix up positional arrays and reference statistics.

What the randomization test asks

For each candidate, the test finds its cover on the validation data. It then draws random subsets without replacement within each draw, all with the same size as that validation cover. The statistic is evaluated on these subsets using the validation population as reference.

The null model asks whether the observed rows have an unusually high statistic compared with random sets of the same size. This is not a new search on shuffled data and not a test of whether every individual condition is independently useful.

Loading...

The example explicitly chooses plus_one=True. The package default follows SubROC’s b/Rb/R: if bb of RR subsets score at least as high, p=b/Rp=b/R. That can be zero. The alternative (b+1)/(R+1)(b+1)/(R+1) prevents zero Monte Carlo p-values Phipson & Smyth, 2010. Its minimum is 1/(R+1)1/(R+1), so the number of draws affects the resolution available after multiple-testing correction.

Keep the test setup fixed before interpreting validation outcomes. Seeded randomization makes the example reproducible; it does not remove Monte Carlo uncertainty.

Multiple testing

adjust_p_values supports:

MethodIntended control
"by" (default)Benjamini–Yekutieli false discovery rate under arbitrary dependency
"bh"Benjamini–Hochberg false discovery rate under its dependency assumptions
"bonferroni"Family-wise error through Bonferroni adjustment
"holm"Step-down family-wise error adjustment
"none"No multiple-testing correction

Subgroup tests overlap and are dependent. BY is the package’s default for that reason Benjamini & Yekutieli, 2001. Correction applies to the fixed candidate family tested on the holdout data. Repeatedly revising the search after inspecting validation results changes the statistical procedure.

Stratification changes the null model

stratify_target=True also preserves the subgroup’s class counts. This is useful for comparing classifier performance at fixed class balance. For WRAcc, however, size and class counts completely determine the score: all randomized scores then equal the observed score and the p-value is one.

WRAcc with fixed size and class counts: [1. 1. 1. 1. 1.]

The default preserves only size, deliberately differing from SubROC’s class-balanced null. stratify= can preserve counts across user-selected columns; with target stratification, these form combined strata. More constraints mean a different scientific question, not automatically a better test.

Undefined validation statistics receive p-value one. For example, ROC AUC is undefined if a validation subgroup contains only one class. A group that was well supported in discovery can still be too small in validation.

The combined helper

holdout_search performs splitting, target refitting, search, testing, correction, and selection in one call. To keep thresholds independent of the validation rows here, use externally specified cuts rather than cuts fitted on the whole input dataset:

Loading...

The helper keeps the provided search space; it does not regenerate cuts after splitting. Its table combines discovery statistics, validation ... columns, p-values, and the significance decision. candidates=True shows every tested candidate; the default table contains only the final accepted results.

For a generalization-aware measure, the tested statistic is qf.value, since the reference used during discovery depends on the searched description lattice. AreaQF is not supported by these significance tests.

Experiment. Compare unstratified and class-stratified randomization for a ROC AUC measure. Explain which variation each null model allows and why this distinction differs from the WRAcc example above.

References
  1. Webb, G. I. (2007). Discovering Significant Patterns. Machine Learning, 68, 1–33.
  2. Siegl, T., Coşkun, K., Hiller, B. C., Mirzaei, A., Lemmerich, F., & Becker, M. (2025). SubROC: AUC-Based Discovery of Exceptional Subgroup Performance for Binary Classifiers. 10.48550/arXiv.2505.11283
  3. Phipson, B., & Smyth, G. K. (2010). Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn. Statistical Applications in Genetics and Molecular Biology, 9(1). 10.2202/1544-6115.1585
  4. Benjamini, Y., & Yekutieli, D. (2001). The Control of the False Discovery Rate in Multiple Testing under Dependency. The Annals of Statistics, 29(4), 1165–1188. 10.1214/aos/1013699998