Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Numeric targets

When the target is a measurement rather than a label, “unusual” can mean a higher mean, a lower median, greater variability, or a differently shaped distribution. Decide which effect matters before selecting a measure. The numeric measures and ordering-based bounds follow the framework of Lemmerich et al. (2016).

The scikit-learn diabetes dataset contains a numeric disease-progression response and standardized descriptors. We use only four descriptors to keep the search small. This example explores associations, not treatment effects.

Loading...

For a mean target, the standard measure is

qa(G)=∣G∣a(yˉG−yˉD).q_a(G)=|G|^a\left(\bar y_G-\bar y_D\right).

Unlike binary StandardQF, its size term is the absolute count ∣G∣a|G|^a. This changes score units across datasets, though using relative size would multiply all scores within one dataset by the same constant.

Mean versus median

The mean is sensitive to large values. The median describes a typical central value and can be more robust to a few extremes. Both can be searched:

Loading...

Use direction="negative" for decreases and "both" for absolute deviations. The exponent must lie in [0,1][0,1]. A median shift and a mean shift are different questions; do not interpret a high mean score as evidence that most members have high target values.

Beyond location shifts

MeasureWhat a large score emphasizes
StandardQFNumericTscoreMean shift standardized by within-subgroup variation and size
VarianceReductionQFSeparation of subgroup and complement means
InterclassVarianceQFThe same ordering as variance reduction, scaled by dataset size
StandardDeviationQF(a)A change in population standard deviation, size-weighted
MannWhitneyQFStandardized rank difference relative to the complement
RankAUCQFProbability a subgroup value exceeds a complement value, ties counting half
KolmogorovSmirnovQFA two-sided difference between cumulative distributions, size-scaled

For example, a bimodal subgroup may have an ordinary mean but an unusual spread or cumulative distribution. Rank measures reduce sensitivity to target units. The package’s t-score gives exactly zero for a constant subgroup; inspect the raw distribution rather than relying on a degenerate standardized statistic.

Loading...

These extra columns rescore the discovered list. They do not turn it into the best list for the additional measures. To answer a different question, rerun the search with that quality function.

<Figure size 800x450 with 1 Axes>

Compare medians, spread, and extreme values in the plot. The listed subgroups can overlap, so their distributions are not independent samples.

Ordering bounds and NumBSD

For many measures, the best arbitrary subset of a given size consists of the largest target values, or of either the largest or smallest values. Checking these prefixes/suffixes yields a tight bound on arbitrary row subsets. The target is sorted once and subgroup rows are gathered in that order.

NumBSD uses this ordered structure directly, with look-ahead and optional fast numba bounds for suitable mean measures:

Loading...

NumBSD requires a supported numeric measure; it is not a general replacement for DFS on other target families. Faster bounds do not change the quality definition. Measure their cost on the actual task before assuming tighter bounds will make the search faster.

Experiment. Search with direction="both" and with a rank-based measure. Which groups move up? Explain the change in terms of target distributions and reference groups, rather than the scales of incomparable quality scores.

References
  1. Lemmerich, F., Atzmueller, M., & Puppe, F. (2016). Fast Exhaustive Subgroup Discovery with Numerical Target Concepts. Data Mining and Knowledge Discovery, 30(3), 711–762.