Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Your first discovery with pysubgroup

We will ask: which groups in the Adult census-income dataset have a higher share of the >50K income label than the dataset overall? We describe groups through attributes such as age, education, occupation, and working hours. This example demonstrates the discovery workflow on historical records; its patterns describe associations rather than effects of changing an attribute.

Installation

Use Python 3.11–3.14 in a virtual environment. The package requires NumPy 2 or newer, pandas, SciPy, and scikit-learn. The plotting extra supplies matplotlib:

pip install "pysubgroup[plot]"

For a local source checkout, run pip install -e ".[plot]" from the repository root in your chosen environment. Optional acceleration is introduced in Making searches practical.

1. Data and a fitted target

Load Adult with scikit-learn’s OpenML loader. The first run requires an internet connection; subsequent runs reuse the downloaded data. When run from the book folder, data_home=Path("data-cache") keeps that cache inside user_guide/. We take a reproducible 3,000-row sample to keep the introductory search small.

Rows: 3000; >50K rows: 729
Dataset >50K share: 24.3%
Loading...

BinaryTarget turns membership in label ">50K" into positive/negative information for each row. It does not train a classifier. The fitted target retains the reference statistics needed by quality functions; the printed dataset share is the baseline against which subgroups will be compared.

The selected columns exclude the census sampling weight fnlwgt, which is not a descriptor of a person. Here each record counts equally; the example is not a survey-weighted population estimate. We also retain education rather than its closely related numeric encoding education-num.

2. Candidate descriptions

Candidate conditions: 60
First six conditions (preview, not the complete search space):
age ≤ 33
age > 33
age ≤ 50
age > 50
age ≤ 67
age > 67

Only descriptor attributes enter X; the income label class stays out of the descriptions. Categorical attributes generate tests such as education=Bachelors. Numeric attributes provide thresholds such as age ≤ ... and their > counterparts. n_bins controls discretization, not the maximum number of conditions in a rule. Missing descriptor values can be matched by an explicit is_missing condition.

Loading...

The task asks for up to five descriptions, each with at most two conditions and at least one hundred rows. DFS exhaustively searches this permitted space and uses optimistic estimates to prune branches. Results are (quality, Rule) pairs in decreasing quality order. With min_quality=0, reported scores must be positive.

Read a row of the table in this order:

  1. Description: which attribute values define membership?

  2. Size: how many records support the finding?

  3. Target share: what fraction of the group has the >50K label?

  4. Quality: how does the chosen size–deviation trade-off rank the finding?

A high target share in a large subgroup is rewarded by WRAcc. Several top descriptions may be variations on the same pattern.

4. Inspect actual members

[age ≤ 67 ∧ marital-status=Married-civ-spouse]
WRAcc: 0.1005; rows: 1323
Loading...

Inspecting rows is useful for catching mistakes in units, labels, or the description language. subgroup_data preserves their original DataFrame index.

5. Draw a first plot

<Figure size 800x385 with 1 Axes>

The first row is the dataset. For each subgroup, the bar to the right counts positives and the bar to the left counts negatives. Compare both the total length and the balance; a visually pure but tiny group may have limited scope.

What to remember

The same pattern—fit a target, generate conditions, configure a task, execute a search, inspect the results—works for many target types. The target and quality function express the analysis question. The search algorithm determines how the permitted descriptions are explored.

Experiment. Change MinSizeFilter(100) to MinSizeFilter(1000), then change depth=2 to depth=1. Which differences are caused by excluding small groups, and which by excluding combinations? Keep everything else fixed while comparing.