Chapter 2: From Data to a First Model

By the end of this chapter, you should be able to:

Define examples, features, targets, predictions, training, and error;
Represent a supervised learning problem with \(X\) and \(y\);
Distinguish classification from regression
Use the scikit-learn fit–predict–score workflow;
Build a simple baseline and explain why it matters;
Explain how a decision tree learns rules and makes predictions;
Distinguish model parameters from hyperparameters; and
Interpret tree depth and decision boundaries as measures of model complexity.

We begin our machine learning journey with supervised learning. In supervised learning, each training example has a target (e.g., temperature, grade, housing price), and the goal is to predict that target for new examples. In unsupervised learning, there is no designated target; instead, we look for structure in the data, such as customers with similar tastes or habits. This chapter focuses on supervised learning: it introduces the core vocabulary, demonstrates a first learned model, a decision tree, and builds intuition for how models learn from data. Along the way, we will establish the basic scikit-learn fit–predict–score workflow, compare the model with a simple baseline, and connect the terminology to parameters, hyperparameters, and decision boundaries.

Show imports and setup

import os
import sys
from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

sys.path.append(os.path.abspath("code"))

import mglearn
from IPython.display import HTML, display
from plotting_functions import *
from sklearn.dummy import DummyClassifier
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
from utils import *

plt.rcParams["font.size"] = 16
pd.set_option("display.max_colwidth", 200)

%matplotlib inline

DATA_DIR = Path("data")

Core terminology

See also

An accompanying video introduces the terminology used in this section.

Running example: quiz-grade prediction

For our first model, imagine that students want to anticipate whether they will receive an A+ on Quiz 2. They collect data from a previous offering containing attendance, lab grades, Quiz 1 performance, and the eventual Quiz 2 result.

Pause and reflect

Before looking at the data, consider two questions: What do you want to predict? What information could be useful for making that prediction?

We will use two small datasets in this and the next chapter:

These deliberately small datasets let us inspect every step. Later chapters use larger, messier datasets.

The classification version asks whether a student will receive an A+ rather than predicting the exact grade. Each row describes one student from a previous offering.

classification_df = pd.read_csv(DATA_DIR / "quiz2-grade-toy-classification.csv")
print(classification_df.shape)
classification_df.head()

(21, 8)

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1	quiz2
0	1	1	92	93	84	91	92	A+
1	1	0	94	90	80	83	91	not A+
2	0	0	78	85	83	80	80	not A+
3	0	1	91	94	92	91	89	A+
4	0	1	77	83	90	92	85	A+

Examples, features, and targets

A supervised learning dataset contains examples for which the outcome is known. Each row is an example, the input columns are features, and the column we want to predict is the target. A learning algorithm uses these examples to construct a model that can make predictions for new cases.

Feature: An input characteristic available to the model. The feature table is conventionally denoted by \(X\), and the number of features by \(d\).
Target: The outcome we want to predict, conventionally denoted by \(y\).
Example: One observation, represented by a row of feature values. The number of examples is conventionally denoted by \(n\).
Training: The process through which a learning algorithm uses \(X\) and \(y\) to determine a model’s parameters. Training is also called fitting.

Separating the feature table and target

classification_df = pd.read_csv(DATA_DIR / "quiz2-grade-toy-classification.csv")
classification_df.head()

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1	quiz2
0	1	1	92	93	84	91	92	A+
1	1	0	94	90	80	83	91	not A+
2	0	0	78	85	83	80	80	not A+
3	0	1	91	94	92	91	89	A+
4	0	1	77	83	90	92	85	A+

Before fitting a supervised model, we separate the feature table X from the target vector y. Keeping this distinction explicit helps prevent the answer from accidentally becoming an input to the model.

X = classification_df.drop(columns=["quiz2"])
y = classification_df["quiz2"]
X.head()

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1
0	1	1	92	93	84	91	92
1	1	0	94	90	80	83	91
2	0	0	78	85	83	80	80
3	0	1	91	94	92	91	89
4	0	1	77	83	90	92	85

y.head()

0        A+
1    not A+
2    not A+
3        A+
4        A+
Name: quiz2, dtype: str

A second example: housing-price prediction

The same representation applies when the target is numerical. Here, each row represents a house, the property attributes form X, and price is y.

housing_df = pd.read_csv(DATA_DIR / "kc_house_data.csv")
housing_df.drop(columns = ["id", "date"])
housing_df.head()

	id	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	...	grade	sqft_above	sqft_basement	yr_built	yr_renovated	zipcode	lat	long	sqft_living15	sqft_lot15
0	7129300520	20141013T000000	221900.0	3	1.00	1180	5650	1.0	...	7	1180	0	1955	0	98178	47.5112	-122.257	1340	5650
1	6414100192	20141209T000000	538000.0	3	2.25	2570	7242	2.0	...	7	2170	400	1951	1991	98125	47.7210	-122.319	1690	7639
2	5631500400	20150225T000000	180000.0	2	1.00	770	10000	1.0	...	6	770	0	1933	0	98028	47.7379	-122.233	2720	8062
3	2487200875	20141209T000000	604000.0	4	3.00	1960	5000	1.0	...	7	1050	910	1965	0	98136	47.5208	-122.393	1360	5000
4	1954400510	20150218T000000	510000.0	3	2.00	1680	8080	1.0	...	8	1680	0	1987	0	98074	47.6168	-122.045	1800	7503

5 rows × 21 columns

X = housing_df.drop(columns=["price"])
y = housing_df["price"]
X.head()

	id	date	bedrooms	bathrooms	sqft_living	sqft_lot	floors	condition	grade	sqft_above	sqft_basement	yr_built	yr_renovated	zipcode	lat	long	sqft_living15	sqft_lot15
0	7129300520	20141013T000000	3	1.00	1180	5650	1.0	3	7	1180	0	1955	0	98178	47.5112	-122.257	1340	5650
1	6414100192	20141209T000000	3	2.25	2570	7242	2.0	3	7	2170	400	1951	1991	98125	47.7210	-122.319	1690	7639
2	5631500400	20150225T000000	2	1.00	770	10000	1.0	3	6	770	0	1933	0	98028	47.7379	-122.233	2720	8062
3	2487200875	20141209T000000	4	3.00	1960	5000	1.0	5	7	1050	910	1965	0	98136	47.5208	-122.393	1360	5000
4	1954400510	20150218T000000	3	2.00	1680	8080	1.0	3	8	1680	0	1987	0	98074	47.6168	-122.045	1800	7503

y.head()

0    221900.0
1    538000.0
2    180000.0
3    604000.0
4    510000.0
Name: price, dtype: float64

X.shape

(21613, 20)

Important

Column names are meaningful to people, but most models operate on the feature values. A descriptive name does not make a feature informative, and an ambiguous name does not prevent its values from influencing a model.

Terminology across disciplines

Different communities often use different words for the same role:

examples = rows = samples = records = instances;
features = inputs = predictors = explanatory variables = covariates;
targets = outputs = outcomes = responses; categorical targets are often called labels; and
training = learning = fitting.

The surrounding context determines which meaning is intended.

See also

The MDS terminology guide provides a broader cross-disciplinary glossary.

Classification and regression

Supervised problems are commonly distinguished by the kind of target:

Classification predicts one of two or more discrete classes. For example, liver disease versus no liver disease, or A+ versus not A+.
Regression predicts a numerical quantity. For example, a house price or percentage grade.

The distinction affects which models and evaluation measures are appropriate.

# quiz2 classification toy data
classification_df = pd.read_csv(DATA_DIR / "quiz2-grade-toy-classification.csv")
classification_df

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1	quiz2
0	1	1	92	93	84	91	92	A+
1	1	0	94	90	80	83	91	not A+
2	0	0	78	85	83	80	80	not A+
3	0	1	91	94	92	91	89	A+
4	0	1	77	83	90	92	85	A+
5	1	0	70	73	68	74	71	not A+
6	1	0	80	88	89	88	91	A+
7	0	1	95	93	69	79	75	not A+
8	0	0	97	90	94	99	80	not A+
9	1	1	95	95	94	94	85	not A+
10	0	1	98	86	95	95	78	A+
11	1	1	95	88	93	92	85	A+
12	1	1	98	96	96	99	100	A+
13	0	1	95	94	96	95	100	A+
14	0	1	95	90	93	95	70	not A+
15	1	0	92	85	67	94	92	not A+
16	0	0	75	91	93	86	85	A+
17	1	0	86	89	65	86	87	not A+
18	1	1	91	93	90	88	82	not A+
19	0	1	77	94	87	81	89	not A+
20	1	1	96	92	92	96	87	A+

# quiz2 regression toy data
regression_df = pd.read_csv(DATA_DIR / "quiz2-grade-toy-regression.csv")
regression_df

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1	quiz2
0	1	1	92	93	84	91	92	90
1	1	0	94	90	80	83	91	84
2	0	0	78	85	83	80	80	82
3	0	1	91	94	92	91	89	92
4	0	1	77	83	90	92	85	90
5	1	0	70	73	68	74	71	75
6	1	0	80	88	89	88	91	91

Prediction and statistical inference

Prediction asks about the outcome for a new or unseen example.

Statistical inference asks what we can learn about a population or relationships from the data we have observed, often including how uncertain we are about those conclusions. The table below shows examples of prediction and inference problems in different domains:

Context	Prediction	Statistical inference
Health	How long will this patient take to recover?	What is the average recovery time in the population?
Education	Is this student likely to pass the course?	Is course performance associated with attendance?
Housing	What will this house sell for?	How are house prices related to square footage?

Prediction and statistical inference can complement one another, but good predictive performance does not necessarily mean that we can draw reliable conclusions about the underlying population or relationships between variables.

Note

In deployed machine-learning systems, the word inference is also sometimes used for the act of generating predictions with an already-fitted model; that operational usage is distinct from statistical inference.

Exercise 2.1: Supervised or unsupervised?

Select all examples of supervised learning.

1. Finding groups of similar properties in a real-estate dataset.
1. Predicting heart-attack risk from demographic, dietary, and clinical measurements.
1. Grouping news articles by topic.
1. Detecting credit-card fraud from labeled fraudulent and non-fraudulent transactions.
1. Using measured employee performance to investigate which recorded factors predict it.

Solution

B, D, and E are supervised because each requires an observed target. E emphasizes interpretation, so predictive associations should not automatically be treated as causal effects.

Exercise 2.2: Classification or regression?

Select all regression problems.

1. Predicting the price of a house.
1. Predicting whether a house will sell.
1. Predicting a student’s percentage grade.
1. Predicting whether to bicycle tomorrow.
1. Predicting an appropriate numerical thermostat setting.

Solution

A, C, and E have numerical targets and are therefore regression problems.

Exercise 2.3: Define the representation

For each problem below, identify one possible unit of analysis, a target, and several features that would be available at prediction time:

sentiment analysis;
fraud detection; and
face recognition.

Are any of the proposed features ethically sensitive, unavailable at prediction time, or likely to encode the target indirectly?

Establishing a baseline

See also

An accompanying video develops the baseline example in this section.

We are now ready to build a simple supervised model for the quiz-grade problem. Before trying a sophisticated method, we need a reference point that tells us what performance can be achieved with almost no learning.

classification_df = pd.read_csv(DATA_DIR / "quiz2-grade-toy-classification.csv")
classification_df.head()

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1	quiz2
0	1	1	92	93	84	91	92	A+
1	1	0	94	90	80	83	91	not A+
2	0	0	78	85	83	80	80	not A+
3	0	1	91	94	92	91	89	A+
4	0	1	77	83	90	92	85	A+

classification_df["quiz2"].value_counts()

quiz2
not A+    11
A+        10
Name: count, dtype: int64

The label “not A+” is more common. A strategy that always predicts this majority class ignores every feature, but it may still achieve substantial accuracy. That makes it a useful sanity check.

Why start with a baseline?

Baseline: A simple reference strategy against which more elaborate models can be compared.

For classification, a common baseline always predicts the most frequent training label. A useful learned model should improve on an appropriate baseline in a way that matters for the problem.

A classification baseline with `DummyClassifier`

Throughout the supervised-learning chapters, we will use the Python library scikit-learn as our primary framework for building and evaluating models. It provides a consistent interface for many different models, including DummyClassifier for simple classification baselines. We will begin by fitting a most-frequent baseline to the quiz-grade data.

`scikit-learn` `fit-predict-score` workflow

In scikit-learn, model objects are formally called estimators. Most models follow the same interface:

construct the feature table X and target y;
create a model;
call fit(X, y) to learn from the training data;
call predict(X_new) to predict new examples; and
evaluate predictions with an appropriate measure.

Reading the data

classification_df.head()

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1	quiz2
0	1	1	92	93	84	91	92	A+
1	1	0	94	90	80	83	91	not A+
2	0	0	78	85	83	80	80	not A+
3	0	1	91	94	92	91	89	A+
4	0	1	77	83	90	92	85	A+

Create \(X\) and \(y\)

\(X\) → Feature vectors
\(y\) → Target

X = classification_df.drop(columns=["quiz2"])
y = classification_df["quiz2"]

Create a classifier object

import the appropriate classifier
Create an object of the classifier

from sklearn.dummy import DummyClassifier  # import the classifier

dummy_clf = DummyClassifier(strategy="most_frequent")  # Create a classifier object

`fit` the classifier

The “learning” is carried out when we call fit on the classifier object.

dummy_clf.fit(X, y)
# fit the classifier

DummyClassifier(strategy='most_frequent')

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

DummyClassifier

?Documentation for DummyClassifieriFitted

Parameters

	strategy strategy: {"most_frequent", "prior", "stratified", "uniform", "constant"}, default="prior" Strategy to use to generate predictions. * "most_frequent": the `predict` method always returns the most frequent class label in the observed `y` argument passed to `fit`. The `predict_proba` method returns the matching one-hot encoded vector. * "prior": the `predict` method always returns the most frequent class label in the observed `y` argument passed to `fit` (like "most_frequent"). ``predict_proba`` always returns the empirical class distribution of `y` also known as the empirical class prior distribution. * "stratified": the `predict_proba` method randomly samples one-hot vectors from a multinomial distribution parametrized by the empirical class prior probabilities. The `predict` method returns the class label which got probability one in the one-hot vector of `predict_proba`. Each sampled row of both methods is therefore independent and identically distributed. * "uniform": generates predictions uniformly at random from the list of unique classes observed in `y`, i.e. each class has equal probability. * "constant": always predicts a constant label that is provided by the user. This is useful for metrics that evaluate a non-majority class. .. versionchanged:: 0.24 The default value of `strategy` has changed to "prior" in version 0.24.	'most_frequent'
	random_state random_state: int, RandomState instance or None, default=None Controls the randomness to generate the predictions when ``strategy='stratified'`` or ``strategy='uniform'``. Pass an int for reproducible output across multiple function calls. See :term:`Glossary <random_state>`.	None
	constant constant: int or str or array-like of shape (n_outputs,), default=None The explicit constant as predicted by the "constant" strategy. This parameter is useful only for the "constant" strategy.	None

Fitted attributes

Name	Type	Value
class_prior_ class_prior_: ndarray of shape (n_classes,) or list of such arrays Frequency of each class observed in `y`. For multioutput classification problems, this is computed independently for each output.	ndarray[float64](2,)	[0.48,0.52]
classes_ classes_: ndarray of shape (n_classes,) or list of such arrays Unique class labels observed in `y`. For multi-output classification problems, this attribute is a list of arrays as each output has an independent set of possible classes.	ndarray[object](2,)	['A+','not A+']
feature_names_in_ feature_names_in_: ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings.	ndarray[object](7,)	['ml_experience','class_attendance','lab1',...,'lab3','lab4','quiz1']
n_classes_ n_classes_: int or list of int Number of label for each output.	int	2
n_features_in_ n_features_in_: int Number of features seen during :term:`fit`.	int	7
n_outputs_ n_outputs_: int Number of outputs.	int	1
sparse_output_ sparse_output_: bool True if the array returned from predict is to be in sparse CSC format. Is automatically set to True if the input `y` is passed in sparse format.	bool	False

`predict` the target of given examples

We can predict the target of examples by calling predict on the classifier object.

dummy_clf.predict(X)  # predict using the trained classifier

array(['not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+',
       'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+',
       'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+',
       'not A+', 'not A+', 'not A+'], dtype='<U6')

Evaluating with `score`

A model’s score method provides a convenient default measure, but its meaning depends on the model. For classifiers it is usually accuracy—the proportion of correctly predicted targets:

\[\text{accuracy} = \frac{\text{correct predictions}}{\text{total predictions}}\]

Always check what score returns before interpreting it as model quality.

print("The accuracy of the model on the training data: %0.3f" % (dummy_clf.score(X, y)))

The accuracy of the model on the training data: 0.524

Classification error is often defined as \(1 - ext{accuracy}\). Calling a classifier’s score(X, y) typically runs predict(X), compares the predictions with y, and returns accuracy.

For now we evaluate on the same small dataset used for fitting. This demonstrates the API, but it does not tell us how well the model will work on new data. Chapter 3 addresses that central problem.

print(
    "The error of the model on the training data: %0.3f" % (1 - dummy_clf.score(X, y))
)

The error of the model on the training data: 0.476

The `fit`, `predict`, and `score` pattern

The complete baseline workflow is shown below.

# Create `X` and `y` from the given data
X = classification_df.drop(columns=["quiz2"])
y = classification_df["quiz2"]

clf = DummyClassifier(strategy="most_frequent")  # Create a class object
clf.fit(X, y)  # Train/fit the model
print(clf.score(X, y))  # Assess the model

new_examples = [[0, 1, 92, 90, 95, 93, 92], [1, 1, 92, 93, 94, 92]]
clf.predict(new_examples)  # Predict on some new data using the trained model

0.5238095238095238

array(['not A+', 'not A+'], dtype='<U6')

Note

You will investigate classification baselines more fully in the exercises and lab.

A regression baseline with `DummyRegressor`

For regression, DummyRegressor can predict a constant such as the training-set mean or median. Its default score is \(R^2\), not accuracy.

The code below applies the same fit and predict pattern to the numerical version of the quiz-grade problem.

from sklearn.dummy import DummyRegressor

regression_df = pd.read_csv(DATA_DIR / "quiz2-grade-toy-regression.csv")  # Read data
X = regression_df.drop(columns=["quiz2"])  # Create `X` and `y` from the given data
y = regression_df["quiz2"]
reg = DummyRegressor()  # Create a class object
reg.fit(X, y)  # Train/fit the model
reg.score(X, y)  # Assess the model
new_examples = [[0, 1, 92, 90, 95, 93, 92], [1, 1, 92, 93, 94, 92]]
reg.predict(new_examples)  # Predict on some new data using the trained model

array([86.28571429, 86.28571429])

fit and predict play the same roles for regression and classification, but the default score differs. For regressors, score normally returns \(R^2\):

\(R^2=1\) represents perfect predictions;
a mean-prediction baseline usually has \(R^2=0\) on the data used to evaluate that mean; and
\(R^2<0\) means the predictions are worse than predicting that mean.

We study regression evaluation measures in detail later in the book.

reg.score(X, y)

0.0

Exercise 2.4: Order the modeling steps

Put the following steps in a sensible order:

call score with labeled evaluation data;
call predict on new examples;
create a model;
construct X and y; and
call fit.

Learning rules with decision trees

See also

An accompanying video introduces decision trees.

From hand-written rules to learned rules

Can we do better than the majority-class baseline? One option is to write nested if/else rules using attendance and previous grades. To make the idea easy to visualize, first imagine that the feature values have been converted to binary indicators.

Look for combinations that distinguish A+ from not A+. A hand-written program might begin like this:

if class_attendance == 1 and quiz1 == 1:
    prediction = "A+"
elif class_attendance == 1 and lab3 == 1 and lab4 == 1:
    prediction = "A+"
else:
    prediction = "not A+"

Even with seven binary features, the number of possible combinations becomes difficult to manage.

A decision tree learns a hierarchy of these questions from examples rather than requiring us to enumerate the rules by hand.

Pause and reflect

Before fitting one, inspect the examples above: which single yes/no question seems most effective at separating A+ from not A+? A useful question creates groups whose outcomes are more homogeneous than the original group.

The decision-tree intuition

A decision tree resembles the Twenty Questions game. At each node it asks a question about one feature and follows a branch based on the answer. Training determines which questions to ask; prediction follows the learned path to an answer.

Building decision trees with `sklearn`

Let’s binarize our toy dataset for simplicity.

classification_df = pd.read_csv(DATA_DIR / "quiz2-grade-toy-classification.csv")
X = classification_df.drop(columns=["quiz2"])
y = classification_df["quiz2"]

X_binary = X.copy()
columns = ["lab1", "lab2", "lab3", "lab4", "quiz1"]
for col in columns:
    X_binary[col] = X_binary[col].apply(lambda x: 1 if x >= 90 else 0)
X_binary.head()

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1
0	1	1	1	1	0	1	1
1	1	0	1	1	0	0	1
2	0	0	0	0	0	0	0
3	0	1	1	1	1	1	0
4	0	1	0	0	1	1	0

y.head()

0        A+
1    not A+
2    not A+
3        A+
4        A+
Name: quiz2, dtype: str

`DummyClassifier` on quiz2 grade prediction toy dataset

dummy_clf = DummyClassifier(strategy="most_frequent")
dummy_clf.fit(X_binary, y)
dummy_clf.score(X_binary, y)

0.5238095238095238

`DecisionTreeClassifier` on quiz2 grade prediction toy dataset

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(random_state=123)  # Create a reproducible decision tree
model.fit(X_binary, y)  # Fit a decision tree
model.score(X_binary, y)  # Assess the model

0.9047619047619048

The decision tree has much higher training accuracy than the dummy baseline. That is encouraging, but it does not yet establish that the tree will predict new students well. We will return to this distinction at the end of the chapter.

Note

The following visualization uses custom_plot_tree, a supporting plotting function from book/code/. Keeping this helper separate lets us focus here on interpreting the model, while its implementation remains available for inspection and reuse.

# Call the custom_plot_tree function to visualize the customized tree
width = 12
height = 8
plt.figure(figsize=(width, height))
custom_plot_tree(
    model,
    feature_names=X_binary.columns.tolist(),
    class_names=["A+", "not A+"],
    impurity=False,
    fontsize=10,
)

Reading a decision tree

Root node: The first question asked by the tree.
Branch: A connection representing the result of a question.
Internal node: A subsequent question within the tree.
Leaf node: The prediction produced after following a path through the tree.
Tree depth: The number of edges on the longest path from the root to a leaf.

How does prediction work?

new_example = pd.DataFrame([[0, 1, 0, 0, 1, 1, 1]], columns=X.columns)
new_example

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1
0	0	1	0	0	1	1	1

What’s the prediction for the new example?

model.predict(new_example)

array(['A+'], dtype=object)

To predict for a new example, begin at the root, answer the question at each node, and follow the corresponding branch. The leaf at the end of the path gives the prediction. Features that do not appear along that path do not affect that prediction. For the new example above, the path is:

lab3 <= 0.5 is False \(\rightarrow\) follow the right branch \(\rightarrow\) lab2 <= 0.5 is True \(\rightarrow\) follow the left branch \(\rightarrow\) predict A+.

How does fitting work conceptually?

During fitting, the algorithm searches over many possible feature-and-threshold questions and possible orders in which to ask them. It prefers questions that divide the training examples into increasingly homogeneous groups. The selected questions and thresholds become model parameters.

Choosing splits

To build intuition, consider three possible questions at the root of our toy tree:

Splitting on ml_experience is not very useful: students without ML experience include 5 A+ and 5 not-A+ outcomes, while students with ML experience include 5 A+ and 6 not-A+ outcomes. The two groups have almost the same class mixture.
Splitting on whether lab3 >= 90 is more useful: the below-90 group contains 2 A+ and 7 not-A+ outcomes, while the at-least-90 group contains 8 A+ and 4 not-A+ outcomes.
class_attendance provides a similar separation: the group that did not attend contains 2 A+ and 6 not-A+ outcomes, while the group that attended contains 8 A+ and 5 not-A+ outcomes.

The latter questions produce groups whose outcomes are more homogeneous than the original group. At each node, a classification tree quantifies this idea by comparing candidate questions using a measure of class impurity, such as the Gini index or entropy. The exact search and optimization details are beyond the scope of this course, but the result is a hierarchy of data-derived rules.

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(random_state=123)  # Create a reproducible decision tree
model.fit(X_binary, y)  # Fit a decision tree
plt.figure(figsize=(width, height))
custom_plot_tree(
    model,
    feature_names=X_binary.columns.tolist(),
    class_names=["A+", "not A+"],
    fontsize=10,
)

Note

Decision trees can split both categorical representations and numerical features. For a numerical feature, fitting also learns a threshold such as quiz1 <= 88.5.

Trees with numerical features

X.head()

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1
0	1	1	92	93	84	91	92
1	1	0	94	90	80	83	91
2	0	0	78	85	83	80	80
3	0	1	91	94	92	91	89
4	0	1	77	83	90	92	85

model = DecisionTreeClassifier(random_state=123)
model.fit(X, y)
plt.figure(figsize=(width, height))
custom_plot_tree(
    model,
    feature_names=X_binary.columns.tolist(),
    class_names=["A+", "not A+"],
    impurity=False,
    fontsize=10,
)

Decision trees for regression

Decision trees can also predict numerical targets. A DecisionTreeRegressor uses the same fit and predict interface, but chooses splits using a regression criterion such as squared error. Its default score is \(R^2\), which can be negative when predictions are worse than the mean baseline.

regression_df = pd.read_csv(DATA_DIR / "quiz2-grade-toy-regression.csv")
regression_df.head()

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1	quiz2
0	1	1	92	93	84	91	92	90
1	1	0	94	90	80	83	91	84
2	0	0	78	85	83	80	80	82
3	0	1	91	94	92	91	89	92
4	0	1	77	83	90	92	85	90

X = regression_df.drop(columns = ["quiz2"])
y = regression_df["quiz2"]

depth = 2
reg_model = DecisionTreeRegressor(max_depth=depth)
reg_model.fit(X, y)
regression_df["predicted_quiz2"] = reg_model.predict(X)
print("R^2 score on the training data: %0.3f\n\n" % (reg_model.score(X, y)))
regression_df.head()

R^2 score on the training data: 0.989

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1	quiz2	predicted_quiz2
0	1	1	92	93	84	91	92	90	90.333333
1	1	0	94	90	80	83	91	84	83.000000
2	0	0	78	85	83	80	80	82	83.000000
3	0	1	91	94	92	91	89	92	92.000000
4	0	1	77	83	90	92	85	90	90.333333

Exercise 2.5: Baselines and decision trees

Select all true statements.

1. Changing the feature representation necessarily changes a most-frequent DummyClassifier prediction.
1. predict receives X, whereas fit and score receive both X and y.
1. Decision-tree features must be binary.
1. A decision tree predicts by routing an example from the root to a leaf.

Solution

B and D are true. A most-frequent baseline ignores X, and decision trees can split numerical features by learning thresholds.

Model controls and decision boundaries

We now separate what a tree learns from the choices that control how it learns, then visualize how those choices affect its predictions.

See also

An accompanying video discusses parameters, hyperparameters, and decision boundaries.

Parameters

Fitting a decision tree determines which feature to inspect and which threshold to use at each node. These learned values are the model’s parameters. They are stored by the fitted model and used during prediction.

classification_df = pd.read_csv(DATA_DIR / "quiz2-grade-toy-classification.csv")
X = classification_df.drop(columns=["quiz2"])
y = classification_df["quiz2"]
model = DecisionTreeClassifier(random_state=123)
model.fit(X, y);

plt.figure(figsize=(width, height))
custom_plot_tree(
    model,
    feature_names=X_binary.columns.tolist(),
    class_names=["A+", "not A+"],
    impurity=False,
    fontsize=10,
)

With the default settings, a decision tree can continue splitting until each leaf contains training examples from only one class, or until another stopping condition is reached. A leaf containing examples from only one class is called pure because all the training examples at that leaf have the same target value. For example, a pure leaf in this problem would contain only A+ students or only not-A+ students.

A very deep tree may create rules that describe only one or two training examples. We can control this behaviour with hyperparameters such as the maximum tree depth.

A decision stump: `max_depth=1`

A tree with one split is called a decision stump.

model = DecisionTreeClassifier(max_depth=1, random_state=123)
model.fit(X, y)
width = 8
height = 2
plt.figure(figsize=(width, height))
custom_plot_tree(
    model,
    feature_names=X_binary.columns.tolist(),
    class_names=["A+", "not A+"],
    impurity=False,
    fontsize=12,
)  # custom function defined in code/utils.py

max_depth is a hyperparameter: a setting chosen before fitting that controls which trees the algorithm is allowed to learn.

A deeper tree: `max_depth=3`

model = DecisionTreeClassifier( 
    random_state=123,
    max_depth=3
)  # Let's try another value for the hyperparameter
model.fit(X, y)
width = 10
height = 5
plt.figure(figsize=(width, height))

custom_plot_tree(
    model,
    feature_names=X_binary.columns.tolist(),
    class_names=["A+", "not A+"],
    impurity=False,
    fontsize=12,
)

Parameters and hyperparameters

Parameters: Values learned from data during fit such as split features and thresholds. The fitted model needs them to make predictions.
Hyperparameters: Settings chosen before fit that control the learning process or model family. They may be selected using domain knowledge, heuristics, or a systematic validation procedure.

Important

In scikit-learn, hyperparameters are normally supplied when a model is constructed. Fitted parameters are learned later by calling fit.

Besides max_depth, commonly used decision-tree hyperparameters include min_samples_split, min_samples_leaf, and max_leaf_nodes. Each constrains the complexity of the learned tree in a different way.

See also

See the DecisionTreeClassifier documentation for the model’s complete interface and hyperparameters.

Decision boundaries

A fitted classifier partitions the possible feature values into regions assigned to different classes. The border between those regions is its decision boundary. With two features, we can draw this boundary and see how model complexity changes it.

Quiz-grade example

For visualization, we fit trees using only lab4 and quiz1.

X_subset = X[["lab4", "quiz1"]]
X_subset.head()

	lab4	quiz1
0	91	92
1	83	91
2	80	80
3	91	89
4	92	85

Decision boundary with `max_depth=1`

depth = 1  # decision stump
model = DecisionTreeClassifier(max_depth=depth)
model.fit(X_subset.values, y)
plot_tree_decision_boundary_and_tree(
    model, X_subset, y, x_label="lab4", y_label="quiz1", fontsize=15
)

The two colored regions show the classes predicted for possible combinations of lab4 and quiz1. A depth-one tree creates a single horizontal or vertical boundary because it asks one threshold question.

Decision boundary with `max_depth=2`

model = DecisionTreeClassifier(max_depth=2)
model.fit(X_subset.values, y)
plot_tree_decision_boundary_and_tree(
    model, X_subset, y, x_label="lab4", y_label="quiz1", fontsize=12
)

A second level allows the model to divide another region, producing a more detailed decision boundary.

Decision boundary with `max_depth=5`

model = DecisionTreeClassifier(max_depth=5)
model.fit(X_subset.values, y)
plot_tree_decision_boundary_and_tree(
    model, X_subset, y, x_label="lab4", y_label="quiz1", fontsize=8
)

At depth five, the boundary follows the training examples much more closely. Greater detail lowers training error, but whether it improves predictions for new examples remains unresolved.

Canada–USA cities example

Suppose we observe the longitude, latitude, and country of several cities near the Canada–USA border. Can a decision tree learn a boundary that predicts the country of a new coordinate?

### US Canada cities data
df = pd.read_csv(DATA_DIR / "canada_usa_cities.csv")
df

	longitude	latitude	country
0	-130.0437	55.9773	USA
1	-134.4197	58.3019	USA
2	-123.0780	48.9854	USA
3	-122.7436	48.9881	USA
4	-122.2691	48.9951	USA
...	...	...	...
204	-72.7218	45.3990	Canada
205	-66.6458	45.9664	Canada
206	-79.2506	42.9931	Canada
207	-72.9406	45.6275	Canada
208	-79.4608	46.3092	Canada

209 rows × 3 columns

X = df[["longitude", "latitude"]]

y = df["country"]

mglearn.discrete_scatter(X.iloc[:, 0], X.iloc[:, 1], y)
plt.xlabel("longitude")
plt.ylabel("latitude");

The geographic boundary

In reality, the Canada–USA boundary is known:

Image source

Our model does not receive this boundary. It approximates one using only the limited labeled examples in the training data.

model = DecisionTreeClassifier(max_depth=1)
model.fit(X.values, y)
plot_tree_decision_boundary_and_tree(
    model,
    X,
    y,
    height=6,
    width=16,
    fontsize=15,
    eps=10,
    x_label="longitude",
    y_label="latitude",
)

model = DecisionTreeClassifier(max_depth=2)
model.fit(X.values, y)
plot_tree_decision_boundary_and_tree(
    model,
    X,
    y,
    height=6,
    width=16,
    fontsize=12,
    eps=10,
    x_label="longitude",
    y_label="latitude",
)

Summary

This chapter established the vocabulary and workflow used throughout the book:

examples are represented by feature rows in \(X\), with targets in \(y\);
classification predicts discrete labels, whereas regression predicts numerical values;
baselines provide reference performance that a useful model should improve upon;
scikit-learn models share the fit and predict interface, while score is model-dependent;
decision trees learn hierarchies of feature-threshold questions;
fitted split features and thresholds are parameters, while settings such as max_depth are hyperparameters; and
increasing tree depth creates a more detailed decision boundary and can reduce training error.

The question that remains

A sufficiently deep tree can classify every training example correctly. Does that mean it will predict new examples well? Chapter 3 introduces data splitting, cross-validation, underfitting, and overfitting to answer this question.

Core terminology

Running example: quiz-grade prediction

Examples, features, and targets

Separating the feature table and target

A second example: housing-price prediction

Terminology across disciplines

Classification and regression

Prediction and statistical inference

Establishing a baseline

Why start with a baseline?

A classification baseline with DummyClassifier

scikit-learn fit-predict-score workflow

Reading the data

Create \(X\) and \(y\)

Create a classifier object

fit the classifier

predict the target of given examples

Evaluating with score

The fit, predict, and score pattern

A regression baseline with DummyRegressor

Learning rules with decision trees

From hand-written rules to learned rules

The decision-tree intuition

Building decision trees with sklearn

DummyClassifier on quiz2 grade prediction toy dataset

DecisionTreeClassifier on quiz2 grade prediction toy dataset

Reading a decision tree

How does prediction work?

How does fitting work conceptually?

Choosing splits

Trees with numerical features

Decision trees for regression

Model controls and decision boundaries

Parameters

A decision stump: max_depth=1

A deeper tree: max_depth=3

Parameters and hyperparameters

Decision boundaries

Quiz-grade example

Decision boundary with max_depth=1

Decision boundary with max_depth=2

Decision boundary with max_depth=5

Canada–USA cities example

The geographic boundary

Summary

A classification baseline with `DummyClassifier`

`scikit-learn` `fit-predict-score` workflow

`fit` the classifier

`predict` the target of given examples

Evaluating with `score`

The `fit`, `predict`, and `score` pattern

A regression baseline with `DummyRegressor`

Building decision trees with `sklearn`

`DummyClassifier` on quiz2 grade prediction toy dataset

`DecisionTreeClassifier` on quiz2 grade prediction toy dataset

A decision stump: `max_depth=1`

A deeper tree: `max_depth=3`

Decision boundary with `max_depth=1`

Decision boundary with `max_depth=2`

Decision boundary with `max_depth=5`