Define generalization and explain why it is a fundamental goal of supervised learning.
Distinguish between train, validation, test, and deployment data
Split a dataset into train and test sets using the train_test_split function
Explain why performance on unseen data must be estimated using data that was not used to fit the model.
Explain how cross-validation works and use cross_val_score and cross_validate to obtain training and validation scores.
Diagnose underfitting and overfitting by comparing training and validation scores.
Describe how increasing model complexity affects training error and the gap between training and validation performance.
Apply the Golden Rule: the test set must not influence model development or selection.
Apply a standard supervised-learning workflow: split the data, select hyperparameters using cross-validation, fit the selected model, and evaluate it once on the test set.
Supervised learning is useful because it lets us learn from examples whose targets are known and make predictions for examples whose targets are not yet known.
Consider a spam filter. We train it using past messages labelled as spam or not spam, but its real job is to classify messages that arrive tomorrow. Success on the messages it has already seen is not enough: the model must also work well on new messages.
This ability to perform well on new, relevant examples is called generalization.
This raises the central question of the chapter: How can we estimate whether a model will generalize before we deploy it?
A motivating example: tree depth
Chapter 2 introduced decision boundaries and the max_depth hyperparameter of a decision tree. Increasing max_depth allows a tree to learn more detailed rules. Let us see what that does to its performance on the data used to fit it.
# Toy quiz2 grade dataclassification_df = pd.read_csv(DATA_DIR /"quiz2-grade-toy-classification.csv")classification_df.head(10)
ml_experience
class_attendance
lab1
lab2
lab3
lab4
quiz1
quiz2
0
1
1
92
93
84
91
92
A+
1
1
0
94
90
80
83
91
not A+
2
0
0
78
85
83
80
80
not A+
3
0
1
91
94
92
91
89
A+
4
0
1
77
83
90
92
85
A+
5
1
0
70
73
68
74
71
not A+
6
1
0
80
88
89
88
91
A+
7
0
1
95
93
69
79
75
not A+
8
0
0
97
90
94
99
80
not A+
9
1
1
95
95
94
94
85
not A+
X = classification_df.drop(["quiz2"], axis=1)y = classification_df["quiz2"]
X_subset = X[["lab4", "quiz1"]] # Let's consider a subset of the data for visualizationX_subset.head()
lab4
quiz1
0
91
92
1
83
91
2
80
80
3
91
89
4
92
85
A depth-one tree
With max_depth=1, the model asks one question. Its decision boundary is simple, but it makes several mistakes on the training data.
/Users/kvarada/CS/2026-27/cpsc330-book/.venv/lib/python3.12/site-packages/sklearn/utils/validation.py:2827: UserWarning: X does not have valid feature names, but DecisionTreeClassifier was fitted with feature names
warnings.warn(
max_depths = np.arange(1, 18)errors = []for max_depth in max_depths: error =1- DecisionTreeClassifier(max_depth=max_depth).fit(X_subset, y).score( X_subset, y ) errors.append(error)plt.plot(max_depths, errors)plt.xlabel("max depth")plt.ylabel("error");
The training error reaches zero for sufficiently deep trees. That sounds ideal, but the increasingly detailed boundary may be learning peculiarities of these particular students rather than patterns that will hold for future students.
Pause and think
Would you trust the deepest tree most for predicting the grade of a student who is not in this dataset? Why or why not?
From memorization to generalization
The tree-depth example exposes a limitation of training performance: a flexible model can memorize its training examples. What we actually need is a model that captures patterns that remain useful beyond those examples.
What does it mean to generalize?
A model generalizes when it makes useful predictions on new examples that are reasonably similar to the examples relevant to the problem.
The following image represents a small training set. What visual patterns might a learner use to distinguish the two labels?
Now consider four new images. Which predictions seem reasonable based on the training examples?
Images 2 and 3 differ from the training images, but they share recognizable characteristics with them. Image 4 is quite different, so expecting a reliable prediction may be unreasonable.
Generalization is therefore not the ability to predict anything. It is the ability to perform well on new examples that are representative of the problem we intend to solve.
Important
A performance estimate is meaningful only when the evaluation data resemble the data the model will encounter after deployment.
Training error and generalization error
A model can perform well on examples it has already seen but poorly on new examples. We therefore distinguish between two kinds of error:
Training error: How often the model is wrong on the examples used to fit it.
Generalization error: How often we expect the model to be wrong on new examples from the same problem.
For a spam filter, training error measures mistakes on the messages used for fitting. Generalization error describes expected performance on future messages.
We cannot observe all possible future examples. For instance, we cannot evaluate a spam filter on every email it might encounter. Instead, we estimate generalization error using representative examples that the model did not see during fitting.
Because the model did not fit those held-out examples, its performance on them gives us a more realistic estimate than its training performance does.
A train/test split
The simplest version divides the available data into two parts:
the training set, used to fit the model; and
the test set, held aside to evaluate the final model.
The diagram shows an 80/20 split of ten examples. We usually create this split near the beginning of a project and keep the test set in an imaginary vault until the end.
# Let's demonstrate this with the canada usa cities data# The data is available in the data directorydf = pd.read_csv(DATA_DIR /"canada_usa_cities.csv")X = df.drop(columns=["country"])y = df["country"]
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=123) # 80%-20% train test split on X and y# Print shapesshape_dict = {"Data portion": ["X", "y", "X_train", "y_train", "X_test", "y_test"],"Shape": [ X.shape, y.shape, X_train.shape, y_train.shape, X_test.shape, y_test.shape, ],}shape_df = pd.DataFrame(shape_dict)HTML(shape_df.to_html(index=False))
Data portion
Shape
X
(209, 2)
y
(209,)
X_train
(167, 2)
y_train
(167,)
X_test
(42, 2)
y_test
(42,)
The most commonly used train_test_split arguments are:
test_size or train_size, which controls the proportion assigned to each set; and
random_state, which makes the random split reproducible.
Choosing the split size
There is no universally best split size. An 80/20 split is common, but the choice depends on how much data is available. We need enough training examples to learn useful patterns and enough test examples to obtain a meaningful final estimate.
Reproducible splitting
By default, train_test_split shuffles the rows before splitting them. Setting random_state fixes the random shuffle, so rerunning the code produces the same split.
Note
Random splitting is not appropriate for every dataset. Time-ordered data, grouped observations, and repeated measurements require strategies that respect their structure. We will revisit these cases later.
Why do we also need validation data?
The test set is meant to evaluate the final model. But model development involves choices (e.g., selecting max_depth) that also require feedback on unseen examples. If we repeatedly use the test set to make those choices, we gradually adapt our decisions to that test set.
We therefore use a third role, validation data, for model and hyperparameter selection.
The three roles are:
Training data: passed to fit; the model learns its parameters from these examples.
Validation data: used to compare models and select hyperparameters; not passed to fit for the model being evaluated.
Test data: kept untouched during model development and used once for final evaluation.
The terms vary across books and software. In this course, validation means data used during model development, while test means the final held-out data.
Note
Validation data is also referred to as development data or dev set for short.
Deployment data
After development, the model receives new examples in the real world. We call these deployment data. Their targets are usually unavailable at prediction time—for example, a spam filter must classify a message before a user tells us whether the prediction was correct.
Deployment performance is what we ultimately care about. Validation and test performance are useful only insofar as they help us anticipate it.
The role of each data subset
Data
Used with fit?
Used during model selection?
Main purpose
Training
Yes
Yes
Learn model parameters
Validation
No
Yes
Compare models and select hyperparameters
Test
No
No
Evaluate the selected model once
Deployment
Possibly later
No
Make predictions in the real world
We often expect training performance to be better than validation or test performance, but there is no guaranteed ordering among these quantities. Differences also arise from random sampling.
Exercise 3.1: The roles of data subsets
Select all true statements.
A low training error guarantees a low deployment error.
Data splitting helps estimate performance on unseen examples.
Validation data can be used to select max_depth.
Test performance can be checked repeatedly while tuning a model.
Test data should remain untouched until the model-selection process is complete.
TipSolution
B, C, and E are true. Training performance alone does not establish generalization, and repeatedly checking test performance makes the test set part of model selection.
We could reserve one fixed validation set, but the result may depend heavily on which examples happen to enter it. This is especially troublesome with a small dataset: setting aside more validation data leaves fewer examples for fitting, while setting aside less makes the validation score less stable.
\(k\)-fold cross-validation
Cross-validation reuses the training data through a sequence of temporary train/validation splits:
Divide the training data into \(k\) groups called folds.
Fit on \(k-1\) folds and validate on the remaining fold.
Repeat until every fold has served as the validation fold once.
The diagram illustrates four-fold cross-validation.
Cross-validation produces \(k\) validation scores. We usually report their mean and also examine their variation. The mean is typically more stable than a score from one validation split, while the variation shows how sensitive the result is to the particular fold.
Only the training portion of the original train/test split participates in cross-validation. The test set remains in the vault.
Cross-validation using scikit-learn
from sklearn.model_selection import cross_val_score, cross_validate
Validation scores with cross_val_score
cross_val_score returns one validation score per fold. Here we evaluate a depth-four decision tree using ten folds.
model = DecisionTreeClassifier(max_depth=4)cv_scores = cross_val_score(model, X_train, y_train, cv=10)cv_scores
Average cross-validation score = 0.84
Standard deviation of cross-validation score = 0.09
For each fold, scikit-learn creates a fresh model, fits it on the fold’s training portion, and scores it on the validation portion. The resulting array contains ten validation scores—one for each fitted model.
Training and validation scores with cross_validate
cross_validate can return additional information. Setting return_train_score=True lets us compare performance on each fold’s training and validation portions.
Cross-validation evaluates a model configuration; it does not return a model ready for deployment. After choosing the hyperparameters, we fit a new model on all available training data.
Note
cross_val_score and cross_validate do not shuffle observations themselves. For classification, scikit-learn uses stratified folds by default so that class proportions are approximately preserved. More specialized splitting strategies are needed for time series or grouped data.
Where cross-validation fits in the workflow
At this point, our workflow is:
Split the available data into training and test sets.
Use cross-validation on the training set to compare model configurations.
Choose a configuration using its cross-validation results.
Fit that configuration on the complete training set.
Evaluate it once on the test set.
Before selecting a tree depth, however, we need to understand what training and validation scores tell us about model complexity.
Underfitting, overfitting, the fundamental trade-off, and the golden rule
TipSee also
An accompanying video discusses underfitting, overfitting, and the golden rule.
Reading training and validation error
Within cross-validation, we can examine:
\(E_\textrm{train}\): mean error on the training folds; and
\(E_\textrm{valid}\): mean error on the validation folds.
Their absolute values and the gap between them help diagnose two common problems: underfitting and overfitting. We use validation—not test—performance for this diagnosis because model development is still underway.
Underfitting
A model underfits when it is too simple to capture useful patterns in the training data. A depth-one decision tree illustrates this behaviour.
Both errors are relatively high, and even the training error is unsatisfactory. This is the signature of underfitting: the model cannot fit the training data well enough.
A more flexible model may reduce both training and validation error.
Overfitting
A model overfits when it learns details of the training data that do not generalize. An unrestricted decision tree can keep splitting until it closely fits—or even memorizes—the training examples.
As depth increases, training error steadily decreases. Validation error initially improves, then becomes less stable and may worsen. Deep trees have enough flexibility to encode peculiarities of individual training folds that do not carry over to their validation folds.
The fundamental trade-off
Increasing model complexity tends to have two competing effects:
training error decreases because the model can fit the training data more closely; and
the gap between training and validation error tends to increase because the model becomes more sensitive to details of the training data.
We want enough complexity to capture useful patterns, but not so much that the model relies heavily on accidental ones.
Selecting model complexity
We use cross-validation to look for a configuration that balances underfitting and overfitting. In the idealized diagram below, validation error is smallest at an intermediate level of complexity.
In practice, validation scores contain sampling noise, so there may not be one obviously best value. A common starting point is to choose the hyperparameter with the best mean cross-validation score while also examining variation across folds and preferring simpler models when performance is similar.
results_df
mean_train_error
mean_cv_error
std_cv_error
std_train_error
depth
1
0.165651
0.190074
0.084735
0.008808
2
0.155011
0.195956
0.085624
0.006376
3
0.137033
0.195588
0.089631
0.016135
4
0.093135
0.159926
0.090119
0.006472
5
0.081152
0.154044
0.082789
0.012076
6
0.069183
0.184926
0.060613
0.017382
7
0.045885
0.160294
0.086287
0.017165
8
0.027267
0.178676
0.090253
0.011293
9
0.019951
0.172426
0.091580
0.011527
10
0.005987
0.190809
0.089655
0.005526
11
0.002658
0.190809
0.089655
0.003255
12
0.000000
0.196691
0.101277
0.000000
13
0.000000
0.190809
0.089655
0.000000
14
0.000000
0.184926
0.080262
0.000000
15
0.000000
0.184926
0.080262
0.000000
Final evaluation on the test set
The cross-validation results let us select max_depth without consulting the test set.
best_depth = results_df.index.values[np.argmin(results_df["mean_cv_error"])]print("The minimum validation error is %0.3f at max_depth = %d "% ( np.min(results_df["mean_cv_error"]), best_depth, ))
The minimum validation error is 0.154 at max_depth = 5
We now fit a new decision tree with the selected depth on the full training set. Only after fitting this final model do we unlock the test set.
model = DecisionTreeClassifier(max_depth=best_depth)model.fit(X_train, y_train)print(f"Error on test set: {1- model.score(X_test, y_test):.2f}")
Error on test set: 0.19
The test error is our final estimate of performance on new, representative examples. It may differ from the cross-validation error because both are estimates based on finite samples.
This estimate is not a guarantee of deployment performance. It depends on the test set being representative and on our having kept it independent of model development.
The Golden Rule
Important
The test data must not influence model development or selection in any way.
Every time we use test results to change features, preprocessing, hyperparameters, or model choice, we adapt the workflow to that test set. Its score then becomes less trustworthy as an estimate of future performance.
This does not mean we ignore validation data: validation data exists specifically to guide model development. The rule is to keep the test data separate until the end.
Protecting the test set
The safest habit is to create the train/test split near the start of a project and treat the test set as though it were locked in a vault.
A reusable supervised-learning workflow
Split: Create training and test sets before model development.
Develop: Explore the training data and build the preprocessing and modeling pipeline without using the test set.
Validate: Compare hyperparameters or models using cross-validation on the training set.
Refit: Fit the selected configuration on the complete training set.
Test once: Evaluate the final model on the test set.
Deploy and monitor: Use the model on new examples and watch for changes between development and deployment data.
We will refine this workflow throughout the course, but its separation of model development from final evaluation remains essential.
Exercise 3.2: Cross-validation and model complexity
Select all true statements.
\(k\)-fold cross-validation fits \(k\) separate models.
Cross-validation should be performed using the training set, not the held-out test set.
Much better training performance than validation performance is evidence of overfitting.
Validation error must always increase when training error decreases.
A decision stump may underfit a complicated classification problem.
TipSolution
A, B, C, and E are true. Validation error does not have to move in the opposite direction from training error; both may initially improve as complexity increases.
Exercise 3.3: Diagnose the model
Match each result to the most likely diagnosis.
Training accuracy: 0.61; validation accuracy: 0.59
Training accuracy: 0.99; validation accuracy: 0.68
Training accuracy: 0.88; validation accuracy: 0.87
Choose from underfitting, overfitting, or neither is clearly indicated.
TipSolution
Underfitting: even training performance is poor.
Overfitting: there is a large training–validation gap.
Neither is clearly indicated: the scores are similar and reasonably high, although judging whether 0.87 is adequate requires context and a baseline.
Exercise 3.4: Protecting the test set
You compare five values of max_depth using cross-validation and select max_depth=4. You then obtain disappointing test performance. Which action preserves the Golden Rule?
Try the other four depths on the test set and report the best result.
Return to the training data, revise the workflow using cross-validation, and acknowledge that another untouched test set would be needed for a fully unbiased final estimate.
Move the easiest test examples into the training set and evaluate again.
Report the cross-validation score as though it were the test score.
TipSolution
B. Once the test result guides further development, that test set no longer provides a clean final estimate for the revised workflow.
Exercise 3.5: Put the workflow in order
Arrange these steps in a sensible order:
evaluate the final model on the test set;
select hyperparameters using cross-validation;
fit the selected configuration on the full training set;
split the available data into training and test sets.
TipSolution
Split the data.
Select hyperparameters using cross-validation on the training set.
Fit the selected configuration on the full training set.
Evaluate once on the test set.
Summary
This chapter introduced the central goal and evaluation workflow of supervised learning:
Generalization means performing well on new, relevant examples—not merely on the training data.
A train/test split reserves independent data for final evaluation.
Validation data and cross-validation guide model and hyperparameter selection without exposing the test set.
High training and validation error suggests underfitting; a large gap between them suggests overfitting.
Increasing complexity usually lowers training error but can widen the training–validation gap.
The Golden Rule keeps test data from influencing model development.
The standard workflow is: split, develop, cross-validate, refit, test once, and then deploy and monitor.
