Lecture 2: Terminology, Baselines, Decision Trees¶

UBC 2025-26

Imports, Announcements, LOs¶

Imports¶

Learning outcomes¶

By the end of this lesson, you will be able to:

• Define key machine learning terminology:
  features, targets, predictions, training, error, classification vs. regression, supervised vs. unsupervised learning, hyperparameters vs. parameters, baselines, decision boundaries

• Build a simple machine learning model in scikit-learn, explaining the fit–predict workflow and evaluating performance with the score method

• Describe at a high level how decision trees are trained (fitting) and how they make predictions

• Implement and visualize decision trees in scikit-learn using DecisionTreeClassifier and DecisionTreeRegressor

Terminology [video]¶

You will see a lot of variable terminology in machine learning and statistics. Let’s familiarize ourselves with some of the basic terminology used in ML.

Big picture and datasets¶

In this lecture, we’ll talk about our first machine learning model: Decision trees. We will also familiarize ourselves with some common terminology in supervised machine learning.

Toy datasets¶

Later in the course we will use larger datasets from Kaggle, for instance. But for our first couple of lectures, we will be working with the following three toy datasets:

• Quiz2 grade prediction classification dataset

• Quiz2 grade prediction regression dataset

• Canada USA cities dataset

I’ll be using the following grade prediction toy dataset to demonstrate the terminology. Imagine that you are taking a course with four home work assignments and two quizzes. You and your friends are quite nervous about your quiz2 grades and you want to know how will you do based on your previous performance and some other attributes. So you decide to collect some data from your friends from last year and train a supervised machine learning model for quiz2 grade prediction.

Recap: Supervised machine learning¶

Tabular data¶

In supervised machine learning, the input data is typically organized in a tabular format, where rows are examples and columns are features. One of the columns is typically the target.

Features

Features are relevant characteristics of the problem, usually suggested by experts. Features are typically denoted by $X$ and the number of features is usually denoted by $d$.

Target

Target is the feature we want to predict (typically denoted by $y$).

Example

A row of feature values. When people refer to an example, it may or may not include the target corresponding to the feature values, depending upon the context. The number of examples is usually denoted by $n$.

Training

The process of learning the mapping between the features ($X$) and the target ($y$).

Example: Tabular data for grade prediction¶

The tabular data usually contains both: the features (X) and the target (y).

So the first step in training a supervised machine learning model is separating X and y.

Example: Tabular data for the housing price prediction¶

Here is an example of tabular data for housing price prediction. You can download the data from here.

Alternative terminology for examples, features, targets, and training¶

• examples = rows = samples = records = instances

• features = inputs = predictors = explanatory variables = regressors = independent variables = covariates

• targets = outputs = outcomes = response variable = dependent variable = labels (if categorical).

• training = learning = fitting

Supervised learning vs. Unsupervised learning¶

In supervised learning, training data comprises a set of features ($X$) and their corresponding targets ($y$). We wish to find a model function $f$ that relates $X$ to $y$. Then use that model function to predict the targets of new examples.

In unsupervised learning training data consists of observations ($X$) without any corresponding targets. Unsupervised learning could be used to group similar things together in $X$ or to provide concise summary of the data. We’ll learn more about this topic in later videos.

Supervised machine learning is about function approximation, i.e., finding the mapping function between X and y whereas unsupervised machine learning is about concisely describing the data.

Classification vs. Regression¶

In supervised machine learning, there are two main kinds of learning problems based on what they are trying to predict.

• Classification problem: predicting among two or more discrete classes

  • Example1: Predict whether a patient has a liver disease or not

  • Example2: Predict whether a student would get an A+ or not in quiz2.

• Regression problem: predicting a continuous value

  • Example1: Predict housing prices

  • Example2: Predict a student’s score in quiz2.

❓❓ Questions for you¶

Exercise 2.1 Select all of the following statements which are examples of supervised machine learning¶

1. How many examples and features are there in the housing price data above? You can use df.shape to get number of rows and columns in a dataframe.

2. For each of the following examples what would be the relevant features and what would be the target?

   1. Sentiment analysis

   2. Fraud detection

   3. Face recognition

iClicker Exercise 2.2 Supervised vs unsupervised¶

Select all of the following statements which are examples of supervised machine learning

• (A) Finding groups of similar properties in a real estate data set.

• (B) Predicting whether someone will have a heart attack or not on the basis of demographic, diet, and clinical measurement.

• (C) Grouping articles on different topics from different news sources (something like the Google News app).

• (D) Detecting credit card fraud based on examples of fraudulent and non-fraudulent transactions.

• (E) Given some measure of employee performance, identify the key factors which are likely to influence their performance.

iClicker Exercise 2.3 Classification vs regression¶

Select all of the following statements which are examples of regression problems

• (A) Predicting the price of a house based on features such as number of bedrooms and the year built.

• (B) Predicting if a house will sell or not based on features like the price of the house, number of rooms, etc.

• (C) Predicting percentage grade in CPSC 330 based on past grades.

• (D) Predicting whether you should bicycle tomorrow or not based on the weather forecast.

• (E) Predicting appropriate thermostat temperature based on the wind speed and the number of people in a room.

Baselines [video]¶

Supervised learning (Reminder)¶

• Training data $\rightarrow$ Machine learning algorithm $\rightarrow$ ML model

• Unseen test data + ML model $\rightarrow$ predictions

Let’s build a very simple supervised machine learning model for quiz2 grade prediction problem.

Seems like “not A+” occurs more frequently than “A+”. What if we predict “not A+” all the time?

Baselines¶

Baseline

A simple machine learning algorithm based on simple rules of thumb.

• For example, most frequent baseline always predicts the most frequent label in the training set.

• Baselines provide a way to sanity check your machine learning model.

DummyClassifier¶

• sklearn’s baseline model for classification

• Let’s train DummyClassifier on the grade prediction dataset.

Steps to train a classifier using sklearn¶

1. Read the data

2. Create $X$ and $y$

3. Create a classifier object

4. fit the classifier

5. predict on new examples

6. score the model

Reading the data¶

Create $X$ and $y$¶

• $X$ → Feature vectors

• $y$ → Target

Create a classifier object¶

• import the appropriate classifier

• Create an object of the classifier

fit the classifier¶

• The “learning” is carried out when we call fit on the classifier object.

predict the target of given examples¶

• We can predict the target of examples by calling predict on the classifier object.

score your model¶

• How do you know how well your model is doing?

• For classification problems, by default, score gives the accuracy of the model, i.e., proportion of correctly predicted targets.

  $accuracy = \frac{\text{correct predictions}}{\text{total examples}}$

• Sometimes you will also see people reporting error, which is usually $1 - accuracy$

• score

  • calls predict on X

  • compares predictions with y (true targets)

  • returns the accuracy in case of classification.

fit, predict , and score summary¶

Here is the general pattern when we build ML models using sklearn.

DummyRegressor¶

You can also do the same thing for regression problems using DummyRegressor, which predicts mean, median, or constant value of the training set for all examples.

• Let’s build a regression baseline model using sklearn.

• The fit and predict paradigms similar to classification. The score method in the context of regression returns somethings called $R^2$ score. (More on this in later videos.)

  • The maximum $R^2$ is 1 for perfect predictions.

  • For DummyRegressor it returns the mean of the y values.

❓❓ Questions for you¶

Exercise 2.4¶

1. Order the steps below to build ML models using sklearn.

   • score to evaluate the performance of a given model

   • predict on new examples

   • Creating a model instance

   • Creating X and y

   • fit

Break (5 min)¶

• We will try to take a 5-minute break half way through every class.

Decision trees [video]¶

Writing a traditional program to predict quiz2 grade¶

• Can we do better than the baseline?

• Forget about ML for a second. If you are asked to write a program to predict whether a student gets an A+ or not in quiz2, how would you go for it?

• For simplicity, let’s binarize the feature values.

• Is there a pattern that distinguishes yes’s from no’s and what does the pattern say about today?

• How about a rule-based algorithm with a number of if else statements?

  if class_attendance == 1 and quiz1 == 1:
      quiz2 == "A+"
  elif class_attendance == 1 and lab3 == 1 and lab4 == 1:
      quiz2 == "A+"
  ...

• How many possible rule combinations there could be with the given 7 binary features?

  • Gets unwieldy pretty quickly

Decision tree algorithm¶

• A machine learning algorithm to derive such rules from data in a principled way.

• Have you ever played 20-questions game? Decision trees are based on the same idea!

• Let’s fit a decision tree using scikit-learn and predict with it.

• Recall that scikit-learn uses the term fit for training or learning and uses predict for prediction.

Building decision trees with sklearn¶

Let’s binarize our toy dataset for simplicity.

DummyClassifier on quiz2 grade prediction toy dataset¶

DecisionTreeClassifier on quiz2 grade prediction toy dataset¶

The decision tree classifier is giving much higher accuracy than the dummy classifier. That’s good news!

Some terminology related to trees¶

Here is a commonly used terminology in a typical representation of decision trees.

A root node

represents the first condition to check or question to ask

A branch

connects a node (condition) to the next node (condition) in the tree. Each branch typically represents either true or false.

An internal node

represents conditions within the tree

A leaf node

represents the predicted class/value when the path from root to the leaf node is followed.

Tree depth

The number of edges on the path from the root node to the farthest away leaf node.

How does predict work?¶

What’s the prediction for the new example?

In summary, given a learned tree and a test example, during prediction time,

• Start at the top of the tree. Ask binary questions at each node and follow the appropriate path in the tree. Once you are at a leaf node, you have the prediction.

• Note that the model only considers the features which are in the learned tree and ignores all other features.

How does fit work?¶

• Decision tree is inspired by 20-questions game.

• Each node either represents a question or an answer. The terminal nodes (called leaf nodes) represent answers.

How does fit work?¶

• Which features are most useful for classification?

• Minimize impurity at each question

• Common criteria to minimize impurity: gini index, information gain, cross entropy

Decision trees with continuous features¶

Decision tree for regression problems¶

• We can also use decision tree algorithm for regression.

• Instead of gini, we use some other criteria for splitting. A common one is mean squared error (MSE). (More on this in later videos.)

• scikit-learn supports regression using decision trees with DecisionTreeRegressor

  • fit and predict paradigms similar to classification

  • score returns somethings called $R^2$ score.

    • The maximum $R^2$ is 1 for perfect predictions.

    • It can be negative which is very bad (worse than DummyRegressor).

❓❓ Questions for you¶

iClicker Exercise 2.5: Baselines and decision trees¶

Select all of the following statements which are TRUE.

• (A) Change in features (i.e., binarizing features above) would change DummyClassifier predictions.

• (B) predict takes only X as argument whereas fit and score take both X and y as arguments.

• (C) For the decision tree algorithm to work, the feature values must be binary.

• (D) The prediction in a decision tree works by routing the example from the root to the leaf.

More terminology [video]¶

• Parameters and hyperparameters

• Decision boundary

Parameters¶

• The decision tree algorithm primarily learns two things:

  • the best feature to split on

  • the threshold for the feature to split on at each node

• These are called parameters of the decision tree model.

• When predicting on new examples, we need parameters of the model.

• With the default setting, the nodes are expanded until all leaves are “pure”.

• The decision tree is creating very specific rules, based on just one example from the data.

• Is it possible to control the learning in any way?

  • Yes! One way to do it is by controlling the depth of the tree, which is the length of the longest path from the tree root to a leaf.

Decision tree with max_depth=1¶

Decision stump

A decision tree with only one split (depth=1) is called a decision stump.

max_depth is a hyperparameter of DecisionTreeClassifier.

Decision tree with max_depth=3¶

Parameters and hyperparameters: Summary¶

Parameters

When you call fit, a bunch of values get set, like the features to split on and split thresholds. These are called parameters. These are learned by the algorithm from the data during training. We need them during prediction time.

Hyperparameters

Even before calling fit on a specific data set, we can set some “knobs” that control the learning. These are called hyperparameters. These are specified based on: expert knowledge, heuristics, or systematic/automated optimization (more on this in the coming lectures).

Above we looked at the max_depth hyperparameter. Some other commonly used hyperparameters of decision tree are:

• min_samples_split

• min_samples_leaf

• max_leaf_nodes

Decision boundary¶

What do we do with learned models? So far we have been using them to predict the class of a new instance. Another way to think about them is to ask: what sort of test examples will the model classify as positive, and what sort will it classify as negative?

Example 1: quiz 2 grade prediction¶

For visualization purposes, let’s consider a subset of the data with only two features.

Decision boundary for max_depth=1¶

We assume geometric view of the data. Here, the red region corresponds to “not A+” class and blue region corresponds to “A+” class. And there is a line separating the red region and the blue region which is called the decision boundary of the model. Different models have different kinds of decision boundaries. In decision tree models, when we are working with only two features, the decision boundary is made up of horizontal and vertical lines. In the example above, the decision boundary is created by asking one question lab4 <= 84.5.

Decision boundary for max_depth=2¶

The decision boundary, i.e., the model gets a bit more complicated.

Decision boundary for max_depth=5¶

The decision boundary, i.e., the model gets even more complicated with max_depth=5.

Example 2: Predicting country using the longitude and latitude¶

Imagine that you are given longitude and latitude of some border cities of USA and Canada along with which country they belong to. Using this training data, you are supposed to come up with a classification model to predict whether a given longitude and latitude combination is in the USA or Canada.

Real boundary between Canada and USA¶

In real life we know what’s the boundary between USA and Canada.

Source

Here we want to pretend that we do not know this boundary and we want to infer this boundary based on the limited training examples given to us.

Practice exercises¶

• If you want more practice, check out module 2 in this online course. All the sections without video or notes symbol are exercises.

Final comments, summary, and reflection¶

What did we learn today?

• There is a lot of terminology and jargon used in ML. Some of the basic terminology includes:

  • Features, target, examples, training

  • Supervised vs. Unsupervised machine learning

  • Classification and regression

  • Accuracy and error

  • Parameters and hyperparameters

  • Decision boundary

• Baselines and steps to train a supervised machine learning model

  • Baselines serve as reference points in ML workflow.

• Decision trees

  • are models that make predictions by sequentially looking at features and checking whether they are above/below a threshold

  • learn a hierarchy of if/else questions, similar to questions you might ask in a 20-questions game.

  • learn axis-aligned decision boundaries (vertical and horizontal lines with 2 features)

  • One way to control the complexity of decision tree models is by using the depth hyperparameter (max_depth in sklearn).