Lecture 10: Regression metrics¶

UBC, 2025-26

Imports and LO¶

Imports¶

Learning outcomes¶

From this lecture, students are expected to be able to:

• Carry out feature transformations on somewhat complicated dataset.

• Visualize transformed features as a dataframe.

• Use Ridge and RidgeCV.

• Explain how alpha hyperparameter of Ridge relates to the fundamental tradeoff.

• Explain the effect of alpha on the magnitude of the learned coefficients.

• Examine coefficients of transformed features.

• Appropriately select a scoring metric given a regression problem.

• Interpret and communicate the meanings of different scoring metrics on regression problems.

  • MSE, RMSE, $R^2$, MAPE

• Apply log-transform on the target values in a regression problem with TransformedTargetRegressor.

After carrying out preprocessing, why it’s useful to get feature names for transformed features?

More comments on tackling class imbalance¶

• In lecture 9 we talked about a few rather simple approaches to deal with class imbalance.

• If you have a problem such as fraud detection problem where you want to spot rare events, you can also think of this problem as anomaly detection problem and use different kinds of algorithms such as isolation forests.

• If you are interested in this area, it might be worth checking out this book on this topic. Imbalanced Learning: Foundations, Algorithms, and Applications

Dataset [video]¶

In this lecture, we’ll be using Kaggle House Prices dataset. As usual, to run this notebook you’ll need to download the data. For this dataset, train and test have already been separated. We’ll be working with the train portion in this lecture.

• The supervised machine learning problem is predicting housing price given features associated with properties.

• Here, the target is SalePrice, which is continuous. So it’s a regression problem (as opposed to classification).

Let’s separate X and y¶

EDA¶

pandas_profiler¶

We do not have pandas_profiling in our course environment. You will have to install it in the environment on your own if you want to run the code below.

conda install -c conda-forge pandas-profiling

Feature types¶

• Do not blindly trust all the info given to you by automated tools.

• How does pandas profiling figure out the data type?

  • You can look at the Python data type and say floats are numeric, strings are categorical.

  • However, in doing so you would miss out on various subtleties such as some of the string features being ordinal rather than truly categorical.

  • Also, it will think free text is categorical.

• In addition to tools such as above, it’s important to go through data description to understand the data.

• The data description for our dataset is available here.

Feature types¶

• We have mixed feature types and a bunch of missing values.

• Now, let’s identify feature types and transformations.

• Let’s get the numeric-looking columns.

Not all numeric looking columns are necessarily numeric.

MSSubClass: Identifies the type of dwelling involved in the sale.

    20	1-STORY 1946 & NEWER ALL STYLES
    30	1-STORY 1945 & OLDER
    40	1-STORY W/FINISHED ATTIC ALL AGES
    45	1-1/2 STORY - UNFINISHED ALL AGES
    50	1-1/2 STORY FINISHED ALL AGES
    60	2-STORY 1946 & NEWER
    70	2-STORY 1945 & OLDER
    75	2-1/2 STORY ALL AGES
    80	SPLIT OR MULTI-LEVEL
    85	SPLIT FOYER
    90	DUPLEX - ALL STYLES AND AGES
   120	1-STORY PUD (Planned Unit Development) - 1946 & NEWER
   150	1-1/2 STORY PUD - ALL AGES
   160	2-STORY PUD - 1946 & NEWER
   180	PUD - MULTILEVEL - INCL SPLIT LEV/FOYER
   190	2 FAMILY CONVERSION - ALL STYLES AND AGES

Also, month sold is more of a categorical feature than a numeric feature.

We’ll treat the above numeric-looking features as categorical features.

• There are a bunch of ordinal features in this dataset.

• Ordinal features with the same scale

  • Poor (Po), Fair (Fa), Typical (TA), Good (Gd), Excellent (Ex)

  • These we’ll be calling ordinal_features_reg.

• Ordinal features with different scales

  • These we’ll be calling ordinal_features_oth.

We’ll pass the above as categories in our OrdinalEncoder.

• There are a bunch more ordinal features using different scales.

  • These we’ll be calling ordinal_features_oth.

  • We are encoding them separately.

The remaining features are categorical features.

• We are not doing it here but we can engineer our own features too.

• Would price per square foot be a good feature to add in here?

Applying feature transformations¶

• Since we have mixed feature types, let’s use ColumnTransformer to apply different transformations on different features types.

Examining the preprocessed data¶

We went from 80 features to 263 features!!

Other possible preprocessing?¶

• There is a lot of room for improvement ...

• We’re just using SimpleImputer.

  • In reality we’d want to go through this more carefully.

  • We may also want to drop some columns that are almost entirely missing.

• We could also check for outliers, and do other exploratory data analysis (EDA).

• But for now this is good enough ...

Model building¶

DummyRegressor¶

Let’s try a linear model: Ridge¶

• Recall that we are going to use Ridge() instead of LinearRegression() in this course.

• Similar to linear regression, ridge regression is also a linear model for regression.

• So the formula it uses to make predictions is the same one used for ordinary least squares.

• But it has a hyperparameter alpha which controls the fundamental tradeoff.

Let’s carry out cross-validation with Ridge.

• Quite a bit of variation in the test scores.

• Performing poorly in fold 8. Not sure why.

  • Probably it contains the outliers in the data which we kind of ignored.

Feature names of transformed data¶

• If you want to get the column names of newly created columns, you need to fit the transformer.

Tuning alpha hyperparameter of Ridge¶

• Recall that Ridge has a hyperparameter alpha that controls the fundamental tradeoff.

• This is like C in LogisticRegression but, annoyingly, alpha is the inverse of C.

• That is, large C is like small alpha and vice versa.

• Smaller alpha: lower training error (overfitting)

• It seems alpha=100 is the best choice here.

• General intuition: larger alpha leads to smaller coefficients.

• Smaller coefficients mean the predictions are less sensitive to changes in the data. Hence less chance of overfitting.

• Smaller alpha leads to bigger coefficients.

With the best alpha found by the grid search, the coefficients are somewhere in between.

To summarize:

• Higher values of alpha means a more restricted model.

• The values of coefficients are likely to be smaller for higher values of alpha compared to lower values of alpha.

RidgeCV¶

Because it’s so common to want to tune alpha with Ridge, sklearn provides a class called RidgeCV, which automatically tunes alpha based on cross-validation.

Let’s examine the tuned model.

So according to this model:

• As OverallQual feature gets bigger the housing price will get bigger.

• Neighborhood_Edwards is associated with reducing the housing price.

  • We’ll talk more about interpretation of different kinds of features next week.

❓❓ Questions for you¶

iClicker Exercise 10.1¶

Select all of the following statements which are TRUE.

• (A) Price per square foot would be a good feature to add in our X.

• (B) The alpha hyperparameter of Ridge has similar interpretation of C hyperparameter of LogisticRegression; higher alpha means more complex model.

• (C) In Ridge, smaller alpha means bigger coefficients whereas bigger alpha means smaller coefficients.

Can we use the metrics we looked at in the previous lecture for this problem? Why or why not?

Regression scoring functions¶

• We aren’t doing classification anymore, so we can’t just check for equality:

We need a score that reflects how right/wrong each prediction is.

There are a number of popular scoring functions for regression. We are going to look at some common metrics:

• mean squared error (MSE)

• $R^2$

• root mean squared error (RMSE)

• MAPE

See sklearn documentation for more details.

Mean squared error (MSE)¶

• A common metric is mean squared error:

Perfect predictions would have MSE=0.

This is also implemented in sklearn:

• MSE looks huge and unreasonable. There is an error of ~$1 Billion!

• Is this score good or bad?

• Unlike classification, with regression our target has units.

• The target is in dollars, the mean squared error is in $dollars^2$

• The score also depends on the scale of the targets.

• If we were working in cents instead of dollars, our MSE would be $10,000 \times (100^2$) higher!

Root mean squared error or RMSE¶

• The MSE above is in $dollars^2$.

• A more relatable metric would be the root mean squared error, or RMSE

• Error of $30,000 makes more sense.

• Let’s dig deeper.

• Here we can see a few cases where our prediction is way off.

• Is there something weird about those houses, perhaps? Outliers?

• Under the line means we’re under-predicting, over the line means we’re over-predicting.

$R^2$ (not in detail)¶

A common score is the $R^2$

• This is the score that sklearn uses by default when you call score()

• You can read about it if interested.

• $R^2$ measures the proportion of variability in $y$ that can be explained using $X$.

• Independent of the scale of $y$. So the max is 1.

• The denominator measures the total variance in $y$.

• The amount of variability that is left unexplained after performing regression.

Key points:

• The maximum is 1 for perfect predictions

• Negative values are very bad: “worse than DummyRegressor” (very bad)

(Optional) Warning: MSE is “reversible” but $R^2$ is not:

• When you call fit it minimizes MSE / maximizes $R^2$ (or something like that) by default.

• Just like in classification, this isn’t always what you want!!

MAPE¶

• We got an RMSE of ~$30,000 before.

Question: Is an error of $30,000 acceptable?

• For a house worth $600k, it seems reasonable! That’s 5% error.

• For a house worth $60k, that is terrible. It’s 50% error.

We have both of these cases in our dataset.

How about looking at percent error?

These are both positive (predict too high) and negative (predict too low).

We can look at the absolute percent error:

And, like MSE, we can take the average over examples. This is called mean absolute percent error (MAPE).

Let’s use sklearn to calculate MAPE.

• Ok, this is quite interpretable.

• On average, we have around 10% error.

Transforming the targets¶

• When you have prices or count data, the target values are skewed.

• Let’s look at our target column.

• A common trick in such cases is applying a log transform on the target column to make it more normal and less skewed.

• That is, transform $y\rightarrow \log(y)$.

• Linear regression will usually work better on something that looks more normal.

We can incorporate this in our pipeline using sklearn.

Why can’t we incorporate preprocessing targets in our column transformer?

We reduced MAPE from ~10% to ~8% with this trick!

• Does .fit() know we care about MAPE?

• No, it doesn’t. Why are we minimizing MSE (or something similar) if we care about MAPE??

• When minimizing MSE, the expensive houses will dominate because they have the biggest error.

Different scoring functions with cross_validate¶

• Let’s try using MSE instead of the default $R^2$ score.

If you are finding greater_is_better=False argument confusing, here is the documentation:

greater_is_better(bool), default=True Whether score_func is a score function (default), meaning high is good, or a loss function, meaning low is good. In the latter case, the scorer object will sign-flip the outcome of the score_func.

Since our custom scorer mape gives an error and not a score, I’m passing False to it and it’ll sign flip so that we can interpret bigger numbers as better performance.

Are we getting the same alpha with mape?

Using multiple metrics in GridSearchCV or RandomizedSearchCV¶

• We could use multiple metrics with GridSearchCV or RandomizedSearchCV.

• But if you do so, you need to set refit to the metric (string) for which the best_params_ will be found and used to build the best_estimator_ on the whole dataset.

What’s the test score?

Using regression metrics with scikit-learn¶

• In sklearn, you will notice that it has negative version of the metrics above (e.g., neg_mean_squared_error, neg_root_mean_squared_error).

• The reason for this is that scores return a value to maximize, the higher the better.

❓❓ Questions for you¶

iClicker Exercise 10.2¶

Select all of the following statements which are TRUE.

• (A) We can use still use precision and recall for regression problems but now we have other metrics we can use as well.

• (B) In sklearn for regression problems, using r2_score() and .score() (with default values) will produce the same results.

• (C) RMSE is always going to be non-negative.

• (D) MSE does not directly provide the information about whether the model is underpredicting or overpredicting.

• (E) We can pass multiple scoring metrics to GridSearchCV or RandomizedSearchCV for regression as well as classification problems.

What did we learn today?¶

• House prices dataset target is price, which is numeric -> regression rather than classification

• There are corresponding versions of all the tools we used:

  • DummyClassifier -> DummyRegressor

  • LogisticRegression -> Ridge

• Ridge hyperparameter alpha is like LogisticRegression hyperparameter C, but opposite meaning

• We’ll avoid LinearRegression in this course.

• Scoring metrics

• $R^2$ is the default .score(), it is unitless, 0 is bad, 1 is best

• MSE (mean squared error) is in units of target squared, hard to interpret; 0 is best

• RMSE (root mean squared error) is in the same units as the target; 0 is best

• MAPE (mean absolute percent error) is unitless; 0 is best, 1 is bad