Lecture 13: Feature engineering and feature selection¶

UBC 2025-26

Imports¶

Learning outcomes¶

From this lecture, students are expected to be able to:

• Explain what feature engineering is and the importance of feature engineering in building machine learning models.

• Carry out preliminary feature engineering on numeric data.

• Explain the general concept of feature selection.

• Discuss and compare different feature selection methods at a high level.

• Use sklearn’s implementation of model-based selection and recursive feature elimination (RFE)

Feature engineering: Motivation¶

❓❓ Questions for you¶

iClicker Exercise 14.1¶

Select the most accurate option below.

Suppose you are working on a machine learning project. If you have to prioritize one of the following in your project which of the following would it be?

• (A) The quality and size of the data

• (B) Most recent deep neural network model

• (C) Most recent optimization algorithm

Discussion question

• Suppose we want to predict whether a flight will arrive on time or be delayed. We have a dataset with the following information about flights:

  • Departure Time

  • Expected Duration of Flight (in minutes)

Upon analyzing the data, you notice a pattern: flights tend to be delayed more often during the evening rush hours. What feature could be valuable to add for this prediction task?

Garbage in, garbage out.¶

• Model building is interesting. But in your machine learning projects, you’ll be spending more than half of your time on data preparation, feature engineering, and transformations.

• The quality of the data is important. Your model is only as good as your data.

Activity: How can you measure quality of the data? (~3 mins)¶

• Write some attributes of good- and bad-quality data in this Google Document.

What is feature engineering?¶

• Better features: more flexibility, higher score, we can get by with simple and more interpretable models.

• If your features, i.e., representation is bad, whatever fancier model you build is not going to help.

Feature engineering is the process of transforming raw data into features that better represent the underlying problem to the predictive models, resulting in improved model accuracy on unseen data.

- Jason Brownlee

Some quotes on feature engineering¶

A quote by Pedro Domingos A Few Useful Things to Know About Machine Learning

... At the end of the day, some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used.

A quote by Andrew Ng, Machine Learning and AI via Brain simulations

Coming up with features is difficult, time-consuming, requires expert knowledge. "Applied machine learning" is basically feature engineering.

Better features usually help more than a better model.¶

• Good features would ideally:

  • capture most important aspects of the problem

  • allow learning with few examples

  • generalize to new scenarios.

• There is a trade-off between simple and expressive features:

  • With simple features overfitting risk is low, but scores might be low.

  • With complicated features scores can be high, but so is overfitting risk.

The best features may be dependent on the model you use.¶

• Examples:

  • For counting-based methods like decision trees separate relevant groups of variable values

    • Discretization makes sense

  • For distance-based methods like KNN, we want different class labels to be “far”.

    • Standardization

  • For regression-based methods like linear regression, we want targets to have a linear dependency on features.

Domain-specific transformations¶

In some domains there are natural transformations to do:

• Spectrograms (sound data)

• Wavelets (image data)

• Convolutions

Source

In this lecture, I’ll show you an example of feature engineering on text data.

Feature interactions and feature crosses¶

• A feature cross is a synthetic feature formed by multiplying or crossing two or more features.

• Example: Is the following dataset (XOR function) linearly separable?

• For XOR like problems, if we create a feature cross $x1x2$, the data becomes linearly separable.

Let’s look at an example with more data points.

The interaction feature has the biggest coefficient!

Feature crosses for one-hot encoded features¶

• You can think of feature crosses of one-hot-features as logical conjunctions

• Suppose you want to predict whether you will find parking or not based on two features:

  • area (possible categories: UBC campus and Rogers Arena)

  • time of the day (possible categories: 9am and 7pm)

• A feature cross in this case would create four new features:

  • UBC campus and 9am

  • UBC campus and 7pm

  • Rogers Arena and 9am

  • Rogers Arena and 7pm.

• The features UBC campus and 9am on their own are not that informative but the newly created feature UBC campus and 9am or Rogers Arena and 7pm would be quite informative.

• Coming up with the right combination of features requires some domain knowledge or careful examination of the data.

• There is no easy way to support feature crosses in sklearn.

Demo of feature engineering with numeric features¶

• Remember the California housing dataset we used earlier in the course?

• The prediction task is predicting median_house_value for a given property.

Suppose we decide to train ridge model on this dataset.

• What would happen if you train a model without applying any transformation on the categorical features ocean_proximity?

  • Error!! A linear model requires all features in a numeric form.

• What would happen if we apply OHE on ocean_proximity but we do not scale the features?

  • No syntax error. But the model results are likely to be poor.

• Do we need to apply any other transformations on this data?

In this section, we will look into some common ways to do feature engineering for numeric or categorical features.

We have total rooms and the number of households in the neighbourhood. How about creating rooms_per_household feature using this information?

Let’s start simple. Imagine that we only three features: longitude, latitude, and our newly created rooms_per_household feature.

• The scores are not great.

• Let’s look at the distribution of the longitude and latitude features.

• Suppose you are planning to build a linear model for housing price prediction.

• If we think longitude is a good feature for prediction, does it makes sense to use the floating point representation of this feature that’s given to us?

• Remember that linear models can capture only linear relationships.

• How about discretizing latitude and longitude features and putting them into buckets?

• This process of transforming numeric features into categorical features is called bucketing or binning.

• In sklearn you can do this using KBinsDiscretizer transformer.

• Let’s examine whether we get better results with binning.

The results are better with binned features. Let’s examine how do these binned features look like.

How about discretizing all three features?

• The results have improved further!!

• Let’s examine the coefficients

• Does it make sense to take feature crosses in this context?

• What information would they encode?

Check out apeendixA for a demo of feature engineering on text data.

Interim summary¶

• Feature engineering is finding the useful representation of the data that can help us effectively solve our problem.

• For numeric data, some common operations to create new features are:

  • Discretization: Converting continuous data into discrete buckets. For example, age could be split into categories like 0-20, 21-40, etc. to learn non-linear relationships

  • Interaction Features: Creating new features by combining two or more existing features.

• In the context of text data, if we want to go beyond bag-of-words and incorporate human knowledge in models, we carry out feature engineering. Some common features include:

  • ngram features

  • part-of-speech features

  • named entity features

  • emoticons in text

• These are usually extracted from pre-trained models using libraries such as spaCy.

• Now a lot of this, especially in the context of text and images, has moved to deep learning.

• Despite the rise of deep learning, many industries still rely on manual feature engineering.

• The best features are application-dependent.

• It’s hard to give general advice. But here are some guidelines.

  • Ask the domain experts.

  • Go through academic papers in the discipline.

  • Often have idea of right discretization/standardization/transformation.

  • If no domain expert, cross-validation will help.

• If you have lots of data, use deep learning methods.

The algorithms we used are very standard for Kagglers ... We spent most of our efforts in feature engineering...

- Xavier Conort, on winning the Flight Quest challenge on Kaggle

Break (5 min)¶

Feature selection: Introduction and motivation¶

• With so many ways to add new features, we can increase dimensionality of the data.

• More features means more complex models, which means increasing the chance of overfitting.

What is feature selection?¶

• Find the features (columns) $X$ that are important for predicting $y$, and remove the features that aren’t.

• Given $X = \begin{bmatrix}x_1 & x_2 & \dots & x_n\\ \\ \\ \end{bmatrix}$ and $y = \begin{bmatrix}\\ \\ \\ \end{bmatrix}$, find the columns $1 \leq j \leq n$ in $X$ that are important for predicting $y$.

Why feature selection?¶

• Interpretability: Models are more interpretable with fewer features. If you get the same performance with 10 features instead of 500 features, why not use the model with smaller number of features?

• Computation: Models fit/predict faster with fewer columns.

• Data collection: What type of new data should I collect? It may be cheaper to collect fewer columns.

• Fundamental tradeoff: Can I reduce overfitting by removing useless features?

Feature selection can often result in better performing (less overfit), easier to understand, and faster model.

How do we carry out feature selection?¶

• There are a number of ways.

• You could use domain knowledge to discard features.

• We are briefly going to look at two automatic feature selection methods from sklearn:

  • Model-based selection

  • Recursive feature elimination

  • Forward/backward selection

• Very related to looking at feature importances.

Model-based selection¶

• Use a supervised machine learning model to judge the importance of each feature.

• Keep only the most important once.

• Supervised machine learning model used for feature selection can be different that the one used as the final estimator.

• Use a model which has some way to calculate feature importances.

• To use model-based selection, we use SelectFromModel transformer.

• It selects features which have the feature importances greater than the provided threshold.

• Below I’m using RandomForestClassifier for feature selection with threahold “median” of feature importances.

• Approximately how many features will be selected?

Can we use KNN to select features?

No KNN won’t work since it does not report feature importances.

What about SVC?

Only with a linear kernel but not with RBF kernel

We can put the feature selection transformer in a pipeline.

Similar results with only 15 features instead of 30 features.

Recursive feature elimination (RFE)¶

• Build a series of models

• At each iteration, discard the least important feature according to the model.

• Computationally expensive

• Basic idea

  • fit model

  • find least important feature

  • remove

  • iterate.

RFE algorithm¶

1. Decide $k$, the number of features to select.

2. Assign importances to features, e.g. by fitting a model and looking at coef_ or feature_importances_.

3. Remove the least important feature.

4. Repeat steps 2-3 until only $k$ features are remaining.

Note that this is not the same as just removing all the less important features in one shot!

• How do we know what value to pass to n_features_to_select?

• Use RFECV which uses cross-validation to select number of features.

• Slow because there is cross validation within cross validation

• Not a big improvement in scores compared to all features on this toy case

(iClicker) Exercise 14.1¶

Select all of the following statements which are TRUE.

• (A) Simple association-based feature selection approaches do not take into account the interaction between features.

• (B) You can carry out feature selection using linear models by pruning the features which have very small weights (i.e., coefficients less than a threshold).

• (C) The order of features removed given by rfe.ranking_ is the same as the order of original feature importances given by the model.

Warnings about feature selection¶

• A feature is only relevant in the context of other features. Check out a

  • Adding/removing features can make features relevant/irrelevant.

• Confounding factors can make irrelevant features the most relevant.

• If features can be predicted from other other features, you cannot know which one to pick.

• Relevance for features does not have a causal relationship.

• Is feature selection completely hopeless?

  • It is messy but we still need to do it. So we try to do our best!

General advice on finding relevant features¶

• Try forward selection.

• Try other feature selection methods (e.g., RFE, simulated annealing, genetic algorithms)

• Talk to domain experts; they probably have an idea why certain features are relevant.

• Don’t be overconfident.

  • The methods we have seen probably do not discover the ground truth and how the world really works.

  • They simply tell you which features help in predicting $y_i$.

Relevant resources¶

• Genome-wide association study

• sklearn feature selection

• PyData: A Practical Guide to Dimensionality Reduction Techniques