Lecture 17: Introduction to natural language processing¶

UBC 2025-26

Imports¶

Learning objectives¶

By the end of this lecture, you will be able to:

• Differentiate between common similarity metrics, including Euclidean distance, dot product similarity, and cosine similarity.

• Explain what Natural Language Processing (NLP) is and identify common applications.

• Describe the idea behind topic modeling and apply it to uncover hidden structure and meaning in text data.

• Distinguish between topic modeling and clustering in terms of goals and outputs.

• Describe key text representations, from bag-of-words to word and sentence embeddings, and explain why richer representations are needed.

• Use word embeddings to explore similarities and analogies between words.

• Explain the difference between word embeddings and sentence embeddings and when each is useful.

• Interpret similarities between words, sentences, or documents using cosine similarity.

• Explain what a language model is and how it predicts or generates text.

• Summarize the main ideas behind large language models (LLMs) and their role in modern NLP.

• Recognize potential biases and ethical concerns when using pretrained language models.

What is Natural Language Processing (NLP)?¶

• What should a search engine return when asked the following question?

Natural Language Processing (NLP) is a branch of machine learning focused on enabling computers to understand, interpret, and generate human language.

Everyday NLP applications

And of course, general-purpose conversational chatbots

• ChatGPT (by OpenAI)

• Claude (by Anthropic)

• Gemini (by Google DeepMind; formerly Bard)

• Copilot (by Microsoft, integrated into Office and Windows)

• Perplexity AI (search-augmented conversational assistant)

• ...

Why is NLP hard?

• Language is complex and subtle.

• Language is ambiguous at different levels.

• Language understanding involves common-sense knowledge and real-world reasoning.

• All the problems related to representation and reasoning in artificial intelligence arise in this domain.

Exmaple: Lexical ambiguity

Example: Referential ambiguity

Example: Ambiguous news headlines

PROSTITUTES APPEAL TO POPE

• appeal to means make a serious or urgent request or be attractive or interesting?

KICKING BABY CONSIDERED TO BE HEALTHY

• kicking is used as an adjective or a verb?

MILK DRINKERS ARE TURNING TO POWDER

• turning means becoming or take up?

NLP in industry

NLP powers a wide range of real-world applications across industries by enabling machines to understand and generate human language.

• customer service (chatbots, voice assistants)

• marketing (sentiment and trend analysis)

• finance (document summarization, fraud detection)

• healthcare (clinical text analysis, medical coding)

• law (contract review, information extraction)

• ...

Overall goal of this lecture

• Give you a quick introduction to you of this important field in artificial intelligence which extensively uses machine learning.

• This is a huge field. We’ll focus on the following three topics which are likely to be useful for you.

  • Topic modeling

  • Word and text representations

  • Quick introduction to LLMs

Topic modeling¶

Why topic modeling?

Topic modeling introduction activity (~5 mins)

• Consider the following documents.

Discuss the following questions with your neighbour

1. Suppose you are asked to cluster these documents manually. How many clusters would you identify?

2. What are the prominent words in each cluster?

3. Are there documents which are a mixture of multiple clusters?

Last week, we learned about clustering. Let’s try K-Means clustering on this data with BOW representation:

Do you see any problem here?

Topic modeling motivation¶

• K-Means clustering gives each document a hard assignment; each document belongs to exactly one cluster.

• But in reality, many documents are a mixture of topics.

  • A news article might talk about sports and politics.

  • A research paper might cover machine learning and linguistics.

• In general, humans are pretty good at reading and understanding a document and answering questions such as

  • What is it about?

  • Which documents is it related to?

• Imagine that you are given a large collection of documents on a variety of topics.

A corpus of news articles

Example: A corpus of food magazines

A corpus of scientific articles

(Credit: Dave Blei’s presentation)

• It would take years to read all documents and organize and categorize them so that they are easy to search.

• You need an automated way

  • to get an idea of what’s going on in the data or

  • to pull documents related to a certain topic

• Topic modeling gives you an ability to summarize the major themes in a large collection of documents (corpus).

  • Example: The major themes in a collection of news articles could be

    • politics

    • entertainment

    • sports

    • technology

    • ...

• Topic modeling is a great EDA tool to get a sense of what’s going on in a large corpus.

• Some examples

  • If you want to pull documents related to a particular lawsuit.

  • You want to examine people’s sentiment towards a particular candidate and/or political party and so you want to pull tweets or Facebook posts related to election.

How do you do topic modeling?

• A common tool to solve such problems is unsupervised ML methods.

• Given the hyperparameter $K$, the goal of topic modeling is to describe a set of documents using $K$ “topics”.

• In unsupervised setting, the input of topic modeling is

  • A large collection of documents

  • A value for the hyperparameter $K$ (e.g., $K = 3$)

• and the output is

  1. Topic-words association

     • For each topic, what words describe that topic?

  2. Document-topics association

     • For each document, what topics are expressed by the document?

Topic modeling: Example

• Topic-words association

  • For each topic, what words describe that topic?

  • A topic is a mixture of words.

Topic modeling: Example

• Document-topics association

  • For each document, what topics are expressed by the document?

  • A document is a mixture of topics.

Topic modeling: Input and output

• Input

  • A large collection of documents

  • A value for the hyperparameter $K$ (e.g., $K = 3$)

• Output

  • For each topic, what words describe that topic?

  • For each document, what topics are expressed by the document?

Topic modeling: Some applications

• Topic modeling is a great EDA tool to get a sense of what’s going on in a large corpus.

• Some examples

  • If you want to pull documents related to a particular lawsuit.

  • You want to examine people’s sentiment towards a particular candidate and/or political party and so you want to pull tweets or Facebook posts related to election.

Topic modeling examples

Topic modeling: Input

Credit: David Blei’s presentation

Topic modeling: output

(Credit: David Blei’s presentation)

Topic modeling: output with interpretation

• Assigning labels is a human thing.

(Credit: David Blei’s presentation)

LDA topics in Yale Law Journal

(Credit: David Blei’s paper)

LDA topics in social media¶

(Credit: Health topics in social media)

In this lecture, I will demonstrate how to perform topic modeling using the Latent Dirichlet Allocation model implemented in sklearn. We won’t delve into the inner workings of the model, as it falls outside the scope of this course. Instead, our objective is to understand how to apply it to your specific problems and comprehend the model’s input and output.

Topic modeling toy example¶

Let’s work with a toy example.

• Input to the LDA topic model is bag-of-words representation of text.

• Let’s create bag-of-words representation of “text” column.

Let’s try to create a topic model with sklearn’s LatentDirichletAllocation.

• Once we have a fitted model we can get the word-topic association and document-topic association

• Word-topic association

  • lda.components_ gives us the weights associated with each word for each topic. In other words, it tells us which word is important for which topic.

• Document-topic association

  • Calling transform on the data gives us document-topic association.

Let’s look at the words with highest weights for each topic more systematically.

• Here is how we can interpret the topics

  • Topic 0 $\rightarrow$ ML modeling

  • Topic 1 $\rightarrow$ fruit and nutrition

  • Topic 2 $\rightarrow$ fashion

Let’s look at distribution of topics for a document

This document is made up of

• ~83% topic 2

• ~9% topic 0

• ~8% topic 1.

Topic modeling pipeline¶

• Above we worked with toy data. In the real world, we usually need to preprocess the data before passing it to LDA.

• Here are typical steps if you want to carry out topic modeling on real-world data.

  • Preprocess your corpus.

  • Train LDA.

  • Interpret your topics.

Data

Preprocessing the corpus

• Preprocessing is crucial!

• Tokenization, converting text to lower case

• Removing punctuation and stopwords

• Discarding words with length < threshold or word frequency < threshold

• Possibly lemmatization: Consider the lemmas instead of inflected forms.

• Depending upon your application, restrict to specific part of speech;

  • For example, only consider nouns, verbs, and adjectives

Check out AppendixC for basic preprocessing in NLP.

We’ll use spaCy for preprocessing. Check out available token attributes here.

Even though you have spacy installed, you will need to install the following pre-trained spacy model in the course environment.

python -m spacy download en_core_web_md

Check out some recent topic modeling tools

• Topic2Vec

• BERTopic

Text representations and word embeddings¶

Motivation¶

Do large language models such as ChatGPT understand your questions to some extent and provide useful responses?

What would it take for a machine to “understand” language?
A first step is to find a way to represent text, numbers that capture meaning.

So far, we have seen bag-of-words (BoW) representation using CountVectorizer in sklearn.

Let’s quickly recall what that looks like.

What are some limitations of Bag-of-Words?

• Sparse, high-dimensional vectors

• Only capture word frequency

• Ignore word order and context

• Do not put similar words (e.g., happy, joyful) close together

BoW represents documents, but it treats each word as an independent token, there’s no notion of word meaning or relationships between words.

In this part of the lecture, we are going to go one step back and talk about word representations. Why care about word representations?

• Words are the basic building blocks of language; the smallest units that carry meaning.

• To truly understand a document, a model must first understand the meaning of the words in it.

• If we can represent each word in a way that captures its meaning, then we can combine these representations to understand larger pieces of text (sentences, paragraphs, documents).

• In other words, to represent text meaningfully, we must start with word meaning.

This brings us to a key question: How can we represent the meaning of individual words using numbers?

Distributional hypothesis¶

Activity: Context and word meaning

Pair up with a neighbor and try to guess the meanings of the following made-up words: flibbertigibbet and groak.

1. The plot twist was totally unexpected, making it a flibbertigibbet experience.

2. Despite its groak special effects, the storyline captivated my attention till the end.

3. I found the character development rather groak, failing to evoke empathy.

4. The cinematography is flibbertigibbet, showcasing breathtaking landscapes.

5. Sadly, the movie’s potential was overshadowed by its groak pacing.

Discussion:

• How did you infer the meanings of these words?

• Which words or phrases helped you?

• In the previous activity, you guessed the meaning of flibbertigibbet and groak based on surrounding words.

• That’s exactly what machines do when they learn from text.

• This idea is called distributional hypothesis.

You shall know a word by the company it keeps.

Firth, 1957

In other words, words that appear in similar contexts tend to have similar meanings.

Word embeddings: The idea¶

Building on this idea, modern NLP systems learn word embeddings, dense vector representations of words that capture these contextual relationships.

(Attribution: Jurafsky and Martin 3rd edition)

Example:

“The plot twist was totally unexpected, making it a flibbertigibbet experience.”

“The plot twist was totally unexpected, making it a delightful experience.”

The goal: words like flibbertigibbet and delightful should be close in the embedding space.

Word embeddings are built on the distributional hypothesis. They mathematically capture the idea that context defines meaning.

Measuring similarity between vectors¶

• To create a vector space where similar words are close together, we need some metric to measure distances between representation.

• We have used the Euclidean distance before for numeric features.

• For sparse features, the most commonly used metrics are Dot product and Cosine distance.

• Let’s look at an example.

Euclidean distance

Dot product similarity:

Cosine similarity: normalized version of dot product.

Where,

• The L2 norm of $vec1$ is the magnitude of $vec1$

  $$\left\lVert vec1\right\rVert_2 = \sqrt{\sum_i vec1_i^2}$$

  (4)

• The L2 norm of $vec2$ is the magnitude of $vec2$

  $$\left\lVert vec2\right\rVert_2 = \sqrt{\sum_i vec2_i^2}$$

  (5)

Discussion question¶

Suppose you are recommending items based on similarity between items. Given a query vector “Query” in the picture below and the three item vectors, determine the ranking for the three similarity measures below:

• Similarity based on Euclidean distance

• similarity based on dot product

• Cosine similarity

• Adapted from here.

Using pre-trained word embeddings¶

Creating these representations on your own is resource intensive. So people typically use “pretrained” embeddings. A number of pre-trained word embeddings are available. The most popular ones are:

• word2vec

  • trained on several corpora using the word2vec algorithm

• wikipedia2vec

  • pretrained embeddings for 12 languages

• GloVe

  • trained using the GloVe algorithm

  • published by Stanford University

• fastText pre-trained embeddings for 294 languages

  • trained using the fastText algorithm

  • published by Facebook

How to use pretrained embeddings

Let’s try Google News pre-trained embeddings.

• You can download pre-trained embeddings from their original source.

• Gensim provides an api to conveniently load them. You need to install the gensim package in the course environment.

conda install conda-forge::gensim

If you get errors when you import gensim, try to install the following in the course environment.
pip install --upgrade gensim scipy

For the demonstration purpose, we’ll use word2vec-google-news-300. This model used to be available through gensim.downloader, but due to licensing and size (1.5+ GB), it’s no longer hosted on the default Gensim data server.

So I have manually download them from this source and then loaded them locally:

• google_news_vectors above has 300 dimensional word vectors for 3,000,000 unique words/phrases from Google news.

What can we do with these word vectors?

• Let’s examine word vector for the word UBC.

It’s a short and a dense (we do not see any zeros) vector!

Finding similar words

• Given word $w$, search in the vector space for the word closest to $w$ as measured by cosine similarity.

If you want to extract all documents containing words similar to information, you could use this information.

Google News embeddings also support multi-word phrases.

Why “erik” and “costco” show up for “british_columbia”? Note that word embeddings capture how words are used, not what they mean.

Finding similarity scores between words

We are getting reasonable word similarity scores!!

Success of word2vec

• This analogy example often comes up when people talk about word2vec, which was used by the authors of this method.

• MAN : KING :: WOMAN : ?

  • What is the word that is similar to WOMAN in the same sense as KING is similar to MAN?

• Perform a simple algebraic operations with the vector representation of words. $\vec{X} = \vec{\text{KING}} − \vec{\text{MAN}} + \vec{\text{WOMAN}}$

• Search in the vector space for the word closest to $\vec{X}$ measured by cosine distance.

(Credit: Mikolov et al. 2013)

So you can imagine these models being useful in many meaning-related tasks.

Examples of semantic and syntactic relationships

(Credit: Mikolov 2013)

Implicit biases and stereotypes in word embeddings

Embeddings reflect biases in the data they are trained on.

• Embeddings reflect gender stereotypes present in broader society.

• They may also amplify these stereotypes because of their widespread usage.

• See the paper Man is to Computer Programmer as Woman is to ....

Most of the modern embeddings are de-biased for some obvious biases. For example, we won’t see this with glove_wiki_vectors. We changed “computer_programmer” to “programmer” because “computer_programmer” is not in the vocabulary of glove_wiki_vectors.

Other popular methods to get embeddings

fastText

• NLP library by Facebook research

• Includes an algorithm which is an extension to word2vec

• Helps deal with unknown words elegantly

• Breaks words into several n-gram subwords

• Example: trigram sub-words for berry are ber, err, rry

• Embedding(berry) = embedding(ber) + embedding(err) + embedding(rry)

(Optional) GloVe: Global Vectors for Word Representation

• Starts with the co-occurrence matrix

  • Co-occurrence can be interpreted as an indicator of semantic proximity of words

• Takes advantage of global count statistics

• Predicts co-occurrence ratios

• Loss based on word frequency

Beyond words: sentence embeddings¶

• Word embeddings capture individual word meanings.

• But what about sentences or paragraphs?

• Modern deep learning models represent whole texts using sentence embeddings, where the meaning of a sentence is captured in a single vector. This is what you used in HW6.

(Optional) Quick peek: Getting an embedding from an LLM API

Under the hood, all large language models (LLMs) represent words and sentences as vectors, also called embeddings. These embeddings capture the meaning of text based on how words appear in context.

In this high-dimensional space, words or sentences with similar meanings lie close together, while unrelated ideas are farther apart.

LLMs like ChatGPT use these embeddings as the foundation for understanding meaning, context, and intent -- powering applications such as chatbots, search, summarization, and many others.

The quality of these embeddings largely determines how well a model understands your question and how good its responses are.

Assuming that you’ve your OPENAI_API_KEY setup in the course environment, the following code will return embeddings associated with these sentences.

Key takeaways

• BoW $\rightarrow$ simple frequency-based text representation

• Word embeddings $\rightarrow$ capture meaning and similarity

• Sentence embeddings $\rightarrow$ meaning at the sentence level

• LLMs $\rightarrow$ context-aware, dynamic embeddings at scale

Summary¶

• NLP is a big and very active field.

• We broadly explored three topics:

  • Topic modeling

  • Word and text representations embeddings using pretrained models

  • Introduction to large language models

Here are some resources if you want to get into NLP.

• Check out this CPSC course on NLP.

• The first resource I would recommend is the following book by Jurafsky and Martin. It’s very approachable and fun. And the current edition is available online.

  • Speech and Language Processing by Dan Jurafsky and James H. Martin

• There is a course taught at Stanford called “From languages to Information” by one of the co-authors of the above book, and it might be a good introduction to NLP for you. Most of the course material and videos are available for free.

• If you are into deep learning, you may refer to this course. Again, all lecture videos are available on youtube.

• If you want to look at current advancements in the field, you’ll find all NLP related publications here.