Building Bigram Tables

From word counts to word transitions

Word frequency tells you what words appear. Bigrams tell you what follows what. “The cat” is far more common than “the refrigerator”, a bigram table captures that relationship. It’s the simplest form of a language model: given a word, what words tend to come next?

The cells below reuse the load_corpus and tokenize helpers from lessons 01 and 03. Every lesson page starts with a fresh Python session, so run this setup cell first:

import csv
import string

with open("slm-corpus.csv", newline="") as f:
    reader = csv.DictReader(f)
    texts = [row["text"] for row in reader]

def load_corpus(path):
    with open(path, newline="") as f:
        reader = csv.DictReader(f)
        return [row["text"] for row in reader]

def tokenize(text):
    text = text.lower()
    for char in string.punctuation:
        text = text.replace(char, " ")
    return text.split()

tokens = tokenize(" ".join(texts))

Key Concepts

What is a bigram?

A bigram is a pair of consecutive words. In the sentence “the cat sat on the mat”, the bigrams are:

(the, cat), (cat, sat), (sat, on), (on, the), (the, mat)

Each pair represents a transition from one word to the next. By counting all transitions in the corpus, you build a statistical model of word sequences.

Building the nested dict

The bigram table is a dict of dicts. The outer key is the current word; the inner dict maps follower-words to their counts:

def build_bigrams(tokens):
    bigrams = {}
    for i in range(len(tokens) - 1):
        current = tokens[i]
        next_word = tokens[i + 1]
        if current not in bigrams:
            bigrams[current] = {}
        bigrams[current][next_word] = bigrams[current].get(next_word, 0) + 1
    return bigrams

Walk through the token list with a sliding window of size 2. For each pair (tokens[i], tokens[i+1]), increment the count in bigrams[tokens[i]][tokens[i+1]].

Example walk-through

For tokens ["the", "cat", "sat", "the", "dog"]:

i=0: current="the", next="cat" → bigrams["the"]["cat"] = 1
i=1: current="cat", next="sat" → bigrams["cat"]["sat"] = 1
i=2: current="sat", next="the" → bigrams["sat"]["the"] = 1
i=3: current="the", next="dog" → bigrams["the"]["dog"] = 1

Result:

{
    "the": {"cat": 1, "dog": 1},
    "cat": {"sat": 1},
    "sat": {"the": 1},
}

Using defaultdict for cleaner code

from collections import defaultdict

def build_bigrams(tokens):
    bigrams = defaultdict(lambda: defaultdict(int))
    for i in range(len(tokens) - 1):
        bigrams[tokens[i]][tokens[i + 1]] += 1
    return dict(bigrams)

The lambda: defaultdict(int) creates a new inner dict automatically for each new word, so you never need to check if a key exists.

Inspecting the bigram table

Verify your table looks reasonable:

bigrams = build_bigrams(tokens)

# How many words have followers?
print(f"Words with followers: {len(bigrams)}")

# Show the top word's followers
top_word = max(bigrams, key=lambda w: sum(bigrams[w].values()))
print(f"Most connected word: '{top_word}'")
print(f"  Followers: {bigrams[top_word]}")

Sentence boundaries

When building bigrams from multiple sentences, the last word of one sentence and the first word of the next become a bigram. This is usually fine for a small model, the model doesn’t know about sentence structure anyway. But if you want cleaner results, you can add sentence boundary markers:

def build_bigrams(tokens, add_boundaries=True):
    bigrams = defaultdict(lambda: defaultdict(int))
    for i in range(len(tokens) - 1):
        bigrams[tokens[i]][tokens[i + 1]] += 1
    if add_boundaries:
        bigrams["<END>"] = defaultdict(int)
        bigrams[tokens[-1]]["<END>"] = bigrams[tokens[-1]].get("<END>", 0) + 1
    return dict(bigrams)

This lets you track which words commonly end sentences.

Try It

Build a bigram table from the corpus and answer:

1. How many unique bigram pairs exist?
2. What are the top 3 most common (word, follower) pairs?
3. Does “the” have more followers than any other word?

from collections import defaultdict

texts = load_corpus("slm-corpus.csv")
tokens = tokenize(" ".join(texts))
bigrams = build_bigrams(tokens)

total_pairs = sum(sum(f.values()) for f in bigrams.values())
print(f"Unique bigram pairs: {total_pairs}")

Key Takeaways

• A bigram is a pair of consecutive words, the simplest sequence model
• The bigram table is a nested dict: bigrams[word] = {follower: count}
• defaultdict(lambda: defaultdict(int)) simplifies nested counting
• Sentence boundaries can be tracked with special tokens like <END>

Practice Challenge

Write a function most_common_bigram(bigrams) that returns the single most frequent (word, follower) pair as a tuple. Then use it to find the most common bigram in the corpus.

def most_common_bigram(bigrams):
    best = (None, None)
    best_count = 0
    for word, followers in bigrams.items():
        for follower, count in followers.items():
            if count > best_count:
                best = (word, follower)
                best_count = count
    return best, best_count