Normalizing Bigram Counts, PyDA Course

From counts to probabilities

Raw counts tell you that “the” → “cat” appeared 15 times and “the” → “dog” appeared 5 times. But to sample the next word, you need probabilities: “cat” should be chosen 75% of the time and “dog” 25%. Normalizing converts counts into a distribution where all followers sum to 1.0.

The cells below reuse the load_corpus, tokenize, and build_bigrams helpers from lessons 01–05. Every lesson page starts with a fresh Python session, so run this setup cell first:

python

import csv
import string
from collections import defaultdict

with open("slm-corpus.csv", newline="") as f:
    reader = csv.DictReader(f)
    texts = [row["text"] for row in reader]

def load_corpus(path):
    with open(path, newline="") as f:
        reader = csv.DictReader(f)
        return [row["text"] for row in reader]

def tokenize(text):
    text = text.lower()
    for char in string.punctuation:
        text = text.replace(char, " ")
    return text.split()

def build_bigrams(tokens):
    bigrams = defaultdict(lambda: defaultdict(int))
    for i in range(len(tokens) - 1):
        bigrams[tokens[i]][tokens[i + 1]] += 1
    return dict(bigrams)

Key Concepts

Normalizing with a loop

For each word, sum its follower counts, then divide each count by that total:

python

def normalize_bigrams(bigrams):
    normalized = {}
    for word, followers in bigrams.items():
        total = sum(followers.values())
        normalized[word] = {w: c / total for w, c in followers.items()}
    return normalized

Now normalized["the"]["cat"] returns a float between 0 and 1, the probability that “cat” follows “the.”

Example

python

raw_bigrams = {"the": {"cat": 15, "dog": 5, "bird": 10}}
norm = normalize_bigrams(raw_bigrams)

print(norm["the"])
# {'cat': 0.5, 'dog': 0.1667, 'bird': 0.3333}

The probabilities sum to 1.0:

python

print(sum(norm["the"].values()))  # 1.0

Why normalization matters for sampling

random.choices() needs weights that represent relative likelihood. If you pass raw counts (15, 5, 10), it works, but having proper probabilities (0.5, 0.167, 0.333) makes the model portable and comparable across different corpus sizes.

python

import random

followers = list(norm["the"].keys())
weights = list(norm["the"].values())
next_word = random.choices(followers, weights=weights, k=1)[0]
print(f"Next word: {next_word}")

Handling edge cases

Some words have no followers (the last word in the corpus, or words that only appear at the end of a sentence). The bigram table won’t have entries for them:

python

def normalize_bigrams(bigrams):
    normalized = {}
    for word, followers in bigrams.items():
        if not followers:
            continue  # skip words with no followers
        total = sum(followers.values())
        normalized[word] = {w: c / total for w, c in followers.items()}
    return normalized

Skipping empty entries prevents division-by-zero errors.

A complete pipeline

Here’s how normalization fits into the full pipeline:

python

texts = load_corpus("slm-corpus.csv")
tokens = tokenize(" ".join(texts))
bigrams = build_bigrams(tokens)
model = normalize_bigrams(bigrams)

# Check a sample
print(f"Words in model: {len(model)}")
print(f"Followers of 'the': {list(model.get('the', {}).keys())[:5]}")

Saving the model

You might want to save the normalized bigram table for reuse. Since it’s a nested dict of floats, json works well:

python

import json

with open("bigram_model.json", "w") as f:
    json.dump(model, f)

# Reload later
with open("bigram_model.json") as f:
    model = json.load(f)

Try It

Build and normalize the bigram table, then verify:

Do the probabilities for “the” sum to 1.0?
How many words have zero followers?
What’s the most likely word to follow “the”?

python

model = normalize_bigrams(bigrams)
the_followers = model.get("the", {})
top_follower = max(the_followers, key=the_followers.get)
print(f"Most likely after 'the': '{top_follower}' ({the_followers[top_follower]:.3f})")

Key Takeaways

Normalization converts raw counts to probabilities that sum to 1.0 per word
random.choices() uses these probabilities as weights for weighted sampling
Skip words with no followers to avoid division by zero
Save normalized models with json.dump() for reuse across scripts

Practice Challenge

Write a function bigram_stats(model) that prints for each word: the word, number of followers, and the most probable next word. Limit output to the top 10 words by total follower count.

python

def bigram_stats(model, top_n=10):
    words = sorted(model, key=lambda w: sum(model[w].values()), reverse=True)
    for word in words[:top_n]:
        followers = model[word]
        total = sum(followers.values())
        best = max(followers, key=followers.get)
        print(f"'{word}': {len(followers)} followers, best=''{best}'' ({followers[best]:.3f})")

Key Concepts

Normalizing with a loop

Example

Why normalization matters for sampling

Handling edge cases

A complete pipeline

Saving the model

Try It

Key Takeaways

Practice Challenge

🚀 Keep building