Word Frequency Counting

Counting words

Once you have tokens, the next step is counting how often each word appears. These frequency counts tell the language model which words are common (likely to appear anywhere) and which are rare (predictive when they do appear).

The cells below reuse the load_corpus and tokenize helpers from lessons 01 and 03. Every lesson page starts with a fresh Python session, so run this setup cell first:

import csv
import string

with open("slm-corpus.csv", newline="") as f:
    reader = csv.DictReader(f)
    texts = [row["text"] for row in reader]

def load_corpus(path):
    with open(path, newline="") as f:
        reader = csv.DictReader(f)
        return [row["text"] for row in reader]

def tokenize(text):
    text = text.lower()
    for char in string.punctuation:
        text = text.replace(char, " ")
    return text.split()

full_text = " ".join(texts)
tokens = tokenize(full_text)

Key Concepts

Building a frequency dict

The counting pattern uses a dict where each key is a word and the value is its count. The get() method handles the “first time we see this word” case:

def word_frequency(tokens):
    freq = {}
    for token in tokens:
        freq[token] = freq.get(token, 0) + 1
    return freq

tokens = ["the", "cat", "sat", "the", "dog", "sat", "the"]
freq = word_frequency(tokens)
print(freq)
# {'the': 3, 'cat': 1, 'sat': 2, 'dog': 1}

freq.get(token, 0) returns the current count if the word exists, or 0 if it’s the first time we’ve seen it. Adding 1 increments the count.

The defaultdict approach

An alternative uses collections.defaultdict, which automatically creates missing keys:

from collections import defaultdict

def word_frequency(tokens):
    freq = defaultdict(int)
    for token in tokens:
        freq[token] += 1
    return dict(freq)

Both approaches produce the same result. The defaultdict version is slightly cleaner but requires an import.

Vocabulary statistics

With a frequency dict, you can compute useful stats:

freq = word_frequency(tokenize(full_text))

total_tokens = sum(freq.values())
unique_words = len(freq)

print(f"Total tokens: {total_tokens:,}")
print(f"Unique words: {unique_words:,}")
print(f"Vocabulary richness: {unique_words / total_tokens:.4f}")

Vocabulary richness (unique / total) measures how diverse the text is. A value close to 1.0 means almost every word is unique; a value close to 0.0 means heavy repetition.

Most and least frequent words

Sort the frequency dict to find the extremes:

sorted_words = sorted(freq.items(), key=lambda item: item[1], reverse=True)

print("Top 10 words:")
for word, count in sorted_words[:10]:
    print(f"  {word}: {count}")

print("\nBottom 10 words:")
for word, count in sorted_words[-10:]:
    print(f"  {word}: {count}")

In most English text, “the”, “of”, “and”, “to”, and “a” dominate the top of the list. This follows Zipf’s law, the most frequent word appears roughly twice as often as the second, three times as often as the third, and so on.

Why frequency matters for generation

A language model uses frequency to weight predictions. If “the” appears 500 times and “platypus” appears 2 times, “the” should be chosen more often, but not always. The bigram model refines this by conditioning on the previous word, which is what makes generated text readable rather than just a stream of “the the the.”

Try It

Load the corpus, tokenize it, and build a frequency dict. Then answer:

1. How many total tokens are there?
2. What are the top 5 most frequent words?
3. What percentage of the vocabulary consists of words that appear only once?

texts = load_corpus("slm-corpus.csv")
full_text = " ".join(texts)
tokens = tokenize(full_text)
freq = word_frequency(tokens)

total = sum(freq.values())
hapax = sum(1 for w, c in freq.items() if c == 1)
print(f"Total tokens: {total}")
print(f"Words appearing once: {hapax} ({hapax/len(freq)*100:.1f}%)")

Key Takeaways

• dict.get(key, default) is the foundation of frequency counting
• Vocabulary richness (unique / total) measures text diversity
• Zipf’s law: a small number of words dominate the frequency distribution
• Frequency counts are the raw material for bigram probability tables

Practice Challenge

Write a function top_n(freq, n) that returns the top N most frequent words as a list of (word, count) tuples. Then use it to find the top 20 words in the corpus.

def top_n(freq, n):
    return sorted(freq.items(), key=lambda item: item[1], reverse=True)[:n]