Handling Missing Values

Why missing values matter

Almost every real dataset has missing values. If you ignore them, aggregations return NaN, visualizations break, and machine learning models fail. The first step in any analysis is understanding and addressing missing data.

import pandas as pd

# titanic.csv ships with the course — load it from the browser file system.
df = pd.read_csv("titanic.csv")

Detecting missing values

Check a single column:

print(df["Age"].isna().sum())   # 177 missing Age values

Check all columns at once:

print(df.isna().sum())

Output:

PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          687
Embarked         2
dtype: int64

See the percentage missing:

print((df.isna().sum() / len(df) * 100).round(1))

Output:

Cabin          77.1%
Age            19.9%
Embarked        0.2%
...

Cabin is 77% missing, too much to fill meaningfully. Age is 20%, worth attempting to fill. Embarked has only 2 missing, easy to handle.

Dropping missing values

Drop rows with any missing values:

df_clean = df.dropna()
print(df_clean.shape)   # (183, 12) — lost most rows

This is too aggressive for most datasets. You lose 708 of 891 rows.

Drop rows where all values are missing:

df_clean = df.dropna(how="all")

Drop rows missing values in specific columns:

df_clean = df.dropna(subset=["Age", "Embarked"])
print(df_clean.shape)   # (712, 12) — much better

Drop columns with too many missing values:

# Drop columns where more than 50% is missing
threshold = len(df) * 0.5
df_clean = df.dropna(thresh=threshold, axis=1)

Filling missing values

Fill with a constant:

df["Embarked"] = df["Embarked"].fillna("S")   # most common port

Fill with a statistic:

df["Age"] = df["Age"].fillna(df["Age"].median())

Fill forward or backward, useful for time series:

# Use the previous valid value to fill gaps
df["Price"] = df["Price"].ffill()

# Use the next valid value
df["Price"] = df["Price"].bfill()

Fill with different values per column:

fill_values = {"Age": df["Age"].median(), "Embarked": "S", "Cabin": "Unknown"}
df = df.fillna(fill_values)

Choosing a strategy

Common pitfalls

Filling before splitting train/test, this leaks information. Calculate fill values on training data only, then apply to both.

Dropping too aggressively, always check how many rows you lose. dropna() without arguments often removes far more than expected.

Forgetting to check, always run df.isna().sum() after filling to confirm no NaN values remain.

Try It

From the Titanic dataset:

1. Calculate the percentage of missing values for each column
2. Drop the Cabin column (too many missing values)
3. Fill Age with the median age
4. Fill Embarked with the most common value
5. Verify no missing values remain

import pandas as pd

# titanic.csv ships with the course — load it from the browser file system.
df = pd.read_csv("titanic.csv")

print((df.isna().sum() / len(df) * 100).round(1))

df = df.drop(columns=["Cabin"])
df["Age"] = df["Age"].fillna(df["Age"].median())
df["Embarked"] = df["Embarked"].fillna(df["Embarked"].mode()[0])

print(df.isna().sum())

Key Takeaways

• Always inspect missing values first with isna().sum() before deciding on a strategy
• dropna() is powerful but often too aggressive without subset or thresh
• fillna() with median or mode is the most common filling strategy
• Columns with > 50% missing values are usually better dropped than filled

Practice Challenge

Load the Titanic dataset and create a cleaned version: drop Cabin, fill Age with median, fill Embarked with mode. Then compare the survival rate before and after cleaning. Did cleaning change the overall survival rate? Why or why not?