Overfitting & the Bad-Data Lab

Introduction

In Unit 3 you heard "garbage in, garbage out". Today you prove it with numbers. First you'll learn to spot a model that has memorised its training data. Then you'll deliberately damage a dataset (wrong labels, too few balls, blurry pictures), retrain, and measure exactly how much each kind of bad data costs.

By the end of this lesson you'll be able to:

Tell overfitting from underfitting by reading a learning curve
Use early stopping and dropout (and more data) to fight overfitting
Sabotage a dataset in three ways (wrong labels, class imbalance, low resolution) and measure the damage
Say which metric reveals each kind of damage
Write a short evidence-based statement: "this much bad data costs this much accuracy"
Link each result to a rule in their field brief (lesson 15)

Working principle

Overfitting and underfitting

	Underfitting	Good fit	Overfitting
Training accuracy	Low	High	Very high
Validation or test accuracy	Low	High	Much lower than training
Cause	Model too simple, too little training, bad features	Balanced	Model too big for the data, too many epochs, too few images
Picture	Learned nothing useful	Learned the pattern	Learned the noise and the particular images

The learning curve

Plot training accuracy and validation accuracy for each epoch. Early on both rise. If training keeps rising while validation flattens or falls, the model has begun memorising. The best moment to stop is where validation was highest.

Early on, both curves rise together: the model is genuinely learning. Epoch 0.

Fixes for overfitting

Fix	Idea
More (varied) data	Hardest to memorise. Usually the best fix.
Augmentation (lesson 13)	More variety from the same photos
Early stopping	Stop at the best validation epoch
Dropout	Randomly switch off some neurons during training, so none can be relied on alone
A smaller model	Fewer weights to memorise with

The three sabotages

Sabotage	What we do	Expected damage	Which metric shows it
A. Wrong labels	Flip a share of the training labels	Accuracy falls, and past about half it is just noise	Accuracy and F1 both fall
B. Class imbalance	Keep only a handful of ball examples	The model learns to say "no ball"	Recall collapses while accuracy looks fine
C. Low resolution	Shrink the images and enlarge them back	Small details disappear	Depends on object size: small balls suffer first

The golden rule of the lab

We sabotage only the training data. The test set stays clean. That's how we know the damage came from the training data and not from changing the exam.

What you'll need

Your lesson 18 script and crops (or the synthetic crops)
Your cv_course environment with TensorFlow and scikit-learn
Your Field Collection Brief from lesson 15

Build it

Install: pip install numpy opencv-python scikit-learn matplotlib. The first cell is long: it builds the data, defines the helpers and runs every experiment. Read it in sections.

Cell 1: baseline and sabotage

import cv2
import numpy as np
from sklearn.metrics import precision_score, recall_score
from sklearn.neural_network import MLPClassifier
 
rng = np.random.default_rng(1)
SIZE = 32          # small crops keep the lab fast on any laptop
 
 
def one_crop(has_ball):
    img = rng.normal(0, 10, (SIZE, SIZE, 3)) + np.array([45, 125, 50])
    if rng.random() < 0.3:
        cv2.line(img, (0, int(rng.integers(5, 27))), (SIZE, int(rng.integers(5, 27))), (230, 230, 230), 1)
    if rng.random() < 0.3:
        x = int(rng.integers(2, 24))
        cv2.rectangle(img, (x, 2), (x + 5, 25), tuple(int(c) for c in rng.integers(40, 255, 3)), -1)
    if has_ball:
        r = int(rng.integers(4, 8))
        cv2.circle(img, (int(rng.integers(r + 1, SIZE - r - 1)), int(rng.integers(r + 1, SIZE - r - 1))), r, (240, 240, 240), -1)
    return np.clip(img, 0, 255).astype(np.uint8)
 
 
def make_set(n_ball, n_other):
    imgs = [one_crop(True) for _ in range(n_ball)] + [one_crop(False) for _ in range(n_other)]
    y = np.array([1] * n_ball + [0] * n_other)
    order = rng.permutation(len(y))
    return np.array(imgs)[order], y[order]
 
 
def to_features(imgs, resolution=SIZE):
    """Flatten to numbers 0..1. Optionally destroy detail by shrinking and enlarging again."""
    out = []
    for im in imgs:
        if resolution < SIZE:
            im = cv2.resize(cv2.resize(im, (resolution, resolution)), (SIZE, SIZE), interpolation=cv2.INTER_NEAREST)
        out.append(im.ravel() / 255.0)
    return np.array(out)
 
 
def train_and_score(X_tr, y_tr, X_te, y_te, epochs=150, hidden=(64,)):
    model = MLPClassifier(hidden_layer_sizes=hidden, max_iter=epochs, random_state=0)
    model.fit(X_tr, y_tr)
    pred = model.predict(X_te)
    return {
        "train_acc": float(np.mean(model.predict(X_tr) == y_tr)),
        "test_acc": float(np.mean(pred == y_te)),
        "precision": float(precision_score(y_te, pred, zero_division=0)),
        "recall": float(recall_score(y_te, pred, zero_division=0)),
    }
 
 
train_imgs, y_train = make_set(400, 400)
test_imgs, y_test = make_set(150, 150)           # a clean, honest test set. We NEVER sabotage this one.
X_test = to_features(test_imgs)
 
 
def show(name, r):
    print(f"{name:34s} train {r['train_acc']:.0%}  test {r['test_acc']:.0%}  precision {r['precision']:.0%}  recall {r['recall']:.0%}")
 
 
results = {}
 
# 0. Baseline
results["baseline"] = train_and_score(to_features(train_imgs), y_train, X_test, y_test)
show("baseline", results["baseline"])
 
# A. Wrong labels: flip a share of the training labels
for share in (0.1, 0.3, 0.45):
    y_bad = y_train.copy()
    flip = rng.choice(len(y_bad), int(share * len(y_bad)), replace=False)
    y_bad[flip] = 1 - y_bad[flip]
    r = train_and_score(to_features(train_imgs), y_bad, X_test, y_test)
    results[f"A wrong labels {share:.0%}"] = r
    show(f"A. wrong labels, {share:.0%} flipped", r)
 
# B. Class imbalance: keep only a few ball examples
for keep in (100, 30, 8):
    ball_idx = np.where(y_train == 1)[0][:keep]
    other_idx = np.where(y_train == 0)[0]
    idx = np.concatenate([ball_idx, other_idx])
    r = train_and_score(to_features(train_imgs[idx]), y_train[idx], X_test, y_test)
    results[f"B only {keep} balls"] = r
    show(f"B. only {keep} ball examples", r)
 
# C. Low resolution: shrink the training images, then enlarge them again
for res in (16, 8, 4):
    r = train_and_score(to_features(train_imgs, res), y_train, to_features(test_imgs, res), y_test)
    results[f"C resolution {res}"] = r
    show(f"C. resolution {res} x {res}", r)
 
# D. Overfitting: a huge model on a tiny dataset
tiny = slice(0, 40)
r = train_and_score(to_features(train_imgs[tiny]), y_train[tiny], X_test, y_test, epochs=300, hidden=(256, 256))
results["D tiny data, big model"] = r
show("D. 40 images, big model", r)

Record the results in a table like this one. Your numbers will differ: the table shows one typical run.

Experiment	Train acc	Test acc	Precision	Recall
Baseline	98%	90%	92%	87%
A. 10% wrong labels	93%	90%	92%	89%
A. 30% wrong labels	80%	80%	86%	71%
A. 45% wrong labels	96%	51%	51%	60%
B. 100 ball examples (of 400)	100%	86%	98%	74%
B. 30 ball examples	100%	74%	100%	47%
B. 8 ball examples	99%	53%	100%	6%
C. 16 by 16 resolution	95%	89%	90%	88%
C. 8 by 8 resolution	92%	86%	87%	83%
C. 4 by 4 resolution	85%	83%	84%	82%
D. 40 images, big model	100%	83%	88%	77%

What to notice:

A: a few wrong labels (10%) barely matter, 30% hurts, and 45% is close to a coin toss. The model needs some honest signal, and surprisingly tolerates a little noise.
B: with 8 ball examples, training accuracy is 99% and test recall is 6%. The model gave up on balls. Look at the recall column. Accuracy alone hides this.
C: with these big, bright synthetic balls, shrinking the image costs little. A real ball in a pitch-side frame may be only a few pixels wide, so expect more damage on real field crops.
D: 100% train versus 83% test is overfitting: a big model that memorised 40 images.

Questions: which sabotage was the most dangerous? Which could you have spotted without a test set? What would you have to do on the field to prevent B?

Cell 2: watch overfitting happen

Run this after cell 1 in the same session.

# Learning curves: watch overfitting happen epoch by epoch (needs the helper functions from the lab above)
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
 
X_small = to_features(train_imgs[:40])
y_small = y_train[:40]
model = MLPClassifier(hidden_layer_sizes=(256, 256), random_state=0)
train_curve, test_curve = [], []
for epoch in range(120):
    model.partial_fit(X_small, y_small, classes=[0, 1])
    train_curve.append(model.score(X_small, y_small))
    test_curve.append(model.score(X_test, y_test))
 
best = int(np.argmax(test_curve))
print(f"best test accuracy {test_curve[best]:.0%} at epoch {best + 1}; final test accuracy {test_curve[-1]:.0%}; final train accuracy {train_curve[-1]:.0%}")
plt.plot(train_curve, label="train")
plt.plot(test_curve, label="test (honest)")
plt.axvline(best, color="grey", linestyle="--", label="early stopping point")
plt.xlabel("epoch")
plt.ylabel("accuracy")
plt.legend()
plt.savefig("overfit_curve.png", dpi=100)

You should see the training line climb to 100% while the test line stops improving early, at about epoch 30 in my run. Stopping there is early stopping.

In Keras (a short preview, for lesson 18's model)

These two additions apply to the lesson 18 model. They were written for Keras 3 and not run here.

from tensorflow import keras
from tensorflow.keras import layers
 
# 1. Dropout: add it before the dense layers in the lesson 18 model
#    layers.Flatten(),
#    layers.Dropout(0.3),
#    layers.Dense(64, activation="relu"),
 
# 2. Early stopping: pass it to model.fit
early_stop = keras.callbacks.EarlyStopping(
    monitor="val_loss",           # watch the honest number
    patience=3,                   # stop after 3 epochs with no improvement
    restore_best_weights=True,    # go back to the best epoch
)
# history = model.fit(train_ds, validation_data=val_ds, epochs=50, callbacks=[early_stop])

Bad-data lab report: template

Each pair submits one page.

Section	What to write
1. Baseline	Test accuracy, precision and recall with clean data
2. Experiments	The table of results for A, B and C
3. Biggest damage	Which sabotage hurt most? By how many percentage points on which metric?
4. The hidden one	Which damage would a single accuracy number have hidden?
5. Three field rules	Three rules for our field brief that come from today. For each: the experiment and the number that supports it.
6. A limit of this lab	One reason these results might not match real football footage

If it goes wrong

Error	Cause	Fix
`ConvergenceWarning` printed	Not enough iterations for scikit-learn's solver	Safe to ignore here, or raise `epochs`
Different numbers from mine	Random generators and library versions	Expected. Compare the pattern, not the exact digits.
Test accuracy changes when you "improve" the test set	They changed the test data	Keep the test set fixed. Only sabotage training.
Recall is 0% with high accuracy	The model never predicts the small class	That's finding B. Add ball examples.
You conclude "low resolution doesn't matter"	The synthetic balls are big	Shrink the synthetic ball radius, or repeat on real crops
`partial_fit` error about classes	Missing `classes=[0, 1]`	Pass it on the first call
Slow run	Big hidden layers	Cut `epochs` or the training size

Try it

Combine two sabotages (imbalance plus wrong labels) and compare with each alone
Find the smallest number of ball examples that gives recall of at least 70%
Try the fix for B: duplicate the few ball images (or use class_weight) and see whether recall recovers

Check what you learnt

How do you know a model is overfitting?
Why keep the test set clean in the sabotage lab?
Which metric shows the damage from class imbalance?
Which of these fights overfitting?
With very few ball examples, what happens?

Conclusion

Overfitting shows on the learning curve as training and validation pulling apart; early stopping, dropout, augmentation and more data fight it. The bad-data lab turns "garbage in, garbage out" into numbers: wrong labels, imbalance and low resolution each cost a measurable amount, and each one becomes a rule in your field brief.

What next?

Next: Transfer learning: stand on shoulders. You'll borrow a network that has already learned from a million photos.