Automation Stress Test: A Fire Drill for Your System

Introduction

Your system works in the classroom. Will it work at dusk, on a shaky camera, with a player in a yellow bib that looks like the ball? A school runs a fire drill so the real emergency holds no surprises. Today you'll run one for your system: make conditions worse step by step, find where it breaks, and decide what to fix first.

By the end of this lesson you'll be able to:

Design a stress test that changes one thing at a time
Measure detection rate, position error and goal correctness, and say what each hides
Name the breaking point of each stress
Distinguish a missed goal from a false alarm and say which matters more for their product
Explain boundary flicker and one fix
Write a failure policy: stop, warn, or carry on

Working principle

A system is only as good as its worst day

You have built a detector, a trigger, an API and a light. They work on your clean test clip. A match day has low sun, lens smudges, a cheap camera and a yellow bib. A stress test finds out which of those breaks you, before match day.

How to stress test

Start with a clean clip where you know the truth (the ball's real positions, or at least the real goals).
Make it worse in one way at a time, in steps from mild to terrible.
At each step, measure: how often is the ball found, how far off is it, and are the goals right?
Write down the first step where it breaks. That is the breaking point.

One change at a time is the key: otherwise you cannot tell which one broke it.

Three numbers, not one

Number	What it tells you	What it hides
Detection rate	How often something was found	Whether it was the right thing
Position error	How close the found thing is to the truth	Nothing, but needs ground truth
Goals correct	Whether the whole system still works	Why it failed

In our test a yellow bib gave 100% detection and 354 px error. The detection rate alone says "perfect".

Two kinds of failure

	Meaning	Cost
Missed goal	A real goal was not detected	The fans are unhappy
False alarm	A goal was announced that did not happen	Everyone loses trust

Which is worse depends on the product. A goal alarm that cries wolf will be switched off. A highlight reel that misses one goal is forgiven. lesson 19 gave you the words (false positives and false negatives); now you see them in action.

Failures come from different places

Layer	Example	Fix
Picture	Dark, bright, blurry, noisy	Better camera position and settings
Detector	Colour clash with a bib	Different colour, a trained model (lesson 24), shape checks
Trigger	Ball on the zone line	Hysteresis, more frames
Hardware	Cable unplugged	Fall back (lesson 35)

Know when you can't trust the picture

A cheap check can warn you. The health check (Cell 6) looks at brightness and sharpness every second. If the picture is bad, show a warning light rather than a wrong score. A system that says "I'm not sure right now" is more trustworthy than one that is confidently wrong.

Decide the failure policy

For each failure, choose in advance: stop, warn, or carry on. A dark picture: warn. A lost ball for three seconds: carry on. Ten seconds with no ball at all: warn. Write the policy in a table and put it in the product notes (lesson 47, lesson 50).

What you'll need

Your goal detector and trigger from lessons 31 to 35
A match clip with a few known goals
Your cv_course environment

Build it

No video needed: scene.py draws a small synthetic match where the ball's true position is known every frame. You can swap in your own clip and hand-labelled positions later.

Cell 1: the scene (ground truth)

"""A small synthetic match so the stress test needs no video. The ball position is known exactly (ground truth)."""
import cv2
import numpy as np
 
W, H = 640, 360
GOAL = {"x": 270, "y": 10, "w": 100, "h": 50}
 
 
def ball_path(n=300, rest_y=40):
    """Ball travels in from the left, scores in the goal around frame 120, rests, then a second shot at 220."""
    pts = []
    for i in range(n):
        if i < 120:                       # run up to the goal
            x, y = 40 + i * 2.5, 300 - i * 2.0
        elif i < 160:                     # in the net, resting
            x, y = 320, rest_y
        elif i < 220:                     # back out
            f = (i - 160) / 60
            x, y = 320 + 200 * f, rest_y + (280 - rest_y) * f
        elif i < 260:                     # second shot
            f = (i - 220) / 40
            x, y = 520 - 200 * f, 280 - 230 * f
        else:
            x, y = 320, rest_y
        pts.append((x, y))
    return pts
 
 
def render(i, pos, rng):
    img = np.full((H, W, 3), (60, 130, 60), np.uint8)             # green pitch (BGR)
    cv2.rectangle(img, (GOAL["x"], GOAL["y"]), (GOAL["x"] + GOAL["w"], GOAL["y"] + GOAL["h"]), (235, 235, 235), 2)
    for k in range(4):                                             # blue and red players wandering
        px = 100 + 120 * k + 30 * np.sin(i / 20 + k)
        py = 180 + 40 * np.cos(i / 25 + k)
        cv2.circle(img, (int(px), int(py)), 14, (200, 60, 30) if k % 2 else (30, 40, 200), -1)
    cv2.circle(img, (int(pos[0]), int(pos[1])), 9, (0, 200, 255), -1)      # orange-yellow ball
    return img
 
 
def make_clip(n=300, seed=0, rest_y=40):
    rng = np.random.default_rng(seed)
    path = ball_path(n, rest_y)
    return [render(i, p, rng) for i, p in enumerate(path)], path

Cell 2: the thing we will stress (lesson 9's colour finder) and the lesson 31 trigger

detector.py is below. Copy events.py and goal_detector.py from lesson 31 into the same folder.

"""The thing we are going to stress: a simple colour ball finder (lesson 9)."""
import cv2
import numpy as np
 
LOWER, UPPER = np.array([15, 120, 150]), np.array([35, 255, 255])      # orange-yellow in HSV
 
 
def find_ball(frame, min_area=20):
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
    mask = cv2.inRange(hsv, LOWER, UPPER)
    mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, np.ones((3, 3), np.uint8))
    cnts, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    cnts = [c for c in cnts if cv2.contourArea(c) >= min_area]
    if not cnts:
        return None
    c = max(cnts, key=cv2.contourArea)
    (x, y), r = cv2.minEnclosingCircle(c)
    return {"x": float(x), "y": float(y)}

Cell 3: the ways things go wrong

"""Ways the real world makes video worse. Each takes a frame and a strength, and returns a worse frame."""
import cv2
import numpy as np
 
_rng = np.random.default_rng(7)
 
 
def darker(f, k):           # k from 0 (none) to 1 (black)
    return (f * (1 - k)).astype(np.uint8)
 
 
def brighter(f, k):         # k from 0 to 1 (washed out)
    return (f + (255 - f) * k).astype(np.uint8)
 
 
def blur(f, k):             # k = kernel size in pixels
    k = int(k) | 1
    return cv2.GaussianBlur(f, (k, k), 0)
 
 
def noise(f, k):            # k = standard deviation of the noise
    n = _rng.normal(0, k, f.shape)
    return np.clip(f.astype(float) + n, 0, 255).astype(np.uint8)
 
 
def jpeg(f, k):             # k = JPEG quality, lower is worse
    ok, buf = cv2.imencode(".jpg", f, [cv2.IMWRITE_JPEG_QUALITY, int(k)])
    return cv2.imdecode(buf, cv2.IMREAD_COLOR)
 
 
def shrink(f, k):           # k = scale factor (0.5 = camera twice as far away)
    h, w = f.shape[:2]
    small = cv2.resize(f, (int(w * k), int(h * k)))
    return cv2.resize(small, (w, h))                 # back to the same size, but blurrier
 
 
def tint(f, k):             # k = how much blue cast (cheap camera / artificial light)
    out = f.astype(float)
    out[:, :, 0] += k
    out[:, :, 2] -= k * 0.5
    return np.clip(out, 0, 255).astype(np.uint8)
 
 
def bib(f, k):              # k = radius of a ball-coloured object (a yellow bib or cone) on the pitch
    out = f.copy()
    if k > 0:
        cv2.circle(out, (560, 300), int(k), (0, 200, 255), -1)
    return out
 
 
DEGRADATIONS = {
    "darker": (darker, [0.0, 0.3, 0.5, 0.7, 0.85]),
    "brighter": (brighter, [0.0, 0.3, 0.5, 0.7]),
    "blur": (blur, [1, 5, 11, 21, 41]),
    "noise": (noise, [0, 10, 25, 50, 80]),
    "jpeg": (jpeg, [95, 50, 20, 8, 3]),
    "shrink": (shrink, [1.0, 0.5, 0.3, 0.2, 0.1]),
    "tint": (tint, [0, 20, 40, 70]),
    "bib": (bib, [0, 6, 12, 20, 30]),
}

Cell 4: run the stress test

"""Run the detector and the goal trigger on a clip under rising stress. Print a table; save it as CSV."""
import csv
 
import numpy as np
 
from degrade import DEGRADATIONS
from detector import find_ball
from events import EventBus
from goal_detector import GoalDetector
from scene import GOAL, make_clip
 
FPS = 30
TRUE_GOALS = 2
 
 
def run(frames, truth, fn=None, strength=None):
    found, errors, goals = 0, [], []
    bus = EventBus()
    bus.on("goal", lambda zone, t: goals.append(round(t, 1)))
    det = GoalDetector(GOAL, bus, confirm_frames=3, cooldown=3.0)
    for i, (f, (tx, ty)) in enumerate(zip(frames, truth)):
        g = fn(f, strength) if fn else f
        ball = find_ball(g)
        if ball:
            found += 1
            errors.append(float(np.hypot(ball["x"] - tx, ball["y"] - ty)))
        det.update(ball, i / FPS)
    return {"detect_rate": found / len(frames),
            "median_error_px": float(np.median(errors)) if errors else float("nan"),
            "goals_found": len(goals)}
 
 
def main():
    frames, truth = make_clip()
    rows = []
    base = run(frames, truth)
    print(f"baseline: detected {base['detect_rate']:.0%}, median error {base['median_error_px']:.1f} px, goals {base['goals_found']} (true {TRUE_GOALS})\n")
    print(f"{'stress':<10}{'strength':>9}{'detected':>10}{'error px':>10}{'goals':>7}  verdict")
    for name, (fn, levels) in DEGRADATIONS.items():
        for s in levels:
            r = run(frames, truth, fn, s)
            if r["goals_found"] == TRUE_GOALS and r["detect_rate"] > 0.8:
                verdict = "ok"
            elif r["goals_found"] < TRUE_GOALS:
                verdict = "MISSED goal"
            elif r["goals_found"] > TRUE_GOALS:
                verdict = "FALSE alarm"
            else:
                verdict = "weak (few detections)"
            rows.append([name, s, round(r["detect_rate"], 3), round(r["median_error_px"], 1), r["goals_found"], verdict])
            print(f"{name:<10}{s:>9}{r['detect_rate']:>10.0%}{r['median_error_px']:>10.1f}{r['goals_found']:>7}  {verdict}")
    with open("stress_results.csv", "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["stress", "strength", "detect_rate", "median_error_px", "goals_found", "verdict"])
        w.writerows(rows)
    print("\nsaved stress_results.csv")
    print("\nFirst strength where each stress breaks the system:")
    seen = {}
    for name, strength, _, _, _, verdict in rows:
        if verdict != "ok" and name not in seen:
            seen[name] = strength
    for name in DEGRADATIONS:
        print(f"  {name:<9} {'never (in the range tried)' if name not in seen else 'breaks at ' + str(seen[name])}")
 
 
if __name__ == "__main__":
    main()

This is what it printed for us (yours should match, apart from tiny differences):

baseline: detected 100%, median error 0.0 px, goals 2 (true 2)
 
stress     strength  detected  error px  goals  verdict
darker          0.0      100%       0.0      2  ok
darker          0.3      100%       0.0      2  ok
darker          0.5        0%       nan      0  MISSED goal
darker          0.7        0%       nan      0  MISSED goal
darker         0.85        0%       nan      0  MISSED goal
brighter        0.0      100%       0.0      2  ok
brighter        0.3      100%       0.0      2  ok
brighter        0.5      100%       0.0      2  ok
brighter        0.7        0%       nan      0  MISSED goal
blur              1      100%       0.0      2  ok
blur              5      100%       0.0      2  ok
blur             11      100%       0.0      2  ok
blur             21      100%       0.0      2  ok
blur             41      100%       0.2      2  ok
noise             0      100%       0.0      2  ok
noise            10      100%       0.0      2  ok
noise            25      100%       0.0      2  ok
noise            50       99%       1.5      2  ok
noise            80        1%       3.9      0  MISSED goal
jpeg             95      100%       0.0      2  ok
jpeg             50      100%       0.3      2  ok
jpeg             20      100%       0.3      2  ok
jpeg              8      100%       0.4      2  ok
jpeg              3      100%       0.7      2  ok
shrink          1.0      100%       0.0      2  ok
shrink          0.5      100%       0.2      2  ok
shrink          0.3      100%       0.3      2  ok
shrink          0.2      100%       0.7      2  ok
shrink          0.1      100%       1.4      2  ok
tint              0      100%       0.0      2  ok
tint             20      100%       0.0      2  ok
tint             40      100%       0.0      2  ok
tint             70      100%       0.0      2  ok
bib               0      100%       0.0      2  ok
bib               6      100%       0.0      2  ok
bib              12      100%     353.8      0  MISSED goal
bib              20      100%     353.8      0  MISSED goal
bib              30      100%     353.8      0  MISSED goal
 
saved stress_results.csv
 
First strength where each stress breaks the system:
  darker    breaks at 0.5
  brighter  breaks at 0.7
  blur      never (in the range tried)
  noise     breaks at 80
  jpeg      never (in the range tried)
  shrink    never (in the range tried)
  tint      never (in the range tried)
  bib       breaks at 12

Read it as a map of where your system is safe. Darkness at half strength kills it; blur, JPEG damage, tint and a far camera do not (the colour finder doesn't care about sharpness); a yellow bib the same colour as the ball destroys it even though the detection rate says 100%. The detection rate alone fooled us: the finder was confidently finding the wrong thing. The error column and the goal count tell the truth.

Cell 5: a surprise in the trigger

rest_y 40 | clean: 2 | jpeg 50: 2
rest_y 60 | clean: 2 | jpeg 50: 0

Same detector, same video quality. The only change: the ball rests exactly on the goal line (y = 60). A tiny JPEG wobble puts it inside, then outside, then inside, and the "three in a row" rule never fires. It is a boundary flicker. The fix is called hysteresis: use a slightly smaller zone to enter than to leave, or count "inside in 3 of the last 5 frames". Try it.

from stress import run
from scene import make_clip
from degrade import jpeg
 
for rest_y in (40, 60):                       # 60 is exactly ON the bottom edge of the goal zone
    frames, truth = make_clip(rest_y=rest_y)
    print("rest_y", rest_y, "| clean:", run(frames, truth)["goals_found"], "| jpeg 50:", run(frames, truth, jpeg, 50)["goals_found"])

Cell 6: a health check you can run live

Instead of discovering bad video after the match, warn during it.

"""A cheap self-check: is the picture good enough to trust? Run it every second or so, show a warning light."""
import cv2
import numpy as np
 
 
def picture_health(frame, dark_below=70, bright_above=200, blurry_below=20.0, noisy_above=3000.0):
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    brightness = float(gray.mean())
    sharpness = float(cv2.Laplacian(gray, cv2.CV_64F).var())          # low = blurry
    problems = []
    if brightness < dark_below:
        problems.append("too dark")
    if brightness > bright_above:
        problems.append("too bright")
    if sharpness < blurry_below:
        problems.append("blurry")
    if sharpness > noisy_above:                                       # noise makes edges everywhere, so this number shoots UP
        problems.append("noisy")
    return {"brightness": round(brightness, 1), "sharpness": round(sharpness, 1), "problems": problems}
 
 
class BallWatchdog:
    """Warn if the ball has not been seen for `limit` frames in a row (covered, lost, or camera problem)."""
 
    def __init__(self, limit=90):
        self.limit, self.missed = limit, 0
 
    def update(self, ball):
        self.missed = 0 if ball else self.missed + 1
        return self.missed >= self.limit

Try it on degraded frames. It catches darkness, brightness, blur and noise. Noise makes the sharpness score shoot up, not down, because every pixel looks like an edge; that is why there are two thresholds. It cannot catch the bib (nothing is wrong with the picture itself). The watchdog catches a ball that has been missing for too long.

Questions: what is the first test you would add for your own venue? What would you do about the bib problem (hint: lesson 10, lesson 26)? Which failures should stop the automation, and which should only warn?

If it goes wrong

Error	Cause	Fix
Everything is 'ok' in your table	Strengths too mild, or the finder is robust to that stress	Push to extreme values and add a new stress
Results change between runs	Random noise with a different seed	Fix the seed (we do) and report the setting
Brightness test looks worse than reality	Cheap brightening function, not a real camera	Say it is a model of the problem, and test with real footage
Goal count wrong although detection looks fine	The error is in the trigger, not the finder	Print the ball track near the zone and look
`ModuleNotFoundError: events`	lesson 31 files are missing	Copy `events.py` and `goal_detector.py` here
Health check says 'noisy' on a clean video	Thresholds are for this scene	Measure your own clean clip and set the thresholds from it
Slow	300 frames times 30 tests	Use 150 frames, or fewer strengths

Try it

Add a new stress: rain streaks, a moving shadow or a smudge
Add hysteresis to the trigger and show the edge case is fixed
Use your own clip with hand-labelled positions

Check what you learnt

Why change only one condition at a time?
What is a breaking point?
The detector found a 'ball' 100% of the time with a yellow bib on the pitch, yet failed. Why?
Missed goal or false alarm: which is worse?
Why does adding noise raise the sharpness score?

Conclusion

A stress test is a fire drill. Change one condition at a time (light, blur, noise, look-alike objects), find each breaking point, count both misses and false alarms, and decide which failure matters more for your product. Then you know what to fix, and what to warn users about, before the live event.

What next?

Next: LLMs + CV: a Claude match commentator. Unit 8 starts: adding Claude, a language model, as your match commentator.