Live Data Debrief: What Really Happened?

Introduction

Your system said there were five goals. The scribe with a notebook says there were three. Who's right, and why? Today is game-film day: you'll line up what your system claimed against what people actually saw, measure precision and recall on real events, and find the patterns behind the mistakes.

By the end of this lesson you'll be able to:

Treat the scribe's notes as ground truth and the system's rows as claims
Match detections to real goals one-to-one within a tolerance
Compute and explain precision and recall, and say what they cannot show with few events
Estimate the clock offset between the system and the scribe
Cut and watch clips around false alarms and misses
Write a fix table: what happened, why, what to change

Working principle

A debrief is where you learn

The match is over. The only thing that matters now is what the data can teach. A good debrief is calm, specific and fair: we are judging the system, not each other.

Truth from people, claims from the system

The scribe's notes are our ground truth: what really happened, as a person saw it. The system's goal rows are its claims. Comparing the two gives us the same scores as lesson 19:

Result	Name	Meaning
Real goal, announced	True positive	Good
Announced, but no real goal	False positive	False alarm
Real goal, not announced	False negative	Missed

Precision and recall in plain words

Precision: of the goals the system announced, how many were real? (Low precision = it cries wolf.)
Recall: of the real goals, how many did it catch? (Low recall = it misses things.)

In our pretend day: 2 of 4 announced were real (50% precision), and 2 of 3 real goals were caught (67% recall).

Matching in time

A detection and a real goal are "the same goal" if they are close in time. How close? That is the tolerance (3 seconds here). Too small and a late-by-a-second detection looks like two mistakes. Too large and a false alarm could steal a real goal's credit. And we match one to one, so one detection cannot be counted as right for two goals.

Clocks disagree

The scribe's stopwatch and the system's clock don't start at exactly the same moment. If every detection is about a second late, that is a timing offset, not a detection fault. The debrief reports the typical offset so you can separate "late" from "wrong".

Be honest about small numbers

Three real goals cannot support "67% recall" as a fact about the system. With so few examples, one more goal changes the number a lot. Say "it caught 2 of 3 goals in this match" and not "it is 67% accurate". To say more, you need more matches.

Watch the video of every mistake

Numbers hide the cause. Cutting a ten-second clip around each false alarm and miss takes a minute and tells you more than any chart. Keep the clip names to link them from the report.

Turn findings into changes

Every mistake becomes a row: what happened, why, what to change, who does it. Then re-run the same debrief on the next match to see whether it worked. Improvement you cannot measure is just hope.

What you'll need

Your raw recording, system log and the human scribe's record from lesson 39
Your cv_course environment

Build it

No real event data yet? Run make_demo_log.py to write a pretend match-day log, then practise on it. Replace it with your own event_log.csv from lesson 39 afterwards.

Cell 1: a pretend match day

"""Write a pretend match day log so the debrief can be practised before you have a real one."""
import csv
 
rows = [
    ["19:01:10", 0.0, "note", "kickoff"],
    ["19:03:30", 140.0, "goal", "north goal"],           # real goal, detected, 0.8 s late
    ["19:03:29", 139.2, "note", "GOAL red 7"],
    ["19:07:05", 355.0, "goal", "north goal"],           # false alarm: nothing happened (a bib in the net?)
    ["19:09:20", 490.0, "health_dark", "too dark"],
    ["19:11:00", 590.0, "note", "GOAL blue 3, shot from the edge"],   # real goal, missed (picture was dark)
    ["19:12:10", 660.0, "health_ok", "light back"],
    ["19:15:40", 870.0, "note", "camera knocked, recalibrated"],
    ["19:20:05", 1135.0, "goal", "north goal"],          # real goal, detected
    ["19:20:03", 1133.5, "note", "GOAL red 9"],
    ["19:20:09", 1139.0, "goal", "north goal"],          # duplicate: same goal counted twice
]
with open("event_log.csv", "w", newline="") as f:
    w = csv.writer(f); w.writerow(["wall_time", "match_seconds", "kind", "text"]); w.writerows(rows)
print("wrote event_log.csv")

Read the log first, like a detective. Which goals do you think were real? What happened at 490 s?

Cell 2: the debrief tool

The scribe writes a note starting with GOAL whenever a goal really happens. That is the truth. The system writes goal rows. The debrief matches the two.

"""Compare what the system said with what people saw. Input: event_log.csv from lesson 39.
In the log:  kind = goal  -> the system detected a goal.
             kind = note  -> a person wrote it. A note that starts with 'GOAL' means a goal REALLY happened (the scribe's truth)."""
import csv
import statistics
 
 
def load_log(path):
    with open(path, newline="") as f:
        return [{"wall_time": r["wall_time"], "t": float(r["match_seconds"]), "kind": r["kind"], "text": r["text"]}
                for r in csv.DictReader(f)]
 
 
def split_events(rows):
    detected = [r["t"] for r in rows if r["kind"] == "goal"]
    truth = [r["t"] for r in rows if r["kind"] == "note" and r["text"].strip().upper().startswith("GOAL")]
    return sorted(detected), sorted(truth)
 
 
def match_goals(detected, truth, tolerance=3.0):
    """One-to-one matching: each real goal can be claimed by at most one detection, the closest in time.
    Returns (pairs, false_alarms, missed). `tolerance` = how many seconds apart we still call it the same goal."""
    candidates = sorted(((abs(d - g), i, j) for i, d in enumerate(detected) for j, g in enumerate(truth)
                         if abs(d - g) <= tolerance))
    used_d, used_g, pairs = set(), set(), []
    for gap, i, j in candidates:
        if i not in used_d and j not in used_g:
            used_d.add(i); used_g.add(j)
            pairs.append((detected[i], truth[j]))
    false_alarms = [d for i, d in enumerate(detected) if i not in used_d]
    missed = [g for j, g in enumerate(truth) if j not in used_g]
    return pairs, false_alarms, missed
 
 
def scores(pairs, false_alarms, missed):
    tp, fp, fn = len(pairs), len(false_alarms), len(missed)
    precision = tp / (tp + fp) if tp + fp else None        # of the goals we announced, how many were real?
    recall = tp / (tp + fn) if tp + fn else None           # of the real goals, how many did we catch?
    return {"tp": tp, "fp": fp, "fn": fn, "precision": precision, "recall": recall}
 
 
def clock_offset(pairs):
    """Median (detected - truth) in seconds. A steady offset means the clocks disagree, not that detection is wrong."""
    return statistics.median(d - g for d, g in pairs) if pairs else None
 
 
def warnings_summary(rows):
    return {k: sum(1 for r in rows if r["kind"] == k) for k in sorted({r["kind"] for r in rows}) if k.startswith("health")}
 
 
def report(path, tolerance=3.0):
    rows = load_log(path)
    detected, truth = split_events(rows)
    pairs, fa, missed = match_goals(detected, truth, tolerance)
    s = scores(pairs, fa, missed)
    off = clock_offset(pairs)
    lines = [f"Goals the scribe saw: {len(truth)}   Goals the system announced: {len(detected)}",
             f"Correct: {s['tp']}   False alarms: {s['fp']}   Missed: {s['fn']}"]
    if s["precision"] is not None:
        lines.append(f"Precision {s['precision']:.0%} (of announced goals, how many were real)")
    if s["recall"] is not None:
        lines.append(f"Recall {s['recall']:.0%} (of real goals, how many were caught)")
    if off is not None:
        lines.append(f"Typical timing offset: {off:+.1f} s (system minus scribe)")
    for t in fa:
        lines.append(f"  FALSE ALARM at {t:.1f} s: look at the video here")
    for t in missed:
        lines.append(f"  MISSED goal at {t:.1f} s: look at the video here")
    h = warnings_summary(rows)
    if h:
        lines.append("Health warnings: " + ", ".join(f"{k} x{v}" for k, v in h.items()))
    return "\n".join(lines)
 
 
if __name__ == "__main__":
    import sys
    print(report(sys.argv[1] if len(sys.argv) > 1 else "event_log.csv"))

Cell 3: what it says about the pretend day

Goals the scribe saw: 3   Goals the system announced: 4
Correct: 2   False alarms: 2   Missed: 1
Precision 50% (of announced goals, how many were real)
Recall 67% (of real goals, how many were caught)
Typical timing offset: +1.2 s (system minus scribe)
  FALSE ALARM at 355.0 s: look at the video here
  FALSE ALARM at 1139.0 s: look at the video here
  MISSED goal at 590.0 s: look at the video here
Health warnings: health_dark x1, health_ok x1

Two goals were caught, one was missed (at 590 s, in the dark spell), one detection was a false alarm (355 s), and one was a duplicate (the same goal counted twice, 1139 s). Matching is one-to-one, so a detection can only be right once.

Cell 4: tests

from debrief import *
 
bad = 0
def check(name, ok, extra=""):
    global bad; bad += not ok; print("PASS" if ok else "FAIL", name, extra)
 
p, fa, m = match_goals([140.0, 355.0, 1135.0, 1139.0], [139.2, 590.0, 1133.5])
check("two real goals matched", len(p) == 2, p)
check("false alarms found: 355 and the duplicate 1139", fa == [355.0, 1139.0], fa)
check("missed goal found: 590", m == [590.0], m)
s = scores(p, fa, m)
check("precision 2/4, recall 2/3", s["precision"] == 0.5 and abs(s["recall"] - 2/3) < 1e-9, s)
check("offset about +1.15 s", abs(clock_offset(p) - 1.15) < 0.01, clock_offset(p))
 
# one detection must not claim two goals
p, fa, m = match_goals([10.0], [9.0, 11.0], tolerance=3)
check("one detection claims only one goal", len(p) == 1 and len(m) == 1)
# closest wins
p, fa, m = match_goals([10.0, 12.5], [11.0], tolerance=3)
check("the closest detection wins", p == [(10.0, 11.0)] or p == [(12.5, 11.0)], p)
check("the farther one is a false alarm", len(fa) == 1)
# nothing at all
check("no goals at all: scores are None, no crash", scores(*match_goals([], [])) ["precision"] is None)
# tolerance edge
check("exactly at tolerance still counts", len(match_goals([13.0], [10.0], tolerance=3.0)[0]) == 1)
check("just past tolerance does not", len(match_goals([13.1], [10.0], tolerance=3.0)[0]) == 0)
print("failures:", bad)

All eleven should print PASS. Notice the two edge tests: exactly at the tolerance counts, just past it does not. The tolerance is a decision, and it changes the scores.

Cell 5: watch the mistakes

Numbers tell you that it failed. The video tells you why. Cut a short clip around every mistake:

"""Cut a short clip around each mistake so the team can watch exactly what happened."""
import cv2
 
 
def cut_clip(video_path, centre_seconds, out_path, before=5.0, after=5.0, offset=0.0):
    """offset: seconds to ADD to the log time to get video time (see the timing offset in the report)."""
    cap = cv2.VideoCapture(video_path)
    fps = cap.get(cv2.CAP_PROP_FPS) or 30.0
    w, h = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)), int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    start = max(0, int((centre_seconds + offset - before) * fps))
    stop = int((centre_seconds + offset + after) * fps)
    cap.set(cv2.CAP_PROP_POS_FRAMES, start)
    out = cv2.VideoWriter(out_path, cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
    n = 0
    for _ in range(start, stop):
        ok, frame = cap.read()
        if not ok:
            break
        out.write(frame)
        n += 1
    cap.release()
    out.release()
    return n
 
 
def clips_for_mistakes(video_path, false_alarms, missed, offset=0.0):
    made = []
    for kind, times in (("false_alarm", false_alarms), ("missed", missed)):
        for t in times:
            name = f"{kind}_{t:.0f}s.mp4"
            made.append((name, cut_clip(video_path, t, name, offset=offset)))
    return made

Use the timing offset from the report if the system clock and the video clock disagree:

from debrief import load_log, split_events, match_goals, clock_offset
from review_clips import clips_for_mistakes
 
rows = load_log("event_log.csv")
detected, truth = split_events(rows)
pairs, false_alarms, missed = match_goals(detected, truth)
print(clips_for_mistakes("match.mp4", false_alarms, missed, offset=0.0))

Watch each clip as a team. For each mistake, write one line: what happened, why, what to change.

Cell 6: the fix table

What you saw	Likely cause	What to try	Where we learned it
Missed goal in a dark spell	Light changed	Better camera spot, exposure lock, health warning	lesson 36
False alarm, nothing visible	Wrong object detected as the ball	More data of that situation, a bib check	lesson 20, lesson 36
Same goal twice	Cooldown too short	Raise it, or require leaving the zone	lesson 31
Goal missed, ball hidden by the net	Rules too strict	Treat missing ball as neutral (already done), recheck	lesson 31
Everything is a few seconds late	Clocks disagree	Use the offset, or sync clocks at kickoff	lesson 39
Players off the pitch in the map	Camera moved	Recalibrate and log it	lesson 33

Questions: with so few goals, how much can you say about precision? (Three real goals cannot support a percent score. Say what you can and cannot claim.) What is the cost of a false alarm in your product? Would you trade some recall for precision?

If it goes wrong

Error	Cause	Fix
Everything is a false alarm and a miss	Timing offset larger than the tolerance	Look at the offset; correct for it, or enlarge the tolerance
`KeyError: match_seconds`	The CSV header differs	Use the lesson 39 `event_log.py` to write the file
Scribe goals are not found	Notes don't start with GOAL	Edit them, or change `split_events`
Precision 100% with zero goals	Nothing was announced; divide-by-zero guarded	Report 'no announced goals', not 100%
Clip is empty	Wrong offset or time outside the video	Print the start and stop frame numbers
Clips won't play	Codec not supported by the player	Try VLC, or change the fourcc
Scores keep changing as you edit tolerance	They should: it is a decision	Pick one in advance, and report it
Team blames the scribe	Misreading the purpose	Restate: we are judging the system

Try it

Add a plot of detections and truth on a timeline
Compute the cost of errors: set a price for a false alarm and a miss
Compare two matches and report the change

Check what you learnt

What is a false alarm here?
What is recall here?
Why match system events to real events one to one?
Recall is 67% from three goals. Why not trust that number much?
What is the timing offset?

Conclusion

A debrief turns a live event into lessons. Treat the human record as the truth, correct for clock offsets, match events one to one within a time window, and count hits, misses and false alarms. Small numbers are shaky, so look for patterns in the mistakes, not just the score.

What next?

Next: Player heatmap from real event data. You'll turn tracked positions into a player heatmap.