How to evaluate a slide-change detector without fooling yourself

Short summary

A maintainer of video-slide-extractor shares a rigorous evaluation protocol for slide-change detectors, emphasizing manual ground-truth labeling before tuning thresholds. The protocol tracks precision, recall, F1 with tolerance windows, and separately reports failure classes like duplicate captures and transition frames. The key lesson: publish labels, settings, and ugly frames — comparisons that show only winners are marketing, not benchmarks.

• •Manual ground-truth labeling must precede threshold tuning
• •Track precision/recall/F1 with tolerance windows plus failure-class rates separately
• •Reproducible evidence includes labels, config, detected timestamps, metrics, and versions

Generated with AI, which can make mistakes.