The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping
Controlled traces show video models can improve final counting scores without faithfully recovering events, so agents handling video need timestamp-level checks, not answer-only evals.
The study profiles event counting across **2,190 controlled videos** with executable traces. Gemini 3.6 Flash reached 80% reliability for persistent transitions up to **12 events**, but had no reliable positive-count region for transient blinks.
For video agents, evaluate the reported event sequence against timestamps, not just the final count. In the high-count, high-frequency regime, only **0.2%** of counts were correct and **18.1%** of true events were recovered.
The study profiles event counting across **2,190 controlled videos** with executable traces. Gemini 3.6 Flash reached 80% reliability for persistent transitions up to **12 events**, but had no reliable positive-count region for transient blinks. For video agents, evaluate the reported event sequence against timestamps, not just the final count. In the high-count, high-frequency regime, only **0.2%** of counts were correct and **18.1%** of true events were recovered. More frames raised Bounce Ball accuracy from 19.6% to 29.3%, while full sequence agreement remained 3.7%. Prompt changes also brought limited gains, so higher aggregate accuracy may conceal poor event bookkeeping.
This makes temporal bookkeeping a distinct video-agent capability that aggregate counting accuracy can conceal. It reinforces the candidates’ case for inspecting trajectories and partial progress, but supplies a stronger executable oracle: compare every reported event with timestamped ground truth. More frames and prompt changes offer limited relief, narrowing the value of simple inference-time adjustments for transient or dense events.