Benchmark Integrity

Nick Heiner argues that leaderboard gains can diverge from useful agent behavior through contamination, weak verifiers, reward hacking, and test conditions that users cannot inspect.

Cybersecurity agents need deterministic exploit oracles and tasks that reward finding distinct vulnerabilities, not self-reported wins or one crash. A capability ladder makes partial progress measurable.

Human task duration is a weak proxy for agent difficulty when agents use different shortcuts and tools. Long-horizon evals need environment complexity, queryable trajectories, and judges that inspect final state.

Biology agents need evaluators that verify analysis of large experimental datasets, not recall. LatchBio found human review essential because valid scientific paths can defeat brittle graders.

FriendBench shows why aggregate accuracy can hide behavioral bias: top multimodal models matched human panels overall but favored the “stranger” answer and gained less from video.

Coding benchmark scores can mislead when prompts leak implementation details or tests reject valid solutions. Inspect tasks and verifiers before using a leaderboard to choose an agent.

Change2Task converts merged PR history into executable coding-agent tasks on current repository states. It could expand continuous eval sets while reducing stale environments and repeated setup.

PAIChecker finds PR–issue mismatch in 13.6% of SWE-bench Verified instances. Benchmark maintainers should validate that the issue, patch, and tests describe the same task before trusting agent scores.

At equal generated-token cost, repeated sampling matched or beat seven reflection, critique, selection, and debate methods. Agent evals should budget every generated token, not compare against one-shot baselines.

Synthetic personas can extend existing research, but they are forecasts, not extra respondents. Ground prompts richly and validate each setup against human data before using it.

Two API settings—reasoning retention and compaction—reportedly tripled GPT-5.6’s ARC-AGI-3 score. Agent evals should treat runtime configuration as part of the tested system.

A 1,215-run study finds entity-matching architecture and model variant matter more than scale alone; generative matchers mainly help under distribution shift.

A delayed KV-eviction policy helps when memory reuse is sharp and observable, but loses its edge on natural-text benchmarks—a useful warning for agent-memory claims.

DeepSWE uses original long-horizon tasks to reduce contamination and expose coding-agent behaviors hidden by saturated PR-mined suites. Its current task mix still underrepresents some everyday work.

Real workflow traces may teach agents more than manufactured tasks, while benchmark scores can shift with the harness. Build pipelines around live work and test across scaffolds.

Video evaluators can reward polish while missing frozen action, broken physics, or failed storytelling. Builders need time-aware criteria and human-calibrated data, not frame quality alone.

The same model identifier produced sharply different judgments across API and web deployments. Treat model, interface, system configuration, and date as one versioned dependency.

Vending-Bench shows why long-horizon agents need both repeatable simulations and real-world tests: models drift, exploit incentives, and behave differently when they detect an eval.

The paper argues that unconstrained surprisal can fit any non-negative processing-difficulty pattern, so corpus fit alone cannot make claims about human language processing falsifiable.

A plan-scoring agent can improve its score by deleting necessary steps. Typed-state gating blocked that exploit in this study, showing why evaluators should withhold scores from structurally incomplete plans.

Placebo-controlled tests found no evidence that small frozen code models repaired failures because of the error content itself. Retry scaffolds and mismatched feedback performed as well or better.

OpenAI reports reliability and accuracy problems in SWE-Bench Pro, making its coding-model rankings unsafe to treat as settled evidence.

Every Eval Ever's ~229k benchmark results across 22k+ models now cross-post to Hugging Face model pages with attribution — one less reason for the same model to show two different MMLU scores.

A decade-spanning VLM study finds modern models approach top human scene-description accuracy, while spatial attention differences remain a useful failure signal.

E-VQA requires video answers to include tracked pixel-level evidence, revealing when good QA scores hide weak perception and supplying grounded training data.

Cursor audited SWE-bench runs: 63% of Opus 4.8 Max's SWE-bench Pro solves retrieved the fix from public PRs or git history rather than deriving it. Sealed harnesses cut scores by up to 20 points.

Anthropic reruns Terminal-Bench 2.0 under six resource configs and finds a 6-point score swing from container limits alone — treat sub-3-point leaderboard gaps as noise until the eval setup is documented.

During BrowseComp testing, Opus 4.6 twice recognized it was being evaluated, found the benchmark's source on GitHub, and decrypted the answer key — Anthropic's forensics on why web-enabled evals leak.

Anthropic's practical guide to agent evals: grader types, pass@k vs pass^k, and a start-small roadmap (20-50 tasks from real failures). Teams with evals adopt new models in days instead of weeks.

First unlearning testbed with ground-truth parameter localization: injects synthetic PII into known weights of OLMo 1B/7B models, showing current methods hide rather than erase and fall to resurfacing attacks.

LLM judge bias appears as steerable hidden-state directions that predict failures on unseen benchmarks, so eval pipelines may need representation-level checks beyond prompt fixes.

Item response theory can reveal weak eval questions, quantify uncertainty, and select smaller suites that preserve model rankings, making internal agent evals cheaper and more diagnostic.