Benchmark Integrity

Claim-level quotes can look verifiable while failing to support the full claim. Builders of retrieval agents should score citation coverage, verbatim accuracy, and entailment separately.

VLoc Bench isolates whether security agents can find vulnerable files across real repositories. Its low scores argue for testing localization and post-fix restraint separately from patch generation.

For edge vision, specialized training outweighed model size: 300M-parameter BioCLIP beat 2–8B VLMs, while field imagery degraded every model and open-set prompts produced invented species.

A calibrated DeBERTa detector scored well on HaluEval but transferred poorly to biomedical claims. Builders should tune hallucination checks to their domain instead of trusting a general evaluator.

Visual embedding initialization gave a small language model durable object-property knowledge that most standard benchmarks missed, showing how broad eval suites can hide narrow causal gains.

HuBERT and wav2vec 2.0 appear to encode word identity beyond local phonetics in later layers. The paper offers a cleaner probe for builders evaluating speech representations.

Semantically equivalent instructions can make VLM reward models score identical robot behavior differently. Agent evaluators should test paraphrase invariance, not assume model scale or reasoning fixes it.

Molecular benchmark accuracy can reflect retrieval of published values rather than prediction. Contamination audits should test digit-level recall and repeat runs across reasoning settings.

LLM decompilers can produce compilable C that passes shipped tests while changing behavior or hiding crashes. Validate reconstructed code against the original with shared fuzzed inputs.

A 52,988-request audit found black-box LLM judges too unstable for preregistered gates. Measure repeatability first, and pilot the judge before fixing evaluation thresholds.

Readable chain-of-thought is not a reliable map of which steps caused a correct answer. Treat step-level LLM critiques as partial signals, especially on already-correct traces.

SWE-Gate shows why green tests are an incomplete agent-eval signal: 221 of 644 functionally passing repairs still violated constraints derived from code review.

A causal framework separates deceptive-looking model output from the mechanism producing it, giving agent evaluators a stricter basis for claims about intent or agency.

User feedback helps models repair targeted faults, but LLM judges often miss those repairs. Agent evals should retain human feedback and avoid treating model preference as ground truth.

Mechanistic tests suggest two open LLM judges route local errors early and form ratings late, giving eval builders concrete layers and components to probe for judge failures.

BLOOM-WILT makes rare model behaviors easier to surface by adapting both the auditor’s strategy and target decoding. Its results also show that elicitation method can reverse safety rankings.

DeepMind’s panel shows why generative-media evals need task-specific human review: broad preferences can miss repeated artifacts, exact sizing, text errors, and brand consistency.

SWE-Prime finds that filtering coding-agent traces by process and segment quality can beat training on every resolved trajectory, reducing noisy imitation from redundant or risky steps.

A multi-GPU kernel benchmark shows coding agents can improve with tools and iteration, but still struggle with communication scheduling, partitioning, and transfer choices.

Correct answers can hide invalid computation. Trace Integrity proposes executable, replayable contracts for data agents and reports 45.8–59.1% correct-answer/invalid-trace rates across three SQL modes.

Einstein Arena suggests multi-agent environments can outperform fixed workflows when they expose verifiers, shared solutions, forums, and incentives. Its results also show why benchmark shortcuts need active testing.

ConceptGuard tests whether model unlearning blocks harmful uses of a concept while preserving benign ones. Current methods show weak contextual control and sharp forgetting-versus-utility trade-offs.

Per-problem self-improvement claims can arise from inference and evaluation noise. This audit argues every transition statistic needs a measured null from frozen baseline replicates.

TokEval links tokenizer properties to language, math, and code performance, offering cheaper screening signals before committing compute to pretraining sweeps.

A controlled study encoded authenticated internal-state evidence into unchanged answers, suggesting generated text could carry provenance signals, but not that current models reveal them naturally.

Tested compliance guards often ignored the governing rule and classified from scenario cues. Builders should counterfactually swap policies before trusting a detector as an audit control.

Static computer-use benchmarks can reward memorized action scripts rather than adaptation. Vary task state, verify every generated case, and calculate uncertainty across both actions and environments.

QuoteBench shows that shell-command scores can conceal failures introduced by serialization and reparsing. Agent evals should identify the execution path, not attribute every result to the model.

LittleLearner offers a controlled model and corpus for studying knowledge acquisition without unknown prior exposure. Its initial results separate better use of known material from new capability.

A task-agnostic influence measure tracks which examples steer pretraining toward final parameters without choosing a downstream eval. It reveals a literature-to-STEM shift across training.

Cursor’s AIUC-1 certification combines a controls audit with adversarial testing of live agents. For enterprise evaluation, it adds behavioral evidence beyond conventional data-security attestations.

P-Bench tests whether agents choose statistically valid methods, not merely execute code. Fisher-R1-14B improved single-trial success over DeepSeek-V4-Pro by 21% on average.

SABRE turns a Markdown test design into generated VLM stress tests, then filters and repairs candidates. It offers a repeatable pattern for refreshing evals as models improve.

Controlled traces show video models can improve final counting scores without faithfully recovering events, so agents handling video need timestamp-level checks, not answer-only evals.

SocietyBench tests forecasting in anonymized social timelines, exposing gaps that task-completion evals miss and showing agent frameworks did not improve the shared base model.

A live, pre-kickoff benchmark removes answer leakage by construction and finds six frontier models clustered near a bookmaker-favorite baseline, with no gain from majority voting.

This paper separates three test-time scaling regimes and argues that model evals must report the full inference protocol, compute accounting, uncertainty, and replay artifacts.

onepot-Bench 0 evaluates chemistry models with three lab-oriented tests, including private experimental data to reduce contamination risk. It is a useful eval design pattern, though results and reproducibility are absent here.

Nick Heiner argues that leaderboard gains can diverge from useful agent behavior through contamination, weak verifiers, reward hacking, and test conditions that users cannot inspect.

Cybersecurity agents need deterministic exploit oracles and tasks that reward finding distinct vulnerabilities, not self-reported wins or one crash. A capability ladder makes partial progress measurable.