Evals

Nick Heiner argues that leaderboard gains can diverge from useful agent behavior through contamination, weak verifiers, reward hacking, and test conditions that users cannot inspect.

Cybersecurity agents need deterministic exploit oracles and tasks that reward finding distinct vulnerabilities, not self-reported wins or one crash. A capability ladder makes partial progress measurable.

Human task duration is a weak proxy for agent difficulty when agents use different shortcuts and tools. Long-horizon evals need environment complexity, queryable trajectories, and judges that inspect final state.

Biology agents need evaluators that verify analysis of large experimental datasets, not recall. LatchBio found human review essential because valid scientific paths can defeat brittle graders.

For subjective agent output, replace vague requests for quality with decomposed brand constraints, then reserve human preference data for style and creativity that resist deterministic checks.

AgentHPOBench tests whether agents can learn from experiment history, not merely produce code. Its results expose weaknesses in sustained refinement and log diagnosis across sequential ML runs.

Coding benchmark scores can mislead when prompts leak implementation details or tests reject valid solutions. Inspect tasks and verifiers before using a leaderboard to choose an agent.

For agent work without deterministic answers, mine production traces into tasks, build controllable simulators, and use scaled judges to derive rubrics while watching for reward hacking.

DeepSeek V4 Flash’s updated weights replace the preview behind the existing model ID, raising its reported Terminal-Bench score from 56.9 to 82.7 without code changes.

Boundary replaces blanket code review with stable architecture rules, agent-generated tests, transcript inspection, execution traces, and human triage—a concrete harness pattern, not a license to skip verification.

OSReward finds that VLM judges often approve failed computer-use runs. Its benchmark and open reward models offer a more grounded way to evaluate trajectories without paying frontier-model costs.

Change2Task converts merged PR history into executable coding-agent tasks on current repository states. It could expand continuous eval sets while reducing stale environments and repeated setup.

PAIChecker finds PR–issue mismatch in 13.6% of SWE-bench Verified instances. Benchmark maintainers should validate that the issue, patch, and tests describe the same task before trusting agent scores.

At equal generated-token cost, repeated sampling matched or beat seven reflection, critique, selection, and debate methods. Agent evals should budget every generated token, not compare against one-shot baselines.

Socher’s automated-research design combines prior knowledge, measurement data, simulation, physical experiments, and agent orchestration, with early demonstrations in training and CUDA optimization.

Synthetic personas can extend existing research, but they are forecasts, not extra respondents. Ground prompts richly and validate each setup against human data before using it.

Nubank uses simulated multi-turn traces to evaluate agent changes before production, shortening release cycles while checking simulation results against real data and human review.

MindForge trains a 27B coding model on source-free program synthesis, raising ProgramBench pass rate from 37.98% to 49.51% and improving across seven unseen benchmarks.

Two API settings—reasoning retention and compaction—reportedly tripled GPT-5.6’s ARC-AGI-3 score. Agent evals should treat runtime configuration as part of the tested system.

Cognition measures coding-agent deployments by delivery outcomes, not sessions or tokens: engineering capacity, shorter timelines, and accepted PRs tied to customer work.

Ramp pairs strict request scoping with agents across the delivery pipeline. Its intake agent reportedly saves about 20% of scoping time, while humans retain final judgment.

Desktop-Delta Bench tests whether computer-use models can verify what changed after an action, exposing a reliability gap hidden by end-task and single-frame scores.

Frontier models iteratively produced trapped-ion compiler code that beat hand-built baselines in some architectures, showing a specification-to-evaluation loop on a hard domain.

ERUnderstand shows vision-language models can recover common ERD elements but often miss rarer schema constructs, so image-to-schema agent workflows still need structural validation.

DeepSWE uses original long-horizon tasks to reduce contamination and expose coding-agent behaviors hidden by saturated PR-mined suites. Its current task mix still underrepresents some everyday work.

Real workflow traces may teach agents more than manufactured tasks, while benchmark scores can shift with the harness. Build pipelines around live work and test across scaffolds.

SonderMind turns clinician-reviewed failures into release-gating evals, keeping mental-health guardrails modular and testing false positives, false negatives, category, and timing.

Reconstructing production traces as replayable environments lets teams compare agent stacks under fixed conditions and gate releases on task outcomes, cost, latency, and retries.

Video evaluators can reward polish while missing frozen action, broken physics, or failed storytelling. Builders need time-aware criteria and human-calibrated data, not frame quality alone.

Uber’s image-editing agent uses routing, iterative QA, golden-set gates, and production feedback to avoid costly edits, hallucinated food, and quality regressions.

Production agent behavior should be improved through a loop of tool design, small eval sets, trace review, and calibrated judges—not prompt tweaks based on one bad run.

Fixed-rubric judges miss failures that emerge across long, variable agent trajectories. Arize argues for adding agent-based analysis while retaining deterministic and LLM-judge evals.

Procedural skills can make an agent fail tasks it previously solved. Evaluate gains and regressions separately, and design skills to preserve input grounding and output verification.

Harbor frames agent development as an empirical loop: run agents in reproducible sandboxes, verify outcomes, inspect trajectories, and evaluate every harness or model change.

Vending-Bench shows why long-horizon agents need both repeatable simulations and real-world tests: models drift, exploit incentives, and behave differently when they detect an eval.

This security eval tests whether agents can discover and exploit logic flaws across live chained services, using hidden zero-days and deterministic grading instead of source-code pattern matching.

DSPy separates an AI task’s contract from its model, prompt, tools, and harness, letting builders swap or optimize implementations while holding behavior against the same evals.

Three studies suggest LLM compliance varies with opinion distance, claimed source, and coalition structure, so agent tests should probe how framing changes judgment rather than score sycophancy once.

Coding-agent loops can raise throughput without preserving maintainability. Keep human ownership of code, and use upfront alignment to make review affordable instead of trying to automate it away.

Agent skills need regression tests, not manual spot checks. Test triggering and output with and without each skill, across repeated trials and the harnesses your team actually uses.