arXivPaperNeeds Review

QuoteBench: How Matched Scores Can Hide Command-Path Failures

QuoteBench shows that shell-command scores can conceal failures introduced by serialization and reparsing. Agent evals should identify the execution path, not attribute every result to the model.

arXiv · Aug 13, 2026

Open Source Open Markdown Open JSON

Source Summary

QuoteBench tests **56 one-shot tasks** from 14 incident-derived families across eight configurations. Adding one unescaped parser cut replayed-command success by **55.4–73.2 percentage points**.

Practical Implication

When the boundary was disclosed, six configurations recovered **30.4–60.7 points**. Builders should test generated commands through the exact production transport and validate final state, especially where wrappers interpolate or reparse shell text.

Agent-Ready Context

QuoteBench tests **56 one-shot tasks** from 14 incident-derived families across eight configurations. Adding one unescaped parser cut replayed-command success by **55.4–73.2 percentage points**.

When the boundary was disclosed, six configurations recovered **30.4–60.7 points**. Builders should test generated commands through the exact production transport and validate final state, especially where wrappers interpolate or reparse shell text.

Adaptation was uneven: two configurations recovered nothing or declined slightly. GPT-5.6-sol’s **−3.6-point matched gap** concealed −64.3 points of transport damage plus 60.7 points of compensation, so aggregate scores can misstate both model and harness quality.

Connected Context · Feed7 Judgment

This provides quantified evidence that command transport is part of the evaluated system: a model can appear unchanged in aggregate while suffering severe parser-induced failures and partially compensating after disclosure. It strengthens calls for cross-harness testing and final-state validation, and further narrows leaderboard interpretation because matched scores can hide offsetting model adaptation and harness damage.

State of Data — Sean Cai, Independent / State of DataQuoteBench quantifies the earlier claim that scores shift with scaffolding and shows why evaluation should inspect tool trajectories and resulting state rather than only aggregate task success.The Bitter Lesson of Tool CallingBoth establish interface representation as a consequential harness variable; QuoteBench adds that reparsing and escaping along the production command path can dominate apparent model performance.Teaching AI to Find Real Vulnerabilities — David Brumley, BugcrowdBugcrowd’s deterministic exploit oracles reinforce QuoteBench’s requirement to validate concrete final effects instead of trusting generated commands or self-reported completion.Quantifying infrastructure noise in agentic coding evalsAnthropic shows resource configuration can move scores, while QuoteBench demonstrates a different hidden systems effect: transport damage can be masked by model compensation even when the aggregate gap is small.

Context Map

benchmarkcodingsecurity#agent-evals#benchmark-integrity#harness-engineering

Uncertainty

Adaptation was uneven: two configurations recovered nothing or declined slightly. GPT-5.6-sol’s **−3.6-point matched gap** concealed −64.3 points of transport damage plus 60.7 points of compensation, so aggregate scores can misstate both model and harness quality.