An Empirical Study of Harness Design for Coding Agents
Harness components pay off differently by model and budget. Elide before summarizing, use planning selectively, and avoid elaborate tools when the model is already strong with bash.
Researchers tested **176 matched settings** across **four models**, varying planning, action space, and context management on SWE-Bench Verified and Terminal-Bench 2.1. Context handling mattered more as budgets tightened, chiefly by preventing overflow.
Stage rule-based elision before LLM summarization. Use planning as an accuracy scaffold for weaker models and a cost control for stronger ones; offer predefined tools when bash skill is weak, but consider bash-only operation for capable models on CLI-heavy work.
Researchers tested **176 matched settings** across **four models**, varying planning, action space, and context management on SWE-Bench Verified and Terminal-Bench 2.1. Context handling mattered more as budgets tightened, chiefly by preventing overflow. Stage rule-based elision before LLM summarization. Use planning as an accuracy scaffold for weaker models and a cost control for stronger ones; offer predefined tools when bash skill is weak, but consider bash-only operation for capable models on CLI-heavy work. Recoverable elision added machinery without an accuracy gain because models rarely used recovery. The evidence spans **five context strategies** and **four window budgets**, but the abstract provides no effect sizes and covers only two benchmarks.
This supplies comparative evidence for choices that prior harness guidance largely framed as design principles. It narrows coding-agent optimization to model- and budget-dependent decisions: prioritize overflow prevention, begin with rule-based elision, use planning differently by model strength, and avoid recovery machinery whose availability does not translate into use.