Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i
Coding benchmark scores can mislead when prompts leak implementation details or tests reject valid solutions. Inspect tasks and verifiers before using a leaderboard to choose an agent.
G2i found SWE-bench Pro instructions averaged **481 words**. Citing DeepSWE’s analysis, the talk says **8.5%** of its tasks accepted wrong implementations and **more than 24%** rejected correct ones, exposing both leaky prompts and weak tests.
Before selecting a coding model from a leaderboard, inspect representative prompts, graders, and harness constraints. Prefer human-authored, production-relevant tasks with behavioral tests, private novel holdouts, and results that explain failure modes rather than only rank models.
G2i found SWE-bench Pro instructions averaged **481 words**. Citing DeepSWE’s analysis, the talk says **8.5%** of its tasks accepted wrong implementations and **more than 24%** rejected correct ones, exposing both leaky prompts and weak tests. Before selecting a coding model from a leaderboard, inspect representative prompts, graders, and harness constraints. Prefer human-authored, production-relevant tasks with behavioral tests, private novel holdouts, and results that explain failure modes rather than only rank models. These figures come from the examples and comparison presented in the talk, not an independent audit described there. Reward hacking also grows as models become better at finding repository or internet traces, so benchmark maintenance must keep pace with agent capability.
This consolidates several coding-benchmark defects into a model-selection rule: leaderboard rank is weak evidence unless teams inspect prompts, tests, harnesses, and failure modes. It reinforces private novel tasks and behavioral grading, while adding caution that the cited defect rates are presented evidence rather than an independent audit and that maintenance must evolve as agents exploit more traces.