AI EngineerVideoSource Linked

Teaching AI to Find Real Vulnerabilities — Prof. David Brumley, Bugcrowd

Cybersecurity agents need deterministic exploit oracles and tasks that reward finding distinct vulnerabilities, not self-reported wins or one crash. A capability ladder makes partial progress measurable.

AI Engineer · Aug 1, 2026

Open Source Open Markdown Open JSON

Source Summary

Brumley describes reproducible vulnerable containers exposed through MCP, with setup, sandboxed file tools, and deterministic grading. He argues that crash-only checks miss exploit quality and duplicate findings; **50%** of DARPA Cyber Grand Challenge tasks reportedly contained unintended vulnerabilities, while AIxCC produced **18 unintended bugs**.

Practical Implication

For security-agent evals, grade concrete effects rather than model claims, deduplicate proofs by the vulnerabilities they trigger, and measure precision and recall across an audit set. A **16-capability ladder** can show where an agent stalls between triggering a flaw and building an out-of-sandbox exploit.

Agent-Ready Context

Brumley describes reproducible vulnerable containers exposed through MCP, with setup, sandboxed file tools, and deterministic grading. He argues that crash-only checks miss exploit quality and duplicate findings; **50%** of DARPA Cyber Grand Challenge tasks reportedly contained unintended vulnerabilities, while AIxCC produced **18 unintended bugs**.

For security-agent evals, grade concrete effects rather than model claims, deduplicate proofs by the vulnerabilities they trigger, and measure precision and recall across an audit set. A **16-capability ladder** can show where an agent stalls between triggering a flaw and building an out-of-sandbox exploit.

Even deterministic oracles encode a chosen definition of progress, and newly discovered bugs can change the ground truth. Publishing transcripts is also unresolved when a benchmark agent produces weaponized, previously private exploits.

Connected Context · Feed7 Judgment

This specializes benchmark-integrity guidance for a domain where success must be externally observable and ground truth can expand during evaluation. It confirms deterministic grading and staged capability measurement, but narrows their authority: an oracle reflects a chosen success definition, duplicate proofs can inflate results, and valid new discoveries complicate recall. It also adds a security-specific limit on publishing traces.

When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AIThis turns the broader warning about weak verifiers and reward hacking into concrete security rules: verify exploit effects, cover each required capability, and prevent duplicate findings from earning unrelated credit.Verifiable Environments for AI in Biology — Kenny Workman, LatchBioBoth show that deterministic checks are valuable but incomplete when legitimate outcomes exceed anticipated ground truth; biology emphasizes valid analysis paths, while security adds unintended vulnerabilities.Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta SoftwareThe capability ladder complements final-state judging by exposing where partial progress stops, while inspectable logs and artifacts could support review of exploit trajectories without relying on model claims.Win by Silence: Deletion Non-Monotonicity, Autonomous Exploitation, and Typed-State Gating in LLM Plan EvaluationTyped-state gating reinforces the same evaluator-design principle: structurally incomplete work should not receive a misleading success score, analogous to distinguishing a crash from a completed exploit.

Context Map

benchmarksecurity#agent-evals#benchmark-integrity#agent-reliability

Uncertainty

Even deterministic oracles encode a chosen definition of progress, and newly discovered bugs can change the ground truth. Publishing transcripts is also unresolved when a benchmark agent produces weaponized, previously private exploits.