AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement
AI4AI-Bench tests whether agents can rewrite training algorithms, not merely tune runs. The best system scored 0.250, with most submissions never changing how the model learns.
AI4AI-Bench gives agents **4 hours on one B300** to rewrite training algorithms across **10 frozen repositories**. Submitted code is rerun from scratch for up to 12 hours and judged by hidden evaluators on a normalized scale.
Treat this as a harder agent evaluation than ordinary repository repair: inspect whether an agent changes the learning rule itself. Across 29 configurations, the mean score was **0.166** and the best was **0.250**.
AI4AI-Bench gives agents **4 hours on one B300** to rewrite training algorithms across **10 frozen repositories**. Submitted code is rerun from scratch for up to 12 hours and judged by hidden evaluators on a normalized scale. Treat this as a harder agent evaluation than ordinary repository repair: inspect whether an agent changes the learning rule itself. Across 29 configurations, the mean score was **0.166** and the best was **0.250**. Most submissions did not alter how the model learns. More reasoning increased the share that did, but even the best system closed less than one-fifth of the gap from the shipped algorithm to the stated optimum.
AI4AI-Bench narrows coding-agent self-improvement claims to verified changes in learning algorithms, not plausible repository edits or benchmark rhetoric. Its frozen repositories, hidden reruns, and low scores show that long reasoning budgets can increase attempts at substantive changes without making recursive algorithm design dependable. The result establishes a difficult evaluation, not evidence of effective self-improvement.