AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
AgentHPOBench tests whether agents can learn from experiment history, not merely produce code. Its results expose weaknesses in sustained refinement and log diagnosis across sequential ML runs.
AgentHPOBench contains **30 executable ML tasks** across **seven research categories**. Starting from a validated baseline, an agent repeatedly reviews prior configurations, metrics, and logs before choosing its next valid hyperparameter intervention.
Use this evaluation shape when an agent is expected to run experiments: score the sequence of decisions and improvement over time, not only final code or answers. The study compares **12 agents** and conventional hyperparameter-optimization baselines under one protocol.
AgentHPOBench contains **30 executable ML tasks** across **seven research categories**. Starting from a validated baseline, an agent repeatedly reviews prior configurations, metrics, and logs before choosing its next valid hyperparameter intervention. Use this evaluation shape when an agent is expected to run experiments: score the sequence of decisions and improvement over time, not only final code or answers. The study compares **12 agents** and conventional hyperparameter-optimization baselines under one protocol. The abstract reports measurable optimization ability but persistent problems with iterative refinement, complex log diagnosis, and consistent progress toward reference performance. It does not provide task-level scores here, so model or agent rankings cannot be inferred from this material.
This turns broad calls for trajectory-aware agent evaluation into an executable optimization benchmark where each intervention, log interpretation, and improvement over time is observable. It confirms that final performance alone can hide weak iterative behavior, while narrowing the evidence to validated ML hyperparameter tasks; the supplied material supports persistent refinement and diagnosis gaps, not an agent ranking.