arXivPaperNeeds Review
AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement
AI4AI-Bench tests whether agents can rewrite training algorithms, not merely tune runs. The best system scored 0.250, with most submissions never changing how the model learns.
arXiv
Source Summary
AI4AI-Bench gives agents **4 hours on one B300** to rewrite training algorithms across **10 frozen repositories**. Submitted code is rerun from scratch for up to 12 hours and judged by hidden evaluators on a normalized scale.
Practical Implication
Treat this as a harder agent evaluation than ordinary repository repair: inspect whether an agent changes the learning rule itself. Across 29 configurations, the mean score was **0.166** and the best was **0.250**.
Agent-Ready Context
AI4AI-Bench gives agents **4 hours on one B300** to rewrite training algorithms across **10 frozen repositories**. Submitted code is rerun from scratch for up to 12 hours and judged by hidden evaluators on a normalized scale. Treat this as a harder agent evaluation than ordinary repository repair: inspect whether an agent changes the learning rule itself. Across 29 configurations, the mean score was **0.166** and the best was **0.250**. Most submissions did not alter how the model learns. More reasoning increased the share that did, but even the best system closed less than one-fifth of the gap from the shipped algorithm to the stated optimum.
Context Map
benchmarkcodingresearch#agent-evals#reasoning#coding-agentsUncertainty
Most submissions did not alter how the model learns. More reasoning increased the share that did, but even the best system closed less than one-fifth of the gap from the shipped algorithm to the stated optimum.