# AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement

Source: [arXiv](https://arxiv.org/abs/2608.20318v1)  
Feed7 permalink: https://feed7.dev/p/2608-20318v1-15y0195  
Published: 2026-08-20T17:56:59.000Z  
Trust: Needs Review (needs_review)

## Why Included

AI4AI-Bench tests whether agents can rewrite training algorithms, not merely tune runs. The best system scored 0.250, with most submissions never changing how the model learns.

## Source Summary

AI4AI-Bench gives agents **4 hours on one B300** to rewrite training algorithms across **10 frozen repositories**. Submitted code is rerun from scratch for up to 12 hours and judged by hidden evaluators on a normalized scale.

## Practical Implication

Treat this as a harder agent evaluation than ordinary repository repair: inspect whether an agent changes the learning rule itself. Across 29 configurations, the mean score was **0.166** and the best was **0.250**.

## Agent-Ready Context

AI4AI-Bench gives agents **4 hours on one B300** to rewrite training algorithms across **10 frozen repositories**. Submitted code is rerun from scratch for up to 12 hours and judged by hidden evaluators on a normalized scale.

Treat this as a harder agent evaluation than ordinary repository repair: inspect whether an agent changes the learning rule itself. Across 29 configurations, the mean score was **0.166** and the best was **0.250**.

Most submissions did not alter how the model learns. More reasoning increased the share that did, but even the best system closed less than one-fifth of the gap from the shipped algorithm to the stated optimum.

## Connected Context

Feed7 judgment across 525 accumulated Signals:

AI4AI-Bench narrows coding-agent self-improvement claims to verified changes in learning algorithms, not plausible repository edits or benchmark rhetoric. Its frozen repositories, hidden reruns, and low scores show that long reasoning budgets can increase attempts at substantive changes without making recursive algorithm design dependable. The result establishes a difficult evaluation, not evidence of effective self-improvement.

- [Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility](https://feed7.dev/p/2608-04001v1-1gj91hk) — The fixed four-hour compute budget makes inference protocol part of the evaluated system, reinforcing the requirement to report reasoning regime and compute rather than attribute results to the base model alone.
- [Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs](https://feed7.dev/p/benchmarking-coding-agents-on-new-vs-legacy-codebases-denys-linkov-wised-0vhw2s2) — Independent reruns and hidden evaluators implement the acceptance-oriented verification that the production refactor found necessary when agents produce convincing but incomplete scaffolding.
- [ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration](https://feed7.dev/p/scarfbench-1u8lniy) — ScarfBench’s low behavioral success and false build claims reinforce AI4AI-Bench’s finding that repository activity is not equivalent to verified success, although AI4AI-Bench raises the bar from migration to changing the learning rule.
- [Efficient LLM-Generated Shuttling Compilers for Complex Trapped-Ion Architectures](https://feed7.dev/p/2607-24714v1-1jmstqw) — The trapped-ion compiler result provides a bounded contrast: iterative model-generated code can beat hand-built baselines in some specified optimization loops, whereas AI4AI-Bench finds broad learning-algorithm improvement remains rare under its harder multi-repository setup.

## Context Map

- Layer: benchmark
- Domains: coding, research
- Topics: agent-evals, reasoning, coding-agents

## Uncertainty

- Most submissions did not alter how the model learns. More reasoning increased the share that did, but even the best system closed less than one-fifth of the gap from the shipped algorithm to the stated optimum.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.