Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation
SearchGen-Bench shows open image generators score 21–28/100 on long-tail entities, and naive search retrieval only adds noise; a teach-then-search co-training recipe learns when to retrieve versus rely on weights.
**The gist** **SearchGen-Bench** pairs **20,839 prompts** across twelve failure categories with a pre-executed 1M-item multimodal corpus for offline experiments. Frontier open visual generators score only **21–28 out of 100** — a roughly **40-point collapse** existing benchmarks miss — because they confidently fabricate long-tail entities and post-cutoff events.
**Why it matters** The retrieval lesson agents keep re-learning applies to image generation: **naive search fails**, injecting noise into prompts the generator already handles. The paper frames a generator-specific **knowledge boundary** — what lives in weights versus external context — and shows a **teach-then-search co-training** loop discovers it, improving monotonically. If you wire image generation into agent pipelines, gate retrieval on that boundary rather than searching every request.
**The gist** **SearchGen-Bench** pairs **20,839 prompts** across twelve failure categories with a pre-executed 1M-item multimodal corpus for offline experiments. Frontier open visual generators score only **21–28 out of 100** — a roughly **40-point collapse** existing benchmarks miss — because they confidently fabricate long-tail entities and post-cutoff events. **Why it matters** The retrieval lesson agents keep re-learning applies to image generation: **naive search fails**, injecting noise into prompts the generator already handles. The paper frames a generator-specific **knowledge boundary** — what lives in weights versus external context — and shows a **teach-then-search co-training** loop discovers it, improving monotonically. If you wire image generation into agent pipelines, gate retrieval on that boundary rather than searching every request. **Watch out** Scores cover **open generators** only, and only a **minimal version** of the co-training recipe is demonstrated; whether the learned boundary keeps up as generators and trending entities keep moving is the open question.