# Phantom Gains: Auditing Self-Improvement Against a Measured Null

Source: [arXiv](https://arxiv.org/abs/2608.20290v1)  
Feed7 permalink: https://feed7.dev/p/2608-20290v1-13o5gau  
Published: 2026-08-20T17:30:14.000Z  
Trust: Needs Review (needs_review)

## Why Included

Per-problem self-improvement claims can arise from inference and evaluation noise. This audit argues every transition statistic needs a measured null from frozen baseline replicates.

## Source Summary

The study ran three rounds of rank-32 LoRA self-training on Qwen3-8B alongside a frozen control using the same pipeline. It found **seven measurement failures** capable of reversing findings when the control was omitted.

## Practical Implication

For agent and model evaluations, do not treat problem-level gains and losses as ground truth from one decode. Measure each statistic’s null with baseline replicates, then use per-problem tests and false-discovery-rate control.

## Agent-Ready Context

The study ran three rounds of rank-32 LoRA self-training on Qwen3-8B alongside a frozen control using the same pipeline. It found **seven measurement failures** capable of reversing findings when the control was omitted.

For agent and model evaluations, do not treat problem-level gains and losses as ground truth from one decode. Measure each statistic’s null with baseline replicates, then use per-problem tests and false-discovery-rate control.

A single greedy decode gave an untrained model an apparent expansion rate of **0.280**, while the replacement exact test detected **nothing on held-out replicates**. Results for problems the base model never solved remained inconclusive, and the method needs more baseline replicates than many studies possess.

## Connected Context

Feed7 judgment across 525 accumulated Signals:

Phantom Gains converts general benchmark skepticism into a concrete prerequisite for self-improvement claims: estimate a statistic’s null from frozen-control replicates before interpreting per-problem transitions. It shows that one-decode expansion can vanish under a replacement exact test and FDR control. Repeated sampling alone is therefore insufficient unless baseline variability is measured; unsolved-base cases remain unresolved.

- [Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B](https://feed7.dev/p/2607-28576v1-09h2m1u) — The repeated-sampling study establishes a token-matched baseline for reflection claims; Phantom Gains adds that repeated decodes must also estimate frozen-baseline variability and control multiplicity before apparent per-problem changes count as improvement.
- [When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI](https://feed7.dev/p/when-will-the-benchmaxxing-plague-end-nick-heiner-surge-ai-178gqcg) — The benchmaxxing critique broadly warns that leaderboard gains can be artifacts; Phantom Gains supplies a specific statistical mechanism and control protocol for detecting such artifacts in self-training evaluations.
- [SocietyBench: Forecasting Counterfactual Social-World Evolution](https://feed7.dev/p/2608-04009v1-05m5u8w) — SocietyBench warns against generalizing variable aggregate results and recommends event-level reporting; Phantom Gains shows that problem-level reporting itself requires a measured null and false-discovery control to avoid mistaking decode noise for transitions.
- [Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing](https://feed7.dev/p/2608-07437v1-0t4v96m) — Fisher-R1’s reported single-trial improvement is a model-comparison result, while Phantom Gains shows what additional frozen-control replication and per-problem testing would be required before interpreting transition-level changes as reliable self-improvement.

## Context Map

- Layer: benchmark
- Domains: research
- Topics: benchmark-integrity, agent-evals, agent-reliability

## Uncertainty

- A single greedy decode gave an untrained model an apparent expansion rate of **0.280**, while the replacement exact test detected **nothing on held-out replicates**. Results for problems the base model never solved remained inconclusive, and the method needs more baseline replicates than many studies possess.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.