# Phantom Gains: Auditing Self-Improvement Against a Measured Null

Source: [arXiv](https://arxiv.org/abs/2608.20290v1)  
Feed7 permalink: https://feed7.dev/p/phantom-gains-auditing-self-improvement-against-a-measur-b3f2ae3bdf  
Published: 2026-08-20T00:00:00.000Z  
Trust: Needs Review (needs_review)

## Why Included

Add frozen-baseline replicates and a measured null before treating one-decode gains or regressions as real.

## Source Summary

Per-problem self-improvement claims can arise from inference and evaluation noise. This audit argues every transition statistic needs a measured null from frozen baseline replicates.

## Practical Implication

For agent and model evaluations, do not treat problem-level gains and losses as ground truth from one decode. Measure each statistic’s null with baseline replicates, then use per-problem tests and false-discovery-rate control.

## Agent-Ready Context

The study ran three rounds of rank-32 LoRA self-training on Qwen3-8B alongside a frozen control using the same pipeline. It found **seven measurement failures** capable of reversing findings when the control was omitted.

For agent and model evaluations, do not treat problem-level gains and losses as ground truth from one decode. Measure each statistic’s null with baseline replicates, then use per-problem tests and false-discovery-rate control.

A single greedy decode gave an untrained model an apparent expansion rate of **0.280**, while the replacement exact test detected **nothing on held-out replicates**. Results for problems the base model never solved remained inconclusive, and the method needs more baseline replicates than many studies possess.

## Context Map

- Layer: benchmark
- Domains: research
- Topics: benchmark-integrity, agent-evals, agent-reliability

## Uncertainty

- Automatically selected from source material; feed7 has not independently tested the claim.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.