How enabling two settings tripled our scores on the ARC-AGI-3 benchmark
Two API settings—reasoning retention and compaction—reportedly tripled GPT-5.6’s ARC-AGI-3 score. Agent evals should treat runtime configuration as part of the tested system.
OpenAI says enabling **reasoning retention** and **compaction** produced **3× ARC-AGI-3 scores** for GPT-5.6 while also improving efficiency.
Record these settings alongside the model name in agent evaluations. Configuration can materially affect results, so defaults and explicit settings should not be compared as equivalent systems.
OpenAI says enabling **reasoning retention** and **compaction** produced **3× ARC-AGI-3 scores** for GPT-5.6 while also improving efficiency. Record these settings alongside the model name in agent evaluations. Configuration can materially affect results, so defaults and explicit settings should not be compared as equivalent systems. The supplied material provides no absolute scores, token usage, latency, or experimental detail, leaving the size and generality of the efficiency gain unclear.
This makes inference configuration part of the evaluated system: the same named model can produce materially different benchmark results when reasoning retention and compaction change. It reinforces prior warnings that infrastructure and inference protocols can move scores independently of model identity, while the absent absolute scores, compute accounting, and experimental detail prevent judging effect size, efficiency, or generality beyond the reported setup.