Structure--Residual Inductive Biases for Language Model Pretraining
A language-model pretraining study combines two representation biases with different layer preferences and reports a 1.44% NLL improvement on GPT-2 Medium at 300M tokens. The authors say gains persist to 20B tokens.
The paper introduces Boundary Bottleneck Concept (BBC), which routes segment information through boundary carriers, and Continuous Concept (CC), which regularizes representation geometry. BBC worked best in early layers; CC was most effective in later layers. Together they improved GPT-2 Medium NLL by **1.44%** at **300M** training tokens.
For teams training models, the result suggests representation structure may be worth testing alongside next-token prediction, with different mechanisms placed at different depths. Track validation loss and representation diagnostics when reproducing the approach.
The paper introduces Boundary Bottleneck Concept (BBC), which routes segment information through boundary carriers, and Continuous Concept (CC), which regularizes representation geometry. BBC worked best in early layers; CC was most effective in later layers. Together they improved GPT-2 Medium NLL by **1.44%** at **300M** training tokens. For teams training models, the result suggests representation structure may be worth testing alongside next-token prediction, with different mechanisms placed at different depths. Track validation loss and representation diagnostics when reproducing the approach. The authors report that gains persisted in runs up to 20B tokens, but the material gives no detailed results across model families or deployment settings. The finding concerns pretraining, not a direct change to coding-agent workflows.
This adds a representation-design intervention to the prior model-training candidates: different structural biases appear useful at different depths, with reported gains in pretraining loss. It complements optimizer and data choices as a research direction, but the supplied evidence does not establish transfer across model families or coding-agent use.