arXivPaperNeeds Review

Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks

A low-dimensional theory links training data and initialization to whether transformers reason through context or learned weights, but only on a generalized synthetic task class.

arXiv

Open Source Open Markdown Open JSON

Source Summary

The paper proves that attention-model training on a generalized class of inductive tasks can remain on a **low-dimensional invariant manifold**. A handful of interpretable coordinates capture dynamics otherwise spread across millions of parameters.

Practical Implication

For builders studying or training reasoning models, the framework offers concrete probes: examine how data statistics shift competition between **in-context and in-weights learning**, and use the manifold’s coordinate frame to detect learned circuits.

Agent-Ready Context

The paper proves that attention-model training on a generalized class of inductive tasks can remain on a **low-dimensional invariant manifold**. A handful of interpretable coordinates capture dynamics otherwise spread across millions of parameters.

For builders studying or training reasoning models, the framework offers concrete probes: examine how data statistics shift competition between **in-context and in-weights learning**, and use the manifold’s coordinate frame to detect learned circuits.

The theory covers a task class that unifies synthetic settings such as in-context n-grams and multi-hop reasoning. The material does not show that the same compact dynamics predict behavior in production-scale models or open-ended coding tasks.

Context Map

modelresearch#reasoning

Uncertainty

The theory covers a task class that unifies synthetic settings such as in-context n-grams and multi-hop reasoning. The material does not show that the same compact dynamics predict behavior in production-scale models or open-ended coding tasks.