arXivPaperNeeds Review

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

OctoLong builds dependency-rich, cross-repository training contexts by following code references. The results suggest long-context coding models benefit from linked program structure, not just larger text windows.

arXiv · Aug 5, 2026

Open Source Open Markdown Open JSON

Source Summary

OctoLong recursively follows code references using an AST parser, language server and package manager. Models from **600M to 14B parameters** receive mid-training on a **roughly 50B-token mixture**, including about **6.2B OctoLong tokens**, then roughly 10B instruction-tuning tokens.

Practical Implication

For repository agents, this supports evaluating models on dependency traversal and state tracking rather than context-window size alone. It also suggests that context pipelines should preserve symbols, packages and cross-repository relationships.

Agent-Ready Context

OctoLong recursively follows code references using an AST parser, language server and package manager. Models from **600M to 14B parameters** receive mid-training on a **roughly 50B-token mixture**, including about **6.2B OctoLong tokens**, then roughly 10B instruction-tuning tokens.

For repository agents, this supports evaluating models on dependency traversal and state tracking rather than context-window size alone. It also suggests that context pipelines should preserve symbols, packages and cross-repository relationships.

The reported gains come from replacing **12% of traditional context-extension data** and evaluating against 18 open-weight long-context models. The abstract calls the gains substantial but provides no task-level scores, so model-selection conclusions require the full evaluation.

Connected Context · Feed7 Judgment

This makes structured cross-repository data a concrete mid-training intervention rather than treating long-context ability as a function of window size alone. It reinforces task-shaped training mixtures and shifts repository-model selection toward dependency traversal, symbol preservation, and state tracking, but the missing task-level scores prevent concluding that OctoLong-trained models outperform particular available routes.

Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAIOctoLong supplies a specific example of the broader claim that curated, task-matched training data can improve capability without merely increasing generic compute or data volume.The Base Model Is Dead — Varun Singh, Arcee AIIt provides concrete evidence for moving code and agent-task structure into training before RL, while leaving the broader question of how such mixtures should be divided across training stages unresolved.Ling 3.0 Flash is now available on AI GatewayOctoLong shows why Ling’s 256K window should not be treated as evidence of repository competence; dependency traversal and cross-repository state tracking require separate evaluation.Laguna S 2.1 is now available on AI GatewayLaguna’s large context variants become model-selection candidates to test on structured dependency traversal, not conclusions based on context capacity alone.

Context Map

modelcoding#open-models#reasoning#model-selection

Uncertainty

The reported gains come from replacing **12% of traditional context-extension data** and evaluating against 18 open-weight long-context models. The abstract calls the gains substantial but provides no task-level scores, so model-selection conclusions require the full evaluation.