# OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

Source: [arXiv](https://arxiv.org/abs/2608.05141v1)  
Feed7 permalink: https://feed7.dev/p/2608-05141v1-0kai3a6  
Published: 2026-08-05T17:58:15.000Z  
Trust: Needs Review (needs_review)

## Why Included

OctoLong builds dependency-rich, cross-repository training contexts by following code references. The results suggest long-context coding models benefit from linked program structure, not just larger text windows.

## Source Summary

OctoLong recursively follows code references using an AST parser, language server and package manager. Models from **600M to 14B parameters** receive mid-training on a **roughly 50B-token mixture**, including about **6.2B OctoLong tokens**, then roughly 10B instruction-tuning tokens.

## Practical Implication

For repository agents, this supports evaluating models on dependency traversal and state tracking rather than context-window size alone. It also suggests that context pipelines should preserve symbols, packages and cross-repository relationships.

## Agent-Ready Context

OctoLong recursively follows code references using an AST parser, language server and package manager. Models from **600M to 14B parameters** receive mid-training on a **roughly 50B-token mixture**, including about **6.2B OctoLong tokens**, then roughly 10B instruction-tuning tokens.

For repository agents, this supports evaluating models on dependency traversal and state tracking rather than context-window size alone. It also suggests that context pipelines should preserve symbols, packages and cross-repository relationships.

The reported gains come from replacing **12% of traditional context-extension data** and evaluating against 18 open-weight long-context models. The abstract calls the gains substantial but provides no task-level scores, so model-selection conclusions require the full evaluation.

## Connected Context

Feed7 judgment across 368 accumulated Signals:

This makes structured cross-repository data a concrete mid-training intervention rather than treating long-context ability as a function of window size alone. It reinforces task-shaped training mixtures and shifts repository-model selection toward dependency traversal, symbol preservation, and state tracking, but the missing task-level scores prevent concluding that OctoLong-trained models outperform particular available routes.

- [Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI](https://feed7.dev/p/data-quality-is-the-compute-multiplier-ari-morcos-datologyai-0x7k2ve) — OctoLong supplies a specific example of the broader claim that curated, task-matched training data can improve capability without merely increasing generic compute or data volume.
- [The Base Model Is Dead — Varun Singh, Arcee AI](https://feed7.dev/p/the-base-model-is-dead-varun-singh-arcee-ai-02hts76) — It provides concrete evidence for moving code and agent-task structure into training before RL, while leaving the broader question of how such mixtures should be divided across training stages unresolved.
- [Ling 3.0 Flash is now available on AI Gateway](https://feed7.dev/p/ling-3-0-flash-is-now-available-on-ai-gateway-1he7mve) — OctoLong shows why Ling’s 256K window should not be treated as evidence of repository competence; dependency traversal and cross-repository state tracking require separate evaluation.
- [Laguna S 2.1 is now available on AI Gateway](https://feed7.dev/p/laguna-s-2-1-is-now-available-on-ai-gateway-1nkdv05) — Laguna’s large context variants become model-selection candidates to test on structured dependency traversal, not conclusions based on context capacity alone.

## Context Map

- Layer: model
- Domains: coding
- Topics: open-models, reasoning, model-selection

## Uncertainty

- The reported gains come from replacing **12% of traditional context-extension data** and evaluating against 18 open-weight long-context models. The abstract calls the gains substantial but provides no task-level scores, so model-selection conclusions require the full evaluation.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.