{
  "schema_version": "1.1",
  "id": "s13:https://arxiv.org/abs/2608.05141v1",
  "slug": "2608-05141v1-0kai3a6",
  "url": "https://feed7.dev/p/2608-05141v1-0kai3a6",
  "title": "OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling",
  "why_included": "OctoLong builds dependency-rich, cross-repository training contexts by following code references. The results suggest long-context coding models benefit from linked program structure, not just larger text windows.",
  "summary": "OctoLong recursively follows code references using an AST parser, language server and package manager. Models from **600M to 14B parameters** receive mid-training on a **roughly 50B-token mixture**, including about **6.2B OctoLong tokens**, then roughly 10B instruction-tuning tokens.",
  "practical_implication": "For repository agents, this supports evaluating models on dependency traversal and state tracking rather than context-window size alone. It also suggests that context pipelines should preserve symbols, packages and cross-repository relationships.",
  "agent_context": "OctoLong recursively follows code references using an AST parser, language server and package manager. Models from **600M to 14B parameters** receive mid-training on a **roughly 50B-token mixture**, including about **6.2B OctoLong tokens**, then roughly 10B instruction-tuning tokens.\n\nFor repository agents, this supports evaluating models on dependency traversal and state tracking rather than context-window size alone. It also suggests that context pipelines should preserve symbols, packages and cross-repository relationships.\n\nThe reported gains come from replacing **12% of traditional context-extension data** and evaluating against 18 open-weight long-context models. The abstract calls the gains substantial but provides no task-level scores, so model-selection conclusions require the full evaluation.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2608.05141v1",
    "published_at": "2026-08-05T17:58:15.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "model",
  "domains": [
    "coding"
  ],
  "topics": [
    "open-models",
    "reasoning",
    "model-selection"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "The reported gains come from replacing **12% of traditional context-extension data** and evaluating against 18 open-weight long-context models. The abstract calls the gains substantial but provides no task-level scores, so model-selection conclusions require the full evaluation."
  ],
  "connected_context": {
    "meaning": "This makes structured cross-repository data a concrete mid-training intervention rather than treating long-context ability as a function of window size alone. It reinforces task-shaped training mixtures and shifts repository-model selection toward dependency traversal, symbol preservation, and state tracking, but the missing task-level scores prevent concluding that OctoLong-trained models outperform particular available routes.",
    "corpus_size": 368,
    "generated_at": "2026-08-06T10:06:10.366Z",
    "connections": [
      {
        "title": "Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=_PdK6x7PQNM",
        "feed7_url": "https://feed7.dev/p/data-quality-is-the-compute-multiplier-ari-morcos-datologyai-0x7k2ve",
        "reason": "OctoLong supplies a specific example of the broader claim that curated, task-matched training data can improve capability without merely increasing generic compute or data volume."
      },
      {
        "title": "The Base Model Is Dead — Varun Singh, Arcee AI",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=xbPriQWXtWM",
        "feed7_url": "https://feed7.dev/p/the-base-model-is-dead-varun-singh-arcee-ai-02hts76",
        "reason": "It provides concrete evidence for moving code and agent-task structure into training before RL, while leaving the broader question of how such mixtures should be divided across training stages unresolved."
      },
      {
        "title": "Ling 3.0 Flash is now available on AI Gateway",
        "source_name": "Vercel",
        "source_url": "https://vercel.com/changelog/ling-3-0-flash-is-now-available-on-ai-gateway",
        "feed7_url": "https://feed7.dev/p/ling-3-0-flash-is-now-available-on-ai-gateway-1he7mve",
        "reason": "OctoLong shows why Ling’s 256K window should not be treated as evidence of repository competence; dependency traversal and cross-repository state tracking require separate evaluation."
      },
      {
        "title": "Laguna S 2.1 is now available on AI Gateway",
        "source_name": "Vercel",
        "source_url": "https://vercel.com/changelog/laguna-s-2-1-is-now-available-on-ai-gateway",
        "feed7_url": "https://feed7.dev/p/laguna-s-2-1-is-now-available-on-ai-gateway-1nkdv05",
        "reason": "Laguna’s large context variants become model-selection candidates to test on structured dependency traversal, not conclusions based on context capacity alone."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-08-05T17:58:15.000Z",
  "modified_at": "2026-08-05T17:58:15.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2608-05141v1-0kai3a6",
    "json": "https://feed7.dev/p/2608-05141v1-0kai3a6.json",
    "markdown": "https://feed7.dev/p/2608-05141v1-0kai3a6.md"
  }
}