{
  "schema_version": "1.1",
  "id": "s8:https://www.youtube.com/watch?v=XzJD1bvXKjs",
  "slug": "how-long-can-your-skills-be-before-your-agent-forgets-what-you-told-it-l-0i0jeww",
  "url": "https://feed7.dev/p/how-long-can-your-skills-be-before-your-agent-forgets-what-you-told-it-l-0i0jeww",
  "title": "How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI",
  "why_included": "Instruction capacity rose sharply, but long skill files still need evals. Current models can track thousands of constraints, with large differences by model, wording, and order.",
  "summary": "A rerun of IFScale reproduced the old failure point around **200–300 rules**. Current frontier models reached roughly **2,000–5,000 rules** before major decline; GPT-5.5 reportedly held **99% accuracy at 5,000 rules** on this proxy task.",
  "practical_implication": "Revisit prompt-length assumptions before splitting a style guide or policy into many agents. Capacity is less likely to be the first constraint now, so test whether one coherent skill is cheaper and easier to maintain than a handoff-heavy design.",
  "agent_context": "A rerun of IFScale reproduced the old failure point around **200–300 rules**. Current frontier models reached roughly **2,000–5,000 rules** before major decline; GPT-5.5 reportedly held **99% accuracy at 5,000 rules** on this proxy task.\n\nRevisit prompt-length assumptions before splitting a style guide or policy into many agents. Capacity is less likely to be the first constraint now, so test whether one coherent skill is cheaper and easier to maintain than a handoff-heavy design.\n\nIFScale checks whether required words appear in a synthetic report, not whether an agent reasons well. Reported limits ranged from **750 to 9,000-plus rules**, and performance can shift with instruction wording or order, making output verification essential.",
  "source": {
    "name": "AI Engineer",
    "url": "https://www.youtube.com/watch?v=XzJD1bvXKjs",
    "published_at": "2026-09-09T15:00:06.000Z"
  },
  "source_class": "video",
  "content_type": "Video",
  "layer": "benchmark",
  "domains": [
    "coding"
  ],
  "topics": [
    "skills",
    "agent-evals",
    "agent-reliability"
  ],
  "verification": {
    "status": "source_linked",
    "label": "Source Linked",
    "method": "source_feed",
    "verified_at": null
  },
  "uncertainty": [
    "IFScale checks whether required words appear in a synthetic report, not whether an agent reasons well. Reported limits ranged from **750 to 9,000-plus rules**, and performance can shift with instruction wording or order, making output verification essential."
  ],
  "connected_context": {
    "meaning": "This raises the estimated instruction-capacity ceiling and weakens capacity alone as a reason to split coherent guidance across agents. It does not show that long skills improve real agent work: IFScale measures synthetic rule adherence, and ordering and wording still matter. Against the prior candidates, the practical conclusion is to compare one maintained skill with handoff-heavy designs using repeated, architecture-relevant regression tests.",
    "corpus_size": 732,
    "generated_at": "2026-09-10T10:09:05.265Z",
    "connections": [
      {
        "title": "Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=0vphxNt4wyk",
        "feed7_url": "https://feed7.dev/p/don-t-ship-skills-without-evals-philipp-schmid-google-deepmind-0fuh3ko",
        "reason": "The higher rule ceiling makes longer skills plausible, while this candidate supplies the necessary test discipline: compare triggering and outcomes with and without the skill across repeated runs."
      },
      {
        "title": "The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2607.22520v1",
        "feed7_url": "https://feed7.dev/p/2607-22520v1-0mz9wnf",
        "reason": "It limits the capacity result’s interpretation: an agent may retain thousands of instructions yet still regress on tasks it previously solved, so adherence and net utility must be measured separately."
      },
      {
        "title": "Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=nxokqOq1imY",
        "feed7_url": "https://feed7.dev/p/your-agent-evolved-your-evals-didn-t-ameya-bhatawdekar-braintrust-1loaqv2",
        "reason": "Choosing one large skill instead of multiple agents changes the harness architecture, reinforcing the candidate’s requirement that eval coverage follow planning, handoffs, tools, and other system structure."
      },
      {
        "title": "SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.04167v1",
        "feed7_url": "https://feed7.dev/p/2609-04167v1-1vvofgw",
        "reason": "SWE-Gate reinforces that following observable requirements is not equivalent to complete engineering correctness, matching the warning that IFScale’s word-inclusion proxy cannot establish real task performance."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-09-09T15:00:06.000Z",
  "modified_at": "2026-09-09T15:00:06.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/how-long-can-your-skills-be-before-your-agent-forgets-what-you-told-it-l-0i0jeww",
    "json": "https://feed7.dev/p/how-long-can-your-skills-be-before-your-agent-forgets-what-you-told-it-l-0i0jeww.json",
    "markdown": "https://feed7.dev/p/how-long-can-your-skills-be-before-your-agent-forgets-what-you-told-it-l-0i0jeww.md"
  }
}