{
  "schema_version": "1.1",
  "id": "s13:https://arxiv.org/abs/2608.05139v1",
  "slug": "2608-05139v1-0ji039d",
  "url": "https://feed7.dev/p/2608-05139v1-0ji039d",
  "title": "Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning",
  "why_included": "Skill²-Bench measures whether models can switch reasoning skills across dependent steps. Its results expose a failure mode relevant to agents whose plans mix coding, retrieval, calculation and tool use.",
  "summary": "Skill²-Bench covers **558 skills across 9 domains** and assigns cross-skill tasks an entropy-based difficulty level. Tests of **8 frontier and 4 open-source models** found accuracy falling as skill-switching difficulty rose.",
  "practical_implication": "Evaluate long-running agents on transitions between skills, not only isolated task scores. The proposed RL method makes models predict each step's skill and improved Qwen3-4B-Instruct from **34.4% to 68.4%** and Qwen3-1.7B from **14.6% to 40.1%**.",
  "agent_context": "Skill²-Bench covers **558 skills across 9 domains** and assigns cross-skill tasks an entropy-based difficulty level. Tests of **8 frontier and 4 open-source models** found accuracy falling as skill-switching difficulty rose.\n\nEvaluate long-running agents on transitions between skills, not only isolated task scores. The proposed RL method makes models predict each step's skill and improved Qwen3-4B-Instruct from **34.4% to 68.4%** and Qwen3-1.7B from **14.6% to 40.1%**.\n\nThose training gains are shown on two Qwen models and depend on gold skill sequences used by the reward. Whether the taxonomy transfers cleanly to messy, tool-driven software work remains an open question.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2608.05139v1",
    "published_at": "2026-08-05T17:57:16.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "benchmark",
  "domains": [],
  "topics": [
    "reasoning",
    "agent-evals",
    "agent-reliability"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "Those training gains are shown on two Qwen models and depend on gold skill sequences used by the reward. Whether the taxonomy transfers cleanly to messy, tool-driven software work remains an open question."
  ],
  "connected_context": {
    "meaning": "This adds skill transitions as a measurable source of long-horizon failure, complementing benchmarks that score final outcomes or sustained trajectories without explicitly modeling switches between capabilities. The reported training gains suggest step-level skill identification can help two Qwen models, but dependence on gold skill sequences and a fixed taxonomy limits claims about messy tool-driven agents.",
    "corpus_size": 368,
    "generated_at": "2026-08-06T10:06:10.366Z",
    "connections": [
      {
        "title": "AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2607.29626v1",
        "feed7_url": "https://feed7.dev/p/2607-29626v1-1dfg6xz",
        "reason": "AgentHPOBench evaluates whether agents improve across sequential experiments; Skill²-Bench adds a possible explanatory dimension by measuring how transitions between required skills increase difficulty."
      },
      {
        "title": "DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=Yk87oUPVaxU",
        "feed7_url": "https://feed7.dev/p/deepswe-a-contamination-resistant-coding-benchmark-james-shi-datacurve-08p61c0",
        "reason": "DeepSWE reinforces the need for long-horizon behavioral evaluation, while Skill²-Bench narrows that behavior into explicit skill-switching difficulty rather than repository completion alone."
      },
      {
        "title": "AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2607.11849v1",
        "feed7_url": "https://feed7.dev/p/2607-11849v1-1sjdl4n",
        "reason": "AdvancedMathBench exposes unreliable verification within rigorous reasoning, whereas Skill²-Bench isolates switching between skills; together they show that long-horizon reliability needs more than one aggregate reasoning score."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-08-05T17:57:16.000Z",
  "modified_at": "2026-08-05T17:57:16.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2608-05139v1-0ji039d",
    "json": "https://feed7.dev/p/2608-05139v1-0ji039d.json",
    "markdown": "https://feed7.dev/p/2608-05139v1-0ji039d.md"
  }
}