# Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

Source: [arXiv](https://arxiv.org/abs/2608.05139v1)  
Feed7 permalink: https://feed7.dev/p/2608-05139v1-0ji039d  
Published: 2026-08-05T17:57:16.000Z  
Trust: Needs Review (needs_review)

## Why Included

Skill²-Bench measures whether models can switch reasoning skills across dependent steps. Its results expose a failure mode relevant to agents whose plans mix coding, retrieval, calculation and tool use.

## Source Summary

Skill²-Bench covers **558 skills across 9 domains** and assigns cross-skill tasks an entropy-based difficulty level. Tests of **8 frontier and 4 open-source models** found accuracy falling as skill-switching difficulty rose.

## Practical Implication

Evaluate long-running agents on transitions between skills, not only isolated task scores. The proposed RL method makes models predict each step's skill and improved Qwen3-4B-Instruct from **34.4% to 68.4%** and Qwen3-1.7B from **14.6% to 40.1%**.

## Agent-Ready Context

Skill²-Bench covers **558 skills across 9 domains** and assigns cross-skill tasks an entropy-based difficulty level. Tests of **8 frontier and 4 open-source models** found accuracy falling as skill-switching difficulty rose.

Evaluate long-running agents on transitions between skills, not only isolated task scores. The proposed RL method makes models predict each step's skill and improved Qwen3-4B-Instruct from **34.4% to 68.4%** and Qwen3-1.7B from **14.6% to 40.1%**.

Those training gains are shown on two Qwen models and depend on gold skill sequences used by the reward. Whether the taxonomy transfers cleanly to messy, tool-driven software work remains an open question.

## Connected Context

Feed7 judgment across 368 accumulated Signals:

This adds skill transitions as a measurable source of long-horizon failure, complementing benchmarks that score final outcomes or sustained trajectories without explicitly modeling switches between capabilities. The reported training gains suggest step-level skill identification can help two Qwen models, but dependence on gold skill sequences and a fixed taxonomy limits claims about messy tool-driven agents.

- [AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers](https://feed7.dev/p/2607-29626v1-1dfg6xz) — AgentHPOBench evaluates whether agents improve across sequential experiments; Skill²-Bench adds a possible explanatory dimension by measuring how transitions between required skills increase difficulty.
- [DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve](https://feed7.dev/p/deepswe-a-contamination-resistant-coding-benchmark-james-shi-datacurve-08p61c0) — DeepSWE reinforces the need for long-horizon behavioral evaluation, while Skill²-Bench narrows that behavior into explicit skill-switching difficulty rather than repository completion alone.
- [AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification](https://feed7.dev/p/2607-11849v1-1sjdl4n) — AdvancedMathBench exposes unreliable verification within rigorous reasoning, whereas Skill²-Bench isolates switching between skills; together they show that long-horizon reliability needs more than one aggregate reasoning score.

## Context Map

- Layer: benchmark
- Domains: None
- Topics: reasoning, agent-evals, agent-reliability

## Uncertainty

- Those training gains are shown on two Qwen models and depend on gold skill sequences used by the reward. Whether the taxonomy transfers cleanly to messy, tool-driven software work remains an open question.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.