Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning
Skill²-Bench measures whether models can switch reasoning skills across dependent steps. Its results expose a failure mode relevant to agents whose plans mix coding, retrieval, calculation and tool use.
Skill²-Bench covers **558 skills across 9 domains** and assigns cross-skill tasks an entropy-based difficulty level. Tests of **8 frontier and 4 open-source models** found accuracy falling as skill-switching difficulty rose.
Evaluate long-running agents on transitions between skills, not only isolated task scores. The proposed RL method makes models predict each step's skill and improved Qwen3-4B-Instruct from **34.4% to 68.4%** and Qwen3-1.7B from **14.6% to 40.1%**.
Skill²-Bench covers **558 skills across 9 domains** and assigns cross-skill tasks an entropy-based difficulty level. Tests of **8 frontier and 4 open-source models** found accuracy falling as skill-switching difficulty rose. Evaluate long-running agents on transitions between skills, not only isolated task scores. The proposed RL method makes models predict each step's skill and improved Qwen3-4B-Instruct from **34.4% to 68.4%** and Qwen3-1.7B from **14.6% to 40.1%**. Those training gains are shown on two Qwen models and depend on gold skill sequences used by the reward. Whether the taxonomy transfers cleanly to messy, tool-driven software work remains an open question.
This adds skill transitions as a measurable source of long-horizon failure, complementing benchmarks that score final outcomes or sustained trajectories without explicitly modeling switches between capabilities. The reported training gains suggest step-level skill identification can help two Qwen models, but dependence on gold skill sequences and a fixed taxonomy limits claims about messy tool-driven agents.