arXivPaperNeeds Review
The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
When quantizing an open model, spend a small extra precision budget across the network before protecting a few “important” layers; causal tests found the damage was usually diffuse.
arXiv
Source Summary
Researchers raised each layer to 8-bit in turn across **9 open-weight models** from four architecture families. In **8 of 9 models**, recovering 75% of the accuracy gap required roughly half the layers.
Practical Implication
For local agent inference, treat globally finer quantization granularity as the first tuning move. At matched precision, it beat selective layer repair by **21–52 points** across all eight models compatible with group-128 quantization.
Agent-Ready Context
Researchers raised each layer to 8-bit in turn across **9 open-weight models** from four architecture families. In **8 of 9 models**, recovering 75% of the accuracy gap required roughly half the layers. For local agent inference, treat globally finer quantization granularity as the first tuning move. At matched precision, it beat selective layer repair by **21–52 points** across all eight models compatible with group-128 quantization. The result covers one budget setting, and Qwen3-8B showed unusually concentrated damage. **8-bit was near-lossless** under the evaluated RTN, GPTQ, and AWQ setups, but cheap correlates still failed to identify where restored precision would help.
Context Map
modelcoding#open-models#model-selectionUncertainty
The result covers one budget setting, and Qwen3-8B showed unusually concentrated damage. **8-bit was near-lossless** under the evaluated RTN, GPTQ, and AWQ setups, but cheap correlates still failed to identify where restored precision would help.