The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
When quantizing an open model, spend a small extra precision budget across the network before protecting a few “important” layers; causal tests found the damage was usually diffuse.
Researchers raised each layer to 8-bit in turn across **9 open-weight models** from four architecture families. In **8 of 9 models**, recovering 75% of the accuracy gap required roughly half the layers.
For local agent inference, treat globally finer quantization granularity as the first tuning move. At matched precision, it beat selective layer repair by **21–52 points** across all eight models compatible with group-128 quantization.
Researchers raised each layer to 8-bit in turn across **9 open-weight models** from four architecture families. In **8 of 9 models**, recovering 75% of the accuracy gap required roughly half the layers. For local agent inference, treat globally finer quantization granularity as the first tuning move. At matched precision, it beat selective layer repair by **21–52 points** across all eight models compatible with group-128 quantization. The result covers one budget setting, and Qwen3-8B showed unusually concentrated damage. **8-bit was near-lossless** under the evaluated RTN, GPTQ, and AWQ setups, but cheap correlates still failed to identify where restored precision would help.
This replaces a common selective-repair intuition with causal evidence that quantization damage is usually distributed: under the tested budget, spreading added precision broadly was more effective than protecting a few layers. It makes quantization granularity a first local-inference tuning variable, while the exceptional model and single budget setting preserve the need for artifact- and workload-specific evaluation.