Low-Bit N-Gram Embeddings for Language Models
Four-bit n-gram tables retained nearly all measured benefit while sharply reducing table memory, suggesting a practical efficiency lever for open-weight model builders.
Across three model families trained from scratch, **four-bit tables** saved **about 75%** of table memory while preserving **more than 99%** of the validation-loss improvement attributed to n-gram tables.
Model builders can quantize these lookup tables separately from the main network, or reinvest the saved capacity in larger tables. A short fine-tuning phase recovered part of the degradation at lower bit widths.
Across three model families trained from scratch, **four-bit tables** saved **about 75%** of table memory while preserving **more than 99%** of the validation-loss improvement attributed to n-gram tables. Model builders can quantize these lookup tables separately from the main network, or reinvest the saved capacity in larger tables. A short fine-tuning phase recovered part of the degradation at lower bit widths. On two large existing models, savings were **47–55%**, with perplexity increases below 1.2% and downstream performance largely preserved. The reported gains concern n-gram table memory, not total model memory.
This isolates quantization as a high-yield optimization for n-gram lookup tables: four-bit storage retained nearly all of their validation-loss benefit, without implying comparable total-model savings. Against prior evidence that network quantization damage is often diffuse, it identifies a component that can be compressed independently and suggests using the recovered capacity for larger tables, with workload checks still required at lower precision.