Balancing hierarchical representation and quantization: A performance trade-off in text embeddings
An MTEB study finds int8 embeddings can cut storage with limited accuracy loss, while binary quantization degrades results; retrieval is especially sensitive to dimensionality reduction.
The study tests **38 MTEB tasks** across four embedding models and 168 configurations per task. Combining **8× dimensionality reduction with int8** cuts per-vector storage 32× at 94.4% of baseline effectiveness; binary quantization performs significantly worse across task categories.
For retrieval or reranking pipelines, benchmark compression by task category and measure storage separately from quality. The paper finds int8 is statistically indistinguishable from float32 for pair classification and reranking, while sampling intermediate layers shortened evaluation time.
The study tests **38 MTEB tasks** across four embedding models and 168 configurations per task. Combining **8× dimensionality reduction with int8** cuts per-vector storage 32× at 94.4% of baseline effectiveness; binary quantization performs significantly worse across task categories. For retrieval or reranking pipelines, benchmark compression by task category and measure storage separately from quality. The paper finds int8 is statistically indistinguishable from float32 for pair classification and reranking, while sampling intermediate layers shortened evaluation time. Evaluation times were measured once, so the speed findings are indicative. The reported average effectiveness can hide larger retrieval degradation than in other categories.
The embedding study makes compression a task-category trade-off: dimensionality reduction plus int8 cuts storage substantially with a smaller average quality loss, while binary quantization performs worse. It supports measuring storage and quality separately for the target retrieval workload, since average effectiveness can conceal larger retrieval losses. One-time timing measurements limit the speed conclusions.