Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs
Fixed chunk sizes leave query-dependent retrieval quality on the table; multiscale indexes plus reciprocal rank fusion improved recall across four tested datasets at a memory cost.
AI21 tested several chunk sizes on QMsum, NarrativeQA, Seinfeld, and FinanceBench. No fixed size consistently dominated; an oracle choosing per query showed a **20–40%** gap, while multiscale retrieval produced reported gains of roughly **10–40%** depending on the dataset.
Instead of committing to one window, index several sizes, query every index in parallel, map chunks back to document rankings, and merge them with **reciprocal rank fusion**. Treat chunking as a retrieval configuration that should be evaluated against real query shapes.
AI21 tested several chunk sizes on QMsum, NarrativeQA, Seinfeld, and FinanceBench. No fixed size consistently dominated; an oracle choosing per query showed a **20–40%** gap, while multiscale retrieval produced reported gains of roughly **10–40%** depending on the dataset. Instead of committing to one window, index several sizes, query every index in parallel, map chunks back to document rankings, and merge them with **reciprocal rank fusion**. Treat chunking as a retrieval configuration that should be evaluated against real query shapes. The method duplicates indexes and was reported to require roughly **2–5x memory**. The tested sizes were chosen somewhat arbitrarily, and the talk leaves the best number of scales and ranking fusion method open.
This replaces fixed-size chunking as a default with a measured, query-dependent retrieval configuration. Parallel multiscale indexes plus reciprocal rank fusion can recover part of the oracle gap, but reported gains come with roughly 2–5× memory and leave scale selection and fusion design unresolved. The practical consequence is to benchmark chunk strategies against real query distributions and deployment budgets.