Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs
Fixed chunk sizes leave query-dependent retrieval quality on the table; multiscale indexes plus reciprocal rank fusion improved recall across four tested datasets at a memory cost.
AI21 tested several chunk sizes on QMsum, NarrativeQA, Seinfeld, and FinanceBench. No fixed size consistently dominated; an oracle choosing per query showed a **20–40%** gap, while multiscale retrieval produced reported gains of roughly **10–40%** depending on the dataset.
Instead of committing to one window, index several sizes, query every index in parallel, map chunks back to document rankings, and merge them with **reciprocal rank fusion**. Treat chunking as a retrieval configuration that should be evaluated against real query shapes.
AI21 tested several chunk sizes on QMsum, NarrativeQA, Seinfeld, and FinanceBench. No fixed size consistently dominated; an oracle choosing per query showed a **20–40%** gap, while multiscale retrieval produced reported gains of roughly **10–40%** depending on the dataset. Instead of committing to one window, index several sizes, query every index in parallel, map chunks back to document rankings, and merge them with **reciprocal rank fusion**. Treat chunking as a retrieval configuration that should be evaluated against real query shapes. The method duplicates indexes and was reported to require roughly **2–5x memory**. The tested sizes were chosen somewhat arbitrarily, and the talk leaves the best number of scales and ranking fusion method open.
Multiscale retrieval turns chunk size from a fixed ingestion choice into a query-dependent ranking problem. It strengthens the prior case for workload-specific retrieval evaluation by showing that no tested size consistently wins, while supplying a practical hedge through parallel indexes and rank fusion. That hedge has a material storage cost, and the unresolved number of scales and fusion method mean it should be tested against simpler retrieval and full-context baselines.