Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora
Legora’s latency failure shows why tenant activity patterns matter more than schema shape: mixing hot and cold projects caused cache thrashing, while per-project namespaces restored isolation.
Legora’s partitioned Postgres design mixed active and dormant projects, causing cache thrashing as the corpus grew. Search and ingestion P99 rose from **100 milliseconds to 20 seconds** before the company moved at roughly **400 million documents**.
For multi-tenant agent retrieval, align the storage boundary with the unit users query. Legora adopted one Turbopuffer namespace per project, gaining BM25 search, per-namespace separation, and object-storage economics for cold projects across **more than 2 billion documents**.
Legora’s partitioned Postgres design mixed active and dormant projects, causing cache thrashing as the corpus grew. Search and ingestion P99 rose from **100 milliseconds to 20 seconds** before the company moved at roughly **400 million documents**. For multi-tenant agent retrieval, align the storage boundary with the unit users query. Legora adopted one Turbopuffer namespace per project, gaining BM25 search, per-namespace separation, and object-storage economics for cold projects across **more than 2 billion documents**. The design fits workloads with many cold indices and a small hot set; it is not a universal database prescription. Cold reads can still incur object-storage latency, and some Legora workloads disabled the NVMe cache to meet isolation requirements.
This adds a concrete storage lesson to enterprise retrieval: partitioning must follow the user-visible query and isolation boundary, especially when a small hot set sits over many cold corpora. The migration supports separating metadata and retrieval workloads, but its object-storage tradeoffs narrow the design to similarly skewed multi-tenant systems rather than a general database replacement.