# Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora

Source: [AI Engineer](https://www.youtube.com/watch?v=V-isu4eTHgw)  
Feed7 permalink: https://feed7.dev/p/connect-ai-to-billions-of-legal-documents-simon-eskildsen-turbopuffer-ja-1xhg8ow  
Published: 2026-09-16T16:00:19.000Z  
Trust: Source Linked (source_linked)

## Why Included

Legora’s latency failure shows why tenant activity patterns matter more than schema shape: mixing hot and cold projects caused cache thrashing, while per-project namespaces restored isolation.

## Source Summary

Legora’s partitioned Postgres design mixed active and dormant projects, causing cache thrashing as the corpus grew. Search and ingestion P99 rose from **100 milliseconds to 20 seconds** before the company moved at roughly **400 million documents**.

## Practical Implication

For multi-tenant agent retrieval, align the storage boundary with the unit users query. Legora adopted one Turbopuffer namespace per project, gaining BM25 search, per-namespace separation, and object-storage economics for cold projects across **more than 2 billion documents**.

## Agent-Ready Context

Legora’s partitioned Postgres design mixed active and dormant projects, causing cache thrashing as the corpus grew. Search and ingestion P99 rose from **100 milliseconds to 20 seconds** before the company moved at roughly **400 million documents**.

For multi-tenant agent retrieval, align the storage boundary with the unit users query. Legora adopted one Turbopuffer namespace per project, gaining BM25 search, per-namespace separation, and object-storage economics for cold projects across **more than 2 billion documents**.

The design fits workloads with many cold indices and a small hot set; it is not a universal database prescription. Cold reads can still incur object-storage latency, and some Legora workloads disabled the NVMe cache to meet isolation requirements.

## Connected Context

Feed7 judgment across 812 accumulated Signals:

This adds a concrete storage lesson to enterprise retrieval: partitioning must follow the user-visible query and isolation boundary, especially when a small hot set sits over many cold corpora. The migration supports separating metadata and retrieval workloads, but its object-storage tradeoffs narrow the design to similarly skewed multi-tenant systems rather than a general database replacement.

- [How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs](https://feed7.dev/p/how-web-data-infrastructure-powers-the-next-generation-of-ai-patricija-z-1ta4xzh) — Oxylabs establishes retrieval as production-critical infrastructure; Legora identifies corpus layout and cache behavior as another source of latency collapse beyond browsers, proxies, parsing, and retries.
- [Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub — Arek Borucki, Hugging Face](https://feed7.dev/p/serving-2-million-models-without-melting-scaling-the-hugging-face-hub-ar-06hazr6) — Both architectures separate workload paths instead of scaling one undifferentiated store; Legora applies that principle to per-project indices and hot-versus-cold document access.
- [Why Your Enterprise Tech Stack Isn’t Ready for AI Agents — Christopher Lovejoy & Saul Howard](https://feed7.dev/p/why-your-enterprise-tech-stack-isn-t-ready-for-ai-agents-christopher-lov-1uwgsr1) — Legora provides an implementation consequence of treating enterprise constraints as foundational: tenant isolation can determine namespace design and may require disabling a performance cache.
- [Regional inference now available on AI Gateway](https://feed7.dev/p/regional-inference-now-available-on-ai-gateway-0642f58) — Regional routing and per-project namespaces enforce different boundaries—residency versus retrieval isolation—but together show that sensitive workloads constrain infrastructure placement before model execution.

## Context Map

- Layer: infra
- Domains: data
- Topics: retrieval, enterprise

## Uncertainty

- The design fits workloads with many cold indices and a small hot set; it is not a universal database prescription. Cold reads can still incur object-storage latency, and some Legora workloads disabled the NVMe cache to meet isolation requirements.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.