Homa: The End of TCP for AI Clusters — John Ousterhout, Stanford
Homa targets the small-message tail latency that can idle GPUs in distributed agentic inference, using message boundaries, short-message priority, and receiver-led congestion control.
Ousterhout argues that AI traffic is shifting from giant transfers toward latency-sensitive synchronization. In his sample benchmark, short-message P99 was over 1 ms with TCP and under 100 μs with Homa, about **13x faster**; long messages were nearly **2x faster**.
If distributed inference stalls between millisecond-scale compute phases, measure small-message tail latency rather than aggregate throughput alone. Homa's message-aware RPCs, shortest-remaining-time priority, and receiver-issued grants directly target queueing behind large transfers.
Ousterhout argues that AI traffic is shifting from giant transfers toward latency-sensitive synchronization. In his sample benchmark, short-message P99 was over 1 ms with TCP and under 100 μs with Homa, about **13x faster**; long messages were nearly **2x faster**. If distributed inference stalls between millisecond-scale compute phases, measure small-message tail latency rather than aggregate throughput alone. Homa's message-aware RPCs, shortest-remaining-time priority, and receiver-issued grants directly target queueing behind large transfers. These figures come from **one sample benchmark**, not a production comparison across varied clusters. Homa is available as a Linux kernel module and is being worked toward upstream inclusion, so adoption carries kernel and deployment risk.
This adds transport queueing to the agent-inference latency budget. Cache-aware scheduling, engine routing, and faster tokenization can still leave distributed execution stalled by short synchronization messages trapped behind large transfers. Homa therefore suggests measuring message-size-specific tail latency alongside TTFT and whole-task latency, but its single sample benchmark and kernel-module deployment burden make it an infrastructure hypothesis to validate on representative traffic, not a general TCP replacement.