Homa: The End of TCP for AI Clusters — John Ousterhout, Stanford
Homa targets the small-message tail latency that can idle GPUs in distributed agentic inference, using message boundaries, short-message priority, and receiver-led congestion control.
Ousterhout argues that AI traffic is shifting from giant transfers toward latency-sensitive synchronization. In his sample benchmark, short-message P99 was over 1 ms with TCP and under 100 μs with Homa, about **13x faster**; long messages were nearly **2x faster**.
If distributed inference stalls between millisecond-scale compute phases, measure small-message tail latency rather than aggregate throughput alone. Homa's message-aware RPCs, shortest-remaining-time priority, and receiver-issued grants directly target queueing behind large transfers.
Ousterhout argues that AI traffic is shifting from giant transfers toward latency-sensitive synchronization. In his sample benchmark, short-message P99 was over 1 ms with TCP and under 100 μs with Homa, about **13x faster**; long messages were nearly **2x faster**. If distributed inference stalls between millisecond-scale compute phases, measure small-message tail latency rather than aggregate throughput alone. Homa's message-aware RPCs, shortest-remaining-time priority, and receiver-issued grants directly target queueing behind large transfers. These figures come from **one sample benchmark**, not a production comparison across varied clusters. Homa is available as a Linux kernel module and is being worked toward upstream inclusion, so adoption carries kernel and deployment risk.
Homa narrows agent-infrastructure performance work to the network synchronization layer: when distributed inference alternates short compute phases with messages, small-message tail latency may matter more than bulk throughput. The sample results justify measuring that bottleneck, not assuming a production win; the protocol’s kernel module and incomplete upstream path make deployment maturity and operational risk part of any evaluation.