Large clusters for small models — Daniel Svonava, Superlinked
Serving many small specialist models needs pull-based queues, model packing, and per-model tuning; generic routers and managed endpoints can waste their throughput and cost advantage.
Superlinked’s Apache-2.0 serving stack replaces push routing with a **central shared queue** from which workers pull, and the talk reports roughly **2× cluster throughput** from that change. It supports PyTorch, Candle, and SGLang behind Rust sidecars and uses MessagePack at the gateway.
For embeddings, reranking, synthetic data, and other controlled workloads, benchmark a small specialist model on owned GPUs before defaulting to a managed endpoint. Pack multiple models per GPU, combine pinned models with lazy loading and eviction, and ship tuned cluster configurations rather than leaving parameter sweeps to operators.
Superlinked’s Apache-2.0 serving stack replaces push routing with a **central shared queue** from which workers pull, and the talk reports roughly **2× cluster throughput** from that change. It supports PyTorch, Candle, and SGLang behind Rust sidecars and uses MessagePack at the gateway. For embeddings, reranking, synthetic data, and other controlled workloads, benchmark a small specialist model on owned GPUs before defaulting to a managed endpoint. Pack multiple models per GPU, combine pinned models with lazy loading and eviction, and ship tuned cluster configurations rather than leaving parameter sweeps to operators. Small-model serving is still a tuning project: runtimes differ, request patterns can make a centralized router the bottleneck, and the speaker says Candle had not matched their PyTorch performance. The throughput and cost claims are from the project’s own tests and need reproduction on your traffic and hardware.
This moves the small-model decision beyond choosing or owning a specialist model to operating it efficiently at cluster scale. A pull-based shared queue, multi-model GPU packing, and tuned configurations may improve the economics of controlled workloads, but they add a workload-specific serving and eviction problem whose reported gains must be reproduced on the intended hardware and traffic.