Large clusters for small models — Daniel Svonava, Superlinked
Serving many small specialist models needs pull-based queues, model packing, and per-model tuning; generic routers and managed endpoints can waste their throughput and cost advantage.
Superlinked’s Apache-2.0 serving stack replaces push routing with a **central shared queue** from which workers pull, and the talk reports roughly **2× cluster throughput** from that change. It supports PyTorch, Candle, and SGLang behind Rust sidecars and uses MessagePack at the gateway.
For embeddings, reranking, synthetic data, and other controlled workloads, benchmark a small specialist model on owned GPUs before defaulting to a managed endpoint. Pack multiple models per GPU, combine pinned models with lazy loading and eviction, and ship tuned cluster configurations rather than leaving parameter sweeps to operators.
Superlinked’s Apache-2.0 serving stack replaces push routing with a **central shared queue** from which workers pull, and the talk reports roughly **2× cluster throughput** from that change. It supports PyTorch, Candle, and SGLang behind Rust sidecars and uses MessagePack at the gateway. For embeddings, reranking, synthetic data, and other controlled workloads, benchmark a small specialist model on owned GPUs before defaulting to a managed endpoint. Pack multiple models per GPU, combine pinned models with lazy loading and eviction, and ship tuned cluster configurations rather than leaving parameter sweeps to operators. Small-model serving is still a tuning project: runtimes differ, request patterns can make a centralized router the bottleneck, and the speaker says Candle had not matched their PyTorch performance. The throughput and cost claims are from the project’s own tests and need reproduction on your traffic and hardware.
This moves the small-specialist-model case from selection and ownership into cluster design: economical self-hosting depends on pull-based scheduling, model packing, loading policy, and runtime tuning. It reinforces workload-specific evaluation over model size or list price, while narrowing the claimed efficiency gains to Superlinked’s own hardware and traffic until reproduced.