Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI
A multi-GPU kernel benchmark shows coding agents can improve with tools and iteration, but still struggle with communication scheduling, partitioning, and transfer choices.
Parallel Kernel Bench asks models to rewrite distributed PyTorch references as performant multi-GPU CUDA kernels across **87 problems**. A Gemini 3 Pro agent with a local shell improved from **24 to 35 solved tasks**.
Treat tool access and iterative compilation as useful but insufficient. Only **26 tasks exceeded the reference speed**, and models still struggled with collective ordering, data partitioning, scheduling, and choosing transfer mechanisms.
Parallel Kernel Bench asks models to rewrite distributed PyTorch references as performant multi-GPU CUDA kernels across **87 problems**. A Gemini 3 Pro agent with a local shell improved from **24 to 35 solved tasks**. Treat tool access and iterative compilation as useful but insufficient. Only **26 tasks exceeded the reference speed**, and models still struggled with collective ordering, data partitioning, scheduling, and choosing transfer mechanisms. Performance plateaued as more time was added, even when the relevant kernel principles were supplied in context. The results cover this benchmark and harness, so they do not establish performance for every model, topology, or production workload.
Parallel Kernel Bench isolates a demanding boundary for coding agents: shell access and iterative compilation raise solved tasks, but producing kernels that outperform references remains substantially harder, and extra time or supplied principles does not prevent a plateau. This reinforces correctness-plus-performance validation and narrows conclusions to one benchmark, agent, harness, and distributed-kernel workload.