Taking Reinforcement Learning Cross Datacenter — Nan Jiang, Modal
Cross-region RL rollouts become practical if workers receive exact sparse weight deltas instead of full checkpoints. The proposal turns scattered inference GPUs into an elastic rollout fleet.
A frontier-scale rollout checkpoint is about **500 GB**, making cross-region synchronization take minutes or hours. Modal’s approach sends lossless patches of changed served weights, reducing an example transfer to **500 MB** and reconstructing the same rollout version.
Separate tightly coupled training from loosely coupled rollout inference. Keep backpropagation on the fast cluster, make rollout workers version-aware, and exchange policy updates one way with trajectories and metadata the other.
A frontier-scale rollout checkpoint is about **500 GB**, making cross-region synchronization take minutes or hours. Modal’s approach sends lossless patches of changed served weights, reducing an example transfer to **500 MB** and reconstructing the same rollout version. Separate tightly coupled training from loosely coupled rollout inference. Keep backpropagation on the fast cluster, make rollout workers version-aware, and exchange policy updates one way with trajectories and metadata the other. The argument relies on served low-precision weights changing sparsely even when gradients and FP32 updates are dense. Whether the pattern holds with Muon, fully asynchronous RL at larger scale, or training stages beyond post-training remains open.
This adds a concrete distributed-systems pattern for frontier RL: decouple rollout inference from backpropagation and synchronize exact served-policy deltas instead of moving full checkpoints. It makes geographically separated rollout capacity more plausible, but only where low-precision served weights change sparsely. The open questions sharply limit generalization to other optimizers, asynchronous regimes, larger scales, or training beyond post-training.