# Taking Reinforcement Learning Cross Datacenter — Nan Jiang, Modal

Source: [AI Engineer](https://www.youtube.com/watch?v=maRzp4kImJ4)  
Feed7 permalink: https://feed7.dev/p/taking-reinforcement-learning-cross-datacenter-nan-jiang-modal-05t7dwu  
Published: 2026-08-10T17:30:30.000Z  
Trust: Source Linked (source_linked)

## Why Included

Cross-region RL rollouts become practical if workers receive exact sparse weight deltas instead of full checkpoints. The proposal turns scattered inference GPUs into an elastic rollout fleet.

## Source Summary

A frontier-scale rollout checkpoint is about **500 GB**, making cross-region synchronization take minutes or hours. Modal’s approach sends lossless patches of changed served weights, reducing an example transfer to **500 MB** and reconstructing the same rollout version.

## Practical Implication

Separate tightly coupled training from loosely coupled rollout inference. Keep backpropagation on the fast cluster, make rollout workers version-aware, and exchange policy updates one way with trajectories and metadata the other.

## Agent-Ready Context

A frontier-scale rollout checkpoint is about **500 GB**, making cross-region synchronization take minutes or hours. Modal’s approach sends lossless patches of changed served weights, reducing an example transfer to **500 MB** and reconstructing the same rollout version.

Separate tightly coupled training from loosely coupled rollout inference. Keep backpropagation on the fast cluster, make rollout workers version-aware, and exchange policy updates one way with trajectories and metadata the other.

The argument relies on served low-precision weights changing sparsely even when gradients and FP32 updates are dense. Whether the pattern holds with Muon, fully asynchronous RL at larger scale, or training stages beyond post-training remains open.

## Connected Context

Feed7 judgment across 419 accumulated Signals:

This adds a concrete distributed-systems pattern for frontier RL: decouple rollout inference from backpropagation and synchronize exact served-policy deltas instead of moving full checkpoints. It makes geographically separated rollout capacity more plausible, but only where low-precision served weights change sparsely. The open questions sharply limit generalization to other optimizers, asynchronous regimes, larger scales, or training beyond post-training.

- [Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute](https://feed7.dev/p/learning-on-the-job-the-future-of-post-training-raymond-feng-applied-com-17u0m7t) — The production-harness candidate identifies non-replayable interactions as a barrier to RL updates; this Signal supplies transport and versioning infrastructure for separated rollout and training, but does not resolve replayability or feedback quality.
- [TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning](https://feed7.dev/p/2608-04007v1-0gjxuy2) — TurnSight requires execution traces and turn-level hindsight for training; the bidirectional exchange of versioned policy updates, trajectories, and metadata provides a compatible systems boundary for collecting that supervision at distributed rollout workers.

## Context Map

- Layer: infra
- Domains: data
- Topics: tool-use

## Uncertainty

- The argument relies on served low-precision weights changing sparsely even when gradients and FP32 updates are dense. Whether the pattern holds with Muon, fully asynchronous RL at larger scale, or training stages beyond post-training remains open.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.