NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale

• Cross-cluster refits must deliver each updated policy before the next rollout batch, although only about 1% of BF16 values change per step.A full-checkpoint transfer takes 87.5 min at 1T; reports also describe 1.5–2.0% Qwen3 RL change rates and average deltas of 1.98% of a 1,024 GiB checkpoint.Source: §1 · p. 1
• A sparse refit must solve placement, exactness, and efficiency: map sparse changes to receiver storage, preserve updated bits through failures, and overlap delivery without a cross-cluster collective.The next rollout batch waits for the complete update, and a cross-cluster collective couples receiver membership to training and introduces failure timeouts.Source: §1 · p. 2
• Canonical coordinates support direct projection of affine changes and residual conversion for others, while XOR masks encode eligible changes and overwrites encode the rest.Direct projection avoids full-tensor assembly and conversion for affine changes; mixed encoding cuts Qwen3 payload bytes by 38–40% versus overwrites.Source: §1 · p. 2
• Receivers apply updates in place through the native loader; retries overwrite the changed set, and a joint commit binds the policy to its source baseline.Proposition 1 guarantees bit-exactness, and tests with receiver kills follow dense NCCL refits’ mean reward and KL trajectories.Source: §1 · p. 2
• Object storage or a relay tree streams payloads during delta construction, without a cross-cluster collective; synchronous RL can also overlap application.At 3% and 5%, relay-tree transport overlaps construction and its lower bound accounts for 77–94% of refit latency.Source: §1 · p. 2
• 12–40× faster than the transport-only full-checkpoint reference, NeMo-DCR refits 30B–1T models; at 1T and 3%, relay-tree delivery takes 150 s instead of 87.5 min.The 3% relay-tree comparison is 150 s versus 87.5 min at 1T.Source: Figure 1 · p. 1, §1 · p. 2

2.1. Refit across layouts and versions

• Training and serving layouts can shard, combine, or permute tensors differently, so a receiver’s native loader maps canonical checkpoint tensors into rank-local storage.A training shard can overlap several receiver slices, or one receiver slice can combine parts of several training shards.Source: §2.1 · p. 3

2.2. Few BF16 values change per step

• 0.6–1.2% of unique source elements change their stored BF16 values per step on average across six models’ first five GRPO steps.A stored value changes only when an update crosses a rounding boundary; larger learning rates and early steps change more stored values.Source: §2.2 · p. 3

2.3. Requirements and related work

• A delta refit must match dense-refit bits, scale transfer with changed values, use native placement, avoid cross-cluster collectives, and recover partial writes under one authoritative commit.The cited existing systems fully meet at most two of the five requirements, and none fully meets recovery.Source: §2.3 · pp. 3–4

• Shard owners compare weights with the source baseline, project affine changes into canonical coordinates, and construct residual tensors for other changes.The direct projection path avoids assembling and converting full tensors for affine changes.Source: §3 · p. 4
• The codec emits XOR entries when projection and the loader preserve stored bits and no write overlaps; otherwise it emits overwrites.The first three refit steps together meet the bit-exactness and delta-volume requirements.Source: §3 · p. 4
• Receivers validate and deduplicate payloads, apply entries in place through the native loader, and retry failed attempts with overwrites before the coordinator commits the policy and source baseline.The joint commit follows acknowledgments from every required receiver.Source: §3 · p. 4
• Object storage or a relay tree delivers payloads without a cross-cluster collective, streaming delivery alongside delta construction and, in synchronous RL, application.The example traces a changed BF16 value from owner-side projection through receiver application and the joint commit.Source: §3 · p. 4

4. Direct Projection and Residual Conversion

• Direct projection covers fixed affine mappings, while residual conversion constructs and compares canonical tensors for changes those mappings cannot represent.Residual tasks include conversions that spread changes across outputs or combine inputs from multiple owners.Source: §4 · pp. 4–5, §4.3 · p. 5

4.1. Ownership and change detection

• One owner per unique source element detects changes against the committed baseline by comparing stored bits, catching changes floating-point equality can miss.Stable name hashing balances ownership across data-parallel ranks; trackers hold one copy of the source weights, alongside canonical residual tensors.Source: §4.1 · p. 5

4.2. Direct projection

• For affine changes, fixed index mappings identify canonical destinations and source projections provide updated values, emitted once for all receivers.Receivers’ native loaders select their own storage slices; owners need not know receiver storage offsets.Source: §4.2 · p. 5

4.3. Residual conversion

• Residual-task flags and conversion dependencies select affected canonical tensors, whose assembled bits are compared with the residual baseline to emit only changed outputs.A shared scale-factor change can affect converted values even when other source values remain unchanged; selected parts must cover each tensor and agree wherever they overlap.Source: §4.3 · pp. 5–6

5. Mixed XOR/Overwrite Encoding with Compression

• Both XOR masks and overwrites reproduce updated bits exactly, unlike arithmetic reconstruction, which can round away from the target value.Applying a mask flips differing bits; an overwrite replaces the whole stored value.Source: §5 · p. 6
• XOR masks encode affine changes only when projection and native loading preserve stored bits; absolute overwrites encode all other records.The mixed mode uses XOR only when no other write touches the same bytes; otherwise it uses overwrites.Source: §5 · p. 6
• In the two MoE checkpoints, default mixed mode uses XOR entries for more than 96% of weight bytes when affine conversions preserve stored bits and no other write touches those bytes.Source: §5 · p. 6
• XOR masks often contain leading zeros because small updates leave high-order bits unchanged, and run-length, gap, and zstd coding compress the payloads.The example mask 0x0003 has 14 leading zero bits.Source: §5 · p. 6
• Because applying an XOR mask twice restores the old bits, receivers deduplicate payloads and retries use overwrites to repair partial writes.The same overwrite rule makes retries safe after incomplete application.Source: §5 · p. 6

6. Recoverable In-Place Refits

• Native-loader placement enables in-place refits without a receiver-side baseline copy, while overwrite recovery repairs partial writes and a joint commit aligns receiver storage with the source baseline.Each transition begins with required receivers at the committed version.Source: §6 · p. 6

6.1. Applying deltas through the native loader

• Receivers scatter each payload item into canonical-shaped scratch, let the native loader transform it, and update active positions at final storage copies.XOR uses zero placeholders so inactive positions remain unchanged; overwrite placeholders must stay distinct from overwrite values until the first intercepted copy.Source: §6.1 · pp. 6–7

6.2. Equivalence to a dense refit

• Under Proposition 1’s assumptions, applying the delta yields rank-local parameter and buffer bits equal to a dense refit.The result requires checks to pass before commit and fixed candidate weights, baseline, ownership, conversion rules, and loader settings through the transition.Source: §6.2 · p. 7

6.3. Overwrite recovery and joint commit

• After a failed attempt, NeMo-DCR recomputes the complete changed set from the same candidate weights and baseline, then retries with absolute overwrites to reach the target bits.New or restarted receivers restore the committed version through a dense refit before joining.Source: §6.3 · p. 7
• A durable, versioned joint commit follows successful application by all required receivers and binds the updated policy to its source baseline for the next delta.Owners update task trackers and the residual baseline in place after commit; a replacement coordinator reads the same commit record before deciding whether to retry.Source: §6.3 · p. 7

7. Pipelined Delivery without a Cross-Cluster Collective

• Object storage and relay-tree delivery overlap refit stages without a cross-cluster collective.Owners emit payloads, and both transports share the refit protocol.Source: §7 · p. 7

7.1. Object-storage and relay-tree delivery

• Object storage uses an expected-payload manifest: owners upload each payload once, while every rollout node downloads each listed payload across the cluster boundary.A retry uploads the complete changed set again.Source: §7.1 · p. 8, Figure 7 · p. 8
• With direct links, a relay-tree root receives owner payloads and forwards them over local links; without retransmission, each payload crosses the cluster boundary once per attempt.By contrast, object-storage downloads make cross-boundary bytes grow with the number of rollout nodes.Source: §7.1 · p. 8

7.2. Overlapping refit stages

• Whole-tensor chunks are compared and their records packed into compressed buckets, amortizing metadata, compression, and transfer overhead.Owners build the next bucket while the previous payload is sent; bounded queues limit decoded payloads held in memory.Source: §7.2 · p. 8, Figure 8 · p. 8
• In synchronous RL, receivers apply payloads during delivery after the request gate closes; in asynchronous RL, they stage the delta while serving the old version, then pause requests to apply it.The gate reopens when the transition ends.Source: §7.2 · p. 8

8. Evaluation

• Six research questions test individual contributions and the combined system, covering compression, construction, exactness, latency, scaling, and transport costs.RQs 4–5 test the combined effect.Source: §8 · p. 8

8.1. Experimental setup and bit-exactness

• Latency runs use 32 GB300 training GPUs and 64 H100 rollout GPUs across AWS regions, with object storage or relay-tree transport.The testbed measured up to 5 Gbps per node’s cross-cluster flow; runs used 3% and 5% change rates as stress cases.Source: §8.1 · pp. 8–9
• Measured latency includes comparison through commit and baseline update, but excludes change injection, post-apply checks, and one-time setup.Each configuration uses ten timed refits after five warm-ups.Source: §8.1 · p. 9

8.2. XOR masks are 1.7–2.2× smaller

• 1.7–2.2× smaller than overwrite values after zstd level-1 compression, XOR masks reduce compressed changed-value size in synthetic BF16 tests.The test generated 20 million BF16 values and added Gaussian noise at three standard deviations.Source: §8.2 · p. 9, Table 2 · p. 9

8.3. Projection saves time, XOR saves bytes

• 1.08–1.16× faster construction than full conversion and 38–40% fewer payload bytes than overwrites result from direct projection and XOR encoding, respectively.Mixed encoding uses 7–13% more construction time than overwrites-only mode, but is the default because transport dominates relay-tree latency.Source: §8.3 · p. 9, Table 3 · p. 10

8.4. Training follows dense NCCL under receiver kills

• 0.41 mean reward for dense NCCL and both NeMo-DCR transports, whose mean reward and per-token KL trajectories follow dense NCCL under receiver kills.Every NeMo-DCR refit and overwrite retry passed its configured post-apply checks.Source: §8.4 · pp. 9–10, Figure 9 · p. 10

8.5. 15–33× faster refits at 120B

• 22.6–49.7 s for 120B refits on both transports, 15.1–33.2× faster than the transport-only full-checkpoint reference.Compressed payloads including locations total 2.4–3.8% of the 247.2 GB checkpoint before replication.Source: §8.5 · p. 10, Table 4 · p. 10

8.6. 12–40× faster from 30B to 1T

• 12–40× faster than the transport-only full-checkpoint reference, 30B–1T refits take 8.3–14.5 s at 30B and 150–300 s at 1T in stress cases.At 5%, relay tree beats object storage at every size, with its advantage growing from 1.13× at 30B to 1.36× at 1T.Source: §8.6 · p. 10, Figure 10 · p. 10

8.7. Relay-tree transport dominates latency

• 77–94% of stress-case refit latency is accounted for by the relay-tree transport lower bound, versus 18–45% by the construction lower bound.Pipelined delivery overlaps delta construction with relay-tree transport.Source: §8.7 · pp. 10–11, Figure 11 · p. 10

9. Conclusion

• NeMo-DCR combines direct projection, residual conversion, mixed XOR/overwrite encoding, recoverable in-place refits, and pipelined delivery without a cross-cluster collective.The source baseline remains in host memory and totals about one checkpoint across training ranks.Source: §9 · p. 11
• 12–40× faster than the transport-only full-checkpoint reference, cross-cluster refits for 30B–1T models take 150 s rather than 87.5 min at 1T and 3%.The authors conclude these results make per-update refits practical for cross-cluster agentic RL at trillion-parameter scale.Source: §9 · p. 11

Appendices

• Appendix A states the conditions for Proposition 1 and proves dense-refit equivalence, while Appendix B specifies protocol operations and recovery rules.The remaining appendices cover integration and measurement details, GRPO settings, affine and XOR coverage, and evidence behind Table 1.Source: Appendices · p. 13, A. Dense-Refit Equivalence · p. 13

A.1. Source conditions and version state

• Each unique source shard has one owner, and owners compare stored bit patterns so signed-zero changes are detected.Floating-point equality treats +0.0 and −0.0 as equal, but their BF16 patterns differ.Source: §A.1 · p. 14
• Direct projection uses supported affine mappings and Bridge metadata; custom export hooks disable this path, while residual tasks cover other conversion dependencies.Tied embeddings, their dependent output head, and sources feeding residual conversions always use residual tasks.Source: §A.1 · p. 14
• For residual tensors, owners assemble changed outputs together when any contributing shard changes, then compare them with the residual baseline.A canonical-name hash selects each residual tensor’s baseline owner.Source: §A.1 · p. 14, Figure A.4 · p. 15
• The committed baseline reconstructs the policy from affine trackers and residual tensors, and candidate weights and source state remain fixed until commit.Training takes its next optimizer step only after the transition; changing source membership requires baseline redistribution or reinitialization.Source: §A.1 · p. 15

A.2. Native-loader conditions

• Loader paths are checked in advance; a reported load without an intercepted copy fails the attempt.Prewarm discovers tensors skipped on each receiver rank.Source: §A.2 · p. 15, Figure A.8 · p. 16
• XOR copies require unchanged scratch bits, existing parameter storage, and loader operations that preserve every possible mask bit pattern.Operations such as nan_to_num, clipping, and casts therefore use overwrite only.Source: §A.2 · pp. 15–16, Figure A.9 · p. 17
• Overwrite paths derive an active mask from non-NaN placeholders, and ordered pointwise writes must preserve that mask and match dense-refit bits.If a loader transform erases the placeholder, the integration needs another validated placeholder scheme.Source: §A.2 · pp. 16–17, Figure A.10 · p. 17

A.3. Equivalence proof

• Under Proposition 1’s assumptions, receiver storage after a successful flush equals the dense-refit result.Every emitted destination range holds target bits, while unchanged ranges retain version-v bits that equal the dense-refit bits.Source: §A.3 · p. 18
• Bitwise change detection ensures changed canonical destinations receive records; XOR masks rely on version-aligned resident bits, while overwrite paths write transformed active values.Projection and the loader preserve stored bits on the XOR path; validated placeholders establish the overwrite active mask.Source: §A.3 · p. 18
• After failed attempts are stopped and drained, an overwrite retry repairs partial writes; a loader write bypassing the dispatch hook defeats this recovery and requires receiver retirement or operator abort.A dense refit restores the receiver after retirement or abort.Source: §A.3 · p. 19

B. Refit Protocol

• Each attempt fixes its candidate-baseline identifier and integration setup; the protocol assumes non-malicious failures and a durable, available control plane for commit state.The control plane accepts prepares only from the active coordinator and rejects later operations from a replaced coordinator.Source: B. Refit Protocol · p. 19

B.1. Admission and delivery

• After admission, rejects or failures trigger overwrite recovery; flush checks expected payloads, drains receiver queues, synchronizes devices, and gathers completion acknowledgments.Object storage and relay trees carry payloads through different transport paths.Source: §B.1 · pp. 19–21
• Receivers validate and deduplicate payloads, then apply them in the attempt’s declared order rather than arrival order.Declared ordering keeps ranks aligned when loader operations use cross-rank collectives.Source: §B.1 · p. 21, Figure B.2 · p. 21

B.2. Joint commit and recovery

• Commit publishes version v+1 only after every required receiver acknowledges; owners then update their baselines before the transition ends.The request gate reopens after all owners report completion.Source: §B.2 · p. 21
• After coordinator failover, the new coordinator drains the old attempt and checks commit state; if version v remains committed, it retries in overwrite mode.A confirmed commit instead leads to conflict handling, not another retry.Source: §B.2 · pp. 22–23

C. Integration and Measurement

• The rollout integration intercepts the native loader’s final storage copies and reserves up to 264 decoded payload slots across its queue and assembly group.In asynchronous RL, compressed payloads wait in host staging until open_apply, when a decoder feeds the queue in declared order.Source: C. Integration and Measurement · p. 24, Figure C.2 · p. 25, Figure C.1 · p. 25
• Refit latency includes construction through baseline update, while the construction and transport lower bounds are the longest per-owner or per-sender interval unions.Mean refit latency exceeds both bounds at every traced size.Source: C. Integration and Measurement · pp. 24–25

D. Coverage and Payload Format

• Affine shards use XOR when their loader path preserves representation and has no overlapping writes; changed residual tensors use overwrites.Scaled and cast values may require overwrites because their converted-bit masks can differ from source masks.Source: D. Coverage and Payload Format · p. 26, Figure D.1 · p. 26
• 18,721 of 18,867 Qwen3-30B-A3B canonical tensors are affine, and XOR covers 96.3% of weight bytes across the reported parallel degrees.At degrees one, four, and eight, XOR covers 18,721 of 18,867, 74,161 of 74,307, and 148,081 of 148,227 items, respectively.Source: D. Coverage and Payload Format · p. 26
• 403,869 of 404,054 items are affine for Nemotron-3-Ultra-550B-A55B at degree eight, with XOR weight bytes rounding to the 97.0% affine share.The few Mamba state parameters transformed by the native loader do not take the XOR path.Source: D. Coverage and Payload Format · pp. 26–27
• Payloads store contiguous changes as a start index and run length, scattered changes as gaps, and values in shared buffers.Each payload also carries identifiers and per-item metadata; a BLAKE2b-128 digest covers its compressed body.Source: D. Coverage and Payload Format · p. 27, Figure D.3 · p. 27
• Element change rate and canonical change density are equal when a representation-preserving bijection maps source elements to canonical elements.The element change rate counts changed source elements; canonical change density counts canonical destinations with changed bits.Source: D. Coverage and Payload Format · p. 27
• Object storage moves about (1 + N_node)V_payload per attempt, while a relay tree without retransmission moves about V_payload across the cluster boundary.Payload volume includes metadata, locations, and values; scattered changes and additional items increase location and metadata bytes.Source: D. Coverage and Payload Format · p. 27

E. GRPO Experiment Settings

• The GRPO change measurements use BF16 weights, FP32 AdamW state, and a constant learning rate of 10^-6 across six listed models.Training runs use 64 prompts with 8 responses each, batch size 512, and probability-ratio clipping to [0.8, 1.28].Source: E. GRPO Experiment Settings · p. 28

F. System Comparison

• NeMo-DCR meets all five requirements, and with receivers killed mid-refit both transports follow dense NCCL’s mean reward and KL trajectories.The comparison cites payload deduplication, overwrite recovery, and joint commit for recovery.Source: F. System Comparison · p. 28
• Arithmetic reconstruction can introduce rounding errors, while NeMo-DCR specifies loader conditions under which its refits produce dense-refit-equivalent bits.The comparison also notes that several systems do not specify supported loader transformations or every intercepted storage write.Source: F. System Comparison · p. 28
• Delta schemes scale with the changed set, but PULSE, SparseRL-Sync, and verl send absolute overwrites of changed elements.The comparison says verl maps training-rank shard changes through block placements and supports conversions only when they permute elements.Source: F. System Comparison · p. 28
• Several systems place updates outside the native loader, while NCCL-Reshard and TensorHub use native placement only partially.NCCL-Reshard directly copies expert feed-forward shards; TensorHub converts weights to rollout layout before publication.Source: F. System Comparison · pp. 28–29
• Some compared systems use a collective spanning training and rollout, whereas PULSE, AReaL-DTE, ROSE, AuroraRL, and TensorHub deliver without one.TensorHub pipelines full-weight replication, while AuroraRL streams each delta checkpoint through one seed actor per region.Source: F. System Comparison · p. 29
• No existing system in Table 1 fully meets recovery requirement G5; described approaches leave gaps in partial-failure repair or cross-receiver commit.For example, TensorHub coordinates version selection within each model-parallel group rather than committing across all required receivers.Source: F. System Comparison · p. 29
• The slime framework meets G1–G4 but reads the full model each refit and does not tie the trainer’s advanced baseline to rollout-node versions.Its rollout nodes patch a full local checkpoint and reload it through the native loader.Source: F. System Comparison · p. 29