PaperlayerView exact revision on arXiv

Source-linked AI summary

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

Changhai Zhou, Kieran Liu, Yuhua Zhou, Qian Qiao, Jun Gao, Harry Zhang, Irvine Lu, Nolan Ho, Lucian Li, Andrew Lei, Cleon Cheng, Steven Chiang, Yihang Zeng, Di Zhang, Rio Yang, Kaijie Chen, Andrew Chen, Pony Ma, Weizhong Zhang, Cheng Jin

arXiv:2607.14952v1cs.LGcs.DC

TL;DR

Long-context RL post-training must handle agent trajectories that exceed typical training contexts without expanding the GPU budget. LongStraw uses a no-grad prompt boundary and serial response replay, validating a 2,097,152-position GLM path across complete 78-layer backwards and terminal optimizer calls.

  • Problem

    Long-context agent trajectories and GRPO workloads remain difficult to fit within fixed GPU memory because prompt, response, state, and communication graphs compete for capacity.

  • Method

    LongStraw captures architecture-specific prompt state without autograd, then serially replays short response branches while making state ownership and execution ordering explicit.

  • Results

    2,097,152 positions: the GLM path carries a prompt through two complete 78-layer backwards and terminal distributed-optimizer calls.

  • Takeaways & Limitations

    LongStraw demonstrates fixed-budget long-context GRPO-shaped execution as an architecture- and ownership-aware systems problem across Qwen and GLM.

  • Takeaways & Limitations

    The execution receipts do not support stronger correctness or training claims, including coherent distributed updates.

Abstract

from arXiv · show

A growing gap separates inference context lengths from RL post-training: inference systems are approaching million-token contexts, while post-training workloads often remain at 256K tokens or below and rely on length generalization at deployment. The gap is especially important for AI agents, whose observations, tool outputs, documents, and prior decisions accumulate over long trajectories. LongStraw is an architecture-aware execution stack for million-token RL post-training under a fixed GPU budget, instantiated with Group Relative Policy Optimization (GRPO). It evaluates the shared prompt without autograd, retains only model-specific state needed by later tokens, and replays short response branches one at a time, reducing the live training graph at the cost of additional replay time. We implement it for the hybrid recurrent and full-attention Qwen3.6-27B and the compressed-attention mixture-of-experts GLM-5.2. On eight H20 GPUs, LongStraw completes grouped Qwen scoring and response backward at 2.1M positions for groups of 2 and 8; increasing the group size adds only 0.21 GB of peak allocated memory, while a separate stress test reaches 4.46M positions. On 32 H20 GPUs, we validate the end-to-end LongStraw execution path for a 2.1M-token prompt across all 78 layers of GLM-5.2. These experiments establish execution capacity rather than complete training correctness because the captured prompt state is detached and some distributed forward and gradient composition paths remain incomplete.

1 Introduction

LongStraw enables fixed-GPU GRPO execution on long prompts by evaluating shared prompt state once and serially replaying short responses. It supports Qwen’s hybrid recurrent/attention stack and GLM’s compressed-attention/MoE stack, while distinguishing completed execution from unverified distributed correctness.

  • Motivation: Long-context agents accumulate evidence, observations, tool outputs, and prior decisions across multi-turn trajectories, making full-prompt response scoring central to post-training.GRPO scores several responses sharing a prompt and backpropagates through them, unlike inference, which can discard the forward graph after caching decoding state.
  • Problem framing: The design targets a gap that memory-efficient attention, FlashAttention, LoRA, QLoRA, and scale-out systems do not independently close for fixed-GPU GRPO.Existing approaches reduce attention workspace, data movement, trainable parameters, base-model storage, or distribute work across larger accelerator fabrics.
  • Execution design: LongStraw evaluates the shared prompt without automatic differentiation, stores conditioning state once, replays one response at a time, and accumulates gradients before optimization.Serial replay increases elapsed time but avoids keeping the prompt graph and every response graph live simultaneously.
  • Model implementations: LongStraw implements architecture-specific state handling for Qwen3.6-27B’s recurrent/attention stack and GLM-5.2’s compressed-attention mixture-of-experts stack.Qwen retains recurrent state and context-sharded key/value pages; GLM stages CPU-resident prompt state one decoder layer at a time.
  • Limitations: Both implementations detach stored prompt state during response backward, and incomplete cross-rank gradient operations limit claims to execution and local optimizer calls rather than correct distributed updates.The Qwen path omits complete key/value-related gradient synchronization, while the GLM path skips usual cross-rank gradient reduction.

2 GRPO Training Dependency Graph

The GRPO execution graph separates shared-prompt capture, frozen scoring, advantage construction, serial policy replay, and one optimizer step across the group. This schedule reduces live activation dependence on group size but uses detached prompt state and has not established distributed-update or full-gradient parity.

  • Objective: GRPO defines conditional log-probability ratios from current and old policies over prompt and response prefixes.The objective also includes a clipped surrogate, group-relative advantages, member normalization, and a reference-policy KL penalty.
  • Execution schedule: The five-phase schedule captures the prompt without autograd, scores frozen policies, constructs advantages, replays one response graph at a time, and steps once after all G members.Serial replay accumulates gradients into the same adapter before synchronization and optimizer execution.
  • Execution schedule: Holding all response graphs makes activation memory scale with G, whereas serial replay makes group cardinality primarily a scheduling and time dimension.G still determines reward normalization and the GRPO advantage set; stepping between members would change importance ratios and prompt state.
  • State and gradient limits: The implementation stores stopgrad(zP(θ)) and computes only the explicit response term, so exact attention reduction does not imply full-sequence gradient equivalence.A captured state remains valid across branches only while parameters stay unchanged; after an optimizer step, the prompt state is stale and must be recaptured or explicitly approximated.
  • Validation levels: Execution receipts establish finite, rank-terminating grouped backward and optimizer events, but neither path has completed distributed-update consistency or full-gradient parity.Qwen reaches response-operator fidelity for full-attention layers, while GLM sparse selection remains local to each CP shard; the acceptance runs exclude online generation and reward-model execution.

3 Architecture Anatomy and Bottleneck Sources

The models expose two independent execution axes: token mixing determines retained prompt state, while FFN structure determines parameter residency, routing, and activation buffers. Their long-context bottlenecks therefore arise from attention-state or sparse-routing communication and are addressed by retaining only state needed by response tokens.

  • Architecture axes: Two independent axes separate token mixing and FFN execution: GDN/full attention versus MLA/DSA, and dense versus MoE.Token mixing controls retained prompt state and response-time collectives, while FFN structure controls parameter residency, token routing, and activation buffers.
  • Qwen: Qwen combines 48 recurrent GDN and 16 full-attention layers with dense FFNs.GDN retains fixed-shape recurrent state, whereas full attention retains key and value pages whose storage grows with P.
  • GLM: GLM combines 21 index-computing and 57 IndexShare layers with three dense and 75 MoE FFNs.IndexShare layers reuse nearby published selections, reducing the layers requiring durable prompt indexer-key pages.
  • GLM MoE bottleneck: 9.66 billion weights reside in the 256 routed experts of one sparse layer, while each token activates only eight experts.At a balanced 65,536-token CP shard, top-8 routing produces 524,288 expert-token rows and one BF16 [524,288, 6144] buffer occupies exactly 6 GiB.
  • Tensor lifetime: Only conditional state required by future response tokens survives prompt execution; dense intermediates, routes, permutations, scratch, and prompt adapter activations die immediately.For Qwen, the surviving state includes compact GDN and KV state; for GLM, it includes MLA latent pages and related state.

4 LongStraw: Long-Context Execution Design

LongStraw executes million-token grouped updates by retaining only model-specific prompt state, releasing transient prompt work, and replaying short response branches serially under autograd. Its design bounds the dominant live activation scale by response length, while detached captured state and incomplete distributed gradient composition limit the demonstrated result to execution capacity.

  • Execution principle: LongStraw retains a tensor across the prompt boundary only when a later response token depends on it, separating durable model state from transient scratch and activations.The rule applies to physical allocations, not merely logical tensor views.
  • Execution phases: Phase 1 captures the full prompt without autograd, saves model-specific state, and releases transient hidden tensors, attention scratch, FFN activations, and MoE routing buffers.The full prompt still passes through every decoder layer, retaining architecture-specific state such as attention pages, recurrent state, and DSA latent values plus index keys.
  • Execution phases: Phase 3 rebuilds each short current-policy response path under autograd, reuses the read-only prompt state, backpropagates its loss, and immediately releases that member’s graph.After all G backwards, worker-local gradients are accumulated and one optimizer call is issued per worker.
  • Memory accounting: The schedule changes the dominant activation scale from P +R to R, while total memory can still grow with G through inputs, labels, rewards, reports, and frozen scores.The live policy graph is bounded by the largest group member rather than the number of members.
  • Architecture-specific placement: Qwen keeps compact GDN and KV state on GPU, whereas GLM keeps CP-local MLA and indexer-key pages on CPU and stages one layer at a time during response replay.Layerwise staging avoids recreating the storage peak that copying the complete 78-layer prefix to GPU would cause.
  • Correctness limitations: The captured prompt state is detached, and distributed update correctness remains incomplete because audited Qwen and GLM paths leave some adapter gradient contributions unreduced.The Qwen audit identifies a missing K/V adapter reduction; GLM bypasses Megatron gradient finalization for CP-replicated non-expert adapter gradients.

5 Qwen: Dense Hybrid Replay within an Eight-H20 Budget

LongStraw’s Qwen implementation combines recurrent-state capture with distributed full-attention KV paging and serialized response replay to execute 2,097,152-context-position workloads on eight H20 GPUs. The results demonstrate execution and update-shaped capacity, not complete distributed training correctness.

  • Architecture: 64 decoder layers comprise 48 GDN layers and 16 full-attention layers in the audited Qwen snapshot.The repeating pattern is three linear_attention entries followed by one full_attention entry.
  • State management: Only 48 compact GDN boundaries and 16 full-attention KV page sets survive prompt capture, eliminating length-dependent state for recurrent layers.Prompt hidden states, FFN intermediates, and temporary mixer work are released after each chunk.
  • Execution layout: 2,097,152 context positions are formed by 2,088,960 prompt tokens followed by 8,192 response-input tokens.The Qwen execution distributes full-attention KV pages across eight ranks while preserving global sequence coordinates.
  • Eight-H20 results: 0.208 GB (0.213%) is the peak-memory increase from group size 2 to group size 8, while end-to-end time rises by 1,586.445 seconds.Reported peaks are 97.503 GB for group size two and 97.711 GB for group size eight.
  • Validation scope: 2,097,152 context positions fit within eight H20 GPUs for global response forwards, response-shaped backward graphs, and eight terminal optimizer calls, without evidence of coherent distributed updates.No terminal receipt records a gradient norm, parameter delta, post-step adapter hash, or next-forward replica comparison.

6 GLM: Paged MLA/DSA Replay within a 32-H20 MoE Budget

The GLM path demonstrates architecture-shaped replay and backward capacity for a 2,097,152-position prompt across 78 layers on 32 H20 GPUs, including all 75 MoE tails under native EP autograd. The receipt establishes execution capacity, not model-faithful training correctness, because distributed attention composition and gradient composition remain incomplete.

  • Paged MLA/DSA replay: 21 index-computing layers publish DSA top-k tensors, while 57 IndexShare consumers reuse them without duplicate prompt-length index state.The cross-layer schedule uses index frequency four and skip offset three.
  • Paged MLA/DSA replay: 186 GiB across 32 ranks is the retained page payload, with 72 MiB for IndexShare layers and 88 MiB for index-computing layers.These figures exclude response activations and kernel workspaces and are not whole-step peaks.
  • MoE execution: All 75 MoE tails execute under the native EP autograd graph, including routed-expert LoRA paths, shared-expert branches, dispatch, exchange, and inverse combination.The trace is therefore not an attention-only or synthetic FFN surrogate.
  • Execution receipt: 396 policy-trace events and 160 old-policy-trace events produce 128 files with 35,584 JSONL events, but no parameter-gradient tensors or cross-rank reductions.Each policy rank records 78 checkpointed layer ends and backward traces for layer, attention-projection, and sparse-attention operations.
  • Limitations: The final run omits selected-K/V exchange, distributed attention output composition, replicated response-hidden-state agreement, and non-expert replica-gradient averaging.These omissions are identified as the primary forward-fidelity gap and prevent claiming model-faithful training or quality evaluation.

7 Making the GLM GRPO Path Fit a Fixed 32-H20 Budget

Under a fixed 32-H20 topology, LongStraw progressively moved the 2,097,152-position GLM prompt out of the differentiable graph, validated staged replay, and resolved cross-layer execution issues. The final grouped run completed two serial 78-layer policy backward passes and optimizer steps, while retaining explicit limitations around detached state and CP-local semantics.

  • Full-graph failure: At 2,097,152 positions, extending the conventional full-sequence LoRA graph successively shifted OOMs from DSA scratch to expert-LoRA work and MoE output concatenation.The progression showed that no single allocation could be optimized in isolation, motivating removal of the long prompt from the differentiable graph.
  • Prefix capture: 2,097,152 positions completed through all 78 prefix layers without autograd, using CP-sharded MLA and DSA pages while releasing prompt MoE work per layer.This established prompt-processing and state-capacity execution, not trainability; state restoration, response logits, routing under autograd, and optimization remained untested.
  • Differentiable replay: 2,097,152 positions enabled layer-0 backward plus optimizer-call canaries after CPU page residency and chunked staging controlled GPU state and MoE costs.The layer-0 result demonstrated both prompt-state fit and response consumption under autograd for one layer, but not cross-layer IndexShare or 78-layer activation lifetime.
  • All-layer correctness gates: 32K and 64K all-layer gates executed backward through all 78 attention and FFN tails, including 21 index producers, 57 IndexShare consumers, three dense FFNs, and 75 MoE FFNs.These tests exposed and closed IndexShare lifetime, resident DSA layout, restored-view mutation, and activation-lifetime issues; complete-layer checkpointing made saved activations scale with response length.
  • Grouped acceptance: The final grouped run used one 2,097,152-position prompt with two deterministic responses, serially executed two 78-layer policy backwards, then one optimizer call and gradient clear on all 32 ranks.It used TP1/CP32/EP32/ETP1/PP1, with rewards [0, 1] producing advantages [−1, 1].
  • Resource and semantic limits: 112.571–145.148 GB was the per-rank capture-window max_memory_- allocated range, compared with 143,771 MiB per NVIDIA H20-3e device; the run remained semantically limited.The capture readings correspond to 74.7–96.3% of device total if the GLM workers expose the same device capacity, while the prompt state remained detached and the DSA response operator remained CP-local.

8 Execution Receipts and Trace Evidence

The receipts demonstrate million-token execution capacity for Qwen and GLM under fixed GPU allocations, including terminal worker-local scoring, backward, and optimizer events. However, the evidence remains execution-level: it does not establish coherent distributed updates, full gradient correctness, or numerical parity.

  • Evidence limits: Execution receipts do not establish a coherent distributed parameter update, complete parameter gradients, synchronized reductions, or full-gradient parity.The Qwen audit lacked post-step parameter hashes or replica comparisons; traces record execution events rather than complete gradient tensors, and some cross-rank reductions remain unverified.
  • Qwen receipts: 2,097,152 context positions completed for both Qwen probes on eight H20 workers, with peak allocated memory of 97.503 and 97.711 decimal GB per rank.The probes used group sizes G = 2 and G = 8; near-flat peaks validate serial response-graph lifetime at these endpoints.
  • GLM receipts: 32/32 GLM ranks recorded two old-policy phases, two policy backwards, one optimizer call, one zero-grad call, and no execution errors.Supplied rewards were [0, 1], producing normalized advantages [−1, 1] and per-member losses in [−0.5, 0.5].
  • Trace evidence: 4,992 forward and 4,992 backward layer-end events covered all 78 layers across the two GLM policy phases.All policy layers reported activation checkpointing, and traces included attention-projection and sparse-attention backward events.
  • Memory diagnostics: 112.571–145.148 GB per rank was the GLM capture-window peak allocation range, while prompt state stored 5.8125 GiB on CPU per rank.Only 72 MiB or 88 MiB was staged per layer, and the counter was read before full-GRPO response replay.

9 Fixed-Budget Systems Lessons

LongStraw’s fixed-budget capacity comes from freeing prompt-side training state after the long forward, then serializing response branches so live autograd memory follows response length. The resulting receipts demonstrate execution feasibility at million-token scales, while leaving distributed gradient composition and learned-behavior validation incomplete.

  • Prompt-state execution: Prompt capacity comes from allowing attention scratch, FFN intermediates, routing, permutations, and adapter activations to die before response backward.Every prompt token still passes through every decoder layer; after capture, the live autograd working set follows response length rather than prompt length.
  • Observed numeric envelope: 2,097,152 positions are executed for Qwen on eight H20 GPUs and GLM on 32 H20 GPUs under fixed accelerator budgets.Qwen’s receipt is 8× its 262,144 native setting; GLM’s captured prompt is 2× its 1,048,576 published setting.
  • Validation limits: 2.097M execution receipts complete requested scoring, response backward, and terminal optimizer calls on every worker, but these tests do not establish full training correctness.Parity still requires per-LoRA-shard comparisons, per-family outliers, global cosine and relative L2, and optimizer-delta verification; GLM response DSA remains local to one CP shard.
  • Observed numeric envelope: 0.208 GB (+0.213%) is the Qwen peak-allocation increase when group size rises from G = 2 to G = 8.The fourfold group increase raises post-prefix work by 3.93×, while prefix sharing reduces mean total wall time per response by 67.4%, from 2,599.390 to 848.153 seconds.
  • Observed numeric envelope: 4,456,448 Qwen positions are reached on the same eight-H20 budget, including one prefix capture and eight serial response branches.The measured path takes 21,750.133 seconds and peaks at 82.960 GB.
  • Validation limits: The reported result is an accelerator-bounded systems feasibility envelope, not evidence of useful learned behavior or a record claim.No long-context task evaluation, loss curve, or repeated learning result accompanies the receipts.

10 Related Work

Prior systems demonstrate million-token processing through full-sequence training, sequence parallelism, memory-efficient attention, and large-scale infrastructure. LongStraw instead targets detached-prefix GRPO replay, requiring explicit training-state contracts across attention, sparse indexing, MoE routing, activation recomputation, and adapter updates.

  • Million-token systems: 4.096M positions is achieved by Ring Attention for exact-attention training of a 7B model on 32 A100 GPUs.DeepSpeed-Ulysses includes a one-million-token sequence for a 1.2B GPT model while scaling to 256 A100 GPUs.
  • Million-token systems: LongStraw uses fewer devices for a detached-prefix GRPO replay problem, so comparisons motivate budget and evidence axes rather than efficiency ratios.Ring Attention and ByteScale provide stronger full-sequence training semantics.
  • Attention and state: Exact-attention methods stream score blocks or distribute sequence partitions, whereas LongStraw retains prompt state across old, reference, and policy computations.Ring Attention circulates sequence blocks; DeepSpeed-Ulysses exchanges sequence and attention-head partitions with all-to-all collectives, and USP combines ring-style and Ulysses-style parallelism.
  • Attention and state: LongStraw extends serving-oriented memory abstractions by making training pages compact physical allocations, read-only during replay, and invalid after adapted parameters change.A view into a larger parent chunk does not release memory.
  • Sparse attention and MoE: Sparse-attention training requires more than saved MLA latents: index reuse imposes producer-consumer lifetimes, while distributed selection needs candidate merging, value movement, and output composition.Local sparse selection is not global selection over a context-parallel prompt.
  • Sparse attention and MoE: GLM’s CP32 and EP32 placement keeps context and expert parallelism semantically distinct: CP preserves attention, while EP handles routed-token computation.Neither collective can replace the other.

11 Conclusion

LongStraw frames long-context GRPO under fixed GPU budgets as a tensor-lifetime and ownership problem, establishing architecture-aware execution envelopes for Qwen and GLM. The results also expose incomplete gradient composition and detached prompt state, so they do not establish complete training correctness.

  • Fixed-budget execution: LongStraw fixes the budget at eight H20 GPUs for Qwen and 32 H20 GPUs for GLM, targeting GRPO-shaped execution without adding accelerators.The conclusion characterizes the systems problem as fitting GRPO execution within a fixed accelerator inventory.
  • Architecture-aware execution: Compact KV pages, recurrent GDN state, and global LSE/output reduction make the dense-hybrid Qwen path fit across CP8.For GLM, CPU-resident MLA and indexer-key pages, one-layer staging, complete-layer checkpointing, IndexShare reconstruction, and CP32/EP32 placement support the response backward path.
  • Capacity results: Qwen completes a 4.25M G = 8 response replay within the same eight-H20 envelope.Under prefix-frozen response-only parameterization, the conclusion also reports eight consecutive G = 8 optimizer steps comprising 64 member replays at a peak of 83.894 GB per rank.
  • Limitations: Qwen lacks shard-local K/V adapter-gradient composition, while the historical GLM receipt uses CP-local DSA and bypasses Megatron gradient finalization for CP-replicated non-expert adapters.Both paths detach the prompt state, leaving a budget-conditioned execution envelope rather than complete training correctness.
  • Capacity results: 2,097,152-position prompts pass through two complete 78-layer GLM backwards and terminal optimizer calls on 32 H20 GPUs.Recorded capture-window peak allocation ranges from 112.571 to 145.148 GB per rank, and the GLM point is not an OOM-derived frontier.

12 Limitations and Validation Roadmap · A Model and Run Configuration

LongStraw’s receipts establish execution capacity, not complete correctness, training validity, cost optimality, or useful behavior. The roadmap prioritizes distributed-gradient, attention, prompt-state, rollout, and resource validation, while the GLM configuration remains partially unbound.

  • 12.1 Distributed Gradient Composition Is Incomplete: The receipts do not support stronger correctness or training claims because distributed gradient composition remains incomplete in both Qwen and GLM paths.Qwen leaves page-owner dK/dV local and omits gradient reduction before independent AdamW calls; GLM bypasses normal gradient finalization and synchronization.
  • 12.6 Validation Order: Validation should proceed in dependency order: repair and verify gradients, establish global DSA parity, compare full-sequence gradients, run real rollouts, then repeat and extend capacity measurements.The roadmap specifies 32K–64K reference checks, sampling through checkpoint reload, and a direct > 2M GLM sweep with memory, variance, and phase-separated timing.
  • 12.2 The Historical GLM Receipt Uses CP-Local Response Attention: The historical GLM receipt uses CP-local top-2048 response attention rather than distributed DSA over 2,097,152 global context positions.The current tree adds global candidate selection, selected-value movement, and sparse-output composition, but still requires layerwise parity against an unsharded reference.
  • 12.3 The Prompt-State Gradient Is Detached: Both paths condition response gradients on detached prompt state, omitting Equation 3’s second term and requiring full-sequence parity across every LoRA target family.The proposed reference comparison fits at 32K or 64K and must include all LoRA gradient components, not only selected loss or attention projections.
  • 12.4 The Workload Is an Execution Probe: The workloads are execution probes using synthetic responses and deterministic rewards, without sampling, environment or reward-model execution, checkpoint reload, repeated updates, or policy-improvement measurement.The Qwen probe uses β = 0 and an unclipped live ratio; GLM uses β = 0.01, but first-step ratios make clipping inactive and KL zero.
  • 12.4 The Workload Is an Execution Probe: The exercised scale exceeds native context settings of 262,144 for Qwen and 1,048,576 for GLM, yet no task evaluation establishes useful behavior at this systems scale.The supplied configurations therefore do not themselves demonstrate task-level utility at the reported execution lengths.
  • 12.5 Resource and Reproducibility Gaps: The fixed-budget claim covers accelerator count and device-memory measurements, not total cost, while timings lack matched baselines, broad group-size sweeps, variance, and whole-transaction accounting.The artifacts omit host memory, network traffic, utilization, energy, and monetary cost; Qwen and GLM also report different peak-allocation scopes.
  • A Model and Run Configuration: The canonical configuration binds each architecture to an inspected snapshot and execution receipt, but the final GLM manifest leaves learning rate, checksums, seeds, software versions, and model revision unbound.Earlier environment snapshots are excluded from Table 10 rather than promoted into the canonical receipt.

B GLM Page Mapping and State-Size Derivation · C Representative GLM Trace Contract · D Distributed-Gradient Audit

The GLM implementation maps 2.1M-token prompts across 32 context-parallel ranks, derives CPU state-storage requirements, and specifies representative replay traces. The distributed-gradient audit identifies missing gradient-finalization and composition paths that limit training correctness.

  • B GLM Page Mapping and State-Size Derivation: 32 ranks split 32,768 global pages into 2C contiguous chunks, with each rank owning chunks r and 63 − r.For C = 32 and 64-token pages, each chunk contains 512 pages.
  • B GLM Page Mapping and State-Size Derivation: 72 MiB is the derived footprint of one local MLA layer, while one local DSA index-key set occupies 16 MiB across 78 and 21 states.These values are derived from BF16 page shapes, not measured memory peaks.
  • B GLM Page Mapping and State-Size Derivation: 186 GiB is the derived collective CPU payload; staged shared-index and index-computing layers require 72 MiB and 88 MiB prompt payloads, respectively.The figures exclude metadata, transfer buffers, weights, response tensors, and allocator overhead.
  • C Representative GLM Trace Contract: The representative policy trace starts with response hidden shape [2, 1, 6144] and records 1,024 pages, local prefix 65,536, total local length 65,538, and top-k shape [1, 2, 2048].Compute layers record both state components, CPU offload, whole-layer checkpointing, and runtime_unfused_absorbed; IndexShare consumers retain MLA state only.
  • C Representative GLM Trace Contract: 1,394 parameters disable gradients during capture, while 99 state hooks cover 78 MLA and 21 DSA states; one shared RoPE cache replaces 98 copies.Complete decoder layers use reentrant checkpointing with RNG preservation, and saved sparse tensors may move to CPU.
  • D Distributed-Gradient Audit: The audit finds missing selective K/V and upstream-hidden composition for Qwen and missing finalization for CP-replicated attention, dense/shared, and output-head adapters in GLM.Routed-expert adapters avoid this issue because EP32 and ETP1 give expert data-parallel size one; traces lack per-module hook coverage and gradient diagnostics.
  • D Distributed-Gradient Audit: The GLM custom resident group loop performs two local backward calls and optimizer steps while bypassing finalize_model_grads and DDP finish_grad_sync.With gradient overlap disabled, hooks accumulate only local main_grad, although the distributed optimizer assumes reduction already occurred.

E Detailed GLM Capacity Progression

The GLM capacity progression moved from diagnosing independent memory peaks to increasingly complete million-token execution canaries. Coverage increased monotonically, but semantic fidelity remained limited by detached or skipped gradient-finalization boundaries.

  • Memory failures: 64 GiB was required by the FP32 DSA score scratch for a 2,097,152-position full-sequence attempt, while expert-LoRA, FC2, and concatenation allocations exposed additional peaks.Expert-LoRA scale/add reached 7.80 GiB, FC2 matmul reached 11.70 GiB, and 65,536-token chunk concatenation requested 19.37 GiB.
  • Capacity milestones: 675.657 s completed the final 2,097,152-position prefix-only all-layer no-grad prompt capture after milestones at 128K, 256K, 512K, and 1M.The progression also included a 2.097M layer-0 capture, two local backwards, and an optimizer-call canary in 738.579 s.
  • Capacity milestones: 2042.975 s produced a 2.097M G = 1 all-78-layer canary after all-layer CP32 gates at 32K and 64K.The CP32 gates followed IndexShare, CPU page, shared RoPE, and checkpoint fixes.
  • Capacity milestones: 2975.138 s completed the fresh G = 2 rank-complete transaction, extending execution coverage beyond the 2.097M G = 1 canary.The final two milestones retained the same CP-local DSA and skipped gradient-finalization boundaries.

F Qwen 4.25M Replay within Eight H20s

Within eight H20s, LongStraw completes Qwen replay at 4,456,448 exact positions and also demonstrates eight optimizer steps for a prefix-frozen response-only objective. The prompt-adapted path still OOMs during policy backward, so these results establish execution capacity for specified scopes rather than the original objective.

  • Resident replay: 4,456,448 exact positions: the resident G = 8 run captures 4,448,256 prompt positions once and completes all eight old/reference/policy branches plus four backward blocks per policy branch.It takes 21,750.133 s from prefix start through G = 8 and reaches 82.960 GB per rank.
  • Prefix-frozen response-only: 8 optimizer steps: the prefix-frozen response-only run completes eight G = 8 accumulation cycles, totaling 64 member replays.Peak allocation rises from 82.960 GB on the first cycle to 83.894 GB thereafter, with optimizer_step_applied=true and prefix_stale=false on every rank.
  • Batched scoring: 0.7%: batched old/reference scoring reduces post-prefix time from 4051.240 to 4024.290 seconds at 4,456,448 context, while peak memory rises from 83.894 to 135.128 GB.At 4K, the one-GPU G = 8 total falls from 17.529 to 14.528 seconds.
  • Prompt-adapted limitation: 4,456,448 positions: the clean prompt-adapted path completes prefix capture, old/reference scoring, and policy stage-1 forward before OOMing during response backward.Detached-prefix gradient-page pruning is exact for the detached-prefix objective, but the response-only evidence is not evidence for the original prompt-adapted QLoRA objective.
  • Capacity scope: 4,538,368 positions: a train-block proxy passes at that size and OOMs one 4,096-position chunk later at 4,542,464.These train-block probes are capacity checks, while plotted points represent different execution scopes and are not a fitted memory law.
Loading 2607.14952v1…