PaperlayerView exact revision on arXiv

Source-linked AI summary

DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation

Paul Raj, Simon Caldwell, Daniel Whitmore, Mutlu Mete

arXiv:2610.36009v1cs.LGcs.AI

TL;DR

Sparse expert execution still leaves the states needed to adapt a mixture-of-experts model in storage. DuplexTune replays coupled hidden states through pretrained attention and experts using width bridges and staged adaptation. Its reported H800 configuration uses less peak memory than checkpointed SFT, while showing throughput and adaptation-quality tradeoffs.

  • Problem

    Sparse expert activation does not remove the weights, gradients, optimizer states, and activations that training must store.

    Source: §1 · p. 1

  • Method

    DuplexTune uses coupled-state replay through pretrained attention and expert modules, with bridge warm-up before backbone adaptation and router weights frozen.

    Source: §1 · p. 1

  • Results

    38.8 GB peak memory versus 64.8 GB for checkpointed SFT; DuplexTune reports higher MMLU, GSM8K, and MT-Bench scores but weaker multilingual performance.

    Source: §4.2 · p. 3, §4.3 · p. 4, §5 · p. 4

  • Takeaways & Limitations

    The reported H800 configuration combines reduced memory with throughput and adaptation-quality tradeoffs; further evaluation should measure matched-token quality, reconstruction error, and state residency.

    Source: §5 · p. 4

    Evaluation covers one backbone, one instruction dataset, and one GPU class; the H800 results do not establish single-consumer-GPU feasibility.

    Source: §4.5 · p. 4

Abstract

from arXiv · show

Sparse expert execution does not remove the stored states needed to adapt a mixture-of-experts language model. We present DuplexTune, a framework for replaying coupled hidden states through pretrained attention and expert modules. Trainable width bridges connect two residual streams to the backbone. Bridge warm-up precedes backbone adaptation, with router weights kept frozen. We analyze the implicit state-recovery equation and identify the convergence and replay conditions needed for reconstruction. On Qwen1.5-MoE-A2.7B, the single-H800 evaluation reports 38.8 GB peak memory, versus 64.8 GB for supervised fine-tuning with checkpointing, and throughput of 25.1 versus 19.9 samples/s. DuplexTune achieves 67.2% MMLU, 75.6% GSM8K, and 7.69 MT-Bench, with a weaker multilingual score than the checkpointed baseline. The results characterize a configuration-dependent tradeoff among state storage, throughput, and adaptation quality.

1. INTRODUCTION

Sparse expert computation still leaves substantial model and training states to store. DuplexTune adapts through coupled-state replay, and its reported configuration trades memory and throughput against adaptation quality.

1. INTRODUCTION

  • MoE activation is sparse, but training still stores weights, gradients, optimizer states, and activations; Qwen1.5-MoE-A2.7B has 14.3 billion parameters and activates about 2.7 billion.The introduction argues that active computation and stored-state costs need separate accounting.Source: §1 · p. 1
  • Prior approaches restrict trainable updates, quantize or shard model states, or recompute intermediate features through checkpointing.LoRA, QLoRA, ZeRO, FSDP, and checkpointing represent these approaches.Source: §1 · p. 1
  • DuplexTune uses two half-width residual streams linked to pretrained attention and expert modules, with bridge warm-up before backbone adaptation and router weights fixed.Backbone adaptation updates attention, expert weights, and bridges.Source: §1 · p. 1
  • 38.8 GB peak memory versus 64.8 GB for checkpointed SFT, with higher scores on three tasks but weaker multilingual performance.The comparison is for complete configurations.Source: §1 · p. 1
  • Because attention queries depend on the stream being recovered, replay requires implicit reconstruction, for which the paper states a sufficient convergence condition, residual error bound, and replay requirements.The reported evaluation compares complete configurations rather than isolating reconstruction speed.Source: §1 · p. 1

2. RELATED WORK

The related work spans expert routing and resource allocation, adaptation objectives, reconstruction methods, and memory systems. It distinguishes these approaches by their mechanisms and the costs they target.

2. RELATED WORK

  • Agent-level expertise allocation and token-level state recovery operate at different granularities, while adaptation objectives and coordination mechanisms differ from activation replay.The discussion groups work on routing, resource allocation, adaptation objectives, and optimization.Source: §2 · p. 1
  • Reversible-layer methods recover states through coupling or contraction conditions, whereas FlashAttention, ZeRO-Offload, and visual-token compression target different memory or computation costs.The paper distinguishes these mechanisms from retaining hidden states across layers.Source: §2 · p. 1
  • Figure 1 depicts width-lift and width-return bridges connecting paired states to pretrained attention and sparse experts, with router weights frozen during backbone adaptation.Its schematic sequence shows attention updating one stream before expert processing updates the other, followed by a channel join.Source: Figure 1 · p. 2
  • MMLU, GSM8K, and MT-Bench assess knowledge, mathematical reasoning, and conversational responses; multimodal and multi-agent studies are not memory baselines in this evaluation.The related-work discussion also mentions annotation and visual-reasoning benchmarks.Source: §2 · p. 2

3. COUPLED-STATE REPLAY WITH WIDTH BRIDGES

DuplexTune couples two half-width residual streams through pretrained attention and expert modules, using width bridges to connect them to the backbone. The section specifies conditional state-recovery requirements, a staged adaptation schedule, and resource tradeoffs.

3.1. Paired States and Width Bridges

  • Attention and expert modules operate on paired half-width streams connected by width bridges, but the stream interface has rank at most d/2 and must learn useful alignment.The pretrained matrix dimensions remain unchanged; this does not make the wrapper functionally identical to the original decoder.Source: §3.1 · p. 2

3.2. Conditional State Replay

  • State replay reconstructs the first stream as a fixed point, requiring contraction for a unique solution and matched parameters, masks, random states, and expert dispatch for backward replay.Approximate recovery perturbs gradient evaluation points, so reconstruction residuals and gradient agreement with a stored-activation reference are separate checks.Source: §3.2 · pp. 2–3

3.3. Bridge Warm-Up and Backbone Adaptation

  • Bridge warm-up updates only transitions before backbone adaptation trains transitions, attention, and expert weights while keeping router weights fixed.The schedule extends learning beyond bridges without updating every parameter.Source: §3.3 · p. 3

3.4. Resource Accounting

  • Replay can reduce stored layer-boundary states from O(LBSd) to O(BSd), but weights, optimizer states, attention workspace, temporary expert activations, and replay metadata remain.The single-H800 efficiency comparison uses capacity-tuned batches, and the evaluation does not separately profile activation savings.Source: §3.4 · p. 3, Table 1 · p. 3

4. EXPERIMENTS

The evaluation compares DuplexTune with parameter-efficient and broader-update methods on one MoE backbone, using capacity-adjusted batches. DuplexTune uses less memory than checkpointed SFT and scores higher on three tasks, but has lower multilingual performance; phase ablations and evaluation scope qualify these results.

4.1. Evaluation Setting

  • The evaluation adapts Qwen/Qwen1.5-MoE-A2.7B on databricks-dolly-15k using one 80 GB NVIDIA H800, with methods trained for the same steps and batch sizes adjusted to memory capacity.This measures achievable throughput for each configuration, not processed-token-matched throughput or isolated kernel efficiency; the multilingual dataset composition is unspecified and multi-seed uncertainty estimates are unavailable.Source: §4.1 · p. 3

4.2. Memory–Throughput Tradeoff

  • 38.8 GB peak memory for DuplexTune versus 64.8 GB for checkpointed SFT; throughput is 25.1 versus 19.9 samples/s under capacity-adjusted batching.DuplexTune also uses 3.1 GB less memory than LOMO and 6.0 GB less than GaLore, but is 29.3% slower than GaLore’s 35.5 samples/s; (IA)3 uses 17.6 GB and LoRA reaches 76.2 samples/s.Source: §4.2 · pp. 3–4

4.3. Adaptation Quality

  • DuplexTune has the highest adapted-model scores on MMLU, GSM8K, and MT-Bench, but scores 39.0 on multilingual evaluation, below checkpointed SFT (39.8), DoRA (39.1), and the base model (40.4).Against checkpointed SFT, its gains are 0.9 and 0.5 percentage points on MMLU and GSM8K, and 0.13 MT-Bench points; these are descriptive differences without statistical-significance estimates.Source: §4.3 · p. 4, Table 2 · p. 4

4.4. Training-Phase Ablation

  • 57.5% MMLU without alignment and 54.9% when learning is restricted to transitions, versus the complete schedule; both variants fall below the 62.4% base-model score.The results support using alignment and subsequent backbone updates together in this configuration, but do not establish divergence or explain the decline’s mechanism.Source: §4.4 · p. 4, Table 3 · p. 4

4.5. Scope and Reproducibility

  • The evaluation covers one backbone, one instruction dataset, and one GPU class; its H800 results do not establish single-consumer-GPU feasibility.Reproduction also requires learning rates, sequence lengths, per-method batches, inverse-solver tolerances, and multilingual task composition; memory comparisons depend on precision and state placement.Source: §4.5 · p. 4

5. CONCLUSION

DuplexTune combines coupled-state replay, width bridges, and staged backbone adaptation. In the reported H800 configuration, it reduces memory relative to checkpointed SFT and improves three task scores, with multilingual and throughput tradeoffs.

5. CONCLUSION

  • DuplexTune combines coupled-state replay, width bridges, and staged backbone adaptation.The reported H800 configuration reduces memory relative to checkpointed SFT and improves three task scores, while retaining multilingual and throughput tradeoffs.Source: §5 · p. 4
  • The conclusion recommends further evaluation of matched-token quality, reconstruction error, and state residency.State recovery requires convergence, and weight and optimizer costs remain.Source: §5 · p. 4
  • The reported results characterize a configuration-dependent tradeoff among state storage, throughput, and adaptation quality.Source: §5 · p. 4

Related papers

Loading 2610.36009v1…