PaperlayerView exact revision on arXiv

Source-linked AI summary

Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging

Yuanyi Wang, Yanggan Gu, Su Lu, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, Hongxia Yang

arXiv:2605.29489v1cs.LGeess.SY

TL;DR

At LLM scale, weight-space merging is limited by which expert weights must be read, motivating an explicit expert-access formulation. MergePipe plans and executes budgeted access to expert delta blocks while preserving the merge operator, and across Qwen and Llama workloads it substantially reduces read I/O and runtime with small representative parameter deviation and retained downstream behavior.

  • Problem

    At LLM scale, merging checkpoint families requires addressing expert-read I/O in addition to the algebraic rule for combining expert parameters.

  • Method

    MergePipe treats expert delta access as a budgeted mask, separates access planning from merge semantics, and provides budget-sound execution that recovers the full-read merge at full budget.

  • Results

    Across Qwen and Llama workloads, MergePipe achieves up to an order-of-magnitude expert-I/O reduction and up to 11× speedups, while representative budgeted merges retain downstream behavior.

  • Takeaways & Limitations

    Budgeting expert access turns full-read scans into bounded access and exposes a practical speed–fidelity frontier for weight-space merging.

  • Takeaways & Limitations

    The omitted-update fidelity bound applies to fixed-coefficient additive merges, while TIES and DARE require empirical fidelity evaluation; the method also assumes a shared weight coordinate system.

Abstract

from arXiv · show

Weight-space model merging is usually formulated as an algebraic operation on checkpoints, yet at LLM scale the limiting resource is often the set of expert weights that must be read. We introduce MergePipe, a budget-aware execution layer that casts LLM merging as an \emph{expert access-set} problem: given a merge operator and a checkpoint family in a shared weight coordinate system, choose which expert delta blocks to access under an explicit I/O budget. MergePipe indexes parameter blocks, builds deterministic access plans, and executes the induced budgeted merge with replayable manifests. The plan is budget-sound by construction and recovers the full-read merge at full budget; for fixed-coefficient additive operators, the omitted-update error is bounded by the norm of omitted deltas. Across Qwen and Llama merging workloads, MergePipe reduces expert-read I/O by up to an order of magnitude and achieves up to $11\times$ speedups. Representative budget sweeps show $O(10^{-3})$ parameter deviation from full-read merges and no monotonic degradation on downstream benchmarks.

1. Introduction

At LLM scale, model merging is constrained by the expert weights that must be read, not only by how deltas are combined. MergePipe makes expert access budgeted and separates physical access from merge semantics, recovering full-read merging at full budget while enabling bounded approximations.

  • Motivation: LLM checkpoint families make expert-read I/O a central execution concern because naive merging scans all expert parameters as the number of experts grows.This can make iterative merging I/O-bound rather than compute-bound.
  • Approach: MergePipe treats expert access sets as a budgeted resource distinct from the operator used to combine accessed deltas.It chooses an access mask under an expert-read budget and executes the induced mask-aware operator.
  • Approach: Full-budget execution recovers the standard full-read merge, while lower budgets expose an explicit speed–fidelity frontier.Omitted entries are represented by the access mask and do not trigger storage reads.
  • Contributions: MergePipe provides budget soundness, full-budget consistency, and an omitted-update bound for additive merges.The contribution applies the access-set abstraction to scalable weight-space merging.
  • Results: Across Qwen and Llama checkpoint families, MergePipe achieves up to order-of-magnitude expert-I/O reduction and 11× speedups while representative budget sweeps preserve downstream behavior.The reported results cover both execution efficiency and downstream behavior under bounded access.

2. Budgeted Access Sets

MergePipe reframes weight-space merging as selecting expert delta blocks under an explicit read budget, while preserving the requested merge operator. Full-budget execution recovers the full-read merge, and fixed-coefficient additive operators admit an omitted-update error bound.

  • Access-set formulation: MergePipe represents expert access with a binary mask A over expert, tensor, and block units, assigning each selected unit a physical read cost.The budgeted merge applies a mask-aware operator to the selected deltas while base reads and output writes remain checkpoint-boundary costs.
  • Access-set formulation: Budgeted execution is non-anticipatory: omitted delta contents are represented only by the mask and never trigger expert reads.This permits the executor to read only selected blocks while producing the same output for delta tuples that agree on selected entries.
  • Execution guarantees: Full-budget execution recovers the full-read merge, with pathwise equality for randomized operators when the same seed or mask is fixed.The invariant requires the planner and executor to use the same nonnegative read costs and remain within the expert-I/O budget.
  • Execution guarantees: Under a fixed absolute budget B and bounded average expert-read cost, the expert-read fraction decreases as O(1/K) as the expert count K grows.Full-read cost scales as K times the average expert-read cost, whereas the budget remains fixed.
  • Approximation guarantees: For fixed-coefficient additive merges, the omitted-update error is bounded using the norms of omitted deltas.This supports norm- or sketch-based access scores, but the fidelity bound does not apply to selected-only renormalization, TIES, or DARE.

3. MergePipe

MergePipe implements budget-aware merging as a catalog–plan–execute pipeline that exposes checkpoint blocks, selects deltas within budget, and streams only selected data. It records the resulting execution in a replayable manifest without changing the requested merge rule.

  • System pipeline: MergePipe returns a logical merged checkpoint and replayable manifest containing the access mask, plan hash, touched blocks, realized reads, and lineage.The pipeline consists of cataloging, planning, and execution stages.
  • Catalog: The catalog stores block sizes, layouts, hashes, sketches, and coverage hints so planning can avoid repeatedly scanning all expert checkpoints.These metadata support reusable cost estimation and access planning.
  • Planner: The planner ranks candidate expert deltas and selects blocks satisfying Cexpert(A) ≤ B, using deterministic tensor-level fallback when metadata are missing.It chooses which deltas are materialized but leaves combination to the requested mask-aware operator, recovering the standard operator at A = 1.
  • Executor: The executor streams base blocks in checkpoint order and materializes only selected deltas from checkpoints, explicit deltas, or LoRA-style adapters.For each block it applies the mask-aware operator and records touched blocks, contributing experts, and realized I/O.
  • Planning algorithm: PLANGEN builds and scores candidate expert-block pairs, sorts them deterministically, greedily fills the budget, and records operator parameters and the selected-block digest.Stable tensor/block tie-breaking makes the generated plan replayable.

4. Experiments

MergePipe bounds expert access during LLM merging, linking budgeted reads to runtime while preserving near-full-read quality in representative sweeps.

  • Scaling with the number of experts: Full-read expert I/O and wall time grow with the number of experts, whereas MergePipe keeps expert reads nearly flat under a fixed budget.The remaining cost shifts toward the unavoidable checkpoint boundary.
  • Budget-aware planning behavior: Realized expert reads, accessed-block fraction, and wall time increase monotonically as the requested budget grows while reads remain under the cap.The budget therefore exposes a direct throughput–fidelity control.
  • Operator generality: Order-of-magnitude expert-I/O reductions and up to 11× speedups are reported across AVG, TIES, and DARE on Llama-3.1-8B.MergePipe applies the same access-budget abstraction before local operator semantics.
  • Operator generality: At K=8, TIES I/O drops from 79.5GB to 3.46GB and wall time from 614s to 51s.At K=20, TIES still reads 3.46GB while the naive pipeline exceeds 174GB, with a 70.4% wall-time reduction.
  • Quality under bounded access: At 0.5 budget, relative ℓ2 deviation from the full-read TIES output remains O(10^-3) on Qwen3-0.6B with 20 experts.The budget is normalized to the full-read TIES endpoint, whose touched ratio is measured after TIES sparsification.
  • Quality under bounded access: HumanEval, IFEval, and DROP stay close to the full-read baseline and show no monotonic degradation in representative budget sweeps.Planning and metadata overhead are small compared with tensor streaming, but gains shrink for small expert sets, dense full-read regimes, or GPU-resident fusion.

5. Related Work

Prior work studies how models and checkpoints are merged, managed, and deployed; MergePipe instead focuses on which expert weights must be accessed during merging.

  • Model merging: Model-merging research combines checkpoints or task vectors using dense, sparse, interference-aware, low-rank, activation-informed, and sensitivity-informed methods.Examples include model soups, TIES, DARE, and related variants.
  • Model merging: Recent LLM work also examines post-merge feature calibration, quantization, continual post-training conflicts, and domain-specific expert composition.
  • Positioning MergePipe: MergePipe is complementary to these areas because it studies physical expert-weight access after checkpoints are in a shared coordinate system.Its focus is the execution layer rather than the merge rule or general checkpoint management.
  • Machine Learning Management Systems: Machine learning management systems address experiment tracking, artifact logging, versioning, workflow orchestration, and provenance for checkpoint-heavy development.These systems improve reproducibility and traceability.

6. Conclusion

MergePipe reframes scalable weight-space merging as budgeted expert access, separating physical reads from logical merge rules while retaining downstream behavior in representative merges.

  • Conclusion: MergePipe treats expert access as a first-class budgeted resource and separates the logical merge rule from its physical access pattern.This yields budget-sound and full-budget-consistent execution.
  • Conclusion: Across Qwen and Llama checkpoint families, bounded expert access produces large I/O and runtime reductions while retaining downstream behavior in representative budgeted merges.
  • Conclusion: The paper suggests pairing future weight-space methods with execution layers that treat model weights as structured, budgeted data rather than opaque checkpoint files.

A. Formalization and Algorithms

MergePipe formalizes budget-aware merging around executable expert access masks, combining block-level indexing, deterministic planning, mask-aware operators, and replayable manifests. Its cost model constrains expert reads while preserving operator semantics and explicitly handling nonlinear-operator limitations.

  • A. Formalization and Algorithms: MergePipe indexes checkpoints as block-level weight data, plans budget-feasible expert-delta access, executes a mask-aware merge, and publishes a replayable manifest.The manifest records the executable state needed to reproduce the logical merged checkpoint.
  • A.1. Cost Accounting and Operator Semantics: The total merge cost separates base reads, expert reads, output writes, and metadata, while the budget constrains only expert reads.The executor uses the same physical accounting unit for planning and realized reads; in uncompressed aligned runs, this is the stored byte length of a block.
  • A.1. Cost Accounting and Operator Semantics: DARE and TIES remain supported through masking, but no global smooth error bound is claimed because masks can alter random drops, sparsification, or sign election.The paper instead evaluates parameter deviation and downstream quality empirically for these nonlinear sparse operators.
  • A. Formalization and Algorithms: The runtime stores tensor and block metadata, touchmaps, cache-hit information, planner fallbacks, and manifest-backed committed state for deterministic execution and replay.The implementation view includes catalog analysis, block-level records, conflict-aware budget planning, tensor fallback, and a visibility/durability boundary.
  • A. Formalization and Algorithms: Figure 4’s runtime pipeline consists of cataloging, access-set planning, mask-aware execution, and manifest-based replay, with the executor streaming only selected expert blocks.The planning and execution sketches are presented as Algorithms 1 and 2.
  • A.1. Cost Accounting and Operator Semantics: For sparse plans, selected-only coefficients become zero for unaccessed expert blocks, whose logical zero entries do not trigger storage reads.Fixed-coefficient additive merges can therefore execute through a zero-completed tuple while retaining explicit access semantics.
  • A.1. Cost Accounting and Operator Semantics: Selected-only averaging requires a coefficient-drift bound, and renormalizing over selected experts invalidates the main omitted-delta bound.The paper distinguishes this case from fixed-coefficient additive operators.

A.3. Proofs

The proofs establish that MergePipe respects the expert-read budget and recovers full-read merging at full budget, with randomized equality conditioned on a shared seed or mask. They also derive a blockwise omitted-update deviation bound.

  • A.3. Proofs: Budget soundness follows when planner and executor use the same nonnegative read costs and the access mask satisfies the expert-read budget.The invariant directly connects planned cost and executed cost.
  • A.3. Proofs: Full-budget execution selects the all-ones access mask and equals the full-read merge blockwise.For randomized operators, equality is pathwise under the same seed or mask, and otherwise holds in distribution.
  • A.3. Proofs: The full-read expert cost scales as K times the average expert cost, while budgeted execution remains bounded by B.Dividing by the full-read cost yields the stated relative cost guarantee.
  • A.3. Proofs: The deviation proof compares each full-read block with its budgeted counterpart and bounds the block difference before aggregating across disjoint parameter coordinates.The triangle inequality supplies the per-block bound used in Equation (11).
  • A.3. Proofs: Atomic visibility ensures execution either publishes one snapshot/manifest pair or leaves no externally visible state.This property follows from the execution algorithm.

B. Additional Experimental Evidence

Additional experiments isolate the source of MergePipe’s gains: reducing expert reads rather than metadata effects, while checkpoint-boundary costs and streaming overhead remain comparatively fixed or small.

  • B. Additional Experimental Evidence: Baseline comparisons use the same merge operator with full-read expert access and disabled OS-level file caching.This setup isolates budgeted expert access from operator differences and operating-system cache effects.
  • B. Additional Experimental Evidence: Budget-controlled planning results are reported in Figure 3.The figure is identified as the main-text location for these planning experiments.
  • B. Additional Experimental Evidence: The I/O gains come from reducing expert reads, while base reads and output writes remain checkpoint-boundary costs and planning or transactional overhead stays small relative to tensor streaming.This decomposition attributes the savings to the expert-read term rather than metadata effects.

C. Limitations

MergePipe applies to shared-coordinate weight-space merging, not alignment problems, and lower-budget quality is empirical, especially for nonlinear sparse operators. Its largest benefits occur when disk-resident expert-read I/O dominates.

  • C. Limitations: Figure 5 compares planning, flush, commit, expert reads, base reads, and output writes to identify where budgeting changes execution cost.The caption states that tightening the budget primarily removes expert reads while base reads and output writes remain nearly fixed.
  • C. Limitations: MergePipe assumes experts share a common weight coordinate system and does not address permutation, symmetry, or representation alignment.It is complementary to behavior-level fusion methods rather than a replacement for them.
  • C. Limitations: Lower budgets are approximate, with only full-budget execution recovering the standard full-read merge.Quality therefore requires empirical evaluation, particularly for TIES and DARE.
  • C. Limitations: The benefits are largest when disk-resident expert-read I/O dominates the merge workload.This identifies the operational setting in which the system’s optimization has the greatest scope.
Loading 2605.29489v1…