PaperlayerView exact revision on arXiv

Source-linked AI summary

Compass: Dissecting Communication and Computation Operators for Efficient LLM Training

Guangyu Xiang, Lin Zhang, Haoxuan Yu, Xinglin Pan, Shaohuai Shi, Xiaowen Chu

arXiv:2609.10549v1cs.PF

TL;DR

Compass addresses communication bottlenecks in tensor-parallel LLM training, where existing fusion underuses hybrid-network topology and decomposition lacks principled degree selection. It combines topology-aware IntraFusion, analytical InterDecom modeling, and unified strategy selection, achieving up to 1.42× end-to-end speedup over Megatron-LM.

  • Problem

    Existing IntraFusion methods underuse hybrid network topology, while InterDecom lacks analytical guidance for choosing its decomposition degree.

  • Method

    Compass combines topology-aware ring-based IntraFusion, an overhead-aware model for optimal InterDecom degree selection, and a unified strategy-selection framework.

  • Results

    Compass consistently selects the optimal strategy and achieves up to 1.42× end-to-end speedup over the Megatron-LM baseline across 288 test cases.

  • Takeaways & Limitations

    Systematic topology-aware optimization and performance modeling can improve communication-computation overlap across varying workloads and hardware configurations.

  • Takeaways & Limitations

    Performance remains sensitive to workload, hardware topology, and decomposition degree, making fixed strategies or heuristic choices unreliable.

Abstract

from arXiv · show

Overlapping communication and computation operators is a common practice to hide communication overheads, accelerating large language models (LLMs) training on GPU clusters. Existing systems achieve this through either intra-operator fusion (IntraFusion), which packs operators into a single large kernel, or inter-operator decomposition (InterDecom), which splits a tensor into multiple parts for pipelined execution. However, current IntraFusion methods underutilize network topology, causing suboptimal bandwidth usage on multi-GPU systems, while InterDecom struggles to determine the optimal number of decomposed parts for peak performance. To address these issues, we introduce Compass, which employs systematic optimization and comprehensive modeling. First, we design a novel IntraFusion algorithm leveraging double-ring communications to maximize bandwidth utilization in hybrid NVLink-PCIe systems, achieving 1.5x-2.5x speedups. Second, we develop a decomposition model that mathematically derives the optimal tensor decomposition degree for InterDecom, improving performance by up to 1.3x. Finally, we develop a unified performance framework that accurately determines the best strategy for different scenarios. We validate Compass through extensive evaluation across 288 configurations and end-to-end experiments on real-world applications. The results demonstrate that Compass consistently selects the optimal strategy, achieving up to a 1.42x end-to-end speedup compared to the Megatron-LM baseline.

I. INTRODUCTION

Tensor parallelism introduces frequent communication that can consume 40–75% of execution time, motivating overlap strategies that hide latency. Compass combines topology-aware fusion, analytically selected decomposition, and unified strategy selection, achieving substantial operator- and end-to-end speedups.

  • I. INTRODUCTION: Tensor parallelism reduces model memory through partitioning but introduces frequent collective communication that can dominate execution time.AllGather and ReduceScatter create fine-grained dependencies, leaving GPUs stalled during naive sequential execution.
  • I. INTRODUCTION: Existing overlap methods use either IntraFusion, which reorders GEMM tiles within one operator, or InterDecom, which pipelines d decomposed sub-operators.Both approaches divide communication and computation into smaller interleaved units, but they expose different performance trade-offs.
  • I. INTRODUCTION: IntraFusion can underperform on hybrid interconnects, while InterDecom lacks principled guidance for selecting its decomposition degree and no framework automatically chooses between them.These limitations make the best strategy dependent on workload and hardware characteristics.
  • I. INTRODUCTION: Compass introduces topology-aware single- and double-ring IntraFusion, an overhead-aware decomposition predictor, and a unified framework for selecting the faster strategy.The design targets hybrid NVLink-PCIe systems and replaces heuristic decomposition tuning with analytical modeling.
  • I. INTRODUCTION: Up to 1.42× end-to-end speedup over Megatron-LM was achieved across 288 test cases using Compass’s selected strategy.The evaluation also reports 1.5×–2.5× operator-level speedups for TA-IntraFusion and up to 1.3× for ODP-InterDecom.

B. Computation-Communication Overlapping Strategies

Computation-communication overlap uses either IntraFusion, which reorders GEMM tiles within a fused operator, or InterDecom, which pipelines partitioned sub-operators. Their practical performance depends on hardware topology and decomposition degree, making reliable selection difficult.

  • IntraFusion: IntraFusion reorders GEMM computation tiles inside a fused communication-computation operator to conceal communication latency.In AllGather+GEMM, locally available data is processed first; in GEMM+ReduceScatter, the final local computation is scheduled to overlap ongoing transfers.
  • InterDecom: InterDecom partitions computation and communication into d independent sub-operators that are pipelined, but startup and drain phases leave some communication exposed.Sequential execution is the special case d = 1.
  • IntraFusion limitation: Flux’s hybrid-interconnect fusion underperforms NCCL, reaching only 25% of NCCL effective bandwidth with 4 GPUs and 50% with 8 GPUs.Its AlltoAll algorithm congests lower-bandwidth cross-NUMA links because every GPU communicates simultaneously with all others.
  • InterDecom limitation: InterDecom’s decomposition degree is difficult to tune because concurrent sub-operators contend for shared GPU resources, so increasing d does not necessarily improve performance.The supplied passages identify resource contention as a significant overhead but do not provide a separate quantitative value for that mechanism.
  • InterDecom limitation: Across 72 workloads, a suboptimal decomposition degree incurred an average 57.44% penalty and reached 281.90%, while the default d = 4 was optimal in only 30% of cases.These results motivate analytical prediction rather than empirical tuning or fixed heuristics.

3) The Strategy Selection Limitation:

The best overlap strategy depends on workload and hardware, motivating topology-aware communication and systematic strategy selection. Compass’s topology-aware design uses ring algorithms and pipelined fusion to exploit hybrid interconnects while overlapping communication with computation.

  • 3) The Strategy Selection Limitation:: IntraFusion was superior in 78 of 144 workloads, while InterDecom was superior in the remaining 66, showing no universal strategy winner.The benchmark compares AllGather+GEMM across diverse workloads.
  • A. Topology-Aware Communication Algorithm: TA-IntraFusion uses single and double rings matched to hybrid hardware, addressing the topology dependence that makes strategy choice workload-specific.The single ring targets cross-NUMA communication, while the double ring targets PCIe-NVLink connectivity within one NUMA domain.
  • A. Topology-Aware Communication Algorithm: The cross-NUMA single ring serializes communication across 7 steps while restricting NUMA-boundary traffic to links 0–7 and 3–4.With eight GPUs each holding data of size m, each step transfers a chunk of size m/8.
  • A. Topology-Aware Communication Algorithm: The double ring concurrently routes forward and backward traffic over NVLink and PCIe, yielding (8/3)B_PCIe versus (4/3)B_PCIe for a single ring.The passage reports a 2x improvement over a single ring and nearly 4x over standard AlltoAll on this topology.
  • B. Mechanics of TA-IntraFusion: TA-IntraFusion partitions AllGather output into forward- and backward-ring components, then triggers GEMM as remote partitions arrive in a rolling-wave pipeline.DMA engines, push-based execution, and data-readiness scheduling support asynchronous overlap; the GEMM+ReduceScatter case reverses the dependency and schedules local computation last.

IV. ODP-INTERDECOM: OPTIMAL DEGREE PREDICTOR

ODP-InterDecom models decomposition execution time with both overlapped stages and non-overlapped startup and drain phases, while accounting for degree-dependent coordination overhead.

  • ODP-InterDecom: Optimal Degree Predictor: The optimal decomposition degree balances overlap potential against increasing overhead from resource contention as d grows.Smaller d limits overlap, whereas larger d increases overhead and produces diminishing returns.
  • ODP-InterDecom: Optimal Degree Predictor: The execution model separates overhead-affected overlapped stages from a final non-overlapped portion covering pipeline startup and drain.The startup and drain process the first and last sub-operators, whose computation and communication times are summed.
  • ODP-InterDecom: Optimal Degree Predictor: The model represents each overlapped stage as the larger of computation time and communication time per decomposed part.For each of the d −1 overlapped stages, Tstage = max(Tcomp/d, Tcomm/d).
  • ODP-InterDecom: Optimal Degree Predictor: 5.43 ms to 0.96 ms: incorporating degree-dependent overhead reduces InterDecom prediction MAE compared with an overhead-agnostic model.The overhead-aware model captures performance degradation that grows with decomposition degree.

C. Optimal Decomposition Degree Selection

Compass analytically selects the decomposition degree that minimizes execution time, comparing the derived candidate against sequential execution before applying decomposition.

  • C. Optimal Decomposition Degree Selection: The execution-time model is strictly convex, guaranteeing a unique minimum for decomposition degrees d > 1.Within each regime, the model has form f(d) = Ad + B + C/d with positive A and C.
  • C. Optimal Decomposition Degree Selection: The final degree selection retains the candidate only when its predicted execution time beats the sequential baseline d = 1.Otherwise, Compass selects d = 1.
  • C. Optimal Decomposition Degree Selection: The predictor calibrates overhead coefficients offline and computes the workload’s computation-to-communication ratio online.The power-of-two rounding function constrains the candidate degree, while λ = Tcomp/Tcomm is calculated from workload characteristics.
  • C. Optimal Decomposition Degree Selection: Compute-bound workloads favor smaller decomposition degrees, and higher overlap overhead likewise lowers the preferred degree.Smaller d preserves GEMM efficiency by reducing fragmentation.

A. Problem Formulation

Compass formulates strategy selection as comparing overhead-aware execution models for fusion and decomposition, with fusion modeled around ring-aligned chunks and synchronization costs.

  • A. Problem Formulation: Compass selects between IntraFusion and InterDecom by minimizing their predicted execution times for each workload and hardware topology.The decomposition prediction uses its analytically derived optimal degree d∗.
  • A. Problem Formulation: IntraFusion preserves one operator and reorders internal tiles, whereas InterDecom partitions operators into d pipelined sub-operators.The two strategies therefore incur different overhead sources.
  • B. IntraFusion Model: The fusion model uses nchunks logical pipeline steps aligned with ring communication, with double-ring chunks split into forward- and backward-ring components.nchunks is typically set to the number of GPUs.
  • B. IntraFusion Model: The fusion execution model combines overlapped pipeline stages, one exposed computation chunk, and internal synchronization overhead.Communication is hidden during nchunks −1 stages, while Tfusion overhead captures coordination at synchronization points.
  • B. IntraFusion Model: Fusion coordination overhead scales with both the number of synchronization points and message volume.The model uses (nchunks −1) · (αfusion + βfusion · m), with m defined by the collective operation.
  • B. IntraFusion Model: 10.87 ms to 2.66 ms: the overhead-aware fusion model reduces prediction MAE by more than 4x versus an overhead-agnostic baseline.This validates additive coordination overhead as part of the fusion performance model.

C. Adaptive Strategy Selection Algorithm

Compass uses closed-form performance models and calibrated overhead parameters to compute the optimal decomposition degree, predict both strategies, and select the faster one at runtime.

  • C. Adaptive Strategy Selection Algorithm: The adaptive algorithm selects the strategy with the lower predicted execution time for a given workload-topology pair.It returns both the selected strategy and predicted execution time.
  • C. Adaptive Strategy Selection Algorithm: Compass first computes base computation and communication metrics and their ratio from workload and topology parameters.These quantities provide inputs for degree calculation and strategy-time prediction.
  • C. Adaptive Strategy Selection Algorithm: The algorithm loads calibrated fusion and decomposition overhead parameters before analytically determining d∗.This replaces heuristic degree selection with model-based calculation.
  • C. Adaptive Strategy Selection Algorithm: Compass predicts Tdecomp at d∗ and Tfusion, then chooses TA-IntraFusion when Tfusion < Tdecomp and ODP-InterDecom otherwise.All predictions use closed-form calculations.
  • C. Adaptive Strategy Selection Algorithm: O(1) time complexity makes the strategy selector suitable for runtime decision-making in production environments.The constant complexity follows from closed-form performance predictions.

VI. EVALUATION

Compass is evaluated across 288 configurations, with experiments covering its operator optimizations and strategy-selection system. TA-IntraFusion substantially outperforms both Flux and sequential execution in representative tensor-parallel workloads.

  • Evaluation scope: Compass is evaluated both through individual operator benchmarks and through a complete strategy-selection system for end-to-end effectiveness.The evaluation first validates TA-IntraFusion and ODP-InterDecom, then assesses selection accuracy and practical gains.
  • Evaluation setup: 288 distinct test cases span representative distributed deep-learning configurations used to evaluate Compass and its individual optimizations.Experiments ran on an 8-GPU server with the software stack specified by the authors.
  • 1) Operator-Level Performance Comparison of TA-IntraFusion:: 1.5x-2.5x speedups are achieved by TA-IntraFusion for AllGather+GEMM across TP=4 and TP=8, outperforming Flux and sequential PyTorch+NCCL baselines.The evaluation also considers GEMM+ReduceScatter and uses normalized execution-time comparisons.
  • 1) Operator-Level Performance Comparison of TA-IntraFusion:: 1.5x-2.0x speedups are achieved for GEMM+ReduceScatter, with resource competition between GEMM and reduction computations slightly reducing overlap efficiency.The results still validate effective communication-latency hiding.

2) Optimal Decomposition Degree Selection:

Compass models decomposition degree selection and evaluates its adaptive strategy against fixed and true-optimal baselines, alongside end-to-end and communication micro-benchmarks. The model tracks workload-dependent optima closely, while Compass selects effective strategies across diverse cases.

  • 2) Optimal Decomposition Degree Selection:: The optimal decomposition degree varies with sequence length and tensor parallelism size, so a fixed d = 4 heuristic can be suboptimal.The analytical model dynamically predicts d* for each workload across all 288 test cases.
  • 2) Optimal Decomposition Degree Selection:: Over 1.2x speedups for TP=4 and nearly 1.3x for TP=8 are achieved at the largest sequence lengths over the fixed d = 4 baseline.The predicted decomposition degree performs nearly identically to the empirically true optimum across the evaluated workloads.
  • 1) Strategy Selection Performance:: 92.67% accuracy is achieved when Compass selects the optimal strategy across 288 test cases.Strategy choices can produce a 2x speedup over sequential execution in some cases, while the worst strategy can make execution 5 times slower.
  • 2) TA-IntraFusion Communication Micro-benchmark:: Bandwidth becomes comparable to or higher than NCCL as message size grows, although NCCL has an advantage for the smallest messages.The authors attribute this pattern to lower startup overhead for NCCL's SM copy mechanism and higher large-message utilization of the DMA copy engine.

VII. RELATED WORK

Related work addresses distributed-training communication through topology-aware algorithms, operator overlap, decomposition, and system-level scheduling. Compass combines topology-aware fusion, analytical decomposition selection, and unified strategy choice across these design spaces.

  • Communication primitives: Hybrid communication methods decompose communication into topology-aware stages to optimize heterogeneous interconnects.Tree-based and hybrid approaches emerged because early ring-based algorithms struggled at scale.
  • Communication-overlap strategies: IntraFusion reorders computation within one operator, whereas InterDecom pipelines sub-operators using decomposition degrees that are typically tuned heuristically.Prior IntraFusion systems often use topology-agnostic designs, while InterDecom methods rely on heuristic degree selection.
  • System-level scheduling: System-level methods improve training efficiency through tensor-fusion scheduling, model-component pipelining, or global task and dataflow management.The cited examples include ACP-WFBP, DualPipe, and Centauri.
  • Compass: Compass unifies topology-aware TA-IntraFusion, analytical decomposition-degree prediction, and dynamic strategy selection for different workloads and hardware configurations.The paper reports up to a 1.42x end-to-end speedup on real-world applications.
Loading 2609.10549v1…