Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections

1 Introduction

• GPU allocations change for elasticity, redeployment for maintenance or preemption, and recovery from failures that leave jobs with fewer devices.These changes can also make the original multi-dimensional parallelization strategy no longer optimal.Source: §1 · pp. 1–2
• Tenplex's state-management approach addresses the need to retrieve and transform job state with little system change while deploying new parallelization with low overhead.The introduction frames these as design questions for supporting dynamic multi-dimensional parallelism.Source: §1 · p. 2
• Tenplex externalizes dataset and model partitions in a parallelizable tensor collection (PTC), then transforms that state when GPU resources change.The PTC represents dataset partitions and partitioned model checkpoints; its partitioning depends on the job's data, pipeline, and tensor parallelism.Source: §1 · p. 2
• State transformations preserve dataset sample access order and repartition model layers and tensors into checkpoints for the new GPU devices.Tenplex exposes dataset state through a virtual file system and stores partitioned model checkpoints in a hierarchy that mirrors model tensors.Source: §1 · p. 2
• 24% less training time than approaches that scale only along data parallelism; reconfiguration is 43% faster than migrating all GPU state and 75% faster than maintaining state centrally.The state-change optimizations parallelize PTC transformations, minimize data movement, and overlap dataset-partition transfers with training.Source: §1 · pp. 2–3

2.1 Deep learning jobs with parallelism

• Data parallelism partitions training data while replicating the model; tensor parallelism splits model operators and parameters, and pipeline parallelism partitions the model into stages.Combining these strategies lets parallelizers navigate trade-offs in scalability, device utilization, and memory consumption.Source: §2.1 · p. 3

2.2 Need for dynamic resource changes

• Schedulers may change GPU allocations for elasticity, redeployment around maintenance or fragmentation, and recovery after failures.Jobs may resume with fewer GPUs, which can change their optimal parallelization configuration.Source: §2.2 · pp. 3–4

2.3 Challenges when changing GPU resources

• GPU changes challenge convergence because dataset samples must retain consistent order and hyperparameters such as batch size must remain consistent.With data parallelism, a fixed local batch size changes the global batch size when the GPU count changes.Source: §2.3 · p. 4
• Throughput differs by over 10× between the best and worst parallelization configurations on the same 16 GPUs.The configurations vary tensor, pipeline, and data parallelism; the best uses tensor parallelism within workers, while the worst uses slower inter-worker links.Source: §2.3 · pp. 4–5

2.4 Current approaches

• Existing approaches either do not support dynamic multi-dimensional parallelism or require users to implement state repartitioning and distribution manually.Virtual-device approaches support dynamic data parallelism but not runtime changes with multi-dimensional parallelism.Source: §2.4 · p. 5

• Tenplex targets consistent training, arbitrary multi-dimensional reconfiguration, and low reconfiguration overhead as GPU resources change.Source: §3 · pp. 5–6
• Tenplex externalizes model and dataset state in a PTC, then modifies it across GPUs after configuration updates, transparently to the DL system.Source: §3 · p. 6
• After a scheduler changes GPU allocation, Tenplex obtains a new parallelization configuration, computes a plan, and redistributes dataset and model state.Source: §3 · p. 6

4.1 PTC overview

• A PTC unifies dataset state, including its iterator, and model parameters as tensors; Tenplex maintains dataset state and retrieves model state from distributed checkpoints before reconfiguration.Source: §4.1 · p. 6
• PTC uses slicing, partitioning, and allocation functions to express multi-dimensional parallelization strategies.Slicing splits tensors; partitioning groups sub-tensors; allocation maps those groups to GPU devices.Source: §4.1 · pp. 6–7

4.2 Reconfiguration plan

• A reconfiguration plan is a minimal sequence that transforms the current PTC into a new one through split, re-partition, and merge operations, moving only necessary tensors.Source: §4.2 · p. 7

4.3 Expanding to new parallelism strategies

• PTC can represent expert parallelism by grouping expert tensors without slicing, and sequence parallelism by slicing data samples along the sequence dimension.Source: §4.3 · p. 8

5 Tenplex Architecture

• The State Transformer and Tensor Store work together to create updated job-state partitions after resources change.The transformer applies a plan for the new PTC; the Tensor Store holds model and dataset partitions.Source: §5 · p. 8

5.1 State Transformer

• Each worker retrieves model checkpoints, obtains a new parallelization configuration, transforms partitions, and restores the job from the new checkpoints.Transformer instances infer local operations from the old and new configurations and fetch required sub-tensors from local or remote stores.Source: §5.1 · p. 8

5.2 Tensor Store

• The Tensor Store organizes model and dataset state in memory and supports sub-tensor access, dataset repartitioning, and streamed data fetching.A dataset index tracks sample file locations and byte ranges; Tenplex prioritizes fetching samples from workers before using remote storage.Source: §5.2 · pp. 8–9

5.3 Fault tolerance

• After failures, Tenplex can resume with fewer GPUs and a new parallelization plan, using surviving model replicas when available.For more frequent failures, it can replicate model state across workers; infrequent checkpointing can still lose progress.Source: §5.3 · p. 9

5.4 Integration with existing training jobs

• Integration requires scheduler notifications, a parallelizer that supplies a configuration, DL-system state APIs, and training programs that use Tenplex’s data and checkpoint interfaces.Programs must also check for reconfiguration and terminate so Tenplex can restart them with transformed state.Source: §5.4 · pp. 9–10

6.1 Experimental setup

• Experiments use 16-GPU on-premise and 32-GPU Azure clusters, comparing Tenplex with elastic systems, a model-library approach, and Tenplex variants.The evaluation uses BERT-large, GPT-3 variants, and ResNet-50 with several datasets.Source: §6.1 · p. 10

6.2 Elastic multi-dimensional parallelism

• The elastic experiment changes GPU allocation among 16, 8, and 4 GPUs over a 538-minute GPT-3 XL training run.Scaling-event timing is based on the average from Microsoft’s Philly trace.Source: §6.2 · p. 10
• 298 mins versus 576 mins for Tenplex-DP and 548 mins for Torch: Tenplex reaches the same training step 46% sooner during elastic scaling.Tenplex scales tensor, pipeline, and data parallelism to configurations reported as achieving the best performance.Source: §6.2 · p. 10

6.3 Job redeployment

• 2.1×, 1.9×, and 2× higher redeployment time for Tenplex-Central than Tenplex on 1.3 B, 2.7 B, and 6.7 B models, respectively.The comparison is shown in Fig. 10.Source: Figure 10 · p. 11, §6.3 · p. 11
• The experiment redeploys GPT-3 jobs among different sets of 8 GPUs while keeping the parallelization configuration unchanged.It tests 1.3 B, 2.7 B, and 6.7 B models on the on-premise cluster.Source: §6.3 · p. 11
• Distributed state management lets workers migrate state directly, avoiding a single worker’s network bandwidth becoming a bottleneck.The baseline centralizes job state at one worker.Source: §6.3 · pp. 10–11

6.4 Failure recovery

• With 8 failed GPUs, Tenplex takes only 5% of the baseline recovery time; it is faster for failures of 4 or 8 GPUs when a model replica remains.The comparison is shown in Fig. 11.Source: §6.4 · p. 11
• Without a redundant model replica, Tenplex uses the last checkpoint and achieves only a slight recovery-time benefit over the baseline.The reported benefit comes from using local rather than remote storage for checkpoint recovery.Source: §6.4 · p. 11

6.5 Reconfiguration overhead

• For scaling up from 8 to 16 GPUs, Tenplex needs 24% less time than DeepSpeed and 10% less than Singularity; scaling down from 16 to 8 GPUs, it needs 64% and 43% less, respectively.The comparisons are shown in Fig. 12.Source: §6.5 · p. 11
• Figures 14 and 15 examine reconfiguration time with different parallelizations and cluster sizes.Source: Figure 14 · p. 12, Figure 15 · p. 12
• Tenplex achieves 430 images/s, close to Horovod’s 438 images/s; the evaluation concludes that Tenplex matches Horovod’s performance despite supporting dynamic reconfiguration.Horovod-Elastic achieves 418 images/s; Fig. 13 reports throughput in samples per second.Source: §6.5 · p. 12

6.6 Impact of parallelization type

• The evaluation compares Tenplex and Tenplex-Central while changing data, pipeline, or tensor parallelism from a shared starting configuration.For each change, the starting configuration is (4, 2, 1); the target varies by parallelism type.Source: §6.6 · p. 12
• For GPT-3 6.7 B, Tenplex-Central takes 4× longer under data parallelism, 3.5× longer under pipeline parallelism, and 3.7× longer under tensor parallelism than Tenplex.The comparison is attributed to the limited bandwidth of a single worker relative to distributed peer-to-peer reconfiguration.Source: §6.6 · p. 12
• With pipeline parallelism on the 1.3 B model, network bandwidth is not a bottleneck because reconfiguration does not split and merge sub-tensors.Source: §6.6 · p. 12
• Centralized state management becomes a bottleneck for many model parameters because one worker performs all state transformations with limited network bandwidth.Source: §6.6 · p. 12

6.7 Impact of cluster size

• The cluster-size experiment uses GPT-3 XL on Wikipedia in a 32-GPU cloud testbed, doubling resources and parallelism degree for each tested change.The GPU changes are 4 to 8 for data, 8 to 16 for tensor, and 16 to 32 for pipeline parallelism; Tenplex is compared with Tenplex-Central.Source: §6.7 · p. 12
• Reconfiguration time rises linearly with GPU count for data parallelism, but decreases with device count for pipeline and tensor parallelism.The model size stays constant for pipeline and tensor parallelism as network bandwidth increases with the number of devices.Source: §6.7 · p. 12
• Data parallelism has the highest reconfiguration time and pipeline parallelism the lowest; data parallelism moves more data, while pipeline parallelism only repartitions.Tensor parallelism also keeps the amount of data constant but must split and merge sub-tensors.Source: §6.7 · p. 13

6.8 Impact on model convergence

• Increasing or decreasing resources at training step 100 does not affect convergence under data, pipeline, or tensor parallelism.For data parallelism, Tenplex maintains consistent hyper-parameters and data order; Fig. 16 shows loss over training steps.Source: §6.8 · p. 13, Figure 16 · p. 13

Elastic ML systems

• Elastic ML systems generally adapt data parallelism, while Tenplex provides generic state abstraction for automatic reconfiguration across multi-dimensional parallelism.TDE supports only data-parallel changes; Varuna additionally supports pipeline parallelism with user-defined partitioning points, but not tensor or sequence parallelism; Spotnik and Hydrozoa support dynamic data parallelism only [65].Source: §7 · p. 13

Reconfiguration performance

• TDE and Varuna centralize state loading and saving, which can bottleneck reconfiguration of large model states [57] [4].Source: §7 · p. 13

Dynamic GPU scheduling

• Dynamic GPU schedulers such as Lyra change job GPU allocations but rely on elastic ML libraries for job reconfiguration, making them complementary to Tenplex.Source: §7 · p. 13

Checkpointing and DL job parallelization

• Checkpointing systems focus on failure recovery, whereas Tenplex supports checkpoint reconfiguration; existing DL systems and frameworks also lack runtime reconfiguration across multi-dimensional parallelism.The passage distinguishes checkpoint frequency, lossy compression, checkpoint placement, and built-in parallelization support, including PyTorch and TensorFlow data parallelism, JAX pmap and xmap, and MindSpore auto parallel search.Source: §7 · p. 13

Parallelization and resharding

• Unity and Alpa search for distribution plans that Tenplex uses in reconfiguration decisions, while XLA and MindSpore resharding operators act on a single GPU rather than a cluster.Source: §7 · p. 13

Distributed data analytics

• MapReduce and stream-processing elasticity do not provide the tensor-splitting abstraction needed for multi-dimensional DL parallelism; cited DL systems address only parts of that need.Elastic Memory supports some data-parallel DL jobs, while Cruise lacks abstractions for tensor splitting.Source: §7 · pp. 13–14

8 Conclusion

• Tenplex represents job state as a parallelizable tensor collection and generates efficient reconfiguration plans when GPU resources change at runtime.Distributed state transformers implement the plan on each GPU with minimum data movement between workers.Source: §8 · p. 14