LinguaLinked: A Distributed Large Language Model Inference System for Mobile Devices

• Centralized server inference raises privacy and bandwidth concerns, while mobile devices impose severe memory and processing constraints that quantization alone may not resolve.A full-precision Vicuna-13B requires over 52GB of runtime memory, and aggressive quantization can trade accuracy for reduced resource use.Source: §1 · p. 1
• LinguaLinked distributes LLM segments across trusted mobile devices so models exceeding one device’s capacity can run locally with less reliance on aggressive quantization.Each device handles only part of the computation and storage, while data and intermediate activations remain local.Source: §1 · pp. 2–3
• Optimized model assignment segments the LLM and uses linear optimization to match segments with device hardware and network conditions while reducing transmission overhead.The assignment also considers model-segment interdependencies and output volume.Source: §1 · p. 2
• Runtime load balancing monitors computational load, network conditions, and available resources, then redistributes segments from overloaded to underutilized devices.This strategy targets bottlenecks caused by changing device workloads and runtime conditions.Source: §1 · p. 2
• Optimized data transmission maps coordinate when and where segment outputs are exchanged while preserving the original model structure.The mechanism is designed to reduce data-transmission latency between devices.Source: §1 · p. 2
• 1.11× to 1.61× acceleration in single-threaded settings and 1.73× to 2.65× with multi-threading were achieved versus the baseline across quantized and full-precision models.Runtime load balancing additionally yielded 1.29× to 1.32× overall inference acceleration.Source: §1 · p. 2

2.1.1 Challenges in LLM Inference with Autoregressive De-

• Autoregressive LLMs generate tokens sequentially from preceding context, creating computational-efficiency and latency challenges for longer text.Centralized serving can additionally increase privacy exposure and latency when data travels to distant or congested servers.Source: §2.1.1 · p. 3

2.1.2 Mobile Device Constraints in LLM Deployment.

• Mobile devices offer lower bandwidth use and enhanced privacy, but limited computation and memory make LLM deployment difficult.Quantization, distillation, and pruning reduce model size but can affect performance and output quality when applied aggressively.Source: §2.1.2 · p. 3

2.2.1 Distributed LLM Inference on Mobile Devices.

• Partitioning LLMs into smaller segments distributes memory and computation across devices, enabling more devices to participate in inference.Prior distributed mobile work has largely focused on smaller computer-vision models, leaving a gap for LLM-specific strategies.Source: §2.2.1 · pp. 3–4
• Local deployment processes data near its source, reducing transmission over potentially untrusted networks and supporting privacy and security.This motivation complements distributed execution across mobile devices.Source: §2.2.1 · p. 3, §2.2.2 · p. 4

2.2.3 Scalability and Responsiveness in Distributed LLM De-

• Dynamic load balancing is needed because heterogeneous device capabilities and changing workloads can otherwise create bottlenecks and inconsistent performance.Redistributing tasks helps maintain resource utilization, scalability, responsiveness, robustness, and reliability.Source: §2.2.3 · p. 4

• The coordinator transforms the LLM into a computational graph, extracts and compiles subgraphs, and assigns sub-modules using device performance metrics.A primary optimizer assigns sub-modules, while a secondary optimizer overlaps selected sub-modules to facilitate load balancing.Source: §3 · p. 4
• Runtime load balancing reallocates tasks using changing device metrics, while optimized communication minimizes transmission during inference.Together these stages are intended to maintain system responsiveness.Source: §3 · p. 4
• Data and intermediate activations remain on mobile devices and are never sent to the server, preserving privacy during distributed execution.Source: §3 · p. 5

• The monitor measures bandwidth, latency, available memory, total memory, and device processing speed in FLOP/s.Memory usage and latency are measured directly, while bandwidth and FLOP/s are calculated from transmission and test-model execution times.Source: §3.1 · p. 5
• Bandwidth between devices is estimated by sending a predefined data chunk and dividing its size by the elapsed transfer time.The sender and receiver establish a direct connection before measurement.Source: §3.1 · p. 5
• Processing speed is estimated by deploying a small test model, recording execution time, and relating the model’s FLOP count to that time.Source: §3.1 · p. 5

3.2.1 Subgraph Extraction from LLMs.

• Subgraphs are extracted from the LLM computation graph at candidate-node boundaries, producing independent and coherent model segments for distribution.Candidate nodes are selected using graph degree conditions, and each subgraph spans between successive candidate-layer boundaries.Source: §3.2.1 · p. 5

3.2.2 Subgraph Dependency Search.

• Dependency search tracks both sequential dependencies between adjacent subgraphs and residual dependencies spanning non-adjacent subgraphs.The sequential dependency map records direct preceding-subgraph dependencies, while the residual dependency map records links to earlier non-adjacent subgraphs.Source: §3.2.2 · pp. 5–6

3.2.3 Model Assignment Optimization.

• The system profiles each compiled sub-module by FLOPs, memory requirement, and output size before optimizing its placement across mobile devices.The profiling data is combined with device performance metrics to initiate model-partitioning optimization.Source: §3.2.3 · p. 6
• The linear optimization minimizes total inference time by combining local computation time with inter-device data transmission time.The formulation models device-sub-module assignments, communication latency, bandwidth, and output sizes while enforcing per-device memory limits.Source: §3.2.3 · p. 6

3.2 Optimized Model Assignment

• Runtime load balancing is illustrated as a mechanism for redistributing model segments when device load or resource availability creates an inference bottleneck.The supplied figure material identifies runtime load balancing, while the section’s optimization formulation provides the operational context.Source: Figure 2 · p. 6

3.3.1 Load Balancing Optimization.

• Movable sub-modules can be dynamically loaded or offloaded across adjacent devices, whereas unmovable sub-modules remain permanently active on their assigned devices.Inactive modules remain on secondary storage until the load balancer triggers their loading into memory.Source: §3.3.1 · pp. 6–7
• The secondary load-balancing optimization refines the primary assignment by minimizing transmission and improving load distribution under device-memory constraints.It uses overlapping sub-module deployment and optimizes memory utilization while preserving fixed assignments for unmovable modules.Source: §3.3.1 · pp. 6–7, §3.3.2 · p. 7
• The load balancer uses runtime device metrics to decide when to trigger rebalancing after detecting an imbalanced load.The revised allocation is derived from the secondary linear program and its overlap variables.Source: §3.3.2 · p. 7

3.3.2 Model Deployment with Load Balancing.

• During deployment, devices transition sequentially to the new allocation after completing current work, while other pipeline devices continue operating.Pausing occurs at the individual-device level rather than across the entire pipeline, minimizing disruption during re-optimization.Source: §3.3.2 · p. 7

3.4.1 Decentralized Device Communication.

• Devices form a ring in which each device receives output, performs inference, and forwards results to the next device.The final device returns results to the Leader, completing the communication cycle.Source: §3.4.1 · pp. 7–8
• Each device alternates between ROUTER and DEALER roles, using asynchronous message queues to distribute and receive data.The communication pattern supports task allocation while devices occupy different positions in the sequence.Source: §3.4.1 · p. 7

3.4.2 Multi-Threaded Inference.

• Multi-threaded inference assigns tasks to independent threads, allowing devices to process subsequent work immediately after completing current tasks.Multiple sockets and ports avoid sharing non-threadsafe queue sockets and help limit communication bottlenecks.Source: §3.4.2 · p. 8
• Dynamic batch-size strategies divide large batches or assign different batch sizes across threads to improve processing flexibility.The system can adjust batch sizes according to the multi-threaded workload.Source: §3.4.2 · p. 8

3.4.3 Sequential & Residual Communication.

• Sequential communication forces residual data to travel through intermediate devices, imposing extra receiving and transmitting latency before reaching its target.Devices may carry unused data across the sequence when the destination is much later in the topology.Source: §3.4.3 · p. 8
• Residual communication uses dependency maps to route outputs directly to target devices and aggregates residual outputs sharing a destination.Shared sockets reduce channel-management complexity and communication overhead, while transmission order prioritizes nearby devices.Source: §3.4.3 · p. 8

4.1 Implementation and Methodology

• The evaluation uses BLOOM models from 1.1B to 3B parameters on Android devices, with text generation and classification workloads and baseline equal sub-module assignment.Experiments include full precision and int8 variants, high-end Google Pixel 7 Pro devices, and a low-end CUBOT X30.Source: Table 1 · p. 8, §4.1 · p. 9

4.2 Performance

• 1.11×–1.61× higher inference throughput results from optimized model assignment across quantized and full-precision BLOOM models.The largest evaluated model benefits most, with improvements increasing as model size grows.Source: §4.2 · pp. 9–10, Figure 4 · p. 10
• Multi-threading improves throughput for quantized and full-precision models in text generation and classification, reaching up to 2.65× over the single-thread baseline.The supplied results cover Figures 5 and 6, including BLOOM models across both tasks and precision formats.Source: §4.2 · pp. 10–11, Figure 5 · p. 10, Figure 6 · p. 11
• Multi-threading exhibits diminishing returns: the largest initial gain occurs from one to two threads, while additional threads continue improving performance less efficiently.The optimal thread count therefore must be determined empirically for computational efficiency.Source: §4.2 · p. 11
• Runtime load balancing reduces BLOOM 1.7b latency from 4.624 to 3.587 seconds per token at full precision and from 2.756 to 2.087 seconds per token when quantized.The measured acceleration is approximately 1.29× for full precision and 1.32× for quantized inference, while reloading relocated sessions is the primary overhead.Source: §4.2 · pp. 11–12, Figure 7 · p. 11

4.3 Sequential and Residual Communication

• Residual communication lowers delay by sending roughly 15 KB directly and in parallel, instead of sequentially forwarding roughly 1.5 MB activation-inclusive hops.Sequential communication requires multiple ordered hops, whereas residual transmission avoids waiting for preceding computation.Source: §4.3 · p. 12, Table 2 · p. 12

5 DISCUSSION

• LinguaLinked’s results show promising distributed LLM inference but underscore load-balancing overheads and current hardware and software framework constraints.The authors identify evolving ONNXRuntime GPU support, advanced quantization, and improved communication mechanisms as potential performance avenues.Source: §5 · p. 12
• Distributed fine-tuning could personalize models from user interactions and local data while preserving data privacy.The authors also envision extending LinguaLinked to multimodal models.Source: §5 · p. 12
• Future partitioning strategies should align task divisions more closely with device capabilities.The proposed direction is to further optimize computational graph partitioning and task division.Source: §5 · p. 12
• Future load-balancing algorithms should account for computational capacity, battery life, and user engagement patterns.This would extend balancing beyond computation alone toward a more holistic mobile-computing strategy.Source: §5 · p. 13
• Continuous intensive inference, especially with full-precision models, drains batteries and causes overheating that can degrade performance.The proposed responses include adaptive load modulation, thermal management, and low-power hardware-specific optimizations.Source: §5 · p. 13

6 CONCLUSION

• LinguaLinked introduces decentralized mobile LLM inference with optimized model assignment, network communication, and runtime load balancing.The system targets both full-precision and quantized LLMs of various sizes.Source: §6 · p. 13