PaperlayerView exact revision on arXiv

Source-linked AI summary

AI Flow: Perspectives, Scenarios, and Approaches

Hongjun An, Wenhan Hu, Sida Huang, Siqi Huang, Ruanjun Li, Yuanzhi Liang, Jiawei Shao, Yiliang Song, Zihan Wang, Cheng Yuan, Chi Zhang, Hongyuan Zhang, Wenhao Zhuang, Xuelong Li

arXiv:2506.12479v3cs.AIcs.CLcs.CVcs.DCeess.SP

TL;DR

Ubiquitous AI faces substantial resource demands, and practical deployment across heterogeneous platforms remains a challenge. AI Flow combines device-edge-cloud collaboration, familial models, and networked model interaction; its reported emergent intelligence surpasses the capacity of any single model.

Abstract

from arXiv · show

Pioneered by the foundational information theory by Claude Shannon and the visionary framework of machine intelligence by Alan Turing, the convergent evolution of information and communication technologies (IT/CT) has created an unbroken wave of connectivity and computation. This synergy has sparked a technological revolution, now reaching its peak with large artificial intelligence (AI) models that are reshaping industries and redefining human-machine collaboration. However, the realization of ubiquitous intelligence faces considerable challenges due to substantial resource consumption in large models and high communication bandwidth demands. To address these challenges, AI Flow has been introduced as a multidisciplinary framework that integrates cutting-edge IT and CT advancements, with a particular emphasis on the following three key points. First, device-edge-cloud framework serves as the foundation, which integrates end devices, edge servers, and cloud clusters to optimize scalability and efficiency for low-latency model inference. Second, we introduce the concept of familial models, which refers to a series of different-sized models with aligned hidden features, enabling effective collaboration and the flexibility to adapt to varying resource constraints and dynamic scenarios. Third, connectivity- and interaction-based intelligence emergence is a novel paradigm of AI Flow. By leveraging communication networks to enhance connectivity, the collaboration among AI models across heterogeneous nodes achieves emergent intelligence that surpasses the capability of any single model. The innovations of AI Flow provide enhanced intelligence, timely responsiveness, and ubiquitous accessibility to AI services, paving the way for the tighter fusion of AI techniques and communication systems.

I. INTRODUCTION · A. The Rise of AI

This section situates AI Flow amid the rise of generative AI and ubiquitous intelligence, outlining resource and communication barriers and the limits of current models in complex collaboration. It also traces the development of LLM-based multi-agent systems for distributed collaboration and the trend toward intelligence at the network edge.

I. INTRODUCTION

  • Large-model demands for computation, storage, and power hinder deployment on constrained IoT and mobile devices, while high-dimensional data transmission creates communication bottlenecks.Source: §I · p. 2
  • AI Flow enables intelligence to emerge across hierarchical networks through inter-agent and human-agent interactions, with model collaboration surpassing any single model’s capability.Source: §I · pp. 3–4
  • AI Flow uses a unified device-edge-cloud architecture to integrate end devices, edge servers, and cloud clusters for scalable, low-latency inference that minimizes communication bottlenecks.Source: §I · p. 3
  • Familial models are feature-aligned, multi-scale architectures that enable knowledge transfer without extra middleware and improve inference efficiency under constrained bandwidth and compute.Source: §I · pp. 3–4

A. The Rise of AI

  • Generative AI moves beyond specialized tasks by synthesizing novel content through open-ended pretraining, while Transformer self-attention addresses long-range dependencies that hinder RNNs and CNNs.Source: A. The Rise of AI · p. 4
  • Large VLMs combine vision encoders with LLMs, using a lightweight adapter to align visual features with the LLM embedding space and transfer visual information.Source: A. The Rise of AI · p. 5
  • Despite content generation and visual understanding, complex reasoning, collective decisions, and multi-agent collaboration remain limited; CoT [22] adds intermediate steps, while RL aligns outputs with human preferences.Source: A. The Rise of AI · p. 5
  • Emergent intelligence from group dynamics has inspired LLM-based multi-agent systems (MASs), for complex real-world problems requiring multi-stage reasoning.Source: A. The Rise of AI · p. 5
  • Heterogeneous agents adaptively collaborate through distributed networks toward common goals, achieving intellectual capacity that outperforms individual agents.Source: A. The Rise of AI · p. 5
  • A key application trend is ubiquitous intelligence services at the network edge, bringing intelligence closer to users.Advances in model capability have enabled a surge of applications across numerous fields.Source: A. The Rise of AI · p. 5

B. Towards Ubiquitous AI Applications … 2) Communication Network Constraints:

Ubiquitous AI applications depend on responsive services across diverse environments, but hardware limitations and communication constraints impede their deployment. AI Flow addresses these challenges through hierarchical networks and coordinated model interaction to support efficient, scalable inference.

B. Towards Ubiquitous AI Applications

  • Ubiquitous AI services are intended to integrate into everyday environments and diverse sectors, with responsive and adaptable service delivery.The passage identifies smart cities, industrial automation, and autonomous systems as application areas.Source: B. Towards Ubiquitous AI Applications · p. 6
  • Collaborative edge agents coordinate in real time, share information, negotiate roles, fuse multimodal data, and adapt to changing environments, but face network and onboard-compute limits.Heterogeneous agents with complementary capabilities also require decentralized control.Source: B. Towards Ubiquitous AI Applications · p. 6

C. Challenges

  • Hardware resource limits and communication network constraints jointly hinder the scalability, responsiveness, and sustainability of emerging AI applications.Source: C. Challenges · p. 7

1) Hardware Resource Limitations:

  • Advanced large models can have inference-phase memory footprints of tens to hundreds of GB, far exceeding typical end-device capacity (∼4–32 GB); model computational complexity also escalates with scale, undermining latency-sensitive applications on low-end devices.Source: 1) Hardware Resource Limitations · p. 7
  • Efficiency techniques such as quantization and pruning can sacrifice model capabilities and reliability, motivating hardware-aware designs that preserve functionality across heterogeneous platforms.Source: 1) Hardware Resource Limitations · p. 7

2) Communication Network Constraints:

  • Wireless-network vulnerability threatens robust connectivity, reliability, and security, particularly in mobile or ultra-dense settings.Source: 2) Communication Network Constraints · p. 8
  • Congestion can cause packet loss and queuing delays, while jitter creates unpredictable latency; high-volume split-inference data and agent reasoning-trace exchanges add bandwidth and scalability bottlenecks.Split inference may generate tens to hundreds of megabytes of activation data per inference step.Source: 2) Communication Network Constraints · p. 8
  • AI Flow combines hierarchical network architectures with cross-domain techniques and familial-model interaction protocols to support scalable, efficient deployment, real-time response, and cross-platform operation.Its approach includes holistic optimization of heterogeneous network resources and strategic co-design of interaction protocols.Source: 2) Communication Network Constraints · p. 8

D. Addressing Challenges with AI Flow · 1) Device-Edge-Cloud Collaboration: · 2) Familial Models:

AI Flow coordinates inference across device, edge, and cloud resources to address hardware constraints and latency, while its familial models and collaborative mechanisms support flexible deployment and intelligence emergence. Its techniques combine hierarchical task placement, communication-efficient inference, and scalable model families.

D. Addressing Challenges with AI Flow

  • AI Flow streamlines model inference through device-edge-cloud collaboration, mitigating hardware constraints and reducing inference latency.The framework uses techniques for collaboration across distributed resources.Source: D. Addressing Challenges with AI Flow · p. 9
  • AI Flow fosters intelligence emergence through interaction and interconnection among agents, achieving superior performance compared with single-model systems.Source: D. Addressing Challenges with AI Flow · p. 9

1) Device-Edge-Cloud Collaboration:

  • The three-tier architecture balances scalability and low-latency inference by prioritizing latency-critical tasks at device and edge tiers while using cloud resources for intensive operations.The hierarchy adapts deployment to heterogeneous hardware capabilities and application requirements.Source: 1) Device-Edge-Cloud Collaboration · p. 10
  • Task-oriented feature compression combines bandwidth-aware coding and adaptive token pruning to reduce communication overhead in device-edge cooperative VLM inference.It targets the high-dimensional intermediate data exchanged in split inference, which constrains scalability and bandwidth-limited deployment.Source: 1) Device-Edge-Cloud Collaboration · p. 10
  • Speculative decoding assigns lightweight device models to predict LLM candidate tokens and larger edge or cloud models to validate and refine them, accelerating inference by shifting larger models to error correction.The mechanism selects verification models according to real-time resource constraints to adapt the quality-efficiency tradeoff.Source: 1) Device-Edge-Cloud Collaboration · p. 11

2) Familial Models:

  • Familial models scale flexibly to arbitrary sizes, using early exit and weight decomposition to match heterogeneous hardware and diverse task demands.Their architectural scalability supports efficient collaboration even under stringent conditions.Source: 2) Familial Models · p. 11
  • Early exit terminates models at earlier layers using pretrained intermediate representations, yielding faster smaller variants with minimal accuracy loss and aligned features that can be reused without middleware.Reusing intermediate results eliminates computational redundancy.Source: 2) Familial Models · p. 11
  • Weight decomposition creates arbitrary-parameter models from low-rank matrices, while tailored initialization limits feature distortion and supports interoperability without severe performance degradation or computational overhead.It complements early exit by enabling finer-grained scalability and decoupling model capability from structural priors.Source: 2) Familial Models · p. 11

3) Connectivity- and Interaction-based Intelligence Emergence: … A. Hierarchical Network Architecture

This section describes hierarchical network connectivity and collaboration among models and network tiers, with a focus on device-server collaboration, diffusion-model collaboration, and device-edge-cloud orchestration. It frames this architecture around latency, throughput, and scalability demands, and its aims of improving response coherence, resource use, and AI-service responsiveness.

3) Connectivity- and Interaction-based Intelligence Emergence:

  • AI Flow connects LLMs, VLMs, and diffusion models through hierarchical networks; a four-step device-server pipeline selects specialized devices, aggregates their responses, and returns a unified answer for possible revision.The server selects devices based on the query and available models; devices contribute domain- or preference-specific responses, which the server synthesizes with global context.Source: 3) Connectivity- and Interaction-based… · p. 12
  • Diffusion models collaborate serially, in parallel, or as networks for motion generation, depth estimation, and multimodal video generation.Serial motion generation composes controlled segments before refining person-to-person interaction; parallel depth estimation uses separate near- and far-region decoders whose outputs are fused using predicted anchor depth.Source: 3) Connectivity- and Interaction-based… · p. 12
  • For multimodal video generation, hierarchical networked collaboration combines RGB, segmentation, depth, and audio features to enforce spatial and temporal alignment, with specialized output heads.Separate encoders may run on distributed hardware, while a unified diffusion transformer captures cross-modal correlations.Source: 3) Connectivity- and Interaction-based… · p. 12

II. DEVICE-EDGE-CLOUD COLLABORATION

  • The device-edge-cloud architecture underpins distributed inference, including task-oriented feature compression for device-edge VLM inference and hierarchical collaborative decoding across network layers for LLM inference.The proposed mechanisms target lower data-transmission overhead and end-to-end latency, and faster inference while maintaining model intelligence.Source: §II · p. 13

A. Hierarchical Network Architecture

  • End devices, edge servers, and cloud clusters combine to address resource limits: edge processing reduces reliance on distant cloud infrastructure, while cloud capacity handles demanding workloads despite higher latency.Devices are computationally constrained, edge servers offer moderate capacity with lower transmission latency, and cloud clusters provide scalable resources but are less suitable for latency-critical operations.Source: A. Hierarchical Network Architecture · pp. 13–15
  • The hierarchical network architecture is framed around strict latency, throughput, and scalability demands.Source: A. Hierarchical Network Architecture · p. 15

B. Efficient Collaboration Paradigms · 1) Task-Oriented Feature Compression:

TOFC addresses VLM edge-inference delays from raw-image transmission and large visual-token workloads by dynamically compressing task-relevant features for device-edge co-inference [29],.

B. Efficient Collaboration Paradigms · 1) Task-Oriented Feature Compression:

  • TOFC dynamically optimizes task-relevant visual-feature transmission according to channel conditions through on-device merging and compression for device-edge co-inference [29],.Source: 1) Task-Oriented Feature Compression · p. 16
  • DPC-KNN clusters CLIP-encoder visual features, then average-pools features within clusters to reduce data size and computational load.Source: 1) Task-Oriented Feature Compression · p. 16
  • A hyperprior-based entropy model encodes and decodes merged features to minimize transmission while maintaining downstream-task performance, with specialized models adapted to input-feature characteristics.Source: 1) Task-Oriented Feature Compression · p. 16
  • For identical RealWorldQA performance, TOFC reduces transmission overhead by 25%–45% versus WebP and 35%–60% versus JPEG.Fig. 4 presents rate-performance curves on RealWorldQA and MME.Source: Figure 4 · p. 17, 1) Task-Oriented Feature Compression · p. 18
  • A router scores entropy models for each merged feature, selecting the highest-scoring model at inference and using softmax-weighted model outputs during training.The loss combines distortion, rate, and balance components.Source: 1) Task-Oriented Feature Compression · p. 17
  • On MME, TOFC saves approximately 25% and 42% of data size versus WebP and JPEG at identical perception score; with feature merging, its data size is 69% and 72% of those baselines, respectively.Source: 1) Task-Oriented Feature Compression · p. 18
  • On RealWorldQA at low communication rates, TOFC halves latency versus WebP and JPEG; on MME, its latency is 32% of server-only inference with minimal performance degradation.Latency reductions are more substantial at higher communication rates.Source: 1) Task-Oriented Feature Compression · p. 18

2) Hierarchical Collaborative Decoding: · III. FAMILIAL MODEL

Hierarchical collaborative decoding combines lightweight device generation with larger-model verification, while familial models align hidden features across differently sized models to support efficient collaboration.

2) Hierarchical Collaborative Decoding:

  • In speculative decoding, a lightweight device model drafts tokens locally and a higher-capacity edge model verifies, refines, and corrects the draft.The verifier applies a soft acceptance strategy to candidate tokens.Source: 2) Hierarchical Collaborative Decoding · p. 19
  • The device handles high-probability token predictions while the edge resolves ambiguities, transmitting only critical updates or corrections to reduce bandwidth use.This draft-then-verify workflow is described as balancing speed, resource use, and output quality.Source: 2) Hierarchical Collaborative Decoding · pp. 19, 21
  • To avoid sequential draft-then-verify latency, the proposed parallel pipeline lets the device generate without blocking while the edge periodically receives and refines tokens.The pipeline aligns the device’s time to decode several tokens with the edge model’s time for one forward pass.Source: 2) Hierarchical Collaborative Decoding · p. 21
  • Both two-tiered and three-tiered collaboration improve response accuracy while decoding faster than edge-only and cloud-only cases, respectively.The passage gives MATH-500 as an example but does not provide further quantitative comparisons here.Source: 2) Hierarchical Collaborative Decoding · p. 21
  • The reported decoding achieves around 40.09 tokens per second generation, about 1.25× faster than standalone edge-model decoding at the same accuracy.Source: 2) Hierarchical Collaborative Decoding · p. 22

III. FAMILIAL MODEL

  • Familial models are differently sized models with aligned hidden features, enabling overhead-free information sharing and effective collaboration.They can directly reuse smaller models’ intermediate results without additional middleware.Source: §III · p. 22
  • Familial-model cooperation is described as particularly efficient under tight communication-bandwidth and hardware constraints, while supporting heterogeneous devices and varied downstream tasks.The architecture supports almost arbitrary parameter counts.Source: §III · p. 22

A. Related Works … 2) Early Exit:

The paper develops weight decomposition and early-exit techniques for familial models, enabling tunable model capacity and collaborative inference across heterogeneous devices. These techniques combine data-whitened decomposition with flexible branches that preserve consistency across configurations.

1) Weight Decomposition for Parameter Efficiency:

  • The proposed decomposition builds on SVD by whitening data and truncating small singular-value components, enabling per-layer rank control while limiting approximation distortion.This enables fine-grained parameter tuning.Source: 1) Weight Decomposition for Parameter… · p. 23
  • The familial architecture extends low-rank decomposition to scalable branch networks, combining decomposed transformer blocks with shared LM heads for multi-branch inference and fine-grained parameter tuning.Source: 1) Weight Decomposition for Parameter… · p. 23

2) Early-Exit Methods:

  • The proposed early-exit framework decomposes branch networks after exit points, supporting flexible parameter control while preserving alignment for collaborative inference on heterogeneous devices.Source: 2) Early-Exit Methods · p. 23

1) Weight Decomposition:

  • Each transformer linear layer can be replaced by two smaller layers with a dynamically tunable hidden dimension, reducing parameters while maintaining structural consistency.The factors have dimensions h×n and m×h for an original m×n weight matrix.Source: 1) Weight Decomposition · p. 24
  • Whitening calibration inputs before SVD makes them orthonormal, while discarded singular values quantify distortion and guide per-layer compression under a fixed parameter budget.This supports improving overall model accuracy within that budget.Source: 1) Weight Decomposition · pp. 24–25

2) Early Exit:

  • Trainable branches refine intermediate features before the LM head, and branches with varying parameter counts and computation allow adaptation to hardware capabilities and task requirements.Source: 2) Early Exit · p. 25
  • Branches with different compression ratios provide almost arbitrary model capabilities while maintaining consistency, letting devices select based on available resources and downstream delay requirements.Source: 2) Early Exit · p. 25
  • Devices can continue processing intermediate features from an exit point on a more powerful edge server, avoiding redundant computation and supporting higher-quality responses with improved efficiency and accuracy.The example assumes sufficient network bandwidth and a complicated downstream task.Source: 2) Early Exit · pp. 25–26

C. Implementation Strategies · 1) Hierarchical Principal Component Decomposition:

HPCD creates a family of compressible models by decomposing network weights into sequentially trained low-rank components. Selecting how many components to retain trades computational cost for model capacity, while later components improve benchmark performance by capturing additional knowledge.

1) Hierarchical Principal Component Decomposition:

  • HPCD compresses deep neural networks by decomposing weight matrices into hierarchies of low-rank components that capture primary and supplementary features.It is proposed as an implementation strategy for familial models.Source: 1) Hierarchical Principal Component… · p. 26
  • Each component progressively fits residual error left by earlier components, preserving learned information while capturing critical features.The method uses iterative residual fitting, with previous components frozen as subsequent ones are trained.Source: 1) Hierarchical Principal Component… · p. 26
  • At inference, retaining the first k components enables adaptive control of FLOPs and memory use according to compute or hardware constraints.Progressive training and adaptive rank selection support trade-offs between model accuracy and computational efficiency.Source: 1) Hierarchical Principal Component… · p. 26
  • The implementation provides four model variants spanning 2.38B to 6.30B parameters, with the number of low-rank components selected from 1 to 4; all variants share a uniform low-rank decomposition structure.Source: 1) Hierarchical Principal Component… · p. 27
  • HPCD training combines language-modeling loss with knowledge distillation from a pre-trained teacher to preserve generation capability and align student behavior.The objective combines cross-entropy on ground-truth tokens and KL divergence between student and teacher logits, weighted by λ.Source: 1) Hierarchical Principal Component… · p. 28
  • Despite limited training tokens, HPCD models perform comparably to LLaMA2-7B, Baichuan2-7B, and ChatGLM2-6B, with each subsequent component improving over its predecessors.The reported progressive improvement indicates that later components capture knowledge not acquired by earlier ones.Source: 1) Hierarchical Principal Component… · p. 28

2) Early Exiting with Scalable Branches:

EESB uses scalable decomposed branches to create familial models with flexible parameter budgets. On VQA benchmarks, it retains more backbone capability with fewer parameters than the baseline.

Method

  • EESB places a decomposed transformer branch between each exit point and the shared LM head, enabling nearly arbitrary parameter counts while keeping a familial model series below twice the main model’s parameters.The branch’s linear layers retain portions of decomposed weight matrices, and the branch block has the parameter count of an original transformer block.Source: 2) Early Exiting with Scalable Branches · pp. 28–29
  • Branch parameters can be dynamically tuned to trade computational cost against performance; at inference, only the layers before the selected exit and its branch are loaded, reducing GPU-memory and storage use.The exit is the last layer that fits the target parameter budget, and branch compression is set to meet that budget together with the shared LM head.Source: 2) Early Exiting with Scalable Branches · pp. 29–30
  • Training combines weighted cross-entropy losses for the main model and branches, with options to freeze the main model, fine-tune it using LoRA, or train it end-to-end.To limit GPU-memory costs when many branches are used, EESB trains a subset per step or accumulates gradients across subsets before updating parameters.Source: 2) Early Exiting with Scalable Branches · p. 30

Evaluation

  • On average, EESB reduces parameters for 60% of backbone capability from 4.43B to 2.76B and exceeds 95% at 4.38B, while the baseline reaches only 75% at 5.44B.The reported averages are across the evaluated benchmarks.Source: Figure 9 · p. 32, 2) Early Exiting with Scalable Branches · p. 32
  • Across the first six general-purpose VQA benchmarks—AI2D, MMBench, MMStar, RealWorldQA, ScienceQA, and SEEDBench—EESB maintains 98.2% of backbone capability.On each benchmark, EESB achieves more than 95% of task performance with less than 45% of the parameter count.Source: 2) Early Exiting with Scalable Branches · p. 31
  • For information-gathering ChartQA, DocVQA, and InfoVQA, EESB achieves 82.8% of task performance with 4.18B parameters, while the baseline struggles to deliver 28.6% with 5.44B parameters.Source: 2) Early Exiting with Scalable Branches · pp. 31–32

D. Discussions · IV. CONNECTIVITY- AND INTERACTION-BASED INTELLIGENCE EMERGENCE · A. Collaboration among LLM Agents

This section describes how AI Flow enables resource- and task-adaptive collaboration through hierarchical interaction among heterogeneous models and LLM agents, including a scheme in which a server synthesizes device-model responses for refinement. It also discusses benchmark findings on collaborative reasoning, where performance gains vary with model strength and assistants’ number and contributions.

D. Discussions

  • Familial models allow fine-grained parameter sizing through transformer-layer weight decomposition, accommodating heterogeneous device capabilities and differing task needs for intelligence and latency.Smaller models can reduce response latency for delay-sensitive tasks.Source: D. Discussions · p. 33
  • Shared intermediate features enable overhead-free split inference between devices and servers; local preliminary responses can continue smoothly through intermittent server disconnections while offloading computation and increasing server capacity.The device executes initial transformer blocks and sends activations to an edge server for further processing.Source: D. Discussions · p. 33
  • Jointly trained familial models produce draft-token distributions closer to verified-token distributions, raising draft-token acceptance and accelerating inference without repeating shared computation.Computation savings persist across separate devices when intermediate results are exchanged.Source: D. Discussions · pp. 33–34
  • Future work on familial models will extend their use to more downstream tasks and deploy trained models on layered infrastructure.The discussion identifies this as the main direction for future work.Source: D. Discussions · p. 34

IV. CONNECTIVITY- AND INTERACTION-BASED INTELLIGENCE EMERGENCE

  • AI Flow uses hierarchical interactions among heterogeneous components to produce coherent, globally consistent outputs with synergistic capabilities beyond individual contributions.The section frames this as collaboration that overcomes the limitations of isolated models through enhanced connectivity and inter-model synergy.Source: §IV · p. 34

A. Collaboration among LLM Agents

  • The proposed device–server paradigm lets users contribute personalized, lightweight on-device LLMs, whose specialized responses a central server selects, aggregates, and returns for revision.The feedback loop lets edge agents refine responses using global context, supporting coherence, accuracy, and consensus across agents.Source: A. Collaboration among LLM Agents · pp. 34–35
  • The reported LLM collaboration benefits weaker models, improves GPT-Conversational’s second-round MT-Bench dialogue scores, and increases most models’ consistency with human preferences on AlpacaEval 2.0.The supplied passages do not provide benchmark values.Source: Table III · p. 36, A. Collaboration among LLM Agents · pp. 36–37
  • Each model first generates a response independently; those outputs are aggregated into shared context and fed back to the original model to refine its answer.Source: A. Collaboration among LLM Agents · p. 37
  • Collaborative setup yields consistent gains across most tasks, especially for weaker models; LLaVA and Janus-Pro improve notably on text recognition and multi-step reasoning.Stronger models GPT-4o and Qwen2.5-VL also improve, modestly, especially on complex reasoning tasks.Source: A. Collaboration among LLM Agents · pp. 37–38
  • As the number of agents increases, benchmark performance improves, with Arena-Hard scaling nearly linearly and supporting the framework’s practical scalability.The results cover AlpacaEval, MT-Bench, and Arena-Hard.Source: A. Collaboration among LLM Agents · p. 38, Table V · p. 38
  • Removing any assistant degrades performance across benchmarks, and no single assistant matches the full system, even though GPT-4o performs strongest individually.The results indicate that specialized assistants contribute complementary insights and make collaboration synergistic rather than merely compensatory.Source: Table VI · p. 38, A. Collaboration among LLM Agents · p. 39

B. Collaboration among Diffusion Models · 1) Serial Collaboration:

Diffusion-model collaboration is presented as a way for connected models to produce capabilities beyond any single model, with serial collaboration combining motion composition and interaction refinement. On InterHuman, the serial method reports strong text-alignment and motion-realism results, alongside reduced coordination distance and near-human diversity.

B. Collaboration among Diffusion Models

  • Across distributed components, structured connections and model interactions can yield capabilities no single model achieves; serial, parallel, and networked paradigms address diverse topologies and hardware.The paradigms use hierarchical network-based collaboration and active model interactions to support richer, more generalized intelligence.Source: B. Collaboration among Diffusion Models · p. 39

1) Serial Collaboration:

  • Serial collaboration suits diverse hardware and network topologies, using a layered motion-generation pipeline in which INS composes motion and REC refines cross-agent synchrony.The modules work together to produce complex, coordinated multi-person behaviors from modular diffusion models.Source: 1) Serial Collaboration · p. 39
  • INS interleaves solo and interactive segments in one temporally controlled trajectory, then uses conditional diffusion guided by text and temporal cues to generate the sequence.The unified sequence captures fluid transitions between self-expression and coordinated social behavior.Source: 1) Serial Collaboration · p. 39
  • INS-generated motion captures fluid transitions between self-expression and coordinated social behavior, reflecting how humans naturally switch between these modes.This result concerns transitions between solo expression and social coordination.Source: 1) Serial Collaboration · p. 40
  • REC uses transformer-based spatial and mutual-influence modeling to refine paired motions, correct inconsistencies from independent generation, and add fine-grained responsiveness.It promotes interaction fidelity, such as gestures being appropriately received or mirrored by another agent.Source: 1) Serial Collaboration · p. 40
  • The serial design separates high-level motion interleaving from low-level inter-agent refinement, allowing complementary module expertise to produce richer interactions.Information flows from composition to coordination, combining modular knowledge in the synthesis process.Source: 1) Serial Collaboration · p. 40
  • Top-1/2/3 R-Precision scores are 0.335, 0.479, and 0.584, with FID 6.332; versus InterGen, Top-1 R-Precision improves 25.3% and FID falls 50.6%.The paper reports state-of-the-art performance across all key metrics.Source: 1) Serial Collaboration · p. 40, Table VII · p. 41
  • REC reduces MM Distance by 37.0% versus single-person MDM baselines; motion diversity is 7.763 versus 7.748, and multimodality is 1.601.The paper describes the diversity as near-human and the multimodality as optimal.Source: 1) Serial Collaboration · p. 41

2) Parallel Collaboration: · 3) Networked Collaboration:

This section examines parallel and networked collaboration in AI Flow, spanning simultaneous inference computation and cross-node multimodal generation and conditioning. It presents OmniVDiff as an example of distributed, modality-specific reasoning for video intelligence, alongside reported depth-estimation and depth-conditioned video-generation performance.

2) Parallel Collaboration:

  • Parallel collaboration runs models simultaneously across devices, using peer models at the same level to tackle challenging tasks and improve resource utilization.For metric depth estimation, sliding anchors and parallel near- and far-depth branches adapt predictions to scene scale, supporting generalization across environments without retraining.Source: 2) Parallel Collaboration · pp. 41–42
  • The parallel framework achieves top performance across all metrics on NYU-V2 and KITTI, outperforming methods fine-tuned individually on each dataset.Evaluation uses standard depth caps of 10 meters for NYU-V2 and 80 meters for KITTI.Source: 2) Parallel Collaboration · p. 42, Table VIII · p. 43

3) Networked Collaboration:

  • OmniVDiff [94] unifies generation and understanding across RGB, depth, segmentation, and edge modalities in a video diffusion framework with arbitrary input-output modality combinations.Its shared diffusion backbone supports cross-modal consistency and joint reasoning across modalities.Source: 3) Networked Collaboration · pp. 43–44
  • Adaptive modality control assigns modalities as generation targets or fixed conditioning inputs, enabling task-specific behavior without retraining or architecture changes.OmniVDiff also uses pseudo-labels from expert models on unannotated Koala-36M videos and trains in stages for modality alignment and cross-modal conditioning.Source: 3) Networked Collaboration · p. 45
  • OmniVDiff outperforms existing baselines on depth-conditioned video generation, improving geometric consistency and fidelity to the input depth.The evaluation compares different condition configurations using Fréchet video distance (FVD) and kernel video distance (KVD).Source: 3) Networked Collaboration · p. 45, Table IX · p. 45
  • OmniVDiff orchestrates distributed, modality-specific reasoning within one system.Source: 3) Networked Collaboration · p. 46
  • OmniVDiff delivers scalable, flexible, and controllable video intelligence within the AI Flow framework.Source: 3) Networked Collaboration · p. 46

C. Discussions · V. APPLICATIONS · A. Embodied AI

AI Flow applies connectivity, interaction, hierarchical collaboration, and familial models to coordinate heterogeneous agents across device–edge–cloud settings. Across the merged sections, it presents these mechanisms as enabling responsive GUI tasks and cooperative embodied systems while reducing redundant computation, bandwidth use, and energy consumption.

C. Discussions

  • AI Flow treats connectivity and interaction as core primitives and generalizes collaboration into a hierarchical device–edge–cloud architecture for heterogeneous agents.The architecture distributes computational workloads to meet diverse latency requirements, spanning lightweight mobile-device models to high-capacity GPU models.Source: C. Discussions · p. 46
  • Token-level interaction can distribute partial inference across hierarchy layers, trading off latency, accuracy, and computational cost in GUI-agent tasks.The GUI decision process—including interface understanding, intent recognition, action planning, and visual grounding—can assign simple latency-sensitive tasks to on-device models and complex reasoning or long-horizon planning to edge or cloud models.Source: C. Discussions · p. 46
  • Coordinated scheduling and orchestration can preserve low response latency while producing high-quality outputs for applications such as personalized assistants and real-time analytics.The discussion also names multimodal interaction platforms as an application.Source: C. Discussions · pp. 46–47

V. APPLICATIONS

  • AI Flow’s applications illustrate seamless transmission and intelligence emergence across hierarchical networks, supported by familial models and connectivity- and interaction-based emergence.The applications section identifies embodied AI, wearable devices, and smart cities as scenarios.Source: §V · p. 47

A. Embodied AI

  • Embodied AI [95], represents a paradigm in which intelligent systems interact physically; heterogeneous agents with varied capabilities and operational constraints collaborate on complex tasks requiring perception.Challenges include real-time decision-making in dynamic settings and constraints involving network bandwidth, memory storage, and processing power.Source: A. Embodied AI · pp. 47–48
  • Familial models tailored to agent hardware enable knowledge sharing across heterogeneous entities and can reduce redundant computation, inference latency, and computational overhead.In the surveillance example, a drone relays motion patterns or object embeddings to a ground robot, which resumes inference without repeating preprocessing.Source: A. Embodied AI · p. 48
  • Sharing task-relevant information rather than exhaustive data transmission curtails bandwidth use and energy consumption for resource-constrained agents.These savings matter in dynamic environments.Source: A. Embodied AI · p. 48
  • AI Flow coordinates heterogeneous agents to accomplish tasks beyond the reach of a single agent, including cooperative object delivery by a robotic arm and drone.The arm identifies and grasps the object while the drone receives and delivers it remotely.Source: A. Embodied AI · p. 48

B. Wearable Devices … A. Federated Learning

AI Flow uses hierarchical device-edge-cloud coordination to address the resource and responsiveness demands of wearable and smart-city applications. The paper identifies federated learning, distributed edge inference, and adaptive network orchestration as future research directions, while noting large-model FL’s efficiency and capability tradeoffs.

B. Wearable Devices

  • For wearables, AI Flow dynamically partitions workloads: devices extract immediate cues, edge or cloud handles intensive recognition and segmentation, and latency-tolerant analytics can be deferred.The framework prioritizes low-latency interactive tasks when edge resources are available.Source: B. Wearable Devices · p. 49
  • AI Flow enables wearable applications to operate within power, memory, and computational limits while minimizing device energy and maintaining inference accuracy and responsiveness.These outcomes are described for wearable devices, without quantitative measurements.Source: B. Wearable Devices · p. 49

C. Smart Cities

  • Smart cities use tier-specific familial models: devices perform localized tasks, edge models aggregate multimodal data for regional decisions, and cloud models train global models and distribute updates.Cloud models also predict system-wide trends such as congestion patterns.Source: C. Smart Cities · p. 50
  • Smart-city AI Flow coordinates heterogeneous systems and distributes computation to support prompt decisions and responsiveness in dynamic urban scenarios.The framework is described as minimizing bottlenecks while addressing critical urban challenges.Source: C. Smart Cities · p. 50

VI. FUTURE WORK

  • Future work identifies federated learning, distributed edge inference, and adaptive network orchestration as priorities for advancing AI Flow in real-world deployment.These directions follow the paper’s discussion of AI Flow across critical applications.Source: §VI · p. 51

A. Federated Learning

  • Federated learning (FL) – enables collaborative model optimization across decentralized devices while preserving privacy by exchanging model updates, such as gradients or parameters, instead of sharing raw data.Source: A. Federated Learning · p. 51
  • Large-model FL faces substantial computation and memory demands on consumer-grade participants, while repeated exchanges of high-dimensional parameters impose communication costs that limit scalability.Pretraining or full-parameter LLM fine-tuning is especially demanding, and FL typically requires multiple synchronization rounds.Source: A. Federated Learning · p. 51
  • Parameter-efficient fine-tuning reduces hardware and communication burdens, but update bias can degrade model capability, leaving the balance between efficiency and capability a key FL research goal.LoRA is cited as an example that constrains updates to compact parameter subsets, distorting gradient information.Source: A. Federated Learning · p. 51

B. Distributed Edge Inference … Authors’ contributions

AI Flow combines distributed edge inference and adaptive network orchestration with device-edge-cloud collaboration, familial models, and connectivity-based intelligence to support AI at the network edge.

B. Distributed Edge Inference

  • Distributed edge inference partitions large AI models into modules executed collaboratively across interconnected devices, using decentralized resources for scalable, low-latency deployment.Networked coordination helps mitigate individual devices’ memory and processing constraints.Source: B. Distributed Edge Inference · p. 52
  • Edge inference must balance workloads across heterogeneous, resource-limited devices despite churn, while inter-device transfers of source data and features can dominate latency and limit scalability.The devices may differ in computational capacity, memory, and energy limits, and may frequently join or disconnect.Source: B. Distributed Edge Inference · p. 52
  • Traditional data and model parallelism often assumes stable, homogeneous infrastructure, but edge heterogeneity and disconnections hinder synchronization and load balancing, motivating dynamic reconfiguration.Future work is directed toward distributed inference systems that reconfigure in response to heterogeneous environments.Source: B. Distributed Edge Inference · p. 52

C. Adaptive Network Orchestration

  • Heterogeneous network conditions, latency and bandwidth fluctuations, and intermittent disconnections complicate coordination, motivating network-aware intelligence that bridges hierarchical resources.Edge-cloud links can also experience jitter and variable delays that degrade data exchange and model synchronization.Source: C. Adaptive Network Orchestration · pp. 52–53

VII. CONCLUSIONS

  • AI Flow integrates device-edge-cloud collaboration, familial models, and connectivity- and interaction-based intelligence, with agent-model interactions producing intelligence beyond any single model.The conclusion also describes timely responsiveness and ubiquitous accessibility for intelligent systems on resource-constrained edge devices.Source: §VII · p. 53

ABBREVIATIONS

Availability of data and material

  • Some HPCD training data come from proprietary datasets restricted by legal and contractual terms, while source data are provided with the paper.The restricted datasets were used only for training and are stated not to affect reproducibility of the core findings.Source: Availability of data and material · p. 55

Authors’ contributions

  • Xuelong Li initiated and led the project, Jiawei Shao facilitated it, named contributors conducted experiments, and named authors wrote the paper.Experiments were conducted with assistance from Jiawei Shao, Hongyuan Zhang, and Yuanzhi Liang under Chi Zhang’s guidance.Source: Authors’ contributions · p. 55

Related papers

Loading 2506.12479v3…