PaperlayerView exact revision on arXiv

Source-linked AI summary

FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

Krish Agarwal, Zhuoming Chen, Yanyuan Qin, Zhenyu Gu, Atri Rudra, Beidi Chen

arXiv:2607.18171v2cs.LG

TL;DR

Real-time multimodal pipelines require application-specific deployment decisions that existing serving systems handle only with limited transformations and fixed assumptions. FlashRT guides coding agents through structured intermediate representations and measurement-gated validation to produce efficient multi-GPU deployments, achieving competitive low-latency performance across diverse applications and hardware.

  • Problem

    Existing serving systems and auto-parallelism compilers offer limited transformations and fixed workload assumptions for complex multimodal deployment pipelines.

  • Method

    FlashRT uses a chain-of-program workflow combining intermediate-representation conversion, static analysis, and application-grounded iterative implementation, verification, and measurement.

  • Results

    Across diverse multimodal applications and NVIDIA B200 and AMD MI355X hardware, FlashRT synthesizes low-latency deployments competitive with expert-designed systems while trading off latency and throughput.

  • Takeaways & Limitations

    FlashRT suggests agent-driven deployment can scale across applications without human intervention, with each application costing $75–$167 in API usage and agent time.

  • Takeaways & Limitations

    FlashRT does not integrate LLM kernel-optimization agents and assumes an underlying model capable of organizing a hierarchical IR and maintaining a hypothesis queue.

Abstract

from arXiv · show

Real-time multimodal applications, including voice agents and interactive video generation, compose heterogeneous models into pipelines whose efficient deployment requires application-specific decisions about placement, streaming, and intra-model parallelism. Existing serving systems and auto-parallelism compilers commit to limited transformations and fixed workload assumptions, so achieving high performance on a new application requires hand-crafting an efficient implementation. We present FlashRT, an agent harness that guides coding agents to lift simple developer-written reference implementations into optimized multi-GPU deployments that flexibly weigh target metrics like latency and throughput. Using a new chain-of-program paradigm, FlashRT directs a generic coding agent through a multi-pass transformation process where an agent transforms the reference into an intermediate representation (IR) to capture data dependencies and persistent-state scopes, validates this IR via a sequential interpreter, and performs static analyses to identify candidate transformations. Then, the agent iteratively implements, verifies, and benchmarks each candidate under a measurement-gated optimization loop to produce effective deployments that span different hardware budgets. Across various applications, including video world models and multimodal LLMs, FlashRT converts reference implementations into highly efficient deployments, delivering up to ~70x latency reduction and 2.8x throughput improvement on NVIDIA B200 GPUs. On AMD MI355X GPUs, FlashRT matches the peak latency reduction while increasing peak throughput improvement to 3.6x, demonstrating that agent-driven optimization can be more scalable on platforms with less mature expert optimization. In fact, for Qwen3-Omni text-to-audio inference, FlashRT reduces response latency by 65% compared to the expert vLLM-Omni implementation on AMD MI355X.

1 Introduction

FlashRT addresses the difficulty of efficiently deploying diverse real-time multimodal applications by guiding a generic coding agent from flexible reference implementations to specialized deployments. Its chain-of-program workflow and application-grounded validation loop enable automated, hardware-generalizable optimization without human intervention beyond providing the reference implementation.

  • Real-time multimodal applications combine diverse modalities and create a deployment design space fundamentally different from conventional LLM serving.
  • Existing systems fall short through restricted deployment policies, fixed workload coverage, and operator-level optimization granularity.Multimodal serving frameworks constrain placement, auto-parallelism systems specialize to fixed workloads, and compilation frameworks target operator-level improvements.
  • An ideal system would transform intuitive, unoptimized, single-GPU implementations into application-specific infrastructure with unconstrained deployment scope, workload flexibility, and adaptive optimization granularity.
  • FlashRT’s chain-of-program paradigm has agents first construct a structured IR, perform static analysis, and then translate the IR into optimized implementations.The staged process is motivated by the limited effectiveness of direct, single-step translation by current agents.
  • FlashRT uses an application-grounded validation loop in which agents explore hypotheses, implement candidates, measure them, and drive simulated user inputs through an application-specific test harness.The loop addresses both incomplete search and incorrect solutions by grounding validation in the target application.
  • Across diverse multimodal applications, FlashRT generalizes across NVIDIA B200 and AMD MI355X hardware while requiring no human intervention beyond supplying a reference implementation.The system is presented as a framework that substantially reduces manual systems effort.

2 Related Work

Prior work spans real-time multimodal serving, auto-parallelism and compiler systems, and coding agents for performance optimization. Existing auto-parallelism systems target dense DNN training rather than heterogeneous inference pipelines, while agents have been explored for code repair and GPU optimization.

  • Real-time multimodal serving: Real-time multimodal applications compose heterogeneous models into pipelines, including voice agents, avatar generation, video world modeling, and embodied vision-language-action systems.These application classes are represented in the cited serving literature.
  • Auto-parallelism and ML compilers: Auto-parallelism systems jointly choose data, model, and pipeline parallelism for distributed deep learning.Examples include FlexFlow, GSPMD, Alpa, and Unity.
  • Auto-parallelism and ML compilers: Their search spaces and cost models target dense DNN training and do not transfer to heterogeneous inference pipelines with disjoint runtimes.The limitation is specifically identified for inference pipelines composed of heterogeneous models.
  • Coding agents for performance optimization: Coding agents paired with environment feedback can navigate large codebases and iteratively repair their own outputs.This line of work includes agent systems for software engineering.
  • Coding agents for performance optimization: GPU kernel-generation and optimization research has combined agents with compiler pass selection, scientific-discovery search, and reward design.The cited approaches explore multiple strategies for agent-driven GPU optimization.

3 Problem Formulation

FlashRT formulates optimization as transforming a synchronous single-GPU reference into a behavior-preserving multi-GPU deployment. The deployment chooses placement and scheduling under dependency, execution-cost, transfer-cost, and resource-capacity constraints to minimize serving latency or maximize throughput.

  • Problem formulation: FlashRT maps a synchronous single-GPU reference Pref into a behavior-preserving multi-GPU deployment Pdep supporting disaggregation, co-location, streaming, and intra-model parallelism.Pref defines application semantics and persistent state scopes such as caches or streaming buffers.
  • Problem formulation: The application is modeled as a task graph G = (V, E), with computation regions as vertices and batch-indexed operation instances.Edges encode intra-batch data dependencies or cross-batch state dependencies, such as KV-cache inheritance.
  • Problem formulation: Each deployment assigns tasks to single- or multi-GPU resources and defines a non-preemptive schedule that respects dependency-induced start times and resource capacity.Execution time depends on placement, while synchronization or transfer costs capture inter-task coordination overhead.
  • Problem formulation: The objective is to minimize application-level serving metrics, including response latency or throughput, subject to correctness constraints.Latency follows the input-to-output critical path, whereas sustainable throughput is governed by the most heavily loaded resource.

4 FlashRT

FlashRT guides coding agents through intermediate representations, validation, analysis, and iterative benchmarking rather than relying on a monolithic reference-to-deployment transformation. This workflow addresses naive agents’ failures to recognize model-level parallelism, explore diverse optimization axes, and compose strategies.

  • Optimization opportunities: The reference conversational pipeline chains ASR, LLM, TTS, and sound-to-video generation, creating application-level and model-level optimization opportunities.Throughput optimization favors inter-module streaming and pipeline parallelism, including streaming TTS audio into autoregressive S2V processing.
  • FlashRT workflow: FlashRT transforms a reference implementation into an intermediate representation, validates it, analyzes candidate transformations, and iteratively tests optimization hypotheses.The IR captures hierarchical pipeline graphs, operations, data dependencies, and cross-batch state dependencies.
  • Naive-agent failure modes: Naive agents fail to recognize model-level parallelization opportunities because effective separation requires reasoning about persistent state, including KV-cache dependencies.For LiveAvatar, diffusion steps have independent KV caches, while the VAE shares no internal state with the DiT, enabling pipeline parallelism.
  • Naive-agent failure modes: Naive agents often terminate after discovering a simple optimization, overlooking additional streaming opportunities and combinations of strategies.In the conversational pipeline, they may stream S2V-generated video chunks while missing TTS-to-S2V streaming and strategy composition.

5 Experiments

Across diverse real-time multimodal applications and GPU platforms, FlashRT automatically discovers deployment structures and optimization strategies that substantially improve latency, throughput, or frame rate. Its IR pass and measurement-gated optimization loop are complementary: the IR identifies optimization axes, while the loop validates and composes them into efficient deployments.

  • Application results: ∼70× lower TTFO latency is achieved with two GPUs by streaming intermediate TTS audio into S2V inference and emitting video chunks incrementally.The deployment co-locates ASR with S2V DiT/VAE and LLM with TTS.
  • Application results: 4.3× higher frame rate is achieved by scaling to six GPUs and applying pipeline parallelism to S2V, with minimal latency impact.The strategy sustains real-time playability while targeting higher frame rate.
  • Cross-platform results: On MI355X, FlashRT reaches ∼70× lower latency and 3.6× higher throughput, while beating vLLM-Omni on Qwen3-Omni by 65%.Across five applications, it recovers the same deployment families and the co-location latency versus disaggregation frame-rate tradeoff found on B200.
  • Ablations: The IR pass and optimization loop are complementary: removing the IR misses pipeline parallelism, while removing the loop prevents effective implementation and composition of discovered strategies.The No Loop prototype reaches 44.69 FPS, whereas the full system isolates strategies into variants before composing them.

6 Conclusion · Appendix

FlashRT presents an agent harness for flexibly deploying real-time multimodal interactive applications, motivated by the difficulty of deriving efficient deployments from arbitrary specifications. Its reported deployment costs, limitations, and future directions frame the approach’s scalability and scope.

  • 6 Conclusion: FlashRT uses a coding agent to flexibly deploy real-time, multimodal interactive applications.The approach is motivated by the intractability of deriving efficient deployments from arbitrary application specifications.
  • 6 Conclusion: $75–$167 in API usage is the reported cost of producing each application’s deployments.This cost also requires a few hours of agent time.
  • 6 Conclusion: FlashRT’s deployment cost is comparable to or below the expert engineering effort it replaces.The passage frames this comparison in terms of producing each application’s deployments.
  • 6 Conclusion: Agent-driven deployment is bounded by compute, whereas manual deployment engineering is bounded by engineers with relevant systems expertise.This contrast addresses the scalability concern motivating the work.
  • 6 Conclusion: FlashRT has not integrated LLM kernel optimization agents.The authors leave integration of such agents as future work to enable a broader range of optimization opportunities.
  • 6 Conclusion: The harness transfers to additional coding agent configurations, but the authors do not claim FlashRT is model-agnostic.The framework assumes an underlying model capable of the native reasoning needed to organize a hierarchical IR and maintain a hypothesis queue.

A Formal Analysis of Pipeline Graphs

The paper formalizes streaming applications as pipeline graphs and derives deployment metrics and bounds for analyzing latency, throughput, and parallelization strategies. It shows that deployment search is NP-hard and characterizes tradeoffs between co-location, disaggregation, sequence parallelism, and stage pipeline parallelism.

  • A Formal Analysis of Pipeline Graphs: The analysis focuses on streaming-generation applications to connect formal deployment analysis with empirical evaluations.The framework is presented as extensible to more general applications.
  • A.1.1 Initial assumptions: Each request produces an indefinitely long stream of batches, with N actions emitted every period T at subinterval spacing ∆ = T/N.Batch j is conservatively ready at Rj = (j + 1)T, and a shared offset d0 determines the display schedule.
  • A.1.2 Pipeline graph: A pipeline graph defines operation types, resources, intra- or inter-batch dependencies, action-gated entries, and the operation releasing each output chunk.Deployment plans assign nonpreemptive schedules, with placement-dependent processing times and synchronization lags.
  • A.1.3 Latency metric: Latency is the smallest feasible display offset d∗_0(ρ, A), while throughput is characterized by the sustainable period Tmin(ρ) and peak displayed-frame rate N Tmin(ρ).Smaller d∗_0 improves latency, whereas smaller Tmin improves throughput.
  • A.2 NP-hardness of deployment search: Even a restricted deployment-search problem without dependencies, state, communication costs, or placement-dependent execution times is NP-hard.The general multimodal deployment problem additionally includes dependencies, streaming, batching, disaggregation, and intra-model parallelism.
  • A.3 Application 1: actions → DiT → VAE: For the DiT–VAE pipeline, co-location is latency-optimal, while disaggregation can be throughput-optimal when it achieves a no-larger sustainable period than co-location.Disaggregation improves throughput by allowing DiT and VAE work from different batches to run concurrently.
  • A.4 Application 2: sequence parallelism vs stage pipeline parallelism: Under the stated assumptions, sequence parallelism has no higher first-frame latency than stage pipeline parallelism.Sequence parallelism reduces the latency-binding path, whereas stage pipeline parallelism can reduce the sustainable period and increase peak frame rate when its inequality is strict.

B Extended Deployment Results and Scaling Analysis

This section extends the deployment evaluation for WorldPlay and LongLive, reporting additional FlashRT-found variants and analyzing how latency and frame rate scale across GPU budgets.

  • Extended Deployment Results: The appendix examines extended deployment results for the WorldPlay video world model and LongLive video narrator applications.These applications correspond to Sections 5.3.3 and 5.3.4.
  • Extended Deployment Results: Additional FlashRT deployments are reported across a range of GPU budgets for each application.The deployments include latency- and frame-rate-optimized variants.
  • Scaling Analysis: The analysis identifies factors governing how latency and frame rate scale with the number of GPUs.It analyzes scaling behavior separately for the two applications.

B.1 Video World Model (WorldPlay)

WorldPlay renders continuous video through an autoregressive DiT and streaming VAE, with FlashRT discovering co-located latency-optimized and disaggregated frame-rate-optimized deployments. Co-location minimizes serial latency, while disaggregation pipelines stages to raise frame rate, producing distinct tradeoffs across GPU budgets.

  • Deployment tradeoff: The tradeoff follows from DiT dominance: disaggregated frame rate is bounded by the slower DiT stage, whereas co-location shortens the serial path through both stages.Disaggregation overlaps one batch’s VAE decode with the next batch’s DiT, but incurs an inter-group transfer; higher SP degrees provide sublinear stage speedups.
  • Two GPUs: Co-location at two GPUs reaches 493 ms latency and 25.5 FPS, while disaggregation reaches 31.0 FPS latency at 625 ms.Co-location uses SP-2 for both stages; disaggregation uses SP-1 per stage and pipelines them.
  • Four GPUs: At four GPUs, co-location attains 425 ms latency and 29.7 FPS, while disaggregation reaches 39.5 FPS at 494 ms latency.Both strategies use higher SP degrees; disaggregation matches the two-GPU co-located stage degree and adds only a small transfer.
  • Six GPUs: At six GPUs, disaggregation reaches 44.3 FPS at 447 ms latency by scaling the DiT from SP-2 to SP-4 while keeping the VAE at SP-2.The pipeline is DiT-bound, so added GPUs go to the DiT group and latency remains near the four-GPU co-located minimum.

B.2 Video Narrator (LongLive) · 4 GPU (co-located)

FlashRT analyzes LongLive’s ASR–DiT–VAE pipeline across GPU budgets, distinguishing latency-critical co-location from frame-rate-oriented disaggregation. In the analyzed deployments, co-location minimizes latency at 4 GPUs, while separating stages at 8 GPUs substantially improves frame rate with only marginally higher latency.

  • B.2 Video Narrator (LongLive): LongLive combines ASR transcription with autoregressive DiT generation and streaming VAE decoding for continuous video output.The analysis focuses on 4- and 8-GPU deployments and how additional GPUs affect latency and frame rate.
  • B.2 Video Narrator (LongLive): ASR latency a contributes to time-to-first-output latency L = a + d(k_D) + v(k_V), but not the period Δ because transcription runs asynchronously.ASR fires only when a new spoken prompt arrives, so it does not affect frame-rate analysis.
  • B.2 Video Narrator (LongLive): Sublinear VAE scaling shapes deployment choices: v(k) flattens after a low SP degree, so FlashRT uses placement instead of rewriting the VAE kernel.This constraint influences the LongLive deployments analyzed below.
  • 4 GPU (co-located): The latency-optimized 4-GPU deployment co-locates DiT and VAE on SP-4, shares ASR on G0, and achieves 430 ms latency, 1.6× below baseline.Because latency is the serial sum a + d(4) + v(4), maximizing both SP degrees within the budget yields the shortest critical path.
  • 4 GPU (co-located): Its frame rate follows Δ_co = d(4) + v(4) and falls between the two 2-GPU deployments, because co-location exposes the VAE term on the serial path.A co-located deployment paying two SP-4 latencies cannot match a pipeline that hides VAE work beneath DiT execution.
  • 4 GPU (co-located): The 8-GPU frame-rate deployment keeps DiT at SP-4, assigns VAE to SP-3, and gives ASR a dedicated GPU, disaggregating all three stages.This placement lets VAE decoding pipeline with denoising and transcription without blocking generation.
  • 4 GPU (co-located): Latency remains nearly unchanged after disaggregation: 462 vs. 430 ms, because DiT stays at SP-4 while VAE runs at SP-3 across a group boundary.The frame-rate-optimized deployment’s latency is only marginally higher than the 4-GPU value.

C Additional Analysis of the Agent Harness

This appendix analyzes additional experiments from Sections 5.5 and 5.6, detailing the broader set of deployments implemented by FlashRT rather than only the best-performing ones. The studies use the face-to-face conversational agent application on an 8-GPU B200 node.

  • Scope of the analysis: The appendix expands Sections 5.5 and 5.6 by examining the full set of deployments implemented by the FlashRT agent.The main sections summarize only the best deployments.
  • Experimental setting: The additional studies run FlashRT on the face-to-face conversational agent application using an 8-GPU B200 node.

C.1 Ablation · C.2 Sensitivity to the underlying model

The ablation results show that FlashRT’s IR pass and structured optimization loop enable discovery of streaming and pipeline-parallel variants beyond simpler configurations. With GPT-5.6 Sol, adding the TTS →S2V streaming edge reduces latency, while VAE sharding produces an 8-GPU deployment.

  • C.1 Ablation: Without the IR pass, both configurations initially leave the TTS →S2V edge blocking, keeping latency above 10 s.They stream generated video but wait for full response-audio synthesis.
  • C.1 Ablation: The No IR configuration eventually identifies the remaining streaming edge through enforced-loop exploration.Its continued iteration after the first working variant distinguishes it from configurations without the loop.
  • C.1 Ablation: The No Loop configuration discovers full streaming during the IR phase and identifies an S2V pipeline-parallel strategy.These discoveries occur directly during IR rather than through structured iteration.
  • C.1 Ablation: The full FlashRT agent’s pipeline-parallel-only variant does not stream, so its output is produced as a single burst without a sustained frame rate.This variant is included among the deployments measured in the ablation study.
  • C.1 Ablation: 44.69 FPS is achieved by the No Loop configuration’s single-process pipeline-parallel prototype before it terminates without further iteration.Synchronization on any stage stalls all stages, making the reported result suboptimal.
  • C.2 Sensitivity to the underlying model: Adding the TTS →S2V edge recovers the latency reduction in the final deployment.The comparison concerns deployments produced by Codex with GPT-5.6 Sol for the face-to-face conversational agent.
  • C.2 Sensitivity to the underlying model: The final deployment uses 8 GPUs because it shards the VAE across three ranks alongside four denoising ranks and one shared rank.The shared rank serves the ASR, L… components described in the passage.

C.3 Consistency across runs

Across three independent runs, FlashRT produced the same sequence of streaming, pipeline-parallelism, and composed deployments, selecting the composed deployment as best despite placement differences. The runs achieved closely consistent performance, landing within 0.15 s and 8.2 FPS of one another.

  • Workflow influence: The FlashRT workflow induced this consistency by identifying streaming and pipeline parallelism in the IR and requiring systematic measurement before composition.The runs agreed on how they reached the final deployment, while differing only in placement.
  • Deployment trajectory: All three runs proposed the same three deployments in the same order and selected the composed deployment as best.The sequence isolates streaming, then pipeline parallelism, then composes both.
  • Performance consistency: 0.15 s and 8.2 FPS were the maximum differences among the three runs’ final performance results.Co-locating ASR, LLM, and TTS changed neither the critical path nor the steady-state period because they finish before S2V frame production begins.

D Agent Cost

FlashRT’s five B200 agent sessions cost $75–$167 per application, including API and GPU compute costs, while replacing specialized serving-engineering effort. The search measures nearly every queued candidate and records justifications when candidates are discarded as unviable.

  • Cost: $75 to $167 per application covers an agent session’s API and GPU compute costs, with GPU cost calculated from end-to-end time and allocated GPU budget.These costs are based on current API pricing and the GPUs remaining allocated throughout each session.
  • Cost: Agent-driven deployment replaces specialized engineering effort with compute provisioned per application, scaling beyond the availability of systems experts.The comparison extends beyond direct monetary cost because efficient serving code for new real-time pipelines requires relevant systems expertise.
  • Search process: The agent implements and measures nearly every queued candidate, discarding candidates only in limited cases with recorded justifications for unviability.One observed failure involves an invalid SP degree that does not evenly divide the latent frames processed by a video model.
Loading 2607.18171v2…