PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs are General Asynchronous Agents

George Yakushev, Denis Mazur, Vladimir Bartenev, Vyacheslav Zhdanovskiy, Timofey Byzov, Vladimir Kaurkin, Vadim Pastushenko

arXiv:2609.35427v1cs.LGcs.CL

TL;DR

Many real-world agents must respond to new inputs while reasoning, but current LLM concurrency solutions are largely task-specific. AsyncLLM provides a shared-memory coroutine framework for asynchronous inference; Qwen3.x agents operate across several asynchronous tasks without task-specific training, although self-modifying behavior is not yet reliable.

  • Problem

    LLM agents generally follow turn-based interaction, while real-world assistants, embodied agents, and monitoring systems may receive new inputs during reasoning or other tasks.

    Source: §1 · p. 1

  • Method

    AsyncLLM lets users or agents define concurrent inference coroutines with overlapping memory states, using async/await and shared-memory attention views.

    Source: §1 · pp. 1–2

  • Results

    Qwen3.x agents handled streaming video, videogames, and system monitoring without task-specific fine-tuning, and modern models showed capacity to react to inputs during reasoning.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results suggest asynchronous operation can be treated as a general capability rather than a collection of domain-specific skills, making these agents easier to adapt to new applications.

    Source: §5 · p. 9

    Concurrent execution may produce an incorrect or harmful action before slower reasoning or monitoring coroutines can intervene.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Modern LLMs are increasingly capable as autonomous agents, but they follow sequential interaction cycles: read, think, reply or call tools, repeat. Many real-world use cases are not sequential: voice assistants, embodied agents, and monitoring systems receive new inputs while they think or perform another task. Modern LLMs address this with specialized architectures for voice interaction and video streams, VLAs for robot control, asynchronous tool calling for API usage, and others. In this work, we generalize from different asynchronous tasks to general asynchronous agents that can adapt to different types of concurrency. To achieve this, we develop an asynchronous LLM framework that lets users (or the agents themselves) define inference coroutines with overlapping memory states. We showcase that Qwen 3.x models are capable of asynchronous operation for streaming video understanding, videogames, and monitoring, without task-specific training.

1 INTRODUCTION

Many LLM applications need to handle new inputs while reasoning, but current concurrency solutions are designed separately for different tasks. The introduction presents AsyncLLM as a training-free framework for defining concurrent inference coroutines and tests Qwen3.x agents across several asynchronous settings.

1 INTRODUCTION

  • Modern LLM agents can tackle tasks requiring extended reasoning, programming, and tool use through a Thought-Action-Observation loop.The agent acts and then uses the observed outcome to inform its next step.Source: §1 · p. 1
  • Real-time voice assistants, self-driving cars, and monitoring agents may need to react to new information while thinking or acting.These examples contrast with turn-based problem solving.Source: §1 · p. 1
  • Although specialized concurrency approaches exist for voice, embodied, and tool-using agents, each research area currently designs and trains for its own tasks.Examples include concurrent listening and thinking, dual actor-thinker architectures, and asynchronous function calling.Source: §1 · p. 1
  • AsyncLLM uses async/await to define concurrent inference coroutines with overlapping memory states, enabling asynchronous inputs and outputs without task-specific fine-tuning.The framework extends direct memory sharing to hybrid and multimodal language models.Source: §1 · pp. 1–2
  • Coroutines write to separate memory blocks and use attention views to access one another’s outputs, while the engine batches concurrent requests for inference.This lets coroutines see each other’s progress and respond to new environment input without waiting for current reasoning to finish.Source: §1 · p. 2, Figure 1 · p. 2
  • Qwen3.x agents are tested for streaming video, videogames, and system monitoring without task-specific fine-tuning, though self-modification is not yet reliable.The experiments suggest that models can define and modify their own coroutines within the same inference loop.Source: §1 · p. 2

2 BACKGROUND

The reviewed applications differ in modality and deployment but share recurring concurrency patterns. The section connects those patterns across voice, video, embodied, computer-use, monitoring, and parallel-reasoning agents, motivating a framework that generalizes without task-specific training.

2 BACKGROUND

  • Voice assistants use full-duplex interaction to listen, think, and speak concurrently, and interruption detectors can pause or adjust responses.The passage also describes shared coding conversations and nonverbal cues as settings where concurrent interaction matters.Source: §2 · p. 2
  • Streaming video systems may skip frames with lightweight probes, compress past events, and reason while watching to keep up with dense real-time input.Related techniques also support responses to traffic changes, GUI pop-ups, and nonverbal cues.Source: §2 · p. 3
  • Embodied agents need to adapt to environmental changes, while many Vision-Language-Action models focus on particular robot types and require fine-tuning for new ones.Some newer models are trained to support multiple embodiments.Source: §2 · p. 3
  • Computer-use and monitoring agents face application-specific inputs and concurrency needs, while parallel reasoning and asynchronous tool use overlap inference with subtasks or waiting.Parallel sub-instances with real-time access to one another’s thoughts have been used to coordinate faster and support safety while reasoning.Source: §2 · p. 3
  • Voice, video, embodied, computer-use, and monitoring agents use similar concurrency patterns despite differing modalities and deployment requirements.The proposed framework aims to generalize across these applications without task-specific training or inference software for each setup.Source: §2 · p. 3

3 ASYNCHRONOUS AGENTS

AsyncLLM extends async/await concurrency to LLM inference through coroutines that share composable memory views. Its cache algorithms support hybrid and multimodal models, while the engine batches concurrent requests for responsive execution.

3.1 ASYNCLLM PROGRAMMING MODEL

  • AsyncLLM uses async/await coroutines whose forward passes write separate CacheBlocks and attend to ordered cache views, enabling shared-state parallel routines.Source: §3 · p. 4, §3.1 · p. 4
  • A thinker writes private reasoning while a writer reads that block to produce a running user-facing summary, without explicit token transfer.Source: §3 · p. 4, Figure 2 · p. 4

3.2 INFERENCE WITH MULTIPLE CACHE BLOCKS

  • O(Nblocks) rather than O(Ntokens) matrix operations per head compose GDN cache-block transitions using affine state summaries.Source: §3.2 · p. 6
  • AsyncLLM combines query-rotated full attention with affine linear-attention composition and MRoPE-corrected multimodal views, tracking temporal spans per block.Source: §3.2 · pp. 5–6

3.3 SCHEDULING AND INFERENCE ENGINE

  • The engine batches prefill and generation across coroutine requests, balances active work to prevent quick reactions being clogged, and chunks long prefills.Source: §3.3 · p. 6
  • Paged attention lets CacheBlocks reference attention pages and GDN transitions, while specialized kernels process cache views for full attention and GDN layers.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments test AsyncLLM on asynchronous text and image changes, streaming video, videogames, and system monitoring. Across these settings, the results show responsive operation, including comparisons with specialized or sequential agents.

4 EXPERIMENTS

  • The experiments frame these evaluations as tests of training-free asynchronous operation across text, visual, video, interactive, and monitoring settings.The study begins with single asynchronous inputs, then evaluates streaming video, videogames, and system monitoring.Source: §4 · p. 7, §4.1 · p. 7, §4.2 · p. 7

4.1 SANITY CHECKS: SINGLE ASYNCHRONOUS INPUT

  • AsyncLLM responds to asynchronous inputs in both text and visual tasks, although visual-task accuracy drops faster when clarifications arrive late.The authors attribute this faster drop to visual tasks requiring less reasoning on average, so some late clarifications arrive after the answer has been produced.Source: §4.1 · p. 7

4.2 STREAMING VIDEO UNDERSTANDING

  • The streaming-video agent uses concurrent event detection, background reasoning, output probing, description writing, and result compilation.Its probes use the base LLM with a pre-filled prompt rather than a separately trained submodule.Source: §4.2 · p. 7
  • AsyncLLM with Qwen3.5+ models outperforms Mage-VL on both streaming-video benchmarks.The table covers SoccerNet and four ProactiveVideoQA subsets and totals; the larger Qwen3.5+ models also provide more accurate descriptions on ProactiveVideoQA.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 INTERACTIVE AGENTS FOR VIDEO GAMES

  • In two ViZDoom scenarios, AsyncLLM reacts faster than a sequential agent using the same model while preserving reasoning gains, but the evaluation establishes basic capability rather than state-of-the-art performance.Figure 4 plots average reward against mean observation-to-action delay over 100 episodes.Source: Figure 4 · p. 8, §4.3 · p. 8

4.4 SYSTEM MONITORING AGENTS

  • AsyncLLM monitoring agents detect problems with comparable accuracy but significantly faster than comparable sequential agents.The evaluation uses log chunks and an anti-distraction prompt; it measures response time and inference steps across 34 DevOps-Gym Monitoring tasks.Source: Table 2 · p. 9, §4.4 · p. 9

5 DISCUSSION AND FUTURE WORK

The discussion presents asynchrony as a general LLM capability and AsyncLLM as a way to adapt agents across environments. It identifies better general asynchronous-agent training and agent-led coroutine improvement as future directions.

5 DISCUSSION AND FUTURE WORK

  • AsyncLLM uses the async/await model and efficient shared-memory inference algorithms to support adaptation across asynchronous text and visual updates, streaming video, monitoring, and videogames.The paper suggests this could support asynchronous APIs or agents that write their own harnesses, while noting that modern models do not always outperform purpose-trained domain models but are easier to adapt.Source: §5 · p. 9
  • Future work could train LLMs as better general asynchronous agents and further investigate their ability to improve their own coroutines for specific scenarios.Source: §5 · p. 9

AI USE STATEMENT

The authors report uses of generative AI tools in dataset preparation, evaluation, prototyping, and drafting selected templates, and state that they reviewed the AI-assisted work.

AI USE STATEMENT

  • Generative AI tools supported synthetic-image preparation, MATH-500 LLM-as-a-judge evaluation, early prototyping, one agent implementation, and drafting table and plot templates; the authors say they reviewed the assisted work.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement flags risks from concurrent execution and asynchronous inputs, including premature harmful actions and behavior changes during ongoing computation. It calls for deployment-appropriate guardrails, sandboxing, validation, and monitoring in sensitive settings.

ETHICS STATEMENT

  • Concurrent execution can produce incorrect or harmful actions before slower reasoning or monitoring intervenes, and asynchronous inputs can change behavior during ongoing computation.The authors recommend guardrails, sandboxing, output/action validation, and monitoring for sensitive real-world deployments; self-modifying-agent experiments also required strict limits on what agents could modify.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The appendix provides reproducibility materials, surveys recurring concurrency patterns, and documents prompts used to construct and validate ShardedVQA examples. Its prompt workflow distinguishes corrected source images from erroneous versions and includes separate solving and auditing steps.

REPRODUCIBILITY STATEMENT

  • The supplementary materials include the AsyncLLM implementation, sharded VQA dataset, experiment configurations, and prompts, with code and evaluation data slated for release upon publication.Source: REPRODUCIBILITY STATEMENT · p. 10

A PATTERNS OF CONCURRENCY

  • Table 4 compares representative concurrency mechanisms by pattern and training requirements; patterns can overlap, and AsyncLM appears in both training categories because its variants differ.Source: Table 4 · p. 29, A PATTERNS OF CONCURRENCY · p. 29

B PROMPTING FOR SHARDEDVQA CONSTRUCTION

  • For ShardedVQA construction, the source image is the corrected state, while an editor constructs the erroneous image presented first during evaluation.The prompts also require localized coherent edits and preserve unrelated image content.Source: B PROMPTING FOR SHARDEDVQA CONSTRUCTION · pp. 30–31
  • The construction workflow separately blind-solves normalized images, audits the erroneous–original pair, and uses a text-only answer-equivalence check when basic matching fails.A proposal prompt suggests edits for human review rather than approving them.Source: B PROMPTING FOR SHARDEDVQA CONSTRUCTION · pp. 30–31

C ADDITIONAL IMPLEMENTATION DETAILS

AsyncLLM schedules concurrent coroutines in balanced batches so reactive work can progress alongside larger requests, while shared cache blocks support batched inference. The reference engine is a minimal implementation with room for further optimization.

C ADDITIONAL IMPLEMENTATION DETAILS

  • AsyncLLM modifies first-come-first-served scheduling with chunked prefilling and shared cache blocks to keep reactive coroutines from being clogged by background processing.Source: C ADDITIONAL IMPLEMENTATION DETAILS · p. 32
  • The scheduler balances token processing across concurrent coroutines so a quick reactive request can progress alongside expensive prefills.It treats prefill and decode requests equally, taking equal-sized chunks from large prefills when possible.Source: C ADDITIONAL IMPLEMENTATION DETAILS · p. 32
  • Requests enter a shared deque, are batched up to the simultaneous-batch limit, and unfinished prefills return to the queue; coroutines writing to the same cache block run in separate batches.Source: C ADDITIONAL IMPLEMENTATION DETAILS · p. 32
  • The reference implementation modifies mini-SGLang for mixed prefill/decode scheduling and Qwen 3.5+ GDN and vision support, but remains a minimal implementation with further optimization possible.Despite that scope, the authors report real-time or near-real-time performance through concurrency and memory reuse.Source: C ADDITIONAL IMPLEMENTATION DETAILS · pp. 32–33

D COMPATIBILITY WITH DIFFERENT ARCHITECTURES

Appendix D describes how AsyncLLM’s cache-view approach extends across inference optimizations and architecture variants, with compatibility depending on how attention and recurrent states use logical positions and memory. Some variants require additional kernel work or remain unimplemented.

D COMPATIBILITY WITH DIFFERENT ARCHITECTURES

  • AsyncLLM’s compatibility discussion covers quantization, KV compression, speculative decoding, attention variants, recurrent-state updates, and positional embeddings beyond its Qwen 3.5+ focus.The paper says there is no fundamental limitation to generalizing the approach to other models and inference stacks.Source: D COMPATIBILITY WITH DIFFERENT… · p. 33, §D.1 · p. 33

D.1 EFFICIENT INFERENCE TECHNIQUES

  • Weight quantization, KV compression, and speculative decoding can be integrated, but the experiments omit them to separate their efficiency gains from AsyncLLM’s.The paper describes weight compression as compatible as-is, cache techniques on a per-block basis, and speculative decoding through api.generate(...).Source: §D.1 · p. 33

D.2 FULL ATTENTION VARIATIONS

  • Sliding-window and block-local attention can use logical positions to select visible tokens, while MQA and GQA can share cached key–value heads across query heads.Attention masks must be defined over the logical cache view; mechanisms that require changing information embedded in cached representations may require recomputation and fall outside direct CacheBlock rearrangement.Source: §D.2 · pp. 33–34
  • MLA can reuse position-independent latent state across cache views, but requires model-specific attention-kernel and cache-representation adaptation; an optimized implementation is future work.The paper also describes learned sparse attention as complementary to AsyncLLM, but leaves its implementation and empirical evaluation to future work.Source: §D.2 · p. 34

D.3 LINEAR ATTENTION VARIANTS

  • Affine recurrent transitions can be cached as block summaries and composed in a different order, supporting DeltaNet, Gated DeltaNet, KDA, and standard linear attention.Exact fixed-size summaries do not represent variants whose updates depend nonlinearly on the previous memory.Source: §D.3 · p. 34
  • GDN’s short causal convolution states are retained as computed when each block is created rather than recomputed after cache-view rearrangement.The paper gives a receptive field of 4 as an example for Qwen 3.5.Source: §D.3 · p. 35

D.4 POSITIONAL EMBEDDING VARIANTS

  • Cache rearrangement works for positional schemes whose score contribution depends on relative displacement and can be adjusted efficiently at query time, including MRoPE-style, ALiBi-style, and NoPE-style cases.For multidimensional RoPE variants, this includes temporal, height, and width coordinates when the corresponding rotations remain known at inference time.Source: §D.4 · p. 35

D.5 NON-EARLY-FUSION MULTI-MODAL LLMS

  • The current implementation targets early-fusion multimodal LLMs; late-fusion cross-attention requires a separate visual-memory view, which the paper leaves for future work.The CacheBlock abstraction could in principle expose those memories, but that implementation is not provided.Source: §D.5 · p. 35

E GPU THROUGHPUT EXPERIMENTS IN CONTROLLED ENVIRONMENTS

The section measures AsyncLLM inference speed with synthetic concurrent decode, prefill, and probe workloads, then reports throughput and latency across GPU configurations and coroutine loads.

E GPU THROUGHPUT EXPERIMENTS IN CONTROLLED ENVIRONMENTS

  • The synthetic benchmarks measure concurrent decoding, prefilling, and probe coroutines, with each reported value as the median of three measurements.The decode coroutine generates into a growing cache block, prefill iteratively encodes chunks, and probe performs a single forward pass before clearing its block.Source: E GPU THROUGHPUT EXPERIMENTS IN… · p. 36
  • Table 5 reports total decoding throughput across coroutines for different models and active-coroutine counts, with and without CUDA graphs on one H200 GPU.The prose describes significant gains similar to conventional sequential decoding.Source: Table 5 · p. 36, E GPU THROUGHPUT EXPERIMENTS IN… · p. 36
  • Figure 5 compares GPU throughput and decode-step and prefill latency under synthetic load on one H200, while Table 6 reports prefill, probe, and decode latency across CUDA-graph setups.The agents in Section 4 average 1.5 to 3.5 simultaneous active coroutines.Source: Figure 5 · p. 36, Table 6 · p. 36

F DATASET CONSTRUCTION FOR ASYNCHRONOUS VISUAL REASONING

The appendix describes how ShardedVQA image pairs are constructed and evaluated, then details a streaming-video agent that uses concurrent probes and reasoning to produce event descriptions.

F DATASET CONSTRUCTION FOR ASYNCHRONOUS VISUAL REASONING

  • 513 source-derived image pairs cover mathematical diagrams, charts, tables, maps, and synthetic scenes, pairing an edited image with its corrected original and state-specific answers.Each pair keeps the question fixed; the corrected image is the original benchmark image.Source: F DATASET CONSTRUCTION FOR… · p. 37, Table 7 · p. 37
  • Edits change answer-relevant image content while preserving the question and unrelated semantics; blind solves, answer checks, and audits screen examples before ambiguous cases and duplicates are excluded.Both images are normalized to matching dimensions and aspect ratio, converted to RGB PNG, and stripped of metadata.Source: F DATASET CONSTRUCTION FOR… · p. 37
  • At decoding step k, the edited image is replaced by the corrected image with a notice, while the two images are never shown together and answer metadata stays evaluator-only.The notice says the earlier picture contained an error and asks the model to recheck its reasoning.Source: F DATASET CONSTRUCTION FOR… · p. 37

G STREAMING VIDEO UNDERSTANDING AGENT DESIGN

  • The video agent uses an event probe to trigger background event reconstruction, an output probe to detect new events, and a writer to describe them; utility coroutines gather descriptions and track time.For the ProactiveVideoQA TV subset, speech recognition converts audio to text inputs for the thinker.Source: G STREAMING VIDEO UNDERSTANDING AGENT… · p. 38

H ADDITIONAL STREAMING VIDEO UNDERSTANDING EVALUATIONS

The appendix reports additional streaming-video evaluation settings and prompts, then describes an agent that can modify its own inference code while running. It specifies tuning sweeps and evaluation procedures but supplies no outcome values in these passages.

H ADDITIONAL STREAMING VIDEO UNDERSTANDING EVALUATIONS

  • The Mage-VL sweep varies segment length, FPS, frame count, and codec around defaults; additional ProactiveVideoQA evaluations vary reply-time coefficient ω, and speed is reported in seconds per second.The captions identify the selected SoccerNet-Caption configuration as a balance of metrics and describe the ProactiveVideoQA subsets and ω settings.Source: H ADDITIONAL STREAMING VIDEO… · p. 39, Table 8 · p. 39, Table 9 · p. 40

I STREAMING VIDEO PROMPTING

  • The AsyncLLM event probe uses a prompted LLM to detect noteworthy frame changes, unlike Mage-VL’s trained visual gate; the thinker is instructed to track changes relevant to the viewer’s question.The ProactiveVideoQA judge scores how well accumulated predicted answers cover key information in the ground truth.Source: I STREAMING VIDEO PROMPTING · pp. 40, 42
  • The streaming prompts instruct the thinker to record changes rather than static scene contents and avoid repeating observations; the speaker answers from the current moment.The current-moment answer prompt asks for one plain sentence about the visible action.Source: I STREAMING VIDEO PROMPTING · pp. 41–42

J SELF-DEFINING ASYNCLLM AGENTS

  • A self-defining agent can edit its generate() inference method through tool calls and patch changes into the running instance, while its asynchronous act() method runs each environment step.Cache blocks, background tasks, and other allocated states persist across rewrites; the harness provides evaluation results but no task-specific guidance on the inference structure.Source: J SELF-DEFINING ASYNCLLM AGENTS · p. 43

Related papers

Loading 2609.35427v1…