PaperlayerView exact revision on arXiv

Source-linked AI summary

Jarvis: Coordination Deserves a Brain of Its Own in Agentic Interactive Models

Yuxiang Wang, Shengbo Cai, Zhixian Zhao, Yingda Shen, Yuancheng Wang, Xueyao Zhang, Liqiang Zhang, Teddy Sun, Steve Yves, Zhizheng Wu

arXiv:2610.23804v1cs.SD

TL;DR

Agentic assistants split conversation and background work across fast and slow brains, but a user can change a request while it runs. Jarvis coordinates them by interpreting turns against live task state and enforcing decisions against current versions; with the front model and back end fixed, it substantially improves task success across benchmarks and front models.

  • Problem

    Realtime models can sustain dialogue but lack the capacity for complex tool use and multi-step reasoning, motivating a division of labor with background agents.

    Source: §1 · p. 2

  • Method

    Jarvis uses JEV to select runtime-built operations on a versioned Task Ledger, while code checks choices against current state and rechecks plans and results.

    Source: §1 · p. 4, §2.1 · p. 5

  • Results

    192 of 278 τ-Voice tasks were solved by Jarvis versus 29 for QAA on Venus-Audio, with gains holding across every benchmark and front model.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Jarvis carries its coordination design to vision by treating visual evidence as time-bounded state and rechecking it before and during playback.

    Source: §1 · p. 4, §8 · p. 21

Abstract

from arXiv · show

Interactive models increasingly hand complex reasoning and tool use to a slow brain in the background, leaving a realtime front model free to sustain full-duplex dialogue. Yet the user may refine, cancel or replace a request while its task runs, so a plan or result that made sense a moment ago goes stale. Each new utterance must therefore be read against the work in flight, but the front model follows only the conversation and the back end sees only its own task. Coordination deserves a brain of its own, one that sees both. We introduce Jarvis, a coordination harness where models judge and code enforces. Its typed decision model, JEV, reads each turn against the live tasks and picks among operations built from a versioned task record. Code checks every choice against the current state before applying it, so no decision can act on a task version the user has replaced. Jarvis also rechecks plans before they execute and results before they are spoken, and counts an answer as delivered only once the user has heard it in full. We build two instances of this design. Jarvis-Audio runs unchanged on three full-duplex speech models. Jarvis-Omni adds vision and rechecks the evidence behind an answer before and during playback, since a scene can change while nobody speaks. With the front model and back end held fixed, replacing Qwen Audio Agent with Jarvis raises tau-Voice task success more than sixfold, from 29 to 192 of 278 tasks, and the gains hold on every benchmark and front model. Against published systems, Jarvis-Audio reaches 69% Pass1 on Full-Duplex-Bench v3 with an open 9B front model, versus 60% for GPT-Realtime. These results suggest that how the two brains are coordinated matters as much as how strong they are.

1. Introduction

Agentic assistants split conversation and background work across fast and slow brains, but users can change a request while it runs. Jarvis adds a coordination harness that reads turns against live tasks and checks actions against current state.

Motivation

  • Background work can go stale when users refine, cancel, or replace tasks, while the front model and backend each see only part of the live context.A task manager alone may still map an utterance to a nonexistent or outdated task version.Source: §1 · p. 2

Instances

  • Jarvis-Omni adds time-bounded visual facts and rechecks answer evidence before and during playback because scenes can change while nobody speaks.Jarvis-Audio also runs unchanged on three full-duplex speech models.Source: §1 · pp. 3–4

Results

  • 192 of 278 τ-Voice tasks were solved by Jarvis versus 29 for QAA on Venus-Audio, with gains on every benchmark and front model.With the front model, admission path, and back end held fixed, replacing QAA with Jarvis produced these gains.Source: §1 · p. 4
  • Jarvis-Audio reaches 69% Pass@1 on Full-Duplex-Bench v3 with an open 9B front model, versus 60% for GPT-Realtime.These are published-system comparison results, distinct from the controlled QAA comparison.Source: §1 · p. 4

Jarvis

  • JEV selects runtime-built operations on a versioned Task Ledger, and code checks each choice against the current task version before applying it.Jarvis also rechecks plans before execution and results before delivery, which is recorded only after full playback.Source: §1 · pp. 3–4

2. Jarvis at a Glance

Jarvis treats coordination as a state-tracking and enforcement problem: it makes live task state explicit and checks actions against it. Its fast and slow brains operate asynchronously, with the slow brain interpreting turns and the fast brain sustaining conversation.

2.1. What Goes Wrong Without Coordination

  • The six failure modes arise when the system acts on a picture of the conversation that is no longer true; Jarvis makes state explicit and checks every action against it.Five failure modes occur with audio alone, and a sixth appears when the agent can see.Source: §2.1 · p. 5

2.2. Design Principles

  • Jarvis learns when to hand off while keeping speech weights fixed, lets models judge among runtime-prepared operations that code validates, rechecks versioned plans and results, and separates task completion from delivery.These are the four design principles: learn when, keep how; models judge, code enforces; nothing is assumed current; done is not delivered.Source: §2.2 · p. 5

2.3. How a Request Flows

  • The Native Gate can keep listening, let the front model answer, or hand a turn to the slow brain; JEV selects operations, the Ledger runs workers, and Delivery returns accepted results in the front model’s voice.Jarvis-Omni reuses the Ledger, workers, and Delivery while adding a perception layer that timestamps visual facts.Source: §2.3 · p. 6

3. The Jarvis Harness

Jarvis coordinates interactive work through a Native Gate, typed decisions grounded in a versioned Task Ledger, and runtime checks that keep task state, plans, and delivery current. Its memory separates durable personal facts from task-specific conditions.

3.1. Native Gate: Learning When to Hand Off

  • The Native Gate uses front-model control tokens to choose listening, direct response, or delegation, and reaches 98.8% correct first admissions on held-out scenario families.Prefixes ending before a request is complete are labeled LISTEN; the speech-producing weights remain frozen while a separate adapter is trained for routing.Source: §3.1 · pp. 6–7

3.2. Decision Brain: Typed Judgments with JEV

  • JEV answers typed questions using runtime-built options from live tasks, so accepted operations name existing task versions and stale-version choices become no-ops.Below the 0.8 confidence threshold, the runtime avoids guessing and sends the turn to a worker, leaving existing tasks unchanged.Source: §3.2 · p. 8

3.4. Delivery: From Done to Heard

  • A task becomes DELIVERED only after a playback receipt confirms the full answer played without interruption in the current generation epoch.Stopping speech opens a new epoch and holds pending announcements without cancelling their tasks; cancelling invalidates a task version and stops its worker.Source: Figure 6 · p. 9, §3.4 · p. 10

3.3. Task Ledger and Workers

  • The Task Ledger versions requests and enforces one live version, rejects late results, and rechecks worker plans against the current conversation before execution.Workers run asynchronously, treat tool results as evidence rather than instructions, and ask for missing details instead of guessing.Source: §3.3 · p. 9

3.5. Memory: Durable Facts Without Drift

  • Memory separates persistent profile facts and preferences from past task records, and only saves a durable personal fact after JEV approves it.Task-specific conditions stay with the task; the assistant reports a save only after the write is confirmed.Source: §3.5 · pp. 10–11

4. Jarvis-Omni: When Evidence Expires

Jarvis-Omni extends Jarvis to visual interaction, where evidence can become stale even without a new utterance. It tracks the lifetime of visual claims and rechecks them before and during playback.

4. Jarvis-Omni: When Evidence Expires

  • Jarvis-Omni treats visual facts as versioned state and checks evidence behind an answer again before it reaches the user.An answer whose evidence expires is dropped or cut off.Source: §4 · p. 11

4.1. Evidence Timeline and Belief State

  • The Evidence Timeline aligns screen, game-audio, and microphone tracks to one clock, and limits each decision to media available at that time.A new scene epoch clears the window so earlier-scene media cannot leak into the current one.Source: §4.1 · p. 11
  • Belief State claims record their supporting observations, validity periods, and invalidating events; expired claims cannot support current advice.Responses are checked again before playback, and playback stops if evidence expires mid-answer; questions about past advice use historical snapshots.Source: §4.1 · p. 11, Figure 7 · p. 12

4.2. A Multimodal Decision Brain

  • Jev-Omni is a 12B multimodal typed judgment model that reviews visual claims and selects tactical advice suited to the current screen.A fast path starts speaking about 2.2 s after a question; the gate also supports unsolicited screen analysis and was trained to miss very few actions.Source: §4.2 · p. 12, §4.3 · pp. 12–13

5. Experiments

With the front model, admission path, and back end fixed, Jarvis substantially improves voice-agent benchmark results over QAA. Experiments also test its runtime checks, decision timing, memory, and visual evidence handling.

5.1. Setup

  • The evaluation uses 278 regular 𝜏-Voice tasks, 100 real FDB-v3 recordings, 200 FDB-v2 tasks at two paces, and ContextDialog conversations.On 𝜏-Voice, a task counts as solved only when the official reward for final database state and required actions equals 1.Source: §5.1 · p. 13

5.2. Does the Harness Matter?

  • 192 of 278 𝜏-Voice tasks are solved by Jarvis on Venus-Audio, versus 29 for QAA, with Jarvis improving on QAA across all nine FDB-v2 front-model/category combinations.The comparison holds the front model, admission path, and back end fixed; the reported gains include fewer lost requests and better results on state-changing work.Source: §5.2 · pp. 13–15, Figure 8 · p. 14

5.3. Comparison with Leading Voice Systems

  • 69% Pass@1 on FDB-v3 is reported for Jarvis-Audio, nine points above GPT-Realtime; its 𝜏-Voice macro success rate is 69.2%, above GPT-Realtime 2 at 42.4%.Jarvis-Audio uses an open 9B front model; the two systems ranked above it on the leaderboard use proprietary frontier background models.Source: §5.3 · p. 15

5.4. Why It Works

  • The full runtime is consistent in all 240 race runs, while removing either result checks or plan checks lets superseded work reach the speaker in 72 runs.Without playback receipts, 36 runs falsely mark an answer delivered even though the user did not hear it in full.Source: §5.4 · p. 16
  • JEV makes a correct, confident decision within 1 s in 91% of test calls, a rate neither general LLM controller achieves.In runtime replays, JEV binds the right operation to the right task in 252 of 256 cases and stops obsolete work in time in 150 of 224.Source: §5.4 · pp. 17–18

5.5. Memory

  • On ContextDialog, both Jarvis memory back ends score close to 3 points above QAA’s default memory, with a clustered interval of [2.43, 3.05] for JEV selection’s gain.JEV selection scores 4.69 versus 4.36 for the hybrid retriever on 45 questions where it retains records, but discards every record on eight questions.Source: §5.5 · p. 18

5.6. Does It Carry Over to Vision?

  • Removing screen frames drops correct answers by 22.7 and 22.5 points in two independent cohorts; evidence checks reduce expired audio reaching users from 12.3 s and 9.9 s to zero and 0.125 ms.Both configurations stay quiet in silent controls, and utterances with valid evidence play in full.Source: §5.6 · p. 19

6. Jarvis in Action

The examples show Jarvis handling changing tasks, user control, and visual game context across everyday assistance and game-companion sessions.

Daily assistance

  • The game-night client shows seven task cards with versions, separates the cancelled welcome sign, and preserves the usual $40 snack budget after a one-night $60 request.In a later session, Jarvis recalls the usual budget and the saved light-theme preference.Source: §6 · pp. 19–20

Playing alongside the user

  • In Plants vs. Zombies and Luanti, Jarvis gives game advice, yields and resumes control, and routes moves through revocable grants tied to game state.Reported outcomes, including plantings and a harvested carrot, come from game-engine receipts.Source: §6 · pp. 19–20

Game companion

  • Jarvis-Omni answers questions about earlier frames, reacts to a victory screen when asked to watch quietly, and switches to relaxed conversation when requested.Source: §6 · p. 19

7. Related Work

Related work spans full-duplex speech, fast-and-slow voice agents, dialogue state, typed decisions, and memory. Jarvis shares components with these lines of work but focuses on coordinating background tasks during live conversation.

Full-duplex spoken dialogue

Fast and slow voice agents

  • Fast-and-slow voice agents include systems with task managers, front-model task calls, or retrieval; Jarvis instead uses a dedicated decision brain and task-version-bound operations.The hand-off is learned on the front model’s native control branch instead of being prompted.Source: §7 · p. 20

Dialogue state, typed decisions and memory

  • Prior dialogue systems track slot states, while Jarvis tracks asynchronous task versions and delivery status; JEV turns action selection into distributions over runtime-supplied options.The paper also distinguishes Jarvis from earlier JARVIS systems for tool orchestration or Minecraft play, which do not coordinate background work during live conversation.Source: §7 · p. 21

8. Conclusion

In interactive systems, each new turn can change what ongoing background work should do. Jarvis coordinates the conversation and live tasks by having models judge and code validate decisions against the latest task state.

8. Conclusion

  • Jarvis gives coordination its own brain: models judge each turn while code checks decisions against the latest state of every task.The need arises because a new turn can change what background work should do.Source: §8 · p. 21

A. Implementation Details

The appendix provides implementation details for Jarvis and compares its two instances. It also lists the decisions assigned to JEV.

A. Implementation Details

  • Table 10 lists every decision handed to JEV; all use a typed interface and return no free text.A confidence threshold determines when the runtime falls back to a safe default or requests a second review.Source: A. Implementation Details · p. 25
  • The appendix separates its implementation comparison from its inventory of JEV decisions in Tables 9 and 10.The section describes details left out of the main text.Source: A. Implementation Details · p. 25, Table 9 · p. 25
  • Table 9 compares the two Jarvis instances, with shared-block rows using the same implementation in both.The section collects details omitted from the main text.Source: Table 9 · p. 25, A. Implementation Details · p. 25

A.1. Jarvis-Audio

Jarvis-Audio validates task-aware decisions and protects task state across asynchronous work. Jarvis-Omni extends the design with visual evidence checks, scene-aware gating, and game-specific memory.

A.1. Jarvis-Audio

  • Jarvis-Audio binds delegation to the most recent final transcript that began before the decision sample and uses separate general and game adapter profiles.Its ASR observer ends an utterance after 1.1 seconds of silence, using an ensemble of three Silero VAD instances.Source: §A.1 · p. 25
  • Jarvis-Audio checks each planned operation: actions must be runtime-supported, refinements or cancellations must name an existing task, and its version must still be live.When a turn is classified as a batch, its contiguous transcript spans are interpreted separately and receive the same checks.Source: §A.1 · p. 25
  • After newly heard but unconfirmed speech, a worker proceeds only if a bounded check finds it clearly unrelated and nothing else changes during the check.The ledger has eight states, including terminal CANCELLED, SUPERSEDED, and FAILED states; a history reset fences off older asynchronous writes.Source: §A.1 · p. 25
  • Jarvis-Audio releases routing decisions from older generation epochs, while replaying a result creates a fresh presentation in the current epoch.Memory records track validity and supersession; history clearing prevents a still-running extraction from restoring cleared facts.Source: §A.1 · p. 26

A.2. Jarvis-Omni

  • Jarvis-Omni aligns media tracks to a continuously drift-corrected monotonic clock and requires some facts, such as low health, to come from the screen alone.Its decision model reviews visual claims and tactical advice; open-ended descriptions proposed by Gemma-4-12B-IT require separate review.Source: §A.2 · p. 26
  • All 18 replies were heard with the new Jarvis-Omni gate in live sessions, versus 13 with the previous adapter on the same 36 recordings.The gate uses dialogue context including previously admitted turns, actually played assistant audio, user profile, and task state.Source: §A.2 · p. 27
  • Jarvis-Omni records match events with their source and whether they were observed, inferred, or user-reported; only explicit user corrections can supersede them.Recalled memory can be cited only as a past event, not reintroduced into the Evidence Timeline as current observation.Source: §A.2 · p. 27

B. Training the Native Gate

The Native Gate is trained on authored and synthesized scenarios through a staged curriculum, with checkpoint selection protecting earlier-stage accuracy. In the final stage, it improves accuracy on new development data while preserving earlier performance and task delegation.

Scenarios

  • Training uses 264 authored Chinese and English scenarios spanning direct responses, delegation, listening, continuity, telephony, and conversational requests.The scenarios cover daily-use speech and include no-game, PvZ, and voxel-sandbox contexts.Source: B. Training the Native Gate · p. 27

Objective

  • GRPO samples eight actions per prefix and combines a clipped policy-gradient term, a KL penalty toward the stage-start adapter, and a small entropy bonus.Correct actions earn +2 and wrong actions −1; there is no value network or token-level supervised loss.Source: B. Training the Native Gate · p. 28

Curriculum and selection

  • The four-stage curriculum adds telephony, an expanded scenario pool, and conversational requests after initial causal-timing training, while replaying older material in every batch.Half of the LISTEN slots use complete speech not addressed to the assistant; eligible checkpoints must not lose accuracy on the earlier-stage development set.Source: B. Training the Native Gate · p. 28

Results

  • Accuracy on the new development set rose from 149 of 164 to 162 of 164 (90.9% to 98.8%), while the earlier set stayed at 40 of 40 throughout.After training, all 78 conversational requests were answered directly, all 48 tasks remained delegated, and correct actions fired 0.16 seconds after the request ended at the median.Source: Figure 11 · p. 29, B. Training the Native Gate · p. 29

C. Evaluation Protocol

The evaluation uses fixed benchmark tasks, shared admission paths and budgets, official tools and scoring procedures, and configured comparison systems.

C. Evaluation Protocol

  • The 278 regular τ-Voice tasks cover airline, retail, and telecom; FDB-v3 measures tool selection, argument accuracy, response quality, and turn-taking.ContextDialog uses 80 questions: four questions from each of 20 complete conversations across both official splits.Source: C. Evaluation Protocol · pp. 29–30
  • Paired harnesses share admission paths; each task has a 16,384-token output budget, 12 tool rounds, and a 120-second limit.MiniCPM-o uses a neutral post-transcript controller, while VoiceChat-11B uses its native function head; Venus-Audio uses the trained Native Gate.Source: C. Evaluation Protocol · p. 30
  • Benchmarks run locally with official task sets, tools, environments, scoring, and judges; systems face the same callers, examiners, and real-time PCM audio.The τ-Voice caller uses DeepSeek-V4-Flash and Qwen3-TTS under official policies, noise conditions, and a 200 ms tick.Source: C. Evaluation Protocol · p. 30
  • Baselines use their released or native runtimes and retrieval paths, with DeepSeek replacing selected back ends under the shared budget.QAA retains its gateway and task-management features; Gander uses released weights and lean orchestration, while MoshiRAG uses native retrieval.Source: C. Evaluation Protocol · p. 30

D. Additional Results

Additional results report paired quality gains, stronger performance on difficult or disfluent speech, and median responsiveness across Jarvis stages.

D. Additional Results

  • With identical admissions, audible responses rose from 68/100 to 97/100 on Venus-Audio; VoiceChat argument accuracy rose from 0.470 to 0.583.On Venus-Audio, mean reward rose 0.62 on retail and 0.70 on telecom; neither QAA nor Gander solved the hardest database-write tasks.Source: Table 14 · p. 30, D. Additional Results · p. 30
  • Jarvis-Audio passed 56.7% of hard FDB-v3 scenarios versus GPT-Realtime’s 43.3%, and 64.7% of self-correction recordings versus 58.8%.Table 15 reports Pass@1 by domain, difficulty, and disfluency type.Source: Table 15 · p. 31, D. Additional Results · p. 31
  • Median JEV settling took 0.81 s and memory-write gating 0.32 s; with MiniCPM-o, Jarvis spoke 3.1 s after requests versus QAA’s 3.5 s.The median tool loop took 4.6 s, and speech request to first generated audio took 2.7 s.Source: Figure 12 · p. 31, D. Additional Results · p. 31

Related papers

Loading 2610.23804v1…