PaperlayerView exact revision on arXiv

Source-linked AI summary

JarvisBench: Always-on Intelligence Between Humans and Agents

Chen Chen, Zhehuai Chen

arXiv:2608.14870v1cs.AI

TL;DR

Intermittent human attention leaves ongoing agents without timely guidance and users without continuous access to their work. JarvisBench evaluates an always-on coordination layer and finds consistent task gains, while exposing tradeoffs among attention efficiency, response quality, and latency.

  • Problem

    Long-horizon agents and intermittently available users lack evidence from benchmarks evaluating bidirectional coordination of questions, guidance, and consequential decisions.

  • Method

    JarvisBench evaluates an external attention-coordination layer across 45 single- and multi-agent tasks using separate agent-collaboration and user-interaction tracks.

  • Results

    Jarvis consistently improved completed worker configurations, with gains of 4.9–24.7 points on single-agent tasks and 12.5–28.2 points on multi-agent tasks.

  • Takeaways & Limitations

    Attention coordination is useful and technically demanding, with preferred operating points depending on how task quality, response time, and human attention are valued.

  • Takeaways & Limitations

    The configurable intervention policy exposes a tradeoff between useful user information, human-attention cost, and possible worker disruption.

Abstract

from arXiv · show

Long-horizon agents can execute continuously, but human attention remains intermittent and scarce. This creates a bidirectional coordination problem: users may need immediate access to an agent while work continues in the background, whereas agents may encounter consequential decisions that require user judgment after the user has stopped monitoring execution. We posit an always-on attention-coordination layer---\textit{Jarvis}\footnote{Named after the fictional AI assistant in \textit{Iron Man}.}---that mediates this interface and allocates human attention across one or more working agents. We introduce \textit{JarvisBench} to evaluate both directions of this coordination: whether an intermediary can accurately and promptly answer user-initiated questions about ongoing work, and whether it can recognize when an agent requires user judgment, solicit that judgment at the right moment, and route it back to improve task outcomes. JarvisBench contains 45 agentic task instances: 20 single-agent tasks and 25 workstreams organized into 10 multi-agent projects. The tasks span 19 domains and were selected and adapted from more than 2,000 public candidates. Crucially, the need for user attention arises naturally during execution rather than from an obvious omission in the initial prompt. JarvisBench is designed to integrate with arbitrary agent runtimes without modifying their underlying execution loops. Our reference implementation further provides a full-duplex speech interface, allowing users to reach Jarvis naturally while timely attention coordination supports agents working in the background. By separating agent execution from attention coordination, JarvisBench provides a stable evaluation target as agent capabilities continue to improve.

1 INTRODUCTION

JarvisBench addresses the bidirectional mismatch between continuously executing agents and intermittently available human attention. It evaluates an always-on intermediary that answers user questions, recognizes when agent work needs human judgment, and coordinates attention across ongoing tasks.

  • Motivation: Long-horizon autonomy and parallelism increase the need for timely human attention because agents may otherwise pursue the wrong direction unchecked.Human attention remains intermittent and scarce while agents can execute autonomously over extended tasks.
  • Motivation: The attention mismatch is bidirectional: users must interrupt agents to ask questions, while agents may guess when users are unavailable for consequential decisions.Interruptions reduce efficiency and mix transient conversation into the agent’s working context.
  • Motivation: Current agents are poorly calibrated for proactive attention-seeking because they are trained and evaluated primarily for task completion.Proactivity is treated as a separate capability in the cited studies.
  • Jarvis: Jarvis coordinates both directions by remaining continuously available for real-time voice interaction and directing scarce user attention across ongoing agent work.Unlike proactivity alone, this design also gives users a continuously available interface for questions and guidance.
  • JarvisBench: JarvisBench fills the lack of a benchmark for bidirectional attention coordination with 20 single-agent tasks and tracks for agent collaboration and user interaction.The Agent-Collaboration Track evaluates timely intervention for human judgment, while the User-Interaction Track evaluates always-on question answering without interrupting execution.
  • Results: 4.9–24.7 points on single-agent tasks and 12.5–28.2 points on multi-agent tasks are gained with GPT-5.6-Sol as Jarvis across every completed worker configuration.Gain magnitude varies substantially across worker and Jarvis LLMs, indicating Jarvis must understand unfolding work and recognize when human judgment matters.

2 BENCHMARK OVERVIEW

JarvisBench evaluates attention coordination among users, Jarvis, and working agents while keeping agent runtimes fixed. It covers single-agent trajectories and coupled multi-agent projects where consequential user needs emerge during execution.

  • Roles and interface: JarvisBench separates the user, Jarvis, and one or more working agents, with Jarvis observing bounded events and carrying user guidance to relevant agents.The user owns intent, preferences, authority, private context, and acceptance judgments; controlled evaluation simulates this role with a frozen task-specific profile.
  • Roles and interface: Jarvis attaches outside the agent runtime through bounded task events and narrow guidance interfaces, preserving the worker and its underlying loop across runtimes.User-owned information becomes relevant only after execution exposes a consequential decision.
  • Single-agent suite: 20 tasks span 15 domains and 7 attention-need forms in the single-agent suite, testing recognition of user-owned decisions during complete worker trajectories.When intervention is needed, Jarvis pauses the worker, asks one focused question, cancels the pending action, injects scoped guidance, and resumes execution.
  • Multi-agent suite: 10 projects comprise the multi-agent suite: five with two workstreams and five with three, coupled around shared outcomes rather than unrelated tasks.Each project exposes a project-level decision whose relevance emerges as work develops, requiring Jarvis to return human guidance to the working agents.

3 EVALUATION TRACKS AND METRICS

JarvisBench evaluates attention coordination through Agent-Collaboration and User-Interaction tracks over the same tasks. It scores task outcomes, attention efficiency, response usefulness, and latency using defined evaluation protocols.

  • Evaluation tracks: JarvisBench uses two tracks: Agent-Collaboration tests whether human attention improves agent outcomes, while User-Interaction tests Jarvis’s usefulness when users reach out.Both tracks operate over the same underlying tasks.
  • Outcome metrics: Each episode receives a Task Outcome Score on a 0–100 scale from weighted checkpoints covering execution, user-owned decision alignment, deliverable quality, and safety.Harness or provider failures are marked invalid rather than scored as zero.
  • Agent-Collaboration Track: The Agent-Collaboration Track reports worker-only score, Jarvis score, and mean attention requests, with requests counted whenever Jarvis seeks task-relevant user judgment.A request counts regardless of whether the response ultimately improves the outcome.
  • Agent-Collaboration Track: Attention Efficiency measures the fraction of the full task-score scale gained per requested user turn.It is undefined without requests and may be negative when intervention reduces task quality.
  • User-Interaction Track: The User-Interaction Track uses causal replay at early and late checkpoints, scoring General and Follow-up responses independently on a 0–2 scale and converting averages to 0–100.Latency is reported separately as time to first audio, while failed Jarvis responses receive zero and remain in the denominator.

4 EXPERIMENTAL SETUP

The experimental setup isolates Jarvis as an external attention-coordination sidecar while keeping worker execution unchanged. Evaluations compare worker models, Jarvis language models, intervention conditions, and a speech-interface prototype under controlled replay and latency protocols.

  • Controlled comparison: Jarvis is evaluated without changing the worker model, harness, prompt, tools, task environment, or worker execution loop.Jarvis observes exposed execution events and communicates through the existing interaction boundary as an external sidecar.
  • Controlled comparison: Two complementary comparisons vary worker models with GPT-5.6-Sol fixed as Jarvis, then vary Jarvis LLMs with Claude Opus 4.8 fixed as worker.Scores are macro-averaged over tasks in each suite.
  • Intervention protocol: The worker-only condition makes the user unavailable after the initial request, whereas Jarvis may solicit concise decisions when consequential human judgment emerges.A common intervention policy permits useful requests, discourages unnecessary ones, and fixes the operating point while leaving the attention budget configurable.
  • Jarvis-brain evaluation: Jarvis-brain evaluation replays identical worker trajectories and fixed questions across five language models, with follow-ups based only on visible conversation and responses graded by GPT-5.6-Luna.Latency uses five shared spoken prompts after one discarded warm-up, measured from the end of user speech to the first audible TTS output.
  • Speech interface: The prototype provides always-listening speech through Qwen3-ASR, Silero VAD, and Kokoro-82M, while supporting replaceable turn control and interruption during playback.Interrupting playback does not stop Jarvis reasoning or ongoing execution.

5 RESULTS

Jarvis improves diverse worker agents without modifying their execution harness, with GPT-5.6-Sol achieving the strongest task-outcome and user-facing results. Attention efficiency and latency remain qualified by intervention costs and deployment conditions.

  • Worker-agent comparison: Jarvis improves every completed worker–task configuration, gaining 4.9–24.7 points on single-agent tasks and 12.5–28.2 points on multi-agent tasks.The effect transfers across model families, but its magnitude depends strongly on the worker.
  • Worker-agent comparison: GPT-5.6-Sol produces the largest score gains and highest final scores on both single-agent and multi-agent task topologies.The worker model, harness, environment, and decoding configuration remain unchanged when Jarvis is added.
  • Attention efficiency: DeepSeek V4-Pro is most attention-efficient on multi-agent tasks, obtaining its gain with only 0.30 requests per task.Intervention strength trades off useful user information and task scores against human attention consumption and possible worker disruption.
  • Overall results: The prototype answers all three evaluation questions consistently: external coordination improves workers, GPT-5.6-Sol is the strongest default Jarvis LLM for outcomes, and it provides the best user-facing score.Latency varies across configurations and remains entangled with deployment and speech-serving conditions.

6 RELATED WORK

JarvisBench relates to multi-agent systems and agent proactivity but differs by treating limited human attention as the object of coordination. It evaluates an always-on intermediary connecting users with working agents in both directions without modifying their execution loops.

  • Positioning: JarvisBench treats limited human attention as the object of coordination, unlike multi-agent systems and agent proactivity research.It connects users with one or more working agents in both directions.
  • JarvisBench distinction: The JarvisBench intermediary connects users with one or more working agents in both directions without modifying their execution loops.This separates attention coordination from the underlying agent execution process.
  • Multi-agent systems: Multi-agent systems improve task execution through role specialization, delegation, and structured communication.Examples include AutoGen, MetaGPT, and MultiAgentBench.
  • Agent proactivity: Agent-proactivity benchmarks evaluate interaction beyond static task completion, including tool-mediated interaction and eliciting preferences from underspecified requests.Examples include τ-bench and UserBench.

7 CONCLUSION

JarvisBench evaluates bidirectional attention coordination between users and ongoing agent work across 45 single- and multi-agent tasks. Its results show consistent improvement in completed worker configurations, alongside substantial variation in task gains, attention efficiency, and response quality across LLMs.

  • Benchmark scope: JarvisBench evaluates bidirectional attention coordination between users and ongoing agent work across 45 single- and multi-agent tasks.Its two tracks assess whether Jarvis can use human judgment to improve task outcomes and remain useful when users reach out.
  • Evaluation directions: The benchmark’s two tracks measure both user-initiated assistance and agent-initiated requests for human judgment.These tracks respectively test whether Jarvis remains useful when users reach out and whether human judgment improves task outcomes.
  • Findings: Jarvis consistently improves completed worker configurations, but task gains, attention efficiency, and response quality vary substantially across evaluated LLMs.The conclusion highlights both a consistent configuration-level improvement and substantial model-dependent differences in key evaluation outcomes.

A BENCHMARK CONSTRUCTION · A.1 CANDIDATE COLLECTION · A.2 SELECTION CRITERIA

JarvisBench was constructed from a broad catalog of candidate tasks, then narrowed through criteria requiring naturally emerging, consequential decisions that specifically depend on user judgment. The benchmark excludes tasks whose failures instead reflect prompt omissions, continuous interaction demands, unavailable infrastructure, or wording artifacts.

  • A.1 CANDIDATE COLLECTION: 2,038 candidate tasks were cataloged after auditing 69 existing agent benchmarks and reviewing 12 at the task level.The catalog served as a search space rather than an automatically included benchmark set.
  • A.1 CANDIDATE COLLECTION: Selected tasks retained their upstream benchmark, task identifier, source revision, and adaptation rationale.Every selected task underwent further adaptation and validation beyond catalog inclusion.
  • A.2 SELECTION CRITERIA: The initial request had to provide enough information for an agent to begin and make meaningful progress.This criterion ensures that human attention is not required immediately merely to start execution.
  • A.2 SELECTION CRITERIA: A consequential decision had to become concrete only after the agent inspected task state or produced an intermediate artifact.The benchmark therefore targets attention needs that arise during execution.
  • A.2 SELECTION CRITERIA: The missing decision had to belong to the user, including preferences, authorization, current observations, intended use, or acceptance judgments.The criterion distinguishes unavailable human judgment from facts the agent should retrieve independently.
  • A.2 SELECTION CRITERIA: Tasks were excluded when they omitted an obvious initial-prompt field, required dense interaction, depended on missing tools or unstable services, or relied on grader-specific wording.Task difficulty alone was insufficient; failures had to be attributable to unavailable human judgment.

A.3 TASK ADAPTATION · A.4 VALIDATION

The benchmark adapts selected tasks by preserving their core work while controlling interaction boundaries, private state, dependencies, and decision points. Validation checks ensure workers reach substantial objective work and that graders reward intended decisions rather than specific wording.

  • A.3 TASK ADAPTATION: Task adaptation preserved each candidate’s core work while changing its interaction boundary.Public materials were separated from private user state, unstable dependencies were replaced with reproducible local state when necessary, and private-information relevance was defined.
  • A.3 TASK ADAPTATION: Adaptation did not instruct workers to ask users or select private preferences after observing worker behavior.
  • A.3 TASK ADAPTATION: Single-agent tasks became complete multi-step episodes, while multi-agent tasks became coupled projects sharing entities, constraints, or consequences.Unrelated tasks were not grouped merely to create concurrency.
  • A.3 TASK ADAPTATION: Each task used a frozen environment, worker prompt, private user state, and grader.
  • A.4 VALIDATION: Every task passed static checks, runtime preflight, a worker-only baseline, and manual trace review.
  • A.4 VALIDATION: Validation verified substantial objective work, arrival at the intended decision point, and requester-owned information as the cause of any remaining loss.The process also checked that losses were not caused by harness or evaluator failures.
  • A.4 VALIDATION: Full and partial reference outcomes verified that the grader rewarded the intended decision rather than specific wording.

B TASK DETAILS … C.3 RESPONSE GRADING

JarvisBench spans single-agent and coupled multi-agent tasks across diverse domains, then evaluates attention coordination through causal replay, user follow-ups, and checkpoint-grounded response grading. The protocol separates bounded current state from hidden future information and penalizes unsupported, leaked, or privacy-sensitive responses.

  • B.1 SINGLE-AGENT SUITE: The single-agent suite uses 15 domains, with 15 text-only tasks, 5 text-and-image tasks, 17 workspace-file tasks, and 3 tasks additionally depending on application state.It covers seven attention mechanisms, led by latent context reveal with 9 tasks and authorization boundaries with 6.
  • B.2 MULTI-AGENT SUITE: The multi-agent suite comprises 10 coupled projects, split between five projects with two workstreams and five with three.Distributed evidence in each project exposes a shared user-owned decision affecting the final integrated result.
  • C USER-INTERACTION REPLAY PROTOCOL: JarvisBench replays each complete worker trajectory causally while exposing only the bounded state available at each checkpoint.Future events, final artifacts, and grader information remain hidden.
  • C.1 REPLAY CHECKPOINTS AND GENERAL QUESTIONS: Two fixed replay checkpoints support identical General questions across every Jarvis LLM, including a late checkpoint at approximately 75% completion.The late question asks where work stands and how close it is to completion.
  • C.2 FOLLOW-UP GENERATION: After each General response, GPT-5.6-Luna generates one concise spoken follow-up using only the task brief and visible user–Jarvis conversation.The follow-up must clarify a concrete statement without introducing unstated facts or requesting changes to the read-only replay.
  • C.3 RESPONSE GRADING: GPT-5.6-Luna grades each response independently using the current checkpoint, task brief, visible conversation, and Jarvis answer, while excluding future and private information.Responses receive grades 2, 1, or 0 based on substantive supported usefulness, correctness with incompleteness, or important error or unsupported certainty.
  • C.3 RESPONSE GRADING: Permitted error tags include unsupported claim, future leak, contradiction, vague, nonanswer, and privacy leak, with nonanswers forced to zero.Any future leak or privacy leak sets all four response grades to zero for that trajectory.

D AUDIO IMPLEMENTATION DETAILS

The audio implementation combines modular voice-turn detection, local streaming speech recognition, incremental speech synthesis, and playback-time barge-in handling. These components coordinate full-duplex interaction while preserving ongoing Jarvis responses and worker execution during interruptions.

  • Input and turn detection: The prototype processes 16-kHz microphone audio in 512-sample frames, with Silero VAD opening turns after approximately 96 ms of speech and closing them after approximately 544 ms of silence.Qwen3-ASR-0.6B runs locally in 4-bit MLX format; rolling hypotheses are displayed, but only the final VAD-delimited transcription is sent to Jarvis.
  • Input and turn detection: The modular turn controller can replace Silero VAD with SoulX-Duplug, which jointly performs streaming ASR and semantic dialogue-state prediction.SoulX-Duplug distinguishes completed turns, incomplete pauses, and backchannels without changing Jarvis or worker interfaces.
  • Output streaming: Jarvis accumulates streamed LLM tokens until a sentence boundary or an 80-lexical-unit segment, then submits each segment immediately to Kokoro-82M for incremental PCM synthesis.Text generation, subsequent-segment synthesis, and current-segment playback proceed concurrently through one persistent audio stream.
  • Barge-in: 0.8 seconds of sustained user speech confirms a barge-in and immediately cancels the active TTS stream during playback.The confirmation audio and short pre-roll enter the normal ASR path, while the Jarvis response remains preserved and the worker is neither cancelled nor paused.
Loading 2608.14870v1…