PaperlayerView exact revision on arXiv

Source-linked AI summary

A Voice-Interactive Multi-Agent System for Smart Operating Rooms: Architecture Design and Key Technologies

Tianxiang Zhou

arXiv:2609.11231v1cs.AIcs.CLcs.HC

TL;DR

Operating rooms need safer, coordinated device interaction without the delays and burdens of conventional controls. SurgicalRoomAgent addresses this with a voice-interactive multi-agent architecture and latency-oriented inference techniques. The prototype supports real-time device control and related operating-room functions, but remains limited by clinical validation, concurrency, noise, and context-window constraints.

  • Problem

    Operating-room device control is hindered by sterile-field disruption, contamination risk, cognitive overload, and difficult coordination across proprietary devices.

  • Method

    SurgicalRoomAgent combines a layered voice pipeline and agent core with KV Cache warming, streaming early task execution, progressive prompt disclosure, and DAG task planning.

  • Results

    The system supports end-to-end real-time interaction, reducing KV Cache prefill overhead by 83%–90% when warming hits and operating within a 16,384-token context limit.

  • Takeaways & Limitations

    The prototype integrates voice interaction, multi-device control, intraoperative recording, and surgical report generation in a single operating-room system.

  • Takeaways & Limitations

    Large-scale clinical trials have not been conducted, and the system faces single-LLM concurrency, noisy-environment ASR, and 16,384-token context constraints.

Abstract

from arXiv · show

This paper presents SurgicalRoomAgent, a voice-interactive multi-agent system for smart operating rooms based on large language models (LLMs). The system achieves natural language understanding, device control, intraoperative recording, and surgical report generation through a layered architecture comprising a voice interaction pipeline (wake, ASR, turn detection, agent reasoning, TTS) and an agent core (skill registry, task planner, device manager). Three key technologies are investigated: (1) KV Cache prefix warming for low-latency inference, reducing recomputation overhead from approximately 500 ms to tens of milliseconds via byte-level Longest Common Prefix reuse; (2) streaming partial JSON parsing with early parallel task execution, reducing end-to-end latency by approximately 30%; and (3) progressive skill prompt disclosure, which dynamically filters system prompts based on user role, connected devices, and surgical phase to maximize information density within limited context windows. The system is implemented using the Qwen3-27B model with llama.cpp/sglang inference engines. Experimental analysis demonstrates effective operation within a 16,384-token context limit and multi-device parallel control response times meeting OR real-time requirements.

1 Introduction

Operating rooms require frequent multi-device control, but conventional interfaces disrupt sterile work, increase contamination risk, add cognitive load, and complicate coordination. SurgicalRoomAgent addresses these challenges with a layered voice-interactive multi-agent system and technologies targeting real-time operation.

  • 1.1 Background: SurgicalRoomAgent targets sterile-field disruption, cross-contamination risk, cognitive overload, and multi-device coordination difficulties in operating rooms.These issues arise from reliance on physical controls and proprietary device protocols.
  • 1.2 Related Challenges: Existing research rarely combines real-time voice interaction, coordinated multi-device control, and low-latency inference optimization for operating rooms.Prior systems addressed surgical assistance, robotic surgery, or device control but left parts of this combination unresolved.
  • 1.3 Objectives and Contributions: The paper proposes SurgicalRoomAgent, a voice-interactive multi-agent system for smart operating rooms.The system is designed to address the identified operating-room interaction gaps.
  • 1.3 Objectives and Contributions: Its six-stage voice pipeline spans wake detection, ASR, turn detection, agent reasoning, LLM generation, and TTS, while the agent core manages device control and intraoperative recording.The architecture separates voice interaction from core agent functions.
  • 1.3 Objectives and Contributions: The system combines KV Cache warming, streaming early task execution, progressive prompt disclosure, and DAG scheduling to support low-latency, context-aware multi-device interaction.The contributions include byte-level LCP reuse, partial JSON parsing, contextual skill filtering, and dependency-aware parallel execution.

2 Related Work

Related work demonstrates LLM-based assistance, voice control, medical agents, and low-latency voice processing, but existing systems remain limited in scope or latency. This paper combines these directions with device-state management and structured parallel task planning for operating-room use.

  • Voice-Interactive OR Assistants: GePpeTto provides surgical question answering and intraoperative decision support but lacks device control and real-time voice latency considerations.
  • Related Medical Agents: Other systems address surgical-robot commands, instrument delivery, clinical dialogue, or medical-agent taxonomies, but each focuses on a narrower application or capability.
  • Voice-Interactive OR Assistants: VISA uses a hierarchical multi-agent architecture for voice control of the da Vinci robot, but its applicability is limited to robotic surgery and latency optimization is not publicly detailed.
  • Real-Time Voice Interaction: Low-latency voice-agent research motivates streaming processing, while OR noise and multi-person dialogue make fixed-threshold VAD prone to misjudgment.This work adopts LLM-based binary turn classification for turn detection.
  • LLM Inference Optimization: LLM inference latency includes prefill and decoding, and OR device-status changes can trigger approximately 500 ms of full prefill recomputation.The paper addresses this problem with LCP-based KV Cache reuse and proactive warming.
  • Device Control and Task Planning: DAG scheduling is used to layer dependent tasks for parallel execution, with Kahn’s algorithm and DFS cycle detection supporting valid multi-intent plans.

3 System Architecture

SurgicalRoomAgent separates a voice interaction pipeline from an agent core that understands intents, plans tasks, controls devices, and generates responses. Its architecture adds extensible skills and devices, contextual prompt filtering, permission control, storage, security, and streaming interfaces.

  • 3.1 Overall Architecture: The layered architecture separates voice services from the agent core, which handles intent understanding, task planning, device control, and response generation.Figure 1 presents these two major architectural layers.
  • 3.2 Voice Interaction Pipeline: The voice pipeline uses microservices for wake detection, real-time ASR, agent reasoning, and streaming TTS communication.The agent service exposes REST and WebSocket interfaces, while TTS can begin synthesis from the first text fragment.
  • 3.3 Agent Core: The agent core integrates device management, dialogue management, skill scheduling, LLM inference, recording, reporting, and dialogue state in dependency order.
  • 3.3.1 Skill Registry: Progressive disclosure filters skill prompts by user role, connected devices, and surgical phase before constructing the system prompt.This avoids prompts for unavailable devices or inactive skills.
  • 3.3.2 Task Planning: The task planner builds executable DAGs, using Kahn’s layered topological sorting for parallel independent tasks and DFS three-color detection for circular dependencies.
  • 3.3.3–3.4 Implementation: Template-based device management, decorator-based registration, RBAC, SQLite stores, JWT authentication, field-level encryption, audit logging, and WebSocket streaming support extensibility and operational control.

4 Key Technologies

The key technologies reduce inference overhead and improve execution responsiveness by reusing stable prompt prefixes, starting tasks during streaming generation, and filtering prompts to the active context. PromptWarmer schedules and diagnoses cache reuse around changing device state.

  • 4.1.1 Motivation: Device-status changes can trigger approximately 500 ms of full prefill recomputation, although only a small suffix of the request typically changes.The unchanged system prompt, device definitions, and skill prompts are suitable for reuse.
  • 4.1.2 Byte-Level LCP Reuse Mechanism: Byte-level LCP reuse places user input at the end of the JSON so the prefix remains byte-identical and its tokenized KV Cache can be reused.The warming request uses an empty user-input placeholder before suffix replacement with the real input.
  • 4.1.3 PromptWarmer Component: PromptWarmer uses 10-second deduplication, 300 ms debouncing, single-thread execution, silent failure, and triggers on device changes, dialogue completion, and recording start.
  • 4.1.3 PromptWarmer Component: PromptWarmer asynchronously precomputes shared prefixes, reducing real-request computation to the incremental suffix when the warmed cache is hit.Figure 2 depicts the warming request, cached prefix reuse, and incremental computation.
  • 4.1.4 Diagnostic Mechanism: The system compares short prefix hashes at real inference time to diagnose whether the warming and actual request prefixes are consistent.The diagnostic uses the first eight characters of an MD5 hash for quick log comparison.

4.2 Streaming Partial JSON Parsing and Early Task Execution

The system parses streamed JSON incrementally, launching task execution and voice synthesis as soon as the relevant fields are complete. This creates a generate-while-execute pipeline that overlaps LLM generation with downstream processing.

  • 4.2.1 Streaming Output Architecture: TTS begins when the response field closes, without waiting for later JSON fields or complete generation.Response extraction and TTS triggering proceed concurrently with subsequent LLM output.
  • 4.2.1 Streaming Output Architecture: Streaming output provides real-time response display and enables task execution before the entire JSON response is parsed.These mechanisms reduce waiting by overlapping user-facing output and task processing.
  • 4.2.2 Partial JSON Parsing: Partial JSON parsing extracts field increments and detects when fields or the tasks array are fully closed.The parser provides field extraction, task-array detection, and completion checks during streaming.
  • 4.2.3 Early Task Execution: Complete tasks arrays trigger immediate parallel execution while the LLM continues generating the response.The background executor runs device-control tasks while response generation continues.

4.4 Binary Turn Detection

The system uses an LLM-based binary turn detector to distinguish discardable chitchat or noise from medical commands. Recent dialogue context and constrained outputs support semantic continuity and frontend routing.

  • 4.4.2 Implementation: Binary turn detection classifies input as chitchat/noise or medical command using outputs 0 and 2.The frontend forwards only medical commands to the agent service.
  • 4.4.2 Implementation: GBNF grammar constrains the independent detector LLM to the permitted binary output format.The detector shares the inference engine but uses a separate slot.
  • 4.4.2 Implementation: The detector retains the most recent five dialogue rounds to distinguish chitchat continuation from new medical commands.This context helps account for conversational semantic continuity.
  • 4.3 Task Planning: Multi-task plans encode task identifiers, intents, slots, dependencies, and a response, while the scheduler executes dependency layers in parallel.Tasks in one layer run concurrently; later layers wait for prior layers to complete.
  • 4.3 Task Planning: A fallback single-task method preserves backward compatibility when the LLM output lacks a tasks field.The method wraps slots or multi-intent output as a single-task plan.

5 Implementation and Performance Analysis

The implementation uses a Python-based stack configured for a 16,384-token context and streaming partial-JSON inference. The analysis combines latency optimization, context-window management, and parallel task execution, while noting that reported values are theoretical estimates.

  • 5.1 Implementation Environment: The configured stack enables streaming partial-JSON inference, prompt prefetch, authentication, audit logging, and a 1,024-token output limit.The implementation uses Python 3.10+ and specifies Qwen3 .8-27B-FP8 in the configuration excerpt.
  • 5.3.2 Context Window Utilization: The 16,384-token context leaves about 3,722 tokens beyond the approximately 12,662-token system prompt, with max tokens set to 1,024.The configuration describes this as a sufficient safety margin.
  • 5.3 Performance Analysis: The reported optimization data are theoretical estimates whose realized performance depends on hardware, network environment, and model load.This qualification applies to the presented analysis rather than establishing measured deployment performance.
  • 5.2 Task Execution: DAG scheduling executes dependency-free tasks in parallel and dependent layers sequentially, yielding max(t1, t2) rather than t1 + t2 for independent tasks.The scheduler routes each task to its skill instance and preserves required ordering for dependent sequences.

6 Discussion

SurgicalRoomAgent is distinguished from existing systems by full-pipeline voice interaction, coordinated multi-device control, latency optimization, and production-grade security. Its reported advantages include real-time responsiveness, extensibility, security, and adaptive prompt use, while clinical, concurrency, ASR, device-integration, and context-window limitations remain.

  • 6.1 Comparison with Existing Systems: SurgicalRoomAgent adds full-pipeline voice interaction, coordinated multi-device control, system-level latency optimization, and production-grade security.The comparison is summarized as a distinction from existing systems, including DAG-based cross-device orchestration and security controls.
  • 6.2 Advantages: 83%–90% prefill-overhead reduction and time-to-first-token within 200 ms support low-latency responsiveness when cache warming hits.The system combines KV Cache prefix warming with streaming early task execution for OR scenarios with frequent device-status changes.
  • 6.2 Advantages: The skill registry and device manager are designed for extensibility, allowing new devices or skills without modifying core logic.New integrations require implementing abstract methods and adding decorators.
  • 6.3 Limitations: The prototype lacks large-scale clinical validation, and its single Qwen3-27B instance may bottleneck concurrent multi-user operation.GPU memory constraints limit concurrency despite llama.cpp multi-slot parallelism.
  • 6.3 Limitations: OR noise can reduce ASR robustness, while manufacturer-specific protocols require adaptation layers for new device integrations.The paper also identifies the 16,384-token context window as potentially insufficient for long, complex surgical dialogues.

7 Conclusion

The paper presents SurgicalRoomAgent as a layered voice-interactive multi-agent system for real-time operating-room device control. Its contributions combine latency optimization, prompt management, and DAG scheduling, while future work targets multimodal fusion, model fine-tuning, and clinical evaluation.

  • 7 Conclusion: SurgicalRoomAgent achieves end-to-end real-time interaction from voice input to device control through a layered architecture.The system supports streaming output, real-time device control, intraoperative recording, and surgical report generation.
  • 7 Conclusion: The system combines byte-level LCP KV Cache warming, streaming early execution, progressive prompt disclosure, and Kahn-based DAG scheduling.These mechanisms target prefill latency, execution overlap, context utilization, and multi-intent dependency management.
  • 7 Conclusion: Future work will address multimodal fusion, model fine-tuning, and clinical-trial evaluation to move the prototype toward clinical application.

A System Module Statistics

The section presents a table containing code statistics for each system module.

  • A System Module Statistics: Table 6 presents code statistics for each system module.

B Voice Pipeline Port Allocation

The section presents a table containing port-allocation information for the voice-pipeline services.

  • B Voice Pipeline Port Allocation: Table 7 presents the port allocation for each voice pipeline service.
Loading 2609.11231v1…