PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamingClaw Technical Report

Jiawei Chen, Zhe Chen, Chaoqun Du, Maokui He, Wei He, Hengtao Li, Qizhen Li, Zide Liu, Hao Ma, Xuhao Pan, Chang Ren, Xudong Rao, Xintian Shen, Chenfeng Wang, Tao Wei, Chengjun Yu, Pengfei Yu, Shengyu Yao, Chunpeng Zhou, Kun Zhan, Lihao Zheng, Pan Zhou, Xuhan Zhu, Yufei Zheng

arXiv:2603.22120v2cs.CV

TL;DR

StreamingClaw targets the fragmented capabilities of existing streaming video agents, including limited online reasoning, long-term multimodal memory, and proactive interaction. It presents a unified OpenClaw-compatible framework combining streaming inference, hierarchical memory, future-event reasoning, and perception–decision–action tools. The framework supports real-time streaming interaction and closed-loop embodied operation, while remaining limited in audio input, fine-grained audiovisual alignment, and end-to-end cross-modal reasoning.

  • Problem

    Existing agents often lack low-latency streaming reasoning, long-term multimodal memory, and proactive interaction for continuous real-world video understanding.

  • Method

    StreamingClaw integrates streaming inference, autonomous multi-agent scheduling, hierarchical multimodal memory, proactive interaction, and action-oriented tools within an OpenClaw-compatible framework.

  • Results

    StreamingClaw supports real-time video ingestion and streaming inference, cooperative future-event reasoning, hierarchical memory management, proactive interaction, and perception–decision–action execution.

  • Takeaways & Limitations

    The framework provides a unified basis for real-time streaming interaction and closed-loop embodied intelligence in applications such as embodied systems and autonomous driving.

  • Takeaways & Limitations

    StreamingClaw remains limited to a vision-plus-text input paradigm, with limited audio input, fine-grained audiovisual alignment, and end-to-end cross-modal joint reasoning and generation.

Abstract

from arXiv · show

Emerging applications such as embodied intelligence, AI hardware, autonomous driving, and intelligent cockpits rely on a real-time perception-decision-action closed loop, posing stringent challenges for streaming video understanding. However, current agents mostly suffer from fragmented capabilities, such as supporting only offline video understanding, lacking long-term multimodal memory mechanisms, or struggling to achieve real-time reasoning and proactive interaction under streaming input. These shortcomings have become a key bottleneck for preventing agents from sustaining perception, making real-time decisions, and executing closed-loop actions in complex real-world environments, constraining their deployment and potential in dynamic, open physical worlds. To alleviate these issues, we propose StreamingClaw, a unified agent framework for streaming video understanding and embodied intelligence. Beyond maintaining full compatibility with the OpenClaw framework, it natively supports real-time, multimodal streaming interactions. StreamingClaw integrates five core capabilities: (1) It supports real-time streaming reasoning. (2) It supports reasoning about future events and proactive interaction under the online evolution of interaction objectives. (3) It supports multimodal long-term memory storage, hierarchical memory evolution, efficient memory retrieval, and memory sharing across multiple agents. (4) It supports a closed loop of perception-decision-action. In addition to conventional tools and skills, it also provides streaming tools and action-centric skills tailored for real-world physical environments. (5) It is compatible with the OpenClaw framework, allowing it to leverage the resources and support of the open-source community.

1 Introduction

StreamingClaw addresses the need for continuous perception, long-term memory, proactive interaction, and physical action in real-world streaming environments. It uses a multi-agent framework to connect streaming understanding with executable tools and skills.

  • Embodied systems require low-latency streaming video understanding and continuous spatiotemporal perception to avoid delayed actions or hasty decisions.
  • Streaming perception treats dynamic environments as continuously evolving streams rather than offline videos processed in advance.
  • Long-term multimodal memory preserves information about environmental evolution beyond limited frames or short clips, supporting reliable interaction and task execution.
  • Proactive interaction links visual semantics to executable actions through active perception that gathers information for decision-making and complex-task execution.
  • StreamingClaw represents video as continuous spatiotemporal data and uses autonomous multi-agent scheduling, tools, and skills for instruction-driven embodied intelligence.
  • Its perception–decision–action loop feeds agent-produced instructions to tools and skills, then returns execution results to the agents.

2 Framework of StreamingClaw

StreamingClaw standardizes streaming inputs from multiple endpoints before routing them through collaborative agents for reasoning, memory, proactive decisions, and tool execution. Its pipeline supports different device conditions through timestamp alignment, configurable caching, and multimodal processing.

  • StreamingClaw accepts streaming inputs from handheld devices, vehicles, smart glasses, and embodied robots, which provide continuous spatiotemporal environmental representations.
  • A unified standardization pipeline aligns endpoint inputs by timestamps and absolute-time anchors before main-agent and sub-agent interaction.
  • The video-frame cache varies time windows and frame density to support low-frequency long-term perception, rapid memory updates, and real-time reminders.
  • StreamingClaw converts multi-end signals into a standardized multimodal representation, then feeds it to the main agent for streaming reasoning and output generation.
  • StreamingReasoning plans with feedback from StreamingMemory and StreamingProactivity, while tools and skills execute the resulting action instructions.

3 StreamingReasoning Agent

StreamingReasoning converts continuous video into low-latency incremental inference and autonomously schedules memory, proactive interaction, or direct reasoning. Dynamic caching and self-planning support stable operation over long streaming sessions.

  • Streaming inference: StreamingReasoning maintains a dynamically updated KV-Cache and sliding window for frame-level streaming perception, understanding, and reasoning.
  • Streaming inference: Incremental inference reuses cached KV tokens and computes only newly arrived tokens, while attention-based pruning removes low-contribution cached tokens.
  • Self-planning scheduling: As the main agent, StreamingReasoning parses task type and autonomously schedules memory retrieval, proactive decisions, or direct streaming reasoning.
  • Streaming inference: The streaming inference flow segments incoming video into fine-grained temporal chunks and processes sampled frames within each time window.
  • Streaming inference: Dynamic pruning and caching preserve key information while reducing attention computation and KV-Cache read–write overhead.
  • Streaming inference: Combined sliding-window and KV-cache processing changes full-history recomputation into linear incremental updates, enabling stable real-time reasoning during long streams.
  • Self-planning scheduling: Adaptive scheduling invokes memory or proactive interaction only when needed, otherwise reasoning directly from the current window to reduce overhead and latency.

4 StreamingMemory Agent

StreamingMemory addresses fragmented context in continuous video by storing multimodal memories, evolving them hierarchically, and retrieving them through adaptive parallel and temporal strategies. Its design preserves temporal structure while compressing redundancy and supporting efficient long-horizon reasoning.

  • Motivation: Sliding-window perception can lose global context, while text-only or direct memory injection causes semantic loss, token inefficiency, and retrieval difficulties.
  • Overview: StreamingMemory stores, evolves, and retrieves multimodal streaming information in real time to support temporal question answering and logical relation inference.
  • Memory capabilities: Its memory system supports multimodal storage, flexible updates and deletion, efficient retrieval, and unified memory access across agents.
  • Multimodal memory storage: A memory node combines a compressed video segment, textual summary, detailed description, and ending timestamp: n_t = (z, s, c, τ)_t.
  • Hierarchical Memory Evolution: Hierarchical Memory Evolution summarizes, removes redundancy, and compresses streaming information from short-term memories into long-term memories.
  • Hierarchical Memory Evolution: Online induction merges segments into atomic actions and aggregates temporally contiguous actions into event nodes under semantic, temporal, and scene-consistency criteria.
  • Hierarchical Memory Evolution: HME preserves temporal order, compresses repetitive segments, and stores events as stable memory chunks for scalable retrieval.
  • Memory retrieval: Retrieval combines command-driven stopping, parallel candidate processing, and self-directed forward, reverse, or salience-first traversal to reduce wasted computation and error accumulation.

5 StreamingProactivity Agent

StreamingProactivity supports proactive interaction over streaming video by monitoring evolving objectives, triggering responses, and adapting through training-free or training-based paradigms.

  • Overview: StreamingProactivity predicts future events and supports proactive objectives specified upfront or evolving during streaming interaction.It converts proactive requests into continuously monitored online tasks.
  • Training-free adaptation: The training-free approach uses reminder-node generation, proactive response matching, and proactive objective evolution without additional model training.It supports real-time objective evolution but has relatively constrained generalization in complex and cross-domain scenarios.
  • Training-free adaptation: Reminder nodes encode time-aware and event-grounded trigger conditions for subsequent online matching and triggering.Examples include specified times and detected events such as a scored goal.
  • Training-based adaptation: Training-based adaptation models streaming state changes as visual-language signals and uses scenario-specific trigger tokens to support proactive perception.Silent inference produces proactive signals from video, while non-silent inference also generates reasoning responses for multimodal input.
  • Training-based adaptation: The training data includes normal and changed-state segments, precise event timestamps and signals, and target responses linked to skill execution.This supports learning from trigger-token prediction through response generation, skill invocation, and action execution.
  • Training-based adaptation: Training-based adaptation supports multiple event types, customizable events, higher event-recognition accuracy than non-training-based approaches, and concurrent multi-demand inference in one forward pass.Scenario-specific signal mappings stabilize trigger timing and improve cross-scenario generalization while reducing inference token overhead.
  • Outcome: The resulting agent enables always-on proactive interaction customized to diverse visual state changes.Its interaction loop monitors, triggers, feeds back, evolves objectives, and monitors again.

6 Scalable Tools and Skills

StreamingClaw combines streaming-specific tools, multimodal memory access, and composable skills to connect video understanding with real-world action.

  • Toolbox: StreamingClaw adds specialized video-understanding and streaming-interaction tools and embodied skills while remaining compatible with OpenClaw resources.These components complete the perception–decision–action loop for multimodal embodied interaction.
  • Basic toolbox: The video cut tool extracts and analyzes temporal sub-clips when a task requires fine-grained video perception.The agent predicts start and end timestamps before requesting analysis of the corresponding segment.
  • Basic toolbox: The call memory tool retrieves structured historical memories, enabling cross-time comparison, evolution tracking, and causal reasoning for long-horizon video questions.Retrieved memory nodes can include event summaries, timestamps, and key entities or tags.
  • Basic toolbox: StreamingClaw uses post-training from open-source multimodal foundation models to improve the reliability, controllability, and generalization of tool usage.The pipeline targets tool manipulation through tool-augmented reasoning.
  • Skill library: Its skill library provides composable callable primitives for general-purpose and embodied tasks, enabling orchestration and closed-loop execution through unified scheduling.General skills include question answering, navigation, weather, travel, media control, memory recall, calendars, and reminders.
  • Skill library: Embodied skills cover scenarios including household care, education tutoring, and driver monitoring under continuous multimodal streaming input.Examples include fall detection with emergency intervention, tutoring with proactive interaction, and graded responses to driver fatigue or distraction.
  • Runtime design: Skills are dynamically loaded on demand after the agent selects candidates based on current intent and context.Only the selected interface definitions and minimal descriptions are loaded during inference.

7 Conclusion

StreamingClaw targets embodied-intelligence scenarios requiring real-time perception–decision–action and integrates streaming reasoning, memory, proactive interaction, and executable tools and skills. It remains compatible with OpenClaw while supporting applications such as embodied intelligence and autonomous driving.

  • Conclusion: StreamingClaw provides real-time video ingestion and streaming inference for low-latency responses in closed-loop embodied scenarios.The framework is designed for continuous perception, decision-making, and action.
  • Conclusion: Sub-agents support future-event reasoning and proactive interaction decision-making under continuous input.This allows the system to execute tasks and engage more proactively during streaming operation.
  • Conclusion: Hierarchical multimodal memory supports memory writing, dynamic evolution, and efficient retrieval for long-duration continuous scenarios.The framework uses this memory to handle information accumulation and utilization.
  • Conclusion: A toolbox and skill library translate high-level understanding and decisions into executable actions across diverse application scenarios.The paper presents this as supporting cross-scenario generality and deployability.
  • Conclusion: StreamingClaw integrates online real-time perception and reasoning, long-term memory, proactive interaction, and extensible action capabilities while remaining compatible with OpenClaw.The paper identifies embodied intelligence and autonomous driving as promising application areas.

8 Limitations and Future Work

StreamingClaw is currently focused on streaming-video scenarios with a vision-plus-text input paradigm, while future work targets broader omnimodal interaction and stronger temporal, spatial, memory, and deployment capabilities.

  • StreamingClaw primarily targets streaming-video scenarios and uses MLLMs for streaming video perception and understanding.
  • Audio is mainly supported as a text-to-speech output channel, with limited audio input, audio-visual temporal alignment, and end-to-end cross-modal reasoning and generation.
  • Future work will pursue a unified omnimodal agent handling video, images, audio, and text in a full-duplex perception-to-action loop.
  • Planned improvements include long-horizon temporal modeling, enhanced spatial understanding, cross-modal alignment, low-latency deployment, advanced memory, and long-horizon tool invocation.

Contributor List

The report lists contributors alphabetically by last name and separately identifies project leaders and supervisors.

  • All contributors are listed in alphabetical order by last name.
  • The core contributors are Jiawei Chen, Zhe Chen, Chaoqun Du, Maokui He, Wei He, Hengtao Li, Qizhen Li, Zide Liu, Xuhao Pan, Chang Ren, Xudong Rao, Xintian Shen, Chenfeng Wang, Chengjun Yu, Shengyu Yao, Chunpeng Zhou, Lihao Zheng, Xuhan Zhu, and Yufei Zheng.
  • The project leaders are Hao Ma, Tao Wei, and Pengfei Yu, while the supervisors are Kun Zhan and Pan Zhou.

A. Full Example of Tools

This section presents complete examples of StreamingClaw’s video cut and call memory tools, including their invocation schemas, execution flow, and integration with subsequent reasoning.

  • The section covers the video cut tool and the call memory tool as representative StreamingClaw tool examples.
  • The examples explain each tool’s invocation schema, runtime execution flow, and incorporation of returned results into subsequent agent reasoning.

B. Full Example of Skills

The report specifies representative embodied interaction skills for vehicle monitoring, robot household care, and AI glasses education tutoring, including triggers and structured outputs.

  • The section presents three representative embodied interaction skills intended to demonstrate applicability across hardware platforms and complex real-world scenarios.
  • Embodied Vehicle Driver Monitoring System Skill: The Embodied Vehicle Driver Monitoring System Skill monitors fatigue and distraction through an in-vehicle camera for real-time safety assessment and intervention.
  • Embodied Vehicle Driver Monitoring System Skill: Head-down phone use or gaze deviation triggers fatigue_state 0, yawning triggers fatigue_state 1, and closed eyes trigger fatigue_state 2.
  • Embodied Vehicle Driver Monitoring System Skill: The driver-monitoring skill returns a driver_fatigue_warning object with an integer fatigue state whose maximum level is 2.
  • AI Glasses Education Tutor Skill: The AI Glasses Education Tutor Skill provides real-time tutoring for problem-solving, translation, literature search, and customized proactive interaction.
  • AI Glasses Education Tutor Skill: Tutor triggers include inquiries about problem-solving, translation, or literature search and explicit demands for active interaction.
  • AI Glasses Education Tutor Skill: Its structured output includes solve_problems and create_proactive_node actions with query fields for problem-solving and proactive reminders.
Loading 2603.22120v2…