PaperlayerView exact revision on arXiv

Source-linked AI summary

VisualClaw: A Real-Time, Personalized Agent for the Physical World

Haoqin Tu, Jianwen Chen, Zijun Wang, Siwei Han, Juncheng Wu, Hardy Chen, Haonian Ji, Kaiwen Xiong, Jiaqi Liu, Peng Xia, Jieru Mei, Hongliang Fei, Jason Eshraghian, Zeyu Zheng, Yuyin Zhou, Huaxiu Yao, Cihang Xie

arXiv:2606.16295v1cs.CVcs.CL

TL;DR

VisualClaw addresses deployment costs, static agent scaffolds, and limited workspace-oriented evaluation for multimodal agents. It combines hybrid visual and skill encoding with self-evolving memories and introduces VisualClawArena, reducing API cost while improving accuracy in most video-QA settings and macro accuracy over no-evolution baselines.

  • Problem

    Existing video-QA setups assume full context, fixed agent scaffolds, and static answer selection rather than visual evidence use during tool-using workspace operations.

  • Method

    VisualClaw combines cascaded frame filtering, hot/cold skill injection, memory-guided skill evolution, and the 200-scenario VisualClawArena benchmark for workspace agents.

  • Results

    −98.1% API cost versus full-frame upload, while accuracy improved in most video-QA settings and macro accuracy improved over no-evolution baselines on VisualClawArena.

  • Takeaways & Limitations

    VisualClaw supports efficient, adaptive multimodal deployment and extends evaluation beyond standard video-QA to visual evidence use in tool-using workspaces.

Abstract

from arXiv · show

Vision language models are serving as general-purpose interfaces for complex multimodal tasks. However, deployment still faces three gaps: VLMs typically incur high latency and cost when processing dense video frames and long prompts, the agent scaffold remains static after deployment, and standard video-QA benchmarks do not test whether agents can use visual evidence inside tool-using workspaces. We present VisualClaw, a self-evolving multimodal agent built around two principles. First, hybrid encoding reduces deployment cost by filtering less informative streaming frames with a cascaded gate and compressing the text skill bank through hot/cold top-k injection. Second, skill evolution lets the agent learn from failures: retrieved memories condition an evolver as direct concatenated context or as guided evidence, producing skill-bank updates that help future questions. Across 4 video-QA benchmarks with 2 VLMs, VisualClaw cuts per-question API cost by an average -98% versus full-frame upload and by -25.9% over the offline uniform 8 frame baseline, while boosting accuracy in most settings, e.g., an average +3.85% and a peak +15.80% on EgoSchema with Gemini 3 Flash. To address the gap, we curate VisualClawArena, a 200-scenario multimodal agentic benchmark built through a strict five-stage pipeline; models must use video evidence, documents, dynamic updates, and executable checks inside a workspace. On VisualClawArena, the same framework with computer-use agent backends improves macro accuracy by +2.9% for Codex (GPT-5.5) and +3.2% for Claude Code (Sonnet 4.6) over no-evolution baselines, with a -9.5% cost reduction compared to the uniform-sampled baseline. These properties make VisualClaw a natural fit for edge applications, where the cascade reduces a 1-hour streaming session from ~3,600 API uploads down to only 5-20 calls and the self-evolution makes it a perfect personalized assistant.

1 Introduction

VisualClaw addresses deployment gaps in multimodal agents with hybrid video-and-skill encoding and self-evolving skills, and introduces VisualClawArena to evaluate tool-using agents with grounded multimodal evidence. Across static video-QA experiments, skill evolution improves accuracy in most settings, including substantial gains on EgoSchema with Gemini 3 Flash.

  • Deployment gaps: Standard offline video-QA assumes full multimodal context is available at query time and leaves deployment gaps for multimodal agents.The setup also assumes a fixed model scaffold after deployment, as described in the introduction passage.
  • VisualClaw: VisualClaw combines hybrid encoding with self-evolving skills to reduce deployment overhead and learn from failures.A cascade filters streaming frames before VLM upload, while hot top-k skills receive full text and remaining skills form a compact cold catalogue.
  • VisualClawArena: VisualClawArena is a 200-scenario multimodal agentic benchmark for tool-using execution with multiple-choice questions and executable-check problems.Its five-stage curation pipeline builds timestamp-grounded workspaces containing documents, chat/audio traces, dynamic updates, and executable checkers.
  • Evaluation: +3.85% average lift and +15.80% peak lift are reported on EgoSchema with Gemini 3 Flash under streaming skill evolution.The evaluation covers 4 static video-QA benchmarks and 2 VLM families; streaming skill evolution improves accuracy in most settings.

2 Related Work

Prior work develops reusable skills and memory, continual adaptation, and efficient video-VLM processing, while existing benchmarks cover diverse egocentric and general video-QA tasks. VisualClaw is positioned against limitations in frame-selection assumptions and native agent evolution.

  • Skill-based and memory-augmented agents: Skill-based agents improve performance through reusable libraries or external memory without modifying model weights.Examples include episodic self-reflections, executable code skills, cross-task natural-language rules, and memory systems.
  • Continual and meta-learning: Meta-learning and continual learning study fast adaptation to new tasks, sequential adaptation without forgetting, and online adaptation for non-stationary LLM-agent streams.The cited work includes RL2, PEARL, ProMP, and MetaClaw.
  • Selected-frame and efficient video VLMs: Selected-frame video-VLM methods reduce visual input, but often use an inner LLM for per-frame selection or assume offline access to the full clip.Other approaches use sliding windows, learned memory queries, or uniform/chunked frame stacks to compress long videos.
  • Egocentric and general video-QA benchmarks: Egocentric and general video-QA benchmarks span multiple task formats, durations, and reasoning requirements.The paper reports results on EgoSchema, EgoPlan-Bench, Video-MME long, and NextQA.
  • Selected-frame and efficient video VLMs: Existing multimodal agents are unable to evolve natively.This limitation is stated in the selected-frame VLM comparison table.

3 VisualClaw: An Efficient Multimodal Agent that Evolves Itself

VisualClaw is a three-timescale multimodal agent that reduces streaming-video and prompt costs through edge frame filtering and hot/cold skill injection, while evolving its skill bank from confidence-gated memories and failures. Its evolver supports concatenated or guided memory evidence and uses quality filters to sustain long-term adaptation.

  • Framework overview: VisualClaw combines an edge cascaded gate, a hot/cold skill injector, and a memory-augmented evolver operating per-frame, per-question, and per-session.The gate triages frames, the injector triages skills, and the evolver distils session memories into skill-bank updates.
  • Framework overview: Per-frame processing takes ∼10 µs on the edge, per-question retrieval takes ∼10 ms in the cloud, and per-session evolution runs offline over minutes.The three components are updated at qualitatively different timescales, with session evolution triggered after every Nevo failures.
  • Hybrid encoding: For each question, the system inlines the top-k retrieved skill bodies while representing remaining skills as name-and-description entries, bounding prompt cost by k rather than |S|.Bodies in the cold catalogue can be fetched on demand if needed, decoupling injection cost from skill-bank growth.
  • Skill evolution: After at least Nevo failures, the evolver retrieves high-confidence episodic memories and proposes new skills, using either direct concatenation or guided evidence.The Guide variant instructs the evolver to extract reusable skills while avoiding scenario-specific details; token-Jaccard deduplication and utility-based pruning control bank quality.

4 VisualClawArena: A Multimodal Agentic Benchmark

VisualClawArena is a multimodal agentic benchmark designed to test visual evidence use within tool-using workspaces rather than one-shot video-QA. It contains 200 scenarios curated through strict validation and requires agents to complete workspace operations whose final states can be automatically scored.

  • Benchmark motivation: VisualClawArena evaluates whether agents can use visual evidence while editing files, maintaining workspace state, and recovering from mistakes.It wraps tool-using backends such as Codex and Claude Code behind a staged-workspace interface.
  • Curation pipeline: The five-stage curation pipeline constructs executable, diverse scenarios with timestamped visual facts, workspace files, updates, reference answers, and scoring scripts.Candidate scenarios are created from videos drawn from Indoor/VSI, EgoSchema, and QVHighlights.
  • Benchmark construction: The benchmark contains 200 multimodal scenarios pairing short videos with documents, chat/audio traces, dynamic updates, and executable checks.Agents must reconcile contradictions across video and text records and leave a final workspace rather than emit only an answer.
  • Validation and leakage control: At least 40% of steps require video, and candidates are retained only when fewer than 40% are text-only solvable and clip access does not reduce accuracy.The paired leakage probe compares the same VLM with the clip hidden versus available, repairing candidates when decisive clues are not in the video.
  • Validation and leakage control: The benchmark ranks candidates with s = v + 8 max(∆, 0) −6L + 0.5m, where visual-required steps, leakage, and multimodal workspace dependence determine selection.Here ∆ = Accclip − Acctext, measured by paired clip-hidden and clip-available runs.

5 Experiments

VisualClaw improves static video-QA and agentic benchmark performance through guided or concatenated skill evolution, while its cascade substantially reduces video-processing cost. Gains are strongest on harder scenarios and longer videos, and VisualClaw surpasses frontier and offline baselines on key evaluations.

  • Static video-QA: +3.85% average and +15.80% peak on EgoSchema: FullEvo (Guide) beats Plain with Gemini 3 Flash.Gemini 3 Flash also gains +4.23% on EgoPlan-Bench at the streaming budget.
  • Static video-QA: +13.00% on EgoSchema and +12.15% on EgoPlan-Bench: Uniform-8 + FullEvo (Guide) improves over the offline Plain baseline.The EgoSchema result is 60.6% →73.6%, forming an offline upperbound.
  • Static video-QA: 68.40% on EgoSchema: streaming FullEvo (Guide) beats VideoAgent’s 60.20% and approaches Gemini 1.5 Pro’s 72.20% offline result at lower latency.Uniform-8+FullEvo (Guide) reaches 73.60% on the same leaderboard.
  • VisualClawArena: +2.92 points for Codex and +3.16 points for Claude Code: VisualClaw (Cat.) improves macro accuracy over w/o FullEvo on the matched 3,106-round core.The corresponding macro accuracies are 54.27% for Codex and 52.16% for Claude Code.
  • VisualClawArena: 35.14% vs. 29.78% on Hard scenarios: VisualClaw (Cat.) improves over w/o FullEvo, while Easy performance slightly decreases from 85.61% to 85.05%.The same difficulty-dependent pattern appears for Uniform-8, with gains on Medium and Hard but lower Easy accuracy.
  • Efficiency: −98.1% average cost across four benchmarks and −99.3% at peak: the cascade reduces Gemini 3 Flash cost versus Full-frame @1 fps.Savings range from −95.0% on 3-minute EgoSchema to −99.3% on 30-minute V-MME long and −34.4% on single-frame EgoPlan-Bench.

6 Conclusion

VisualClaw is a self-evolving multimodal agent for efficient, adaptive deployment, paired with VisualClawArena, a benchmark for visual-evidence use in tool-using workspaces. Its design combines hybrid encoding across three timescales with memory routed to an offline evolver.

  • Contributions: VisualClaw targets efficient and adaptive deployment as a self-evolving multimodal agent.
  • Contributions: VisualClawArena evaluates whether agents use visual evidence inside tool-using workspaces.
  • Architecture: Hybrid encoding operates at three timescales: edge frame filtering, hot/cold skill injection, and offline memory evolution.The edge cascade filters frames before upload, hot/cold injection bounds prompt cost as the skill bank grows, and memory is routed into an offline evolver rather than concatenated per question.

A Appendix · A.1 Broader Impact

VisualClaw targets edge deployment by filtering most streaming frames before transmission, while its self-evolving memory introduces drift and reward-hacking risks that require operational safeguards. The appendix also reports benchmark-level memory retrieval rates for FullEvo with Gemini 3 Flash.

  • A Appendix: Table 12 reports memory ingest and retrieval rates per benchmark for FullEvo with Gemini 3 Flash.The table defines retrievals as skill evolver-fusion fetches where memory was never injected into VLMs.
  • A.1 Broader Impact: A 1-hour AI-glasses session at 1 fps emits 3,600 frames, creating roughly 3.9M input tokens under full-frame frontier-VLM upload.The resulting tail-latency budget is incompatible with cellular variance.
  • A.1 Broader Impact: ∼98% of streaming frames are rejected on-device before radio activation, reducing the same hour to 5–20 end-to-end uploads.This reduction is consistent with the per-question keyframe rates measured in the paper.
  • A.1 Broader Impact: Self-evolving memory can drift in directions unavailable to a static system, creating operational risks that require monitoring.The appendix identifies reward hacking and drift as two risks.
  • A.1 Broader Impact: Automatically scored failures can cause reward hacking when answer-key biases are amplified into the skill bank across evolution rounds.Examples include position bias on multiple-choice questions and length-prior bias on free-form answers.
  • A.1 Broader Impact: Confidence-gated memory ingest and F1/F2 hygiene filters are used to mitigate systematic answer-key bias amplification.These safeguards are referenced in Section 3.4 and Appendix A.8.

A.2 Further Analysis

Further analysis shows that VisualClaw’s benefits depend on the target VLM’s failure modes, while skill evolution grows the bank and retrieves memories sparsely; hot/cold injection is backbone-dependent.

  • Capability-conditional Gains: +15.80% gain on EgoSchema with Gemini 3 Flash versus +4.00% with GPT-5.2, despite GPT-5.2’s Plain baseline being 11.40% higher.The Plain baselines are 64.00 for GPT-5.2 and 52.60 for Gemini 3 Flash.
  • Bank Composition and Memory Dynamics: 28–55 new skills are added per run from a 12-seed base, while memory retrieval fires only approximately once per 21–59 questions.This yields a low-frequency conditioning signal for the evolver.
  • Bank Composition and Memory Dynamics: F1 runs on every evolution but records zero rejections at the 0.5 Jaccard threshold, while F2 is opt-in and excluded from the headline grid.The zero-rejection result is attributed to the Haiku evolver’s name diversity remaining below the cutoff at this scale.
  • Hybrid Injection Top-k Trade-off: GPT-5.2 performs best at k=3 with 51.90% and declines to 50.12% under ALL, indicating sensitivity to unrelated skill-bank entries.The comparison holds the cascade frame budget at K=8 and keeps the FullEvo stack fixed.

A.3 Detailed Algorithm

VisualClaw uses a three-tier agentic framework combining hybrid visual encoding with meta-evolution. Its streaming loop filters frames, answers questions, records failures, updates skills periodically, and prunes the skill bank.

  • Framework: The framework combines a 3-tier gate for hybrid visual encoding with meta-evolution.Algorithm 1 specifies the VisualClaw agentic streaming inference loop.
  • Streaming inference: Per-frame processing removes near-duplicate images using dHash and Hamming distance, then applies a lightweight encoder and change gate to retain major frames.The loop operates at the edge with per-frame, sub-millisecond processing.
  • Streaming inference: Per-question processing uses retained frames to answer each question in approximately 10 ms.The algorithm iterates over the question stream after frame filtering.
  • Meta-evolution: After failed responses, the agent accumulates question-answer-memory tuples and evolves the skill bank once failures reach the evolution threshold.Updated skills are deduplicated, while candidates with Jaccard similarity below the stated acceptance criterion are rejected.
  • Meta-evolution: The skill bank is pruned every 100 questions to control its contents over the session.Failure-driven evolution resets the failure buffer after applying the update.

A.4 Compute cost

The paper reports approximately $190 in total experiments across Gemini 3 Flash and GPT-5.2 using an Azure proxy. GPT-5.2 costs may be 1.5–3× higher because proxy reasoning-token billing is not separately exposed.

  • Compute cost: ∼$190 covered all reported experiments across Gemini 3 Flash and GPT-5.2 using an Azure proxy.The total includes experiments conducted with both models.
  • Compute cost: 1.5–3× inflation may affect the GPT-5.2 portion because Azure proxy reasoning-token billing is not separately exposed.The range depends on the Azure deployment configuration.

A.5 Additional VisualClawArena details … A.12 Additional case studies

The appendix clarifies VisualClaw’s matched benchmark scope and reports benchmark-conditional cascade behavior, VLM-specific skill transfer, bank hygiene, per-skill results, cost profiles, and additional case studies.

  • A.5 Additional VisualClawArena details: The matched core scope contains 200 active scenarios, but the broadened video_required selector covers only 100/200, so its runs remain separate from the main-paper scope.The 100/200 scenarios are the EgoSchema and QVHighlights halves; Indoor/VSI use the earlier narrower visual_required subset.
  • A.6 Cascade-fill on long-clip and uniform-activity benchmarks: Cascade-fill is benchmark-conditional rather than a universal replacement for uniform-8, because uniform activity and coarse long-clip fill positions limit recovery of missed content.EgoSchema distributes salient content roughly evenly, while V-MME long uses 60-second minimum gaps that are too coarse between scene transitions.
  • A.7 Cross-VLM transfer dynamics: Format-enforcement skills are VLM-family-bound, whereas reasoning-pattern skills transfer cleanly across VLMs.Removing the format skill recovers GPT-5.2 by +4 to +6 % across three benchmarks, while the remaining reasoning skills deliver +2.60 % on GPT-5.2 EgoSchema.
  • A.8 Bank hygiene activity on supplementary benchmark: F1 deduplication logged zero rejections on the four headline benchmarks, while F2 utility pruning was disabled in the headline grid and enabled only on a supplementary benchmark.The Haiku evolver’s name diversity stayed below the 0.5 Jaccard cutoff during 500–1000-question runs.
  • A.9 Per-skill EgoPlan-Bench numerics: EgoPlan-Bench per-skill differences remain near the noise floor, with top-tier seed skills around 27 % accuracy and evolved entries around 26–29 %.Top seed skills had 225 activations each; evolved entries accumulated 148–204 activations.
  • A.10 Cascade per-stage breakdown: +9.5 % is uniform-8’s offline advantage over the cascade at 8 KF/Q on EgoSchema, while cg-adaptive wins cg-static by +3.4 % on TeleEgo.Dhash-only is the worst cascade mode; adaptive thresholds and temporal decay cost ∼4 % versus static thresholds on EgoSchema, but adaptive features are robust to live streaming.
  • A.11 Full per-experiment token / cost / latency profile: Table 17 profiles tokens, cost, latency, and post-cascade keyframes per question across 4 benchmarks × 2-VLM, including cross-VLM ablation runs.The profiles are drawn from each results dump’s vlm_usage field; $/Q denotes average USD per question and lat./Q(s) average seconds per question.
  • A.12 Additional case studies: +6.18% and +4.00% are reported in two additional case studies: Gemini 3 Flash on EgoPlan-Bench and GPT-5.2 on EgoSchema with cross-VLM transfer.The cases show evolved skills correcting “open tap” to “close tap” and a Gemini-evolved bank correcting “making a sweater” to “knitting a scarf.”
Loading 2606.16295v1…