Source-linked AI summary
MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning
Yingying Fan, Penghui Du, Leyan Zhu, Runze He, Zimeng Wu, Yuxuan Zhang, Liang Chen, Jiahao Xie, Jiangtang Wang, Shuai Shao, Anchao Yang, Yutong Bai, Yan Wang
TL;DR
Long surgical videos require temporal reasoning grounded in evidence distributed across a procedure, which one-shot VLMs and trained video agents handle poorly. MedClaw uses an auditable orchestrator with frozen vision sub-agents and reward-gated skill distillation, outperforming one-shot VLMs and general video-agent frameworks across both datasets and all four evaluation dimensions.
Problem
Long-horizon surgical video reasoning requires grounding temporal questions in visual evidence distributed across an entire procedure, a capability current systems fail to deliver.
Method
MedClaw combines an auditable text orchestrator with frozen vision sub-agents and reward-gated skill distillation that evolves retrieval context without updating model weights.
Results
Across both datasets and all four evaluation dimensions, MedClaw consistently outperforms one-shot VLMs and general video-agent frameworks, with the largest margins on long, out-of-domain neurosurgery videos.
Takeaways & Limitations
The results support an auditable, training-free harness with reusable retrieval skills for long-horizon surgical video reasoning.
Takeaways & Limitations
The benchmark contains limited questions, and its neurosurgery portion draws from only a small set of recordings.
Abstract
from arXiv · showhide
Understanding tens-of-minutes surgical videos requires long-horizon temporal reasoning, answering what happens before, after, or across stages of a procedure by grounding the question in visual evidence spread across time. Existing approaches handle this poorly: a one-shot vision-language model (VLM) compresses the whole procedure to fit its context window and loses the detail a "before" or "after" question depends on, while video agents that train the model where to look are data-hungry and transfer poorly to out-of-domain surgery. We build an agent harness that separates reasoning from perception and improves by evolving context rather than optimizing weights. A text-only orchestrator plans which evidence to gather and issues an auditable sequence of tool calls, while frozen vision-language sub-agents execute each call over the pixels, viewing, cropping, inspecting frames, and retrieving external knowledge. We further propose a gradient-free, reward-gated Heuristic Skill Distillation loop that mines the agent's own low-scoring traces and keeps a candidate skill only when it raises a validation reward, yielding reusable retrieval skills, notably directed re-look. Growing an external skill library rather than tuning weights, the loop adapts from only about 100 labeled examples, far fewer than supervised or reinforcement fine-tuning requires. To evaluate this agent, we introduce MedClawBench, a de-leaked, doctor-grounded benchmark of 1,123 questions over self-built long neurosurgery recordings and a held-out public lecture-video test split. Across both datasets and all four evaluation dimensions, our agent consistently outperforms one-shot VLMs and general video-agent frameworks, with the largest gains on the long, out-of-domain neurosurgery videos. Project page: https://fyycs.github.io/medclaw/.
I. INTRODUCTION
Long surgical videos require iterative temporal evidence retrieval because clinically meaningful information is distributed across time, making one-shot VLM compression inadequate for “before” and “after” questions. MedClaw addresses this with a frozen-model agent harness, reward-gated skill distillation, and MedClawBench, achieving consistent gains over one-shot VLMs and general video-agent frameworks.
- Problem: Surgical procedures span tens of minutes, so answering what happens before or after a step requires grounding questions in temporal evolution rather than a single frame.The clinically meaningful evidence is distributed across the procedure.
- Problem: One-shot VLMs compress the whole procedure to fit their context window, losing the detail needed for temporal questions and preventing targeted re-examination.The task instead requires deciding where to look and iteratively inspecting relevant moments.
- Method: MedClaw factorizes reasoning into a text-only orchestrator and frozen vision-language sub-agents, making evidence seeking an auditable sequence of tool calls without updating model weights.Tools include viewing the full video, cropping time windows or frames, selecting frames, and querying a surgical knowledge base.
- Method: Heuristic Skill Distillation uses gradient-free, reward-gated selection to mine low-scoring traces and retain reusable retrieval strategies, notably directed re-look, from roughly one hundred preference examples.The retained skills are stored in a separate library and consulted when their declared triggers match a question.
- Evaluation: MedClawBench contains 1,123 de-leaked, doctor-grounded questions and isolates long-horizon temporal reasoning across self-built neurosurgery recordings and held-out SVU questions.The benchmark includes 259 self-built neurosurgery questions and 864 held-out SVU questions.
- Results: MedClaw consistently outperforms one-shot VLMs and general video-agent frameworks across both datasets and all four evaluation dimensions, with the largest margins on long, out-of-domain neurosurgery videos.These are the settings where long-horizon temporal reasoning matters most.
II. RELATED WORK · A. Agent Harnesses for Video Reasoning
Prior work frames long-video understanding as tool-augmented reasoning, with language models interleaving planning and actions to retrieve and localize visual evidence. Agent harnesses separate text planning from frozen vision–language sub-agents, enabling auditable coordination and clinical video reasoning.
- A. Agent Harnesses for Video Reasoning: Video understanding systems increasingly combine language-model reasoning with external tool calls in an interleaved loop.
- A. Agent Harnesses for Video Reasoning: Long-video agents adaptively retrieve evidence by selecting or sampling informative frames.
- A. Agent Harnesses for Video Reasoning: Other systems localize temporal windows through multi-turn spotlighting or agentic synthesis.
- A. Agent Harnesses for Video Reasoning: Reinforcement learning is often used to shape retrieval or broader spatiotemporal reasoning policies.
- A. Agent Harnesses for Video Reasoning: An agent harness separates a planning text orchestrator from frozen vision–language sub-agents, coordinating them through context.
- A. Agent Harnesses for Video Reasoning: These harnesses support auditable evaluation and task solving on realistic benchmarks.
- A. Agent Harnesses for Video Reasoning: In clinical video reasoning, MedScope interleaves reasoning with coarse-to-fine tool calls to “think with videos.”
- A. Agent Harnesses for Video Reasoning: A broader line of medical agents extends tool-augmented reasoning in the clinical domain.
B. Heuristic Learning beyond Gradients … A. Task Formulation
The paper frames MedClaw as a gradient-free paradigm that improves frozen models by evolving context, while targeting free-form answers to long surgical-video questions grounded in observed instruments, structures, and actions. It situates this task against prior short-clip surgical resources, long-form video models, and medical vision-language agents that are rarely evaluated on long surgical procedures.
- B. Heuristic Learning beyond Gradients: Gradient-free methods improve frozen models by searching over context rather than weights, including self-feedback, skill distillation, memory recall, and prompt optimization.These approaches use verbal or reward signals and avoid back-propagation.
- C. Long Surgical and Clinical Video Understanding: Surgical video understanding evolved from localized recognition of phases, tools, and actions to broader surgical VQA and medical-video corpora.The cited resources include ophthalmic video-language pretraining, granular medical-video annotation, and datasets such as SurgPub-Video, SurgBench, SurgViVQA, and EyePCR.
- C. Long Surgical and Clinical Video Understanding: Long-form video models and open foundation VLMs have advanced, but they are rarely evaluated on long surgical procedures.The passage identifies memory-augmented and long-context models alongside strong open foundation VLMs as prior progress.
- C. Long Surgical and Clinical Video Understanding: Medical vision-language agents provide clinical tools for diagnosis and interpretation, while most medical video QA resources still use short, pre-trimmed clips with temporally local content.This limitation leaves long surgical procedures underrepresented in existing evaluation resources.
- A. Task Formulation: Given a surgical video V and question q, the task is to produce a free-form answer a grounded in what the video actually shows.The formulation requires answers to identify specific instruments, structures, and actions observed in the procedure.
- A. Task Formulation: Free-form answering is chosen over multiple choice because distractors can exploit textual priors, whereas open responses must name procedure-specific visual details.The formulation is intended to test both visual-detail and temporal reasoning in long-horizon surgical video.
B. Overview
MedClaw addresses long-horizon temporal questions with an agent harness that separates text-based evidence planning from pixel-level vision tools. Its external knowledge base and failure-derived skill library support auditable retrieval trajectories and reusable improvements.
- Harness architecture: The harness uses a text orchestrator to plan evidence gathering and stopping, while tools expose the video at multiple granularities and provide retrieval.This design addresses evidence distributed across a few seconds of videos lasting tens of minutes.
- Retrieval skills: Directed re-look retrieves evidence from an adjacent window after locating the event window, rather than relying on the event window itself.The skill explicitly instructs cropping the adjacent window for temporal before/after questions.
- Harness architecture: Frozen vision sub-agents execute auditable tool calls that view the full video, localize a window, crop it, inspect frames, and ground the answer.The overview figure illustrates this end-to-end inference trajectory for a temporal “before” question.
- External knowledge and skills: A distillation loop mines reusable retrieval skills from low-scoring trajectories and stores them in a tunable skill library.The framework also queries an external knowledge base through retrieval-augmented generation and uses surgical priors, anatomy, and procedures.
C. Orchestration
The orchestrator is a frozen text-only policy that reasons through a fixed set of vision-agent tools, iteratively extending textual context until it stops and answers. Because it never sees the video directly, every visual need becomes an explicit, recorded, auditable tool call.
- Orchestration: The text-only orchestrator πθ, with frozen parameters, acts through five fixed tools whose frozen vision sub-agents answer each call over the surgical video.At each step, it emits a tool action conditioned on running context, including arguments such as a time range.
- Orchestration: The reason-act loop appends each action and textual observation to context, repeats until STOP, and then produces the final answer from the accumulated context.The update is ct+1 = ct ⊕(at, ot), with the final answer given by a = πθ(cT ).
- Orchestration: All state resides in the running context without gradient updates or external memory, while the complete trajectory of action-observation pairs is recorded.The recorded trajectory is ρ = (a1, o1), . . . , (aT , oT ).
- Orchestration: Since the orchestrator cannot access video pixels directly, every visual need must be externalized as an explicit tool call, making the trajectory auditable and retrieval gains attributable to specific behaviors.This design links observed improvements to particular retrieval actions rather than hidden visual processing.
D. Tool Suite
The tool suite implements coarse-to-fine temporal and spatial inspection, moving from whole-procedure localization to segment, frame, and region-level verification, with surgical knowledge retrieval for grounding.
- Coarse-to-fine inspection: The tool set progresses from whole clips to segments to frames, enabling the orchestrator to localize first and verify the exact instrument, structure, or action.Each tool accepts the video and typed arguments and returns a textual observation.
- Coarse-to-fine inspection: view full video localizes queried events through a low-resolution timeline followed by a higher-resolution re-watch that identifies the tightest relevant sub-window.Stage 1 proposes a candidate window; Stage 2 returns [s⋆, e⋆] containing the event.
- Temporal inspection: crop video inspects a selected or adjacent range at crop resolution and returns a dense description of what happens within that segment.The orchestrator uses it to read the localized window or an adjacent one.
- Frame-level inspection: select video frame provides an independent single-frame description for cross-model verification of the instrument or action at a specific timestamp.The independent frame model checks the localized visual evidence.
- Spatial inspection: crop video frame extracts and describes a normalized spatial region within a timestamped frame to disambiguate small or overlapping structures.The tool supports fine spatial disambiguation after temporal localization.
- Knowledge grounding: search surgical kb retrieves top-k passages from a surgical knowledge base to ground terms with general surgical background and no benchmark answers.The database combines PubMed abstracts with literature-grounded summaries indexed in FAISS using cosine similarity.
E. Vision Sub-agents · F. Heuristic Skill Distillation
The harness delegates perception to frozen, interchangeable vision sub-agents operating at video and frame granularities. A heuristic skill-distillation loop mines actionable retrieval failures and stores reusable strategies for improved evidence gathering.
- E. Vision Sub-agents: Frozen vision sub-agents execute every tool call, performing perception without planning or task-specific weights.They comprise two experts aligned with the tool set’s granularities.
- E. Vision Sub-agents: The video expert serves full-video and video-cropping tools, returning timelines, interval descriptions, and localized temporal windows.Gemini-3.1-flash-lite/Gemini-3.5-flash handles reasoning over clips or temporal segments.
- E. Vision Sub-agents: The frame verifier serves frame-selection and frame-cropping tools, independently checking specific structures at higher spatial resolution.GPT-5.5 analyzes a single frame or spatial crop and describes that instant.
- E. Vision Sub-agents: The fixed perception interface makes ψ interchangeable, allowing the same orchestrator πθ to run over cheaper or stronger backbones.This interchangeability separates harness contributions from backbone contributions.
- F. Heuristic Skill Distillation: The heuristic skill-distillation loop discovers reusable retrieval strategies from the harness’s mistakes and stores them in an external skill library.The harness cannot assume its preferred retrieval behaviors a priori.
- F. Heuristic Skill Distillation: Only type A gaps yield actionable lessons: the evidence is present but requires a tighter or adjacent crop, higher resolution, another frame, or disambiguation.Type B gaps are unreachable, while type C blind answers are already correct; type A lessons are written through the native skill manage tool after inspecting the library.
IV. MEDCLAWBENCH · A. Data Construction
MedClawBench is designed to test long-horizon temporal reasoning over long surgical videos while removing shortcuts such as narration leakage and single-frame answerability. Its 1,123 questions span self-recorded neurosurgery and externally sourced lecture videos, covering temporal and perceptual question types.
- IV. MEDCLAWBENCH: MedClawBench targets open-ended question answering over long surgical videos, with an 18.4-minute median duration and 41% of videos exceeding twenty minutes.The benchmark is constructed to measure long-horizon temporal reasoning in the medical domain.
- IV. MEDCLAWBENCH: Every question is designed to avoid weak temporal structure, narration leakage, textbook-prior answerability, and single-frame answerability.These four shortcuts are explicitly excluded by construction.
- IV. MEDCLAWBENCH: MedClawBench uniquely combines long surgical videos, explicit de-leaking, and neurosurgery content in one benchmark.SurgViVQA lacks long-horizon structure, while SVU-31K uses shorter narrated lectures and applies no de-leaking.
- A. Data Construction: The benchmark contains 1,123 questions from two complementary sources: a long self-recorded surgical domain and a large externally sourced lecture domain.Its composition is summarized by question type, source-video duration, and surgical type.
- A. Data Construction: Its temporal questions ask about immediately preceding or following steps, distances between phases, or content within a stated interval.These are categorized as temporal-after, temporal-before, long-range-temporal, and segment-content questions.
- A. Data Construction: Its perceptual questions ask about instruments, actions, item counts, findings, comparisons, or case-specific details visible at a single moment.These question types are categorized as VPR tasks.
B. Filtering … B. Baselines
MedClawBench filters for long-horizon, visually grounded questions and evaluates 1,123 questions across neurosurgery and held-out lecture videos using four scored dimensions. The study compares the harness with general, reasoning-oriented, long-video, and surgical video-language baselines.
- B. Filtering: All candidates pass shared duration, blind, and local filters, with SVU questions additionally requiring narration and neurosurgery questions requiring faithfulness.The filters certify that each candidate can be answered only by watching its targeted moment.
- B. Filtering: At least ten minutes of source-video duration is required to emphasize extended-procedure reasoning rather than short-clip perception.The filter removes sub-ten-minute videos and shifts the benchmark toward the long-video tail.
- A. Setup: MedClawBench contains 259 long neurosurgery questions and 864 held-out SVU lecture-video questions, totaling 1,123 questions.The neurosurgery split includes 109 Visual Temporal questions and 150 Visual Perception questions.
- V. EXPERIMENTS: Every answer is scored from 1–5 on correctness of information, detail orientation, contextual understanding, and temporal understanding, with their average reported as Avg.Higher is better for all five reported metrics.
- B. Baselines: The comparison includes general video-language models Qwen2.5-VL-7B-Instruct, InternVL3-8B, and VideoLLaMA3-7B.These models form one of three broad baseline families used in the comparison.
- B. Baselines: Video-reasoning baselines are Video-R1-7B, VideoChat-R1-7B, and Video-RFT, while long-video agents are LongVT-7B-RFT and ReWatch-R1-7B.The study also includes the surgical video-language model SurgVidLM.
C. Main Results · D. Ablation Studies
MedClaw outperforms the strongest tested open-source video model on long neurosurgery videos, while ablations show gains from wrapping diverse visual backbones in the harness. Automatic evaluation also aligns closely with neurosurgeon ratings across both evaluated splits.
- C. Main Results: On the long neurosurgery split, MedClaw achieves a four-dimensional average of 2.90, exceeding VideoChat-R1-7B at 2.14 by +0.76.MedClaw uses the flash visual backbone on neurosurgery.
- C. Main Results: On SVU, MedClaw uses the cheaper lite visual backbone because its clips are shorter and the two visual backbones are comparable.The results are reported separately for Visual Temporal Reasoning and Visual Perception Reasoning, with four scoring dimensions.
- D. Ablation Studies: Across every tested backbone, wrapping the same model as a visual sub-agent in the harness improves its four-dimensional average over a one-shot pass.The ablation uses 259 questions from the long neurosurgery split and reports gains of +0.54 for Qwen2.5-VL and +0.55 for Video-R1.
- C. Main Results: The main-results table breaks down each split by Visual Temporal Reasoning and Visual Perception Reasoning, with correctness, detail, context, and temporal scoring dimensions.Each dimension is scored from 1–5, and the delta row compares against the corresponding one-shot baseline.
- D. Ablation Studies: Component ablation removes or alters one learned component at a time from the full MedClaw harness on the 259-question neurosurgery split.Table IV reports all four dimensions, their average, and the change in average from the full model.
- D. Ablation Studies: On neurosurgery, the GPT-5.5 judge agrees substantially with neurosurgeon ratings, reaching overall quadratic weighted Cohen’s κ = 0.72.The held-out subset contains 30 items scored by a neurosurgeon, with mean scores of 2.95 versus 2.94 overall.
- D. Ablation Studies: Orchestrator ablation changes only the text-only orchestrator while holding the visual sub-agent and all tools fixed.Table V reports the four dimensions, their average, and the change from the Opus-4.8 orchestrator.
- D. Ablation Studies: On SVU, judge–expert agreement is almost perfect, with overall quadratic weighted Cohen’s κ = 0.86 and mean scores of 3.64 versus 3.65.The subset contains 70 items scored by surgical experts, and the paper attributes the higher agreement to the lecture domain being less specialized.
VI. LIMITATIONS AND FUTURE WORK · VII. CONCLUSION
The paper identifies limited benchmark coverage as a key limitation and proposes expanding long neurosurgery recordings and temporal-reasoning questions. It concludes with a training-free agent harness that separates reasoning from perception and distills reusable retrieval skills from about one hundred preference examples.
- VI. LIMITATIONS AND FUTURE WORK: The benchmark contains a limited number of questions, with neurosurgery represented by only a small set of recordings.This limits evaluation coverage for the neurosurgery portion.
- VI. LIMITATIONS AND FUTURE WORK: Future work will annotate a larger collection of neurosurgery videos to expand benchmark coverage.The planned collection specifically targets longer recordings.
- VI. LIMITATIONS AND FUTURE WORK: Future recordings will focus on long procedures beyond twenty minutes.This targets the long-horizon setting that motivates the benchmark.
- VI. LIMITATIONS AND FUTURE WORK: The expanded benchmark will include more temporal-reasoning questions over the newly annotated videos.The stated goal is a substantially larger benchmark for testing performance and generality.
- VII. CONCLUSION: The method addresses long-horizon temporal reasoning on long surgical videos without training a model to decide where to look.Its design avoids training the model for search-location decisions.
- VII. CONCLUSION: The training-free harness separates reasoning from perception: a text orchestrator plans while unmodified vision–language sub-agents inspect the video.This division assigns planning to the orchestrator and visual execution to frozen sub-agents.
- VII. CONCLUSION: The gradient-free, reward-gated Heuristic Skill Distillation loop mines reusable retrieval skills, notably directed re-look, from about one hundred preference examples rather than fine-tuning.The loop extracts skills from experience while avoiding weight updates.