MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning

Motivation

• Long-horizon surgical reasoning requires evidence spread across a procedure, but one-shot VLMs compress long videos and trained video agents require task-specific data and transfer poorly out of domain,, [10]–.The answer to a before-or-after question depends on how the operation evolves, not on a single frame [5],.Source: §I · p. 1

Heuristic Skill Distillation

• Heuristic Skill Distillation mines low-scoring traces and retains candidate retrieval skills only when they raise validation reward across correctness, detail, context, and temporal understanding.From about one hundred preference examples, it discovers reusable skills such as directed re-look and stores their triggers and tool-call instructions in a library, without parameter updates.Source: §I · p. 2

Agent harness

• The harness pairs a text-only orchestrator with frozen vision-language sub-agents, making evidence gathering through video, frame, crop, and knowledge-base tools an auditable, re-examinable trace.Localization proceeds from the whole procedure to the exact moment; no model weights are updated.Source: §I · pp. 1–2

Benchmark

• MedClawBench contains 1,123 de-leaked, doctor-grounded questions: 259 from self-built long neurosurgery recordings and 864 from the held-out SVU-31K test split [5].The automatic judge is validated against expert ratings on both splits using κ.Source: §I · p. 2

Results

• MedClaw consistently outperforms one-shot VLMs and general video-agent frameworks across both datasets and all four evaluation dimensions, with the largest margins on long, out-of-domain neurosurgery videos.The gains are greatest where long-horizon temporal reasoning matters most.Source: §I · p. 2

A. Agent Harnesses for Video Reasoning

• Long-video agents retrieve informative frames –[21], localize temporal windows [9], [22],, and may use reinforcement learning to shape retrieval and spatiotemporal reasoning, –.These systems build on language models that reason and act in an interleaved loop [14] while calling external tools –.Source: A. Agent Harnesses for Video Reasoning · pp. 2–3
• An agent harness separates a planning text orchestrator from frozen vision-language sub-agents, coordinates them through context [29], and can serve as an auditable evaluator or task solver [30], [31].Source: A. Agent Harnesses for Video Reasoning · p. 3
• In clinical video reasoning, MedScope interleaves reasoning with coarse-to-fine tool calls, while other medical agents train tool-augmented reasoning with reinforcement learning [32], [33].Source: A. Agent Harnesses for Video Reasoning · p. 3
• Training backbones to choose where to look requires large task-specific trajectory datasets and transfers poorly to long-horizon, out-of-domain settings such as neurosurgery.MedClaw instead freezes both orchestrator and sub-agents and uses an external skill library, attributing gains to retrieval strategy rather than backbone fine-tuning; the authors describe it as the first training-free harness for long-horizon surgical-video temporal reasoning.Source: A. Agent Harnesses for Video Reasoning · p. 3

B. Heuristic Learning beyond Gradients

• Heuristic Skill Distillation improves a frozen agent by searching its context rather than its weights, producing temporal evidence-retrieval skills for surgical video.The approach adapts context optimization, previously explored mainly for text and general multimodal tasks, to a frozen clinical-video agent.Source: B. Heuristic Learning beyond Gradients · p. 3
• The agent diagnoses blind failures against gold answers, distills operational lessons into candidate skills, and keeps a skill only if it raises validation four-dimensional reward.This reward gate addresses the risk that plausible but unhelpful skills accumulate from the agent’s own judgment in open-ended surgical question answering.Source: B. Heuristic Learning beyond Gradients · p. 3
• Because open-ended surgical question answering lacks automatic success scoring, self-judgment alone can accumulate plausible-but-useless skills.The method therefore uses gold-answer failure diagnosis and validation reward as explicit checks.Source: B. Heuristic Learning beyond Gradients · p. 3

C. Long Surgical and Clinical Video Understanding

• Prior surgical-video work moved from clip-level phase, tool, and action recognition, and single-scene VQA, toward full-procedure resources, including SVU-31K [5].Other corpora broaden surgical VQA coverage, including OphClip, MedGen [46], SurgPub-Video [47], SurgBench [48], SurgViVQA, and EyePCR [50].Source: C. Long Surgical and Clinical Video… · p. 3
• Long-context video models,, –[54] and clinical-tool medical vision-language agents – have advanced, but are rarely evaluated on long surgical procedures.Most medical video QA resources instead use short, pre-trimmed clips with local content, and some include narration that states the answer.Source: C. Long Surgical and Clinical Video… · p. 3
• Long-horizon aggregation of evidence across a full procedure remains largely unaddressed; the benchmark pairs long neurosurgery recordings with held-out lecture videos and filters out questions answerable without video.Fine-grained evidence-annotated clinical data encourages grounding, while the benchmark’s blind filter isolates questions requiring visual evidence.Source: C. Long Surgical and Clinical Video… · p. 3

A. Task Formulation

• Free-form answers must name procedure-specific instruments, structures, and actions, reducing reliance on textual priors that distractor-based multiple choice can permit.The task is to answer a natural-language question about a surgical video with an answer grounded in what it actually shows.Source: A. Task Formulation · p. 3
• Following SVU-31K [5], questions are grouped into visual-detail VPR and process-focused VTR, with VTR more demanding because evidence may lie outside the named event’s window.VPR concerns details such as instruments, anatomy, counts, or findings once a moment is localized; VTR asks what happens before or after a step or how state changes across the procedure.Source: A. Task Formulation · p. 3

B. Overview

• MedClaw separates planning from perception: a text orchestrator selects evidence-gathering actions, tools expose video at three granularities and provide external-knowledge retrieval, and frozen vision sub-agents inspect pixels.Neither the orchestrator nor vision sub-agents are fine-tuned; competence comes from the loop and skill library rather than updated parameters.Source: B. Overview · p. 3
• The Heuristic Skill Distillation loop mines low-scoring traces, diagnoses gaps as operational, unreachable, or already correct, and keeps a candidate skill only if it raises four-dimensional validation reward.The retained reusable retrieval skills populate a tunable library, as illustrated in the framework overview.Source: Figure 2 · p. 4, B. Overview · p. 4

C. Orchestration

• The orchestrator is a frozen, text-only policy (Opus-4.8) that never sees the video directly; it chooses calls from a fixed tool set whose outputs are textual observations from frozen vision sub-agents [14].At each step it conditions on the running context to emit a tool action and arguments, such as a time range.Source: C. Orchestration · p. 4
• After each tool call, the textual action and observation extend the context until the policy stops and answers; because visual needs must be explicit calls, the recorded trajectory is auditable and supports attribution to retrieval behavior.All state lives in the context, without gradient steps or external memory.Source: C. Orchestration · p. 4

D. Tool Suite

• The tools progress from whole-clip localization to segment inspection and single-frame verification, returning textual observations from typed arguments.This coarse-to-fine sequence lets the orchestrator localize first, then verify the relevant instrument, structure, or action.Source: D. Tool Suite · p. 4
• The suite also includes surgical-knowledge retrieval, which returns top-k background passages to ground terms without providing benchmark answers.Its knowledge base indexes PubMed abstracts and literature-grounded summaries of techniques and endoscopic visual cues, embedded with a sentence encoder in FAISS using cosine similarity.Source: D. Tool Suite · pp. 4–5
• Full-video viewing first produces a low-resolution timeline and candidate event window, then re-watches a widened window at higher resolution to identify the tightest relevant sub-window.Temporal localization is performed by the pixel-viewing model rather than the text orchestrator.Source: D. Tool Suite · p. 4
• Video cropping re-reads a chosen range at crop resolution to describe a localized or adjacent segment densely.Source: D. Tool Suite · p. 4
• Single-frame selection provides an independent model’s cross-check of an instrument or action, while frame-region cropping disambiguates small or overlapping structures.Source: D. Tool Suite · p. 4

• Frozen vision sub-agents perform perception without planning or task-specific weights, using two experts matched to the tool set’s temporal and frame-level granularities.Source: E. Vision Sub-agents · p. 5
• The video expert (Gemini-3.1-flash-lite/Gemini-3.5-flash) handles full-video and video-crop tools, returning timelines, interval descriptions, and localized windows for temporal reasoning.Source: E. Vision Sub-agents · p. 5
• The frame verifier (GPT-5.5) describes a single frame or crop for a higher-resolution check, while its fixed interface makes perception interchangeable without changing the orchestrator.This separation lets the authors compare harness contributions independently of the backbone.Source: E. Vision Sub-agents · p. 5

• The loop uses 100 SVU training questions—60 for mining and 40 for validation—kept disjoint from evaluation sets, and evolves an external library rather than model weights.The paper contrasts this with trained agents that fit millions of parameters to much larger corpora.Source: F. Heuristic Skill Distillation · p. 5

Mining candidate skills from failed traces

• From blind-answer failures, the orchestrator classifies gaps as operational, unreachable, or already correct, and mines only actionable operational lessons into at most five deduplicated candidate skills.It updates an existing skill when the same rule is already present; otherwise it adds a new one.Source: F. Heuristic Skill Distillation · p. 5

Reward-gated retention

• A candidate is retained only if its validation reward exceeds the empty-library baseline by more than 0.05 on the 1–5 scale and no question drops by two points or more.Each candidate is tested alone by re-answering and re-judging the full validation set; rejected skills are removed.Source: F. Heuristic Skill Distillation · p. 5

The learned skill library

• The compact learned library is dominated by directed re-look: for before/after questions, it crops and rereads the window adjacent to a localized event, grounding answers outside the event window.It reads the following window for “after” questions and the preceding window for “before” or preparation questions.Source: F. Heuristic Skill Distillation · p. 5

IV. MEDCLAWBENCH

• By construction, questions exclude weak temporal structure, narration leakage, textbook-prior answerability, and single-frame answerability.Together, these exclusions target shortcuts that would avoid long-horizon reasoning.Source: §IV · p. 5
• MedClawBench is the only compared benchmark combining long surgical videos, explicit de-leaking, and neurosurgery; its videos have an 18.4-minute median, with 41% lasting over 20 minutes.The comparison covers surgical-video QA datasets; the paper identifies weak temporal structure, narration leakage, textbook-prior answerability, and single-frame answerability as shortcuts excluded by construction.Source: §IV · pp. 5–6, Table I · p. 6

A. Data Construction

• The benchmark contains 1,123 questions: 259 from 14 neurosurgery recordings and 864 retained from the SVU-31K test split.The SVU questions retain the original Visual Temporal Reasoning (163) and Visual Perception Reasoning (701) labels.Source: A. Data Construction · p. 6
• Neurosurgery questions are grounded in expert neurosurgeon interval annotations, with GPT-5.5 drafting event-anchored questions and a fact-checker checking them against those annotations.The 14 recordings cover microvascular decompression (10 videos, 186 questions), tumour resection (3 videos, 49), and epileptogenic-lesion resection (1 video, 24); their median duration is 28.5 minutes, and 71% exceed 20 minutes.Source: A. Data Construction · p. 6, B. Filtering · p. 6
• The ten fine-grained question types partition into VTR (109) and VPR (150), covering temporal relations or interval content and single-moment perception, respectively.VTR asks about what precedes or follows an event, how phases relate over time, or what occurs within an interval; VPR covers instruments, actions, counts, findings, comparisons, and case-specific details.Source: A. Data Construction · p. 6

B. Filtering

• All candidates must come from videos at least ten minutes long, fail a text-only blind-answer check (sB < 3), and pass a targeted-window answerability check (sC ≥ 3).GPT-5.5 scores restricted answers against the gold on a 1–5 scale, using a fixed threshold τ=3.Source: B. Filtering · p. 6
• SVU questions are discarded if audio-only answers score at least 3, while neurosurgery questions are retained only when a fact-checker confirms alignment with the expert interval annotation.The audio filter excludes questions recoverable from narration; the faithfulness check discards content GPT-5.5 hallucinated beyond the annotation.Source: B. Filtering · p. 6

A. Setup

• 1,123 questions comprise 259 long neurosurgery questions and 864 held-out SVU lecture-video questions, forming the MedClawBench evaluation set.Every answer is scored on correctness of information (CI), detail orientation (DO), contextual understanding (CU), and temporal understanding (TU), with their average (Avg) reported.Source: A. Setup · p. 6
• Answers are scored from 1–5 by an automatic judge on CI, DO, CU, and TU, and the reported Avg is the mean of those four dimensions.Higher scores are better on every metric.Source: A. Setup · p. 6
• The benchmark statistics show 109 Visual Temporal and 150 Visual Perception neurosurgery questions, with sub-ten-minute videos filtered out to emphasize the ≥20-minute tail.The figure also depicts ten fine-grained neurosurgery question types and surgical-type distributions across both splits.Source: Figure 3 · p. 7

B. Baselines

• The general video-language baselines are Qwen2.5-VL-7B-Instruct [55], InternVL3-8B [56], and VideoLLaMA3-7B [57].Source: B. Baselines · p. 7
• Video-reasoning baselines are Video-R1-7B, VideoChat-R1-7B [25], and Video-RFT; long-video agents are LongVT-7B-RFT and ReWatch-R1-7B [10].The comparison also includes the surgical video-language model SurgVidLM [5].Source: B. Baselines · p. 7
• One-shot backends Gemini-3.1-flash-lite, Gemini-3.5-flash, GPT-5.5, and Opus-4.8 answer each question in a single pass, while MedClaw is a frozen-weight agent.MedClaw issues auditable tool calls for viewing, cropping, frame inspection, and knowledge retrieval, with a coarse-to-fine full-video pass and directed re-look skill; it is implemented on the open-source Hermes framework.Source: A. Setup · p. 6, B. Baselines · p. 7

C. Main Results

• 2.90 is MedClaw’s four-dimensional average on neurosurgery, +0.76 over the strongest tested open-source video model, VideoChat-R1-7B (2.14).Table II reports the best deployed configuration for each split: flash on neurosurgery and lite on SVU.Source: C. Main Results · p. 7, Table II · p. 8
• +0.17 to +0.56: MedClaw improves every dimension over the corresponding one-shot backbone in each split.The ∆ comparison uses Gemini-3.5-flash for neurosurgery and Gemini-3.1-flash-lite for SVU; on neurosurgery, every open-source baseline trails all one-shot backends.Source: C. Main Results · p. 7, Table II · p. 8

D. Ablation Studies

• +0.54, +0.55, and up to +1.02: wrapping Qwen2.5-VL, Video-R1, and ReWatch-R1-7B in the harness improves their four-dimensional averages.The ablation compares each backbone in a one-shot pass with that same model serving as the harness’s visual sub-agent on 259 neurosurgery questions.Source: D. Ablation Studies · p. 7, Table III · p. 8
• −0.38, −0.16, and −0.13: removing directed re-look, coarse-to-fine localization, and Heuristic Skill Distillation lowers the four-dimensional average, respectively.Directed re-look targets before/after questions by re-examining the window adjacent to a localized event; distillation outperforms a hand-written looking-discipline skill.Source: D. Ablation Studies · p. 7, Table IV · p. 8
• κ = 0.72 on neurosurgery and κ = 0.86 on SVU: the GPT-5.5 judge has substantial and almost-perfect overall agreement with expert ratings, respectively.On the held-out subsets, overall mean scores were 2.95 vs. 2.94 for neurosurgery and 3.64 vs. 3.65 for SVU; agreement was substantial on every neurosurgery dimension and almost perfect on every SVU dimension.Source: D. Ablation Studies · p. 8, Table VI · p. 8
• −0.33 and −0.34: replacing Opus-4.8 with DeepSeek-V3 and DeepSeek-R1, respectively, reduces the average score while leaving the perception side fixed.Opus-4.8 remains the strongest tested orchestrator, while either open-weight reasoner degrades the harness without breaking it.Source: D. Ablation Studies · p. 7, Table V · p. 8

VI. LIMITATIONS AND FUTURE WORK

• The benchmark has few questions and draws its neurosurgery portion from a small set of recordings; future work plans more long-video annotations and temporal-reasoning questions.The planned recordings will exceed twenty minutes, supporting a substantially larger benchmark for testing the method’s performance and generality.Source: §VI · p. 9

VII. CONCLUSION

• On MedClawBench, the harness surpasses one-shot VLMs and general video-agent frameworks across all four dimensions, with its largest margins on long, out-of-domain neurosurgery videos.The conclusion identifies these videos as the setting where long-horizon temporal reasoning is most stressed.Source: §VII · p. 9
• Ablations show that directed re-look contributes the most, gains persist across diverse frozen backbones, and performance survives swapping the orchestrator.Source: §VII · p. 9
• The harness separates reasoning from perception: a text orchestrator plans retrieval while unmodified vision–language sub-agents inspect the video pixels.The approach does not train a model to decide where to look.Source: §VII · p. 9
• A gradient-free, reward-gated Heuristic Skill Distillation loop mines reusable retrieval skills, notably directed re-look, from about one hundred preference examples rather than fine-tuning.Source: §VII · p. 9
• Explicit retrieval tool calls over unchanged weights expose inspectable traces, allowing clinicians to review where the agent attended and how it derived its conclusion.Source: §VII · p. 9
• Keeping adaptation in an external skill library enables extension to unseen procedures or specialties from a small number of examples without retraining or large-scale annotation.Source: §VII · p. 9