PaperlayerView exact revision on arXiv

Source-linked AI summary

TRACER: A Multimodal Harness for Patient Care in Hospitals

Zane Durante, Paul Tang, Keshigeyan Chandrasegaran, Amit Kaushal, Shrinidhi Kowshika Lakshmikanth, Feolu Kolawole, Yu Jun Zheng, Yifan Geng, Shobhit Agarwal, Kevin Schulman, Juan Carlos Niebles, Ehsan Adeli, Arnold Milstein, Li Fei-Fei

arXiv:2610.21266v1cs.CVcs.AI

TL;DR

Understanding behavior across a hospital stay requires connecting continuous video with clinical records, but systems must balance temporal coverage with access to fine-grained evidence. TRACER builds a minute-to-hour behavioral hierarchy and gives an agent tools to retrieve records and inspect video. It improves longitudinal QA over the strongest same-backbone baseline and identifies a cohort-level association between purposeful activity and delirium status.

  • Problem

    Longitudinal ICU understanding requires connecting brief visual observations to behavior across days and interpreting activity alongside clinical records, while preserving access to fine-grained evidence.

    Source: §1 · p. 1

  • Method

    TRACER uses a training-free harness that builds minute-level action records and captions, combines them with medication records in hourly reports, and lets an agent retrieve evidence through tools.

    Source: §1 · p. 2, §2.1 · p. 4, §2.3 · p. 4

  • Results

    61.0% QA accuracy with TRACER versus 49.3% with the strongest baseline using the same language model and data sources.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    Across 280 patients and approximately 60,000 camera-hours, purposeful actions were less frequent among patients with a positive delirium assessment during their stay.

    Source: §1 · p. 3, §6 · p. 9

    This is a single-center retrospective study, and TRAJECTORY-144 is deep rather than broad, with 144 questions over two ICU stays.

    Source: §6 · p. 9

Abstract

from arXiv · show

Continuous video and clinical records provide complementary information for understanding how patient behavior evolves over a hospital stay. A central challenge is how to represent behavior across the entire stay while preserving access to fine-grained visual and clinical evidence. We introduce TRACER, a training-free multimodal harness that constructs minute-level action records from video and combines them with clinical records into hourly reports. The harness equips an LLM agent with tools to query this hierarchy, inspect relevant video, and retrieve clinical records. To evaluate TRACER, we construct two benchmarks. The first contains 397 annotated one-minute clips spanning 15 atomic actions; the second poses 144 clinician-informed multiple-choice questions, grounded in 264 hours of video and clinical records from two ICU patients. TRACER achieves up to 84.8% accuracy for action detection and 61.0% QA accuracy versus 49.3% for the strongest baseline using the same backbone. Across 280 patients spanning approximately 60,000 camera-hours, TRACER identifies significantly less purposeful activity in patients with a positive delirium assessment during their stay than in those without (p < 0.0001). Our results show that combining continuous video with clinical records can support understanding of patient behavior over time and across patient cohorts without training on protected health data.

1 INTRODUCTION

Understanding ICU behavior over days requires combining video with clinical history and ongoing treatment, while balancing temporal coverage with access to detail. TRACER organizes minute-level video observations and medication records into hourly reports that an agent can query alongside raw video and EHR.

1 INTRODUCTION

  • TRACER addresses the need to connect brief movements to behavioral patterns across days by combining bedside video with clinical records that provide complementary context.Video captures observable activity and interactions; EHRs document clinical assessments and interventions.Source: §1 · p. 1
  • TRACER is a training-free multimodal harness that turns one-minute video segments into action detections and captions, combines them with medication records in hourly reports, and equips an agent to retrieve evidence.The evaluation covers atomic action detection and longitudinal clinical reasoning.Source: §1 · p. 2, Figure 1 · p. 2
  • 76.9% QA accuracy with the best proprietary LLM agent and 67.9% with the best open-weight agent; no long-context VLM or Socratic caption baseline exceeds 50%.On the 397-clip action benchmark, TRACER achieves up to 84.8% accuracy and 68.6 micro-F1 across 15 atomic actions.Source: §1 · p. 2
  • Across 280 patients and approximately 60,000 camera-hours, patients with a positive delirium assessment show significantly lower frequencies of purposeful activities.The study interprets these differences as clinically relevant behavioral differences between patient groups.Source: §1 · p. 3

2 HARNESS DESIGN

TRACER separates continuous construction of a minute-to-hour record hierarchy from question-time evidence retrieval. It combines clinically informed action labels, descriptive reports, and tools for querying the hierarchy and inspecting raw video and EHR records.

2.1 PROBLEM FORMULATION

  • A single day of video can exceed model context limits, while models fitted to protected health information are difficult to share; TRACER keeps problem-specific structure in publicly releasable components.These components include the activity taxonomy, timeline structure and construction, aggregation tools, and prompts.Source: §2.1 · p. 3
  • During temporal abstraction, each minute becomes an action-label vector and caption, and each completed hour combines its 60 minute records with concurrent medication events into a report.During evidence acquisition, an agent uses tools across the hierarchy, including raw video clips and EHR events, until it answers or exhausts its call budget.Source: §2.1 · p. 4

2.2 HARNESS ARCHITECTURE

  • Three replaceable, untrained model roles build and use the hierarchy: a perception VLM creates minute records, a report LM compacts hourly records, and an agent uses tools per question.The perception and report roles run continuously as the stay is recorded; the agent runs per question.Source: §2.2 · p. 4

2.3 MINUTE-LEVEL CLINICAL EVENT RECORDS

  • The clinically developed taxonomy represents each minute as a 15-action vector grouped into patient–clinician interaction and head, torso, arm, and leg movements.Each camera’s one-minute clip is sampled at 1 fps; the two views are merged by logical OR over the action labels, alongside view-specific captions.Source: §2.3 · pp. 4–5

2.4 HOURLY BEHAVIORAL REPORTS

  • Each hourly report fuses 60 minute records with medication administrations and presents movement, interaction, significant moments, medication context, activity–medication dynamics, and a brief narrative.The hourly cadence matches the most frequent cadence of ICU flowsheet entries; report faithfulness is audited against minute records and the medication log.Source: §2.4 · p. 5

2.5 TOOL-AUGMENTED TRAJECTORY REASONING

  • The agent can search reports and timelines, retrieve predefined EHR data, or prompt additional video analysis, repeating tool calls until it answers or reaches 25 calls.The video tool can capture details missed by the initial action labels and captions.Source: §2.5 · p. 5

3 DATA AND EVALUATION

The evaluation uses a retrospective Stanford ICU cohort, a manually annotated action-detection test set, and a clinician-informed benchmark of questions about whole stays. Baselines include long-context video models and caption-based approaches, with and without EHR context.

3.1 ICU DATASET

  • The retrospective Stanford dataset links continuous video from 16 fixed cameras in eight rooms to structured EHR data; eligibility exclusions yielded 280 patients and approximately 60,000 camera-hours.The recordings were collected in cardiovascular and medical ICUs between August 2021 and June 2025.Source: §3.1 · p. 5

3.2 PERCEPTION BENCHMARK

  • The perception benchmark contains 397 one-minute clips from 26 patient stays, labeled for 15 actions by one annotator and verified or corrected by a second.The report specifies micro-averaged accuracy, F1, precision, and recall across nine perception VLMs; Table 2 summarizes average detection performance across five runs.Source: §3.2 · p. 5, Table 2 · p. 6

3.3 TRAJECTORY-144: LONGITUDINAL REASONING BENCHMARK

  • TRAJECTORY-144 comprises 144 clinician-informed, five-way questions grounded in 264 hours of video and health records from two ICU stays, with distractors revised to resist surface cues.Baselines include text-only LLMs, long-context VLMs given uniformly sampled frames, and Socratic captioning systems, evaluated with and without EHR context.Source: §3.3 · p. 6

4 EXPERIMENTS AND RESULTS

The experiments evaluate TRACER at minute-level perception, multi-day reasoning, and cohort-level behavioral analysis. They test both benchmark performance and whether detected activity patterns distinguish patient groups.

4.1 MINUTE-LEVEL PERCEPTION

  • Gemini 3.5 Flash led evaluated models on a benchmark of 397 one-minute clips, achieving 84.8% accuracy and 68.6 micro-F1; Gemini 3.1 Flash Lite achieved 82.3% accuracy and 62.2 micro-F1 at less than one-tenth the cost.Its lower cost motivated using Gemini 3.1 Flash Lite as the perception VLM when applying TRACER across the 280-patient cohort.Source: §4.1 · p. 7

4.2 LONGITUDINAL REASONING ON TRAJECTORY-144

  • 61.0% accuracy with TRACER versus 49.3% for the strongest same-backbone baseline; its hierarchy and tool use outperform long-context and Socratic approaches by 11.7 points.The paired-bootstrap 95% confidence interval for the gain is [4.4, 18.9].Source: §4.2 · p. 7
  • The full harness exceeded the best single toolset by 16 points, with cumulative toolsets sustaining accuracy gains and individual families helping different question types.EHR tools helped EHR questions, video tools short moments, reports 5–30-minute spans, and timeline tools longer horizons; Gemini 3.5 Flash reached up to 76.9% on TRAJECTORY-144.Source: §4.2 · p. 7
  • With perception fixed, Qwen3.5 9B and Gemma-4 31B matched or outperformed Gemini 3.1 Flash Lite as agent models, indicating that evaluated open-weight models’ performance gap primarily reflects perception limitations.The fixed perception setting used Gemini 3.1 Flash Lite for video queries, timeline, and report construction.Source: §4.2 · pp. 7–8, Table 4 · p. 8

4.3 COHORT-LEVEL BEHAVIORAL SIGNAL

  • Across 280 patients and roughly 60,000 camera-hours, seven actions were significantly less frequent with documented delirium, with all but head turning remaining significant after Holm correction.The largest reductions were in purposeful actions, including using non-medical objects (0.41×), leg movements associated with daily living (0.46×), object exchange with a clinician (0.48×), and gesticulation (0.60×).Source: §4.3 · p. 8

5 RELATED WORK

Related work spans ambient sensing and monitoring, long-video understanding, and clinical agents. TRACER is positioned as a shareable harness combining continuous bedside video with EHR information for questions over entire stays.

Ambient intelligence in hospitals

  • Prior ambient-intelligence systems commonly train task-specific models on curated real-patient annotations, making models fitted to protected health information difficult to share publicly.TRACER instead proposes a shareable harness using pretrained models and fusing activity detections with EHR for stay-level question answering.Source: §5 · p. 8

Long-video understanding

  • Long-video systems use frozen-model composition, segment retrieval, or caption memories, while week-long life-log systems use hierarchical summaries with fine-tuned captioners or reinforcement-trained agents.TRACER differs in its ICU setting and fusion of asynchronous EHR data into the temporal hierarchy.Source: §5 · pp. 8–9

Clinical agents

  • Clinical LLM agents have focused on text, structured records, single images, or diagnostic dialogue; TRACER examines continuous bedside video alongside EHR rather than treating the modalities in isolation.The cited work reports that LLM agents struggle to understand ICU stays from EHR alone (Shen et al., 2026).Source: §5 · p. 9

6 DISCUSSION

The discussion highlights TRACER’s reported benchmark and cohort findings while framing the delirium results as associations rather than predictive evidence. It also describes retrospective querying, study limitations, and reproducibility.

Key findings

  • TRACER’s key findings include improved multi-day question answering over same-backbone approaches, lower purposeful activity among patients with documented delirium, and open-weight agents matching proprietary agents when perception is fixed.The discussion identifies perception-VLM performance, rather than agent-model performance, as the greater current constraint on on-premise deployment.Source: §6 · p. 9

Behavioral signal and delirium

  • The delirium findings are retrospective associations based on model predictions and are hypothesis-generating; they do not establish predictive capability.The discussion relates fewer purposeful actions to delirium’s motoric phenomenology while noting that hypoactive delirium is often missed by intermittent screening.Source: §6 · p. 9

User Interface

  • TRACER is designed for clinicians’ retrospective querying rather than continuous alerting, with detected actions shown as a video timeline and cited timestamps linking to video minutes.Clinicians can inspect agent tool calls in the interface; nurses and physicians provided feedback on its design.Source: §6 · p. 9

Limitations

  • The study is single-center and retrospective, and camera placement, population, and documentation practices may not transfer; TRAJECTORY-144 covers 144 questions over two ICU stays.Intermittent delirium tests and per-action detection errors affect delirium frequency ratios; hourly reports may also contain inaccuracies or hallucinations.Source: §6 · p. 9

Reproducibility

  • The harness, tools, prompts, and taxonomy are released, and the hierarchy can be instantiated over other temporally indexed video or health-record collections without training on protected health data.Instructions for running the harness on sample videos and data are included in the repository.Source: §6 · p. 10

AI USE STATEMENT

The authors report AI assistance in manuscript preparation, coding, experiments, and analysis, while attributing the research design and manual annotations to the authors and collaborators.

AI USE STATEMENT

  • AI assistants helped draft and revise manuscript text, implement methods and baselines, and support data processing, debugging, and analysis; authors and clinical collaborators developed the research questions, harness, and taxonomy and produced annotations manually.AI assistance also helped draft and revise some clinician-consulted benchmark questions and distractors, which authors reviewed and verified.Source: AI USE STATEMENT · p. 10

A.1 TRAJECTORY-144 EXAMPLE QUESTIONS

TRAJECTORY-144 illustrates questions that range from specific moments to longer behavioral sequences and clinical-record retrieval. Examples ask about visible activity, event order, medication, and delirium assessments.

Video moment (<5 min) stay B

  • Questions cover brief video moments, including identifying an imaging procedure or describing whether the patient moved during a ten-minute window.Other examples ask about patient behavior during a staff-free interval and changes in movement before and after tests.Source: §A.1 · pp. 14–15

Long-form video (>30 min) stay B

  • Longer video questions test ordering events across a stay, such as sleeve placement, imaging, family visits, and transport for tests.They also ask about wakefulness periods and a specific stretch of purposeful activity.Source: §A.1 · pp. 14–15

Video window (5–30 min) stay A

  • Record-focused questions ask about scheduled medication classes and the result of a delirium assessment.A further example links a video interval to a medication administered within 45 minutes.Source: §A.1 · p. 15

A.2 TOOL INTERFACE AND AGENT PROMPT

The appendix documents the agent’s tool interface and prompting rules. The agent can query activity timelines and clinical records, inspect one-minute video, and is instructed to cross-check evidence.

Tool interface

  • The tool table lists eleven tools by family, with exact names, parameters, and return values.Optional parameters are marked with a question mark, and Table 3 configuration labels identify added tool families.Source: Table 5 · p. 16, §A.2 · p. 16
  • Activity tools support minute-level action lookup, counts over hour ranges, first-occurrence searches, and detection of rises or falls in selected actions.Medication and flowsheet tools retrieve medication-administration events and nursing measurements; the video tool answers a free-form question about one specific minute.Source: §A.2 · p. 17

Agent prompt

  • Video-tool answers are evidence to weigh because brief events may be missed, motion may be misread, and camera views may disagree.The instructions recommend re-querying with another prompt or adjacent minute and cross-checking other tools when the answer matters.Source: §A.2 · p. 17
  • Agent instructions require checking patient-specific time windows, querying whole admissions for stay-wide record questions, and investigating with at least two tools when possible.The evaluation prompt requires a forced choice, while the clinical interface may report insufficient evidence.Source: §A.2 · pp. 16, 18

A.3 TEMPORAL HIERARCHY CONSTRUCTION DETAILS

TRACER builds a temporal hierarchy from video and medication events, retaining minute-level action labels and captions while producing hourly reports. Prompts constrain these records to observable behavior and timed evidence.

Minute-level records

  • Each minute’s two camera views are merged by logical OR across 15 action labels, producing one binary label vector while retaining view-specific captions.The perception model receives 60 frames sampled at 1 fps; UNABLE TO IDENTIFY is treated as negative for the merged labels.Source: §A.3 · p. 20

Perception prompt

  • The perception prompt treats each of the 15 action categories independently and bases judgments only on clearly visible evidence.It allows multiple simultaneous labels and instructs the model to use UNABLE TO IDENTIFY when visibility is poor or confidence is low.Source: §A.3 · p. 21

Hourly reports

  • The hourly report prompt combines up to 60 minutes of caption outputs from two views with medication events relevant to that hour.It requires six sections, including movement, patient-clinician interaction, significant moments, medication context, activity-medication alignment, and a brief summary.Source: §A.3 · pp. 21–22
  • Hourly reports require timing references for behavioral findings and use observational-only activity-medication alignment, without clinical cause-and-effect interpretation.The prompt also requests uncertainty or incomplete-camera coverage to be noted in the brief summary.Source: §A.3 · p. 22

A.4 TRAJECTORY-144 QUESTION CREATION AND FILTERING

TRAJECTORY-144 questions were created, revised, and manually verified to probe temporal video understanding, fine-grained activity, and EHR reasoning. The benchmark spans multiple time references but covers two patient trajectories.

Question creation

  • Questions were manually drafted to assess long-span patient-status changes, shorter-span activities, or accurate retrieval and reasoning over EHR information.Question types were informed by discussions with clinicians, nurses, and domain experts.Source: §A.4 · p. 23

Question filtering

  • Questions were revised for difficulty, given plausible distractors, and manually checked; 25 of 144 remained answerable correctly at least 80% of the time by a blind model.Those questions account for part of text-only baseline accuracy above chance.Source: §A.4 · p. 23

Question coverage

  • The final questions span concrete moments, whole days, and record-based queries, while event localization uses question times, EHR events, action queries, or hourly reports.The 144 questions comprise 60 with a clock time, 34 with an hour or day reference, and 50 with no time reference.Source: §A.4 · p. 23

Benchmark scope

  • The benchmark covers two long patient trajectories and does not test generalization across patients.Accuracy differs between the two stays.Source: §A.4 · p. 23

A.5 PATIENT COHORT

Figure 3 summarizes how patients were selected and describes the final study cohort of 280 patients.

A.5 PATIENT COHORT

  • Figure 3 presents a CONSORT-style patient-selection flow diagram and characteristics of the final cohort (N = 280).The flow diagram covers exclusions for ICU stay duration, sedative exposure, neurological admission, and bilateral video coverage.Source: Figure 3 · p. 25

A.5.1 PATIENT COHORT SELECTION

This appendix describes the cohort and data used, TRACER’s evaluation and implementation details, and the scope and qualifications of the cohort-level delirium analysis.

A.5.1 PATIENT COHORT SELECTION

  • Patients were selected from 2,617 screened by requiring ICU stays of 24 hours to two weeks, sedative or analgesic exposure, non-neurological admission, and specified bilateral-video coverage.Sequential exclusions included 782 patients without continuous bilateral video coverage and 181 whose cameras were not centered on the bed area.Source: §A.5.1 · p. 25

A.5.2 ELECTRONIC HEALTH RECORDS

  • The harness exposes demographics and encounter metadata, ADT events, medication administrations, and nursing flowsheet measurements through distinct tools.TRAJECTORY-144 EHR questions use medication and flowsheet records; cohort delirium labels come from CAM-ICU screens.Source: §A.5.2 · p. 25

A.6 TOKEN COST OF RAW VIDEO

  • 950 uniformly sampled frames still span several minutes between observations, while a single day of one camera exceeds model context by more than an order of magnitude.TRACER’s hierarchy is constructed during the stay, so each question needs substantially fewer tokens than using the entire hierarchy in context.Source: §A.6 · p. 26, Table 7 · p. 26

A.7 BASELINE IMPLEMENTATIONS

  • The strongest baseline scored 49.3; adding EHR to the 475-frame baseline raised long-form video performance from 36.6 to 48.9 and EHR performance from 17.7 to 83.8.Source: §A.7 · p. 28, Table 8 · p. 28

A.8 MODEL INSTANTIATION AND COMPUTATIONAL COST

  • Gemini 3.1 Flash Lite was selected for perception because it offered strong, inexpensive performance at the Pareto frontier.Its perception accuracy–cost comparison is shown against Gemini and open-weight models in Figure 4.Source: §A.9 · p. 30, Figure 4 · p. 30

A.14 DELIRIUM AND ATOMIC ACTION FREQUENCY ANALYSIS

  • The three largest reductions remained similar under equal patient weighting: using non-medical objects (0.41× → 0.39×), leg ADL movement (0.46× → 0.47×), and object exchange (0.48× → 0.47×).The minute-weighted estimator gives more influence to patients with longer stays, which are associated with delirium.Source: §A.14 · p. 35
  • Seven of 13 evaluated actions were reduced in the delirium group under both minute- and patient-weighted estimates; clinician–patient contact was elevated (1.20× and 1.18×).Five actions were not distinguishable from no difference under either weighting; infrequent actions and lower-F1 categories have wider intervals.Source: §A.14 · p. 35
  • 0.053 versus 0.146 was the median purposeful-activity fraction in delirium and non-delirium groups, respectively (p = 1.3 × 10−8, Mann–Whitney).The pooled rates were 0.080 versus 0.161 (log2 FR = −1.01); the analysis defines purposeful activity as four actions and compares patients by any positive CAM-ICU screen during their stay.Source: §A.14 · p. 35

Related papers

Loading 2610.21266v1…