PaperlayerView exact revision on arXiv

Source-linked AI summary

Failure Modes of Vision-Language Models for Human Activity Understanding in Thermal HRI

Han Hu, He Chen

arXiv:2610.29244v1cs.CVcs.RO

TL;DR

Thermal video offers activity sensing in darkness with less facial and household appearance detail than RGB, but VLM failure modes for thermal human activity understanding in HRI remain underexplored. The paper evaluates seven locally deployable models across two thermal datasets and diagnostic tasks. Models often recognize broad activities but struggle with fine-grained actions, interaction timelines, and reliable abstention; tested interventions yield limited or inconsistent gains.

  • Problem

    How VLMs fail at understanding human activities from thermal video for HRI remains underexplored, despite thermal sensing’s potential for dark and privacy-sensitive settings.

    Source: §I · p. 1

  • Method

    Seven locally deployable VLMs are evaluated on two person-centric thermal datasets through action recognition, interaction grounding, and backoff tasks, with paired RGB and controlled observation and inference comparisons.

    Source: §I · p. 1

  • Results

    Models recover only part of the annotated interactions, make inaccurate timeline predictions and object-action hallucinations, and show varied abstention behavior; tested changes produce limited or inconsistent gains.

    Source: §I · pp. 1–2, §VI · p. 7

  • Takeaways & Limitations

    Reliable robotic use requires video-supported event labels and intervals, and backoff that rejects unsupported interpretations while retaining valid answers.

    Source: §VI · p. 7

    The fixed-camera datasets and 5 FPS clip-based evaluation omit active observation and the coupling between activity interpretation and robot behavior.

    Source: §VII · p. 7

Abstract

from arXiv · show

Thermal cameras enable robots to perceive human activities in darkness while revealing less personally identifiable appearance information and fewer details of the surrounding environment than RGB cameras. As vision-language models (VLMs), originally developed for RGB imagery, are increasingly integrated into embodied systems for scene understanding and high-level planning, thermal sensing offers a promising way to extend activity perception to low-light and privacy-sensitive settings. However, the failure modes of VLMs in thermal human activity understanding for human-robot interaction (HRI) remain underexplored. We fill this gap by evaluating seven locally deployable VLMs, including models developed for physical AI and robotics, on two person-centric thermal datasets. Our diagnostic tasks examine fine-grained action recognition, activity timeline reconstruction, and the ability to withhold unsupported answers. Paired RGB recordings serve as a performance reference, while controlled variations in frame sampling, thermal input presentation, and reasoning are used to assess whether the observed errors can be mitigated. Our results show that models can recognize broad activity categories but struggle with fine-grained actions such as coughing, sneezing, and yawning. They also produce inaccurate event timelines and hallucinate actions involving objects. Simple thermal input preprocessing and additional reasoning offer only limited, model-dependent gains. Abstention behavior also varies across models even under explicit instructions, limiting their reliability for practical robotic use.

I. INTRODUCTION

Thermal video may support activity sensing in darkness and privacy-sensitive spaces, but how VLMs fail at understanding human activity for HRI remains insufficiently characterized. The evaluation finds recurring fine-grained action confusions, incomplete and inaccurate interaction timelines, limited gains from interventions, and unreliable abstention.

I. INTRODUCTION

  • Thermal sensing captures body shape and motion in darkness while revealing less facial and household appearance detail than RGB, motivating its use in private spaces.Prior work and isolated failures do not yet provide a concrete basis for selecting HRI activity tasks, configuring perception pipelines, or prioritizing improvements.Source: §I · p. 1
  • Seven locally deployable VLMs are evaluated on two person-centric thermal datasets across action recognition, action–object interaction grounding, and backoff, with paired RGB recordings and controlled observation and inference changes.The comparison includes models developed for physical AI or robotics and their general-purpose base-model references.Source: §I · p. 1
  • 42.9% of pooled exact-action errors fall into five recurring answer labels, including cross-category substitutions such as coughing labeled as touching the face and staggering as walking.These substitutions can present physiological or potentially urgent behavior as routine activity.Source: §I · p. 1
  • 30.2%–51.8% of annotated events are recovered with correct action and object labels in long clips, even without requiring accurate temporal boundaries.Harder joint predictions also have inaccurate boundaries and hallucinated object interactions.Source: §I · p. 1
  • Across two tested models, interventions change exact-action accuracy by only −5.9% to +5.9% point relative to their respective paired baselines.The interventions include pseudocoloring, contrast enhancement, person cropping, denser frame sampling, and additional reasoning.Source: §I · p. 1
  • Abstention varies across models, which struggle to reject unsupported answers while retaining valid ones even under explicit instructions.Qwen3.6-27B correctly abstains on 6.1% of questions without a valid candidate, while RoboBrain2.5 answers 3.2% of questions containing one.Source: §I · p. 2

II. RELATED WORK

Prior work spans robotic activity understanding and thermal sensing, but leaves open how reliably thermal observations support fine-grained actions and temporal structure. This paper builds on that work with controlled experiments designed to locate recurring and model-specific errors.

II. RELATED WORK

  • Prior RGB and RGB-D studies examine activity understanding in robotic settings, while leaving thermal recovery of the same action distinctions and temporal structure open.Related work includes action–object relationships, affordances, fine-grained recognition, temporal segmentation, and caregiving video captioning.Source: §II · p. 2
  • Thermal research spans pedestrian detection, activity classification, privacy-preserving sensing, past-action inference, and VLM evaluations, but limitations remain relevant to robot use.Existing VLM studies report stronger recognition of large body movements than fine gestures and safety-related actions.Source: §II · p. 2
  • Controlled experiments characterize where errors concentrate, which actions are repeatedly confused, and which failure patterns are shared across models or model-specific.Source: §II · p. 2

III. EVALUATION DESIGN

  • The evaluation tests specific-action distinctions, action–object interactions and their temporal extent, and the choice between an answer and backoff using OctoNet and Thermal-IM.Figure 1 provides examples of observations, questions, and reference answers.Source: §III · p. 2

A. Fine-grained Action Recognition

The evaluation pairs short OctoNet clips for fine-grained action recognition with Thermal-IM recordings for interaction recovery and temporal grounding. It scores both interaction identity and temporal agreement, and separately tests interaction presence without requiring temporal boundaries.

A. Fine-grained Action Recognition

  • 865 OctoNet thermal clips cover 62 actions and 20 participants; each question selects the labeled action from 20 candidates, with distractors prioritizing related actions.Random-choice accuracy is 5%, and the evaluation also maps labels into 12 functional activity categories to distinguish fine-label from broader-category errors.Source: A. Fine-grained Action Recognition · p. 2

B. Interaction Recognition and Temporal Grounding

  • Thermal-IM’s official test split contains 118 recordings and 199 annotated action–object events, with synchronized RGB recordings used for modality comparisons.Each event has an action, object, and start and end times; recordings range from 3.0–143.2 seconds, with a median of 46.5 seconds.Source: B. Interaction Recognition and… · p. 2
  • The joint task asks models to return every interaction as a chronological action, object, and time-interval list, testing both interaction recovery and localization.Source: B. Interaction Recognition and… · p. 2, Figure 1 · p. 3
  • Temporal grounding is also tested with annotated action–object identities supplied, asking models to return occurrence intervals or an empty list for absent pairs.Comparison with the joint task tests grounding when identities are known.Source: B. Interaction Recognition and… · p. 3
  • Predicted and annotated intervals are matched one-to-one within each recording for the same action–object pair, maximizing total temporal intersection over union.Pair F1 measures interaction recovery, event F1 additionally requires sufficient temporal overlap, and mean tIoU summarizes temporal agreement over matched events.Source: B. Interaction Recognition and… · p. 3
  • A separate 15-choice presence test measures selection of the target interaction across complete-event clips, controls without a target event, and temporal-order transformations.The question and candidates remain unchanged across these conditions.Source: B. Interaction Recognition and… · p. 3

C. Candidate-set Backoff

The candidate-set backoff evaluation tests whether models answer or abstain when the labeled action is present versus unavailable. Paired OctoNet questions retain the same clips and matched distractors while adding a backoff option.

C. Candidate-set Backoff

  • Paired GT-present and GT-absent OctoNet questions test backoff with the same clips and matched distractors, adding a 21st option to withhold an answer.The GT-absent version replaces the labeled action while retaining the other 19 candidates.Source: C. Candidate-set Backoff · p. 3
  • Correct-backoff and false-backoff rates measure backoff on GT-absent and GT-present questions, respectively.Coverage and answered accuracy characterize answering tendency and exact-action correctness on GT-present questions.Source: C. Candidate-set Backoff · p. 3

IV. EXPERIMENTAL SETUP

The evaluation uses seven locally deployable VLMs without task-specific adaptation, with Qwen3-VL as the base-model reference for Cosmos3 and RoboBrain2.5. Models answer directly by default, while additional reasoning is evaluated separately.

A. Models and Inference

  • Seven locally deployable VLMs are evaluated without task-specific adaptation to the evaluation data.The set includes general-purpose and physical-AI or robotics models.Source: A. Models and Inference · pp. 3–4
  • Qwen3-VL serves as the general-purpose base-model reference for Cosmos3 and RoboBrain2.5.This comparison tests whether physical-AI or robotics training improves thermal activity understanding.Source: A. Models and Inference · p. 4
  • Direct answers without extended reasoning are the primary setting, with additional reasoning evaluated separately.Models receive native video inputs sampled at a target rate of 5 FPS unless otherwise stated; event intervals use continuous seconds.Source: A. Models and Inference · p. 4

B. Controlled Comparisons

Controlled comparisons vary observation, thermal presentation, reasoning, modality, temporal evidence, and backoff behavior. Thermal-input and processing interventions use two representative models, while RGB–thermal and backoff comparisons cover all seven.

B. Controlled Comparisons

  • Thermal-input and processing comparisons use Qwen3.5-4B and Qwen3.6-27B, while RGB–thermal and backoff comparisons cover all seven models.These comparisons modify thermal input or its processing, whereas RGB–thermal and explicit-backoff comparisons leave thermal input unchanged.Source: B. Controlled Comparisons · p. 4

a) Frame sampling

  • Frame sampling compares 2, 5, and 10 FPS on all 865 single-action clips, starting from the Qwen video processor’s default 2 FPS rate.The comparison tests whether denser observations help distinguish actions and whether gains continue as more frames are provided.Source: B. Controlled Comparisons · p. 4

b) Thermal input transformations

  • Thermal transformations test pseudocolor, contrast enhancement, person and upper-body crops, and synchronized skeleton videos against matched baselines.After checking crop and skeleton quality and excluding unreliable outputs, the evaluation covers 293 clips from five participants and all 62 actions.Source: B. Controlled Comparisons · p. 4

c) Reasoning

  • Reasoning-enabled inference is compared with direct answers on 289 clips from the five-participant transformation subset.Four tap clips are excluded because their labels do not precisely describe the recorded motion; 12 changed predictions are inspected.Source: B. Controlled Comparisons · p. 4

d) RGB–thermal comparison

  • Synchronized RGB and thermal recordings are compared on interaction recognition and temporal grounding using the same inference settings.The comparison reports action–object recognition and temporal grounding separately.Source: B. Controlled Comparisons · p. 4

e) Temporal-evidence intervention

  • Temporal-evidence tests pair complete annotated target events with equal-duration clips of the same person performing other activities, then reverse or shuffle target-event blocks.The blocks are at most three seconds, with at least four blocks; shuffling preserves frames and within-block order at the original playback speed and 5 FPS.Source: B. Controlled Comparisons · p. 4

f) Backoff settings

  • Backoff tests evaluate all seven models on 865 paired ground-truth-present and ground-truth-absent questions with explicit instructions to back off when unsupported.Four models are also tested without the instruction, while retaining the backoff option and candidate wording.Source: B. Controlled Comparisons · p. 4

V. RESULTS AND ANALYSIS

Thermal action recognition captures broad movement more readily than subtle actions, and recurring confusions can erase safety-relevant distinctions. Sampling and input changes provide partial, inconsistent relief, while additional reasoning does not reliably resolve errors.

A. Fine-grained Action Recognition

  • Large body movements are often recognized, but short hand-to-face actions, physiological events, and similar-trajectory gestures remain difficult.The results distinguish broad activity recognition from fine-grained action identification.Source: A. Fine-grained Action Recognition · p. 4
  • 42.9% of exact-action errors across seven models are accounted for by five frequent incorrect labels: waving, lifting a hand, touching the face, drinking, and turning.Recurring substitutions include yawning and coughing as touching the face, and brushing teeth as drinking.Source: A. Fine-grained Action Recognition · p. 4
  • Similar movement-level confusions can cross functional categories, with staggering labeled as walking or turning and pushing or punching someone labeled as conversation.These substitutions present potentially urgent behavior as ordinary movement or communication.Source: A. Fine-grained Action Recognition · p. 5
  • Increasing sampling from 2 to 5 FPS improves exact-action accuracy for both tested models, with little further gain at 10 FPS.Person cropping yields small gains, JET and contrast enhancement do not help, and skeleton or upper-body views have model-dependent effects.Source: A. Fine-grained Action Recognition · p. 5
  • Additional reasoning leaves Qwen3.5-4B accuracy nearly unchanged after 25 corrections and 24 new errors, while reducing Qwen3.6-27B accuracy by 5.9 percentage points.For Qwen3.6-27B, 30 new errors outweigh 13 corrections; inspected reasoning chains include misidentified objects, poses, and motion patterns.Source: A. Fine-grained Action Recognition · p. 5

B. Interaction Recognition and Temporal Grounding

On longer thermal recordings, models miss interactions and misplace event boundaries; supplying interaction identities improves event detection scores. RGB recovers more events than thermal input, while temporal-order perturbations do not reliably reduce target selection.

B. Interaction Recognition and Temporal Grounding

  • Joint predictions recover only part of the annotated interactions, with missed or misidentified events and invalid responses reducing pair F1 and boundary errors further reducing event F1.A correctly named interaction can still receive inaccurate temporal boundaries.Source: B. Interaction Recognition and… · p. 5
  • Cosmos3 and Qwen3-VL have similar exact-action accuracy (43.7% and 44.5%), but thermal pair F1 is 0.257 and 0.371, respectively; RoboBrain2.5 reaches 0.248.The comparison does not show a consistent interaction-recovery advantage from physical-AI or robotics specialization under direct answering.Source: B. Interaction Recognition and… · p. 5
  • With action–object pairs given, event F1 rises in all seven model comparisons, with scores ranging from 0.256 to 0.699.For events matched across tasks, given identities yield equal or better boundaries for six of seven models; the joint task skips harder occurrences.Source: B. Interaction Recognition and… · pp. 5–6
  • RGB yields higher action–object pair F1 and recovers more annotated events than thermal input across all seven models.The mean relative change in correctly matched event counts is 59.0%; mean tIoU changes by at most 0.07 except for Cosmos3 and RoboBrain2.5, where it improves.Source: B. Interaction Recognition and… · p. 6
  • Replacing a complete target event with an unrelated activity reduces target-label selection by more than 50 percentage points for both tested models, but correct labels are still missed on complete events.Reversing events or shuffling blocks changes 16.1–19.5% of answers without a reliable decline in target selection; posture and contact cues remain visible after reordering.Source: B. Interaction Recognition and… · p. 6

C. Candidate-set Backoff

Backoff exposes opposing failures: some models answer unsupported questions, while others reject valid ones. Explicit guidance shifts these behaviors differently across models rather than reliably improving abstention.

C. Candidate-set Backoff

  • RoboBrain2.5 rejects 98.5% of GT-absent questions but answers only 3.2% of GT-present questions, despite 36.8% exact-action accuracy without backoff.Its answered accuracy with backoff is 67.9%, but coverage is only 3.2%.Source: C. Candidate-set Backoff · p. 6
  • With explicit guidance, Qwen3.5-4B raises answered accuracy from 44.1% to 55.0% as coverage falls, while other models shift differently.Qwen3.6-27B becomes more willing to answer when the correct action is absent; Qwen3.8-27B and Gemma-4-31B-IT show milder trade-offs.Source: C. Candidate-set Backoff · p. 6
  • Explicit guidance can suppress valid answers or fail to prevent unsupported ones, making backoff difficult to rely on without evaluating the chosen model.Source: C. Candidate-set Backoff · pp. 6–7

VI. CONCLUSION

The evaluation finds failures in fine-grained action recognition, event reconstruction, and backoff. RGB improves action–object pair F1 across models, but tested interventions offer limited gains and reliable robotic use still requires supported labels, intervals, and answers.

VI. CONCLUSION

  • Across two person-centric thermal-video datasets, models recover only part of what happened and when: joint pair F1 ranges from 0.248 to 0.487, and correctly named events may receive unusable intervals.Qwen3.6-27B reaches 52.3% overall accuracy but 16.7% on coughing, sneezing, and yawning; five labels account for 42.9% of action errors.Source: §VI · p. 7
  • With backoff available, Qwen3.6-27B answers 97.2% of GT-present questions but correctly backs off on only 6.1% of GT-absent questions; RoboBrain2.5 answers only 3.2% of GT-present questions.Source: §VI · p. 7
  • RGB improves action–object pair F1 across all seven models, while supplying interaction identities improves event F1 and tested changes yield limited or inconsistent gains.The comparisons point to action and object recognition as a major bottleneck.Source: §VI · p. 7
  • Robotic use requires event labels and intervals supported by video, alongside backoff that rejects unsupported interpretations while retaining valid answers.Source: §VI · p. 7

VII. LIMITATIONS AND FUTURE WORK

The findings are bounded by dataset and evaluation scope, and the behavioral analysis does not identify where models fail internally. The authors propose targeted adaptation, complementary sensing, and active observation as future directions.

VII. LIMITATIONS AND FUTURE WORK

  • OctoNet and Thermal-IM use fixed cameras and activity distributions that differ from household conditions; the 5 FPS clip evaluation omits active observation and robot-behavior coupling.Source: §VII · p. 7
  • The behavioral analysis tracks output changes across conditions but does not locate failures inside models, such as in visual encoding or temporal integration.Layer-wise analyses [33] are identified as a complementary route to mechanistic questions.Source: §VII · p. 7
  • Recognition, localization, and answering tasks remain proxies for robotic applications, and a benchmark of whether robots respond appropriately to human activity is still missing.Source: §VII · p. 7
  • Future work should combine thermal-specific adaptation, complementary sensors, and active observation, evaluating fusion by whether it corrects specific confusions rather than aggregate scores.Active observation could revisit a suspected onset, move closer to an ambiguous object, or select a new viewpoint; its value should be weighed against added observation and computation.Source: §VII · pp. 7–8
Loading 2610.29244v1…