PaperlayerView exact revision on arXiv

Source-linked AI summary

DementiaCare-Bench: A Modality-Validated Video Benchmark

Afrouz Sheikholeslami, Yuankai Qi, Xuyun Zhang, Luping Zhou, Amin Beheshti, Quan Z. Sheng, Ming-Hsuan Yang

arXiv:2609.12929v1cs.CVcs.MA

TL;DR

No existing benchmark evaluates whether video-language models can interpret dementia-care behavior in context, where meaning depends on what happened beforehand. DementiaCare-Bench measures this capability with modality-validated caregiver-video questions, and its evaluation finds that current VLM performance is driven more by clinical knowledge than video understanding.

  • Problem

    No existing benchmark covers dementia-care interactions where a behavior’s clinical meaning depends on the order and cause of events, which can call for different caregiver responses.

    Source: Introduction · p. 1, Conclusion · p. 7

  • Method

    DementiaCare-Bench pairs 56 caregiver-training videos, 94 clips, and 2,023 questions with transcript-grounded clinical claims and verified per-question modality labels.

    Source: Introduction · p. 2

  • Results

    Across 12 current VLMs, high aggregate scores rested on clinical knowledge rather than video understanding, with accuracy falling 17 points on average for sequence-required questions; DemCare-VLM shifted video dependence from −3.3 to +4.5 points.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    Modality validation shows that benchmark claims about visual demand should reflect what questions require when tested, not what their generators intended.

    Source: Benchmark Statistics · p. 5, Conclusion · p. 7

    The source videos portray people with dementia using actors, so their dramatized behaviors are more legible and narratively structured than real caregiving footage.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Dementia affects an estimated 57 million people worldwide, and for most families the hardest part of care is not memory loss but the behavioral and psychological symptoms of dementia (BPSD): agitation, wandering, resistance to care, sundowning. Understanding these symptoms requires more than recognizing the behavior itself; it also requires knowing what happened beforehand. The same behavior may call for a different response depending on its trigger. Video-language models (VLMs) could potentially support caregivers, yet no existing benchmark evaluates this capability. To fill this gap, we present DementiaCare-Bench: 56 professionally produced caregiver training videos segmented into 94 clips across nine BPSD categories, with 2023 questions generated by a multi-agent pipeline that grounds every clinical claim in a verbatim transcript span. Each question is then probed under four visual conditions and labelled by the least it requires, so its visual demand is measured rather than assumed. Measurement contradicts intent: we wrote 77.7% of the questions to require ordered frames, and 34.8% do. Across 12 current VLMs the pattern is uniform. The best reach 85% overall, but that average is carried by questions a language model can answer from clinical knowledge alone; accuracy falls by 17 points on average on questions that require the ordered clip, and a leading open model scores at chance on judging whether a caregiver's response was appropriate. A lightweight LoRA fine-tune, DemCare-VLM, moves video dependence from -3.3 to +4.5 points, so what the benchmark exposes can be repaired and not only measured.

Introduction

DementiaCare-Bench evaluates whether video-language models can interpret caregiving events and their triggers, with each question validated for the least visual information needed to answer it. The benchmark combines clinically grounded questions with clips spanning nine BPSD categories.

Introduction

  • An estimated 57 million people worldwide live with dementia, while BPSD drive caregiver burden and burnout and contribute to decisions about institutional care [Kales et al. 2015].Most dementia care is delivered at home by family members without clinical training [Friedman et al. 2015], and the affected population is projected to nearly triple by 2050 [GBD 2019 Dementia Forecasting Collaborators 2022].Source: Introduction · p. 1
  • The same caregiving action can call for different responses depending on what preceded it, making trigger interpretation central to useful support.The introduction contrasts a push following an unexpected touch with an unprompted push, which may indicate pain or discomfort.Source: Introduction · p. 1
  • No benchmark measures this dementia-care interpretation capability; adjacent work detects agitation without interpreting it or evaluates caregiving language models on text alone [Iaboni et al. 2022] [Parmanto et al. 2024].VideoQA questions may appear to require video while remaining answerable from language priors, a single frame, or co-occurrence statistics [Cheng et al. 2025].Source: Introduction · p. 1
  • Each question is tested with no frames, one frame, shuffled frames, and ordered frames, then labelled by the least visual information that suffices.Success with shuffled frames is labelled Semantic rather than Temporal, and validation occurs within generation before publication.Source: Introduction · p. 1
  • DementiaCare-Bench contains 94 clips across nine BPSD categories and 2,023 multiple-choice questions in 14 categories, with 38 paired videos showing common and expert-recommended responses.Questions are produced by a multi-agent pipeline, with every clinical claim grounded in a verbatim transcript span and each question carrying a verified modality label and per-condition evidence.Source: Introduction · pp. 1–2
  • DementiaCare-Bench separately tests perception, temporal, causal, and clinical-reasoning capabilities because interpreting a BPSD episode requires tracking events, their order, and their trigger.It reports where a model’s competence stops across those steps.Source: Introduction · p. 2

2. Modality Validation.

Modality validation measures each question’s actual visual requirements inside the generation loop, revealing a gap between intended and measured video dependence. A clip-conditioning fine-tune shifts video dependence from negative to positive.

2. Modality Validation.

  • 77.7% of questions written to require ordered video reduce to 34.8% when modality requirements are measured.Validation certifies per question whether answering requires ordered video, a single frame, or text alone, within the generation loop rather than through a posthoc audit.Source: §2 · p. 2

3. DemCare-VLM.

  • Video dependence moves from −3.3 to +4.5 points when DemCare-VLM is trained to condition on clips and penalized for unsupported answers.Each question appears twice in training: with frames from its own clip and with frames from an unrelated clip.Source: §2 · p. 2

Related Work

Temporal reasoning remains weaker than static perception in video-language models, and common benchmark scores can reflect language or static-frame shortcuts. Existing dementia tools and medical video benchmarks leave temporally ordered human interaction and response-appropriateness judgments unmeasured.

Video-Language Models

  • Temporal reasoning lags static perception in video-language models, which often sample frames uniformly; DemCare-VLM responds with lightweight fine-tuning rather than architectural change [Feng et al. 2025, Plizzari et al. 2025].Temporal modeling work includes recurrent memory [Wang et al. 2024], hierarchical token compression [Wang et al. 2025], and training recipes [Zohar et al. 2024].Source: Video-Language Models · p. 2

Video Understanding Benchmarks

  • VideoQA has expanded from short-clip tests [Xiao et al. 2021, Li et al. 2024] to long-video and expert-level suites [Fu et al. 2025, Mangalam et al. 2023, Patraucean et al. 2023], yet language priors and frame shuffling reveal shortcut risks [Cores et al. 2025, Krojer et al. 2025, Feng et al. 2025, Plizzari et al. 2025].Expert-level suites include professional lectures [Hu et al. 2026], skilled physical activity [Yi et al. 2025], and social interaction [Kong et al. 2025].Source: Video Understanding Benchmarks · p. 2
  • Dementia sensing detects agitation without language [Iaboni et al. 2022], while caregiver-support LLM applications use text alone [Parmanto et al. 2024]; DementiaCare-Bench tests ordered interaction and response appropriateness.Response appropriateness is a central caregiver-training judgment [Kales et al. 2015].Source: Video Understanding Benchmarks · p. 2
  • DrVD-Bench structures questions around stepwise clinical reasoning but focuses on diagnostic imaging rather than human interaction [Zhou et al. 2025].Source: Video Understanding Benchmarks · p. 2

DementiaCare-Bench Construction

DementiaCare-Bench combines professionally produced caregiver-training videos with clinically grounded questions whose visual requirements are measured. Its construction spans nine BPSD categories and four reasoning axes, with questions validated against the least visual information needed to answer them.

Data Collection and Clip Structure

  • The corpus comprises 56 professionally produced actor-based training videos across nine BPSD categories, segmented into 94 clips, including paired common- and recommended-response clips.Thirty-eight videos yield paired clips; the other 18 are retained as single full-length clips.Source: Data Collection and Clip Structure · pp. 2–3

Question Categories and Generation Pipeline

  • The benchmark adapts DICE into four axes—visual perception, temporal and causal reasoning, clinical reasoning, and comparative analysis—with fourteen fine-grained categories; trigger identification carries particular clinical weight.Paired clips support judging why one caregiver response is safer or more effective.Source: Question Categories and Generation… · p. 3

Generation pipeline.

  • A five-stage multi-agent pipeline separates scene description, transcript parsing, clinical-record extraction, quota-based category generation, and independent modality validation; clinical claims must trace to transcript lines.A different model validates questions than the one used to generate them.Source: Generation pipeline · p. 3

Modality Validation

  • Questions are labeled by the least of no, one, shuffled, or ordered frames that lets GPT-4o answer correctly across five reordered-option runs; only ordered-clip-answerable items remain.The shuffled-frame condition distinguishes semantic items from those that require temporal order.Source: Modality Validation · p. 4

Benchmark Statistics

  • 77.7% of questions were written to require ordered frames, but validation found 34.8%; language-prior controls shifted from 12.4% to 49.8%, overstating temporal content 2.2-fold.Semantic items moved from 9.9% to 15.9%.Source: Benchmark Statistics · pp. 4–5

Method: DemCare-VLM

DemCare-VLM uses a frozen Qwen3-VL-32B with LoRA adapters and mismatch negatives to train the model to check whether video frames support a question and abstain when they do not.

Method: DemCare-VLM

  • DemCare-VLM adapts Qwen3-VL-32B with a frozen base and small LoRA adapter, aiming to make it read the clip before answering instead of guessing from question text.The training data is designed so that relying on language priors loses points on half of every batch.Source: Method: DemCare-VLM · p. 5

LoRA fine-tuning

  • The frozen model receives rank r=16, scaling α=32 LoRA adapters in four attention projections per block, adding 39.8 M trainable parameters (0.12% of the model).Training runs for three epochs with AdamW, peak learning rate 10^-4, 5% linear warmup, cosine decay, and effective batch size 8; checkpoints store only adapter tensors.Source: LoRA fine-tuning · p. 5

Training data

  • Matched training examples pair each question and answer options with six evenly spaced frames from its source clip, targeting the correct answer letter.These examples teach benchmark vocabulary and the mapping between a specific clip and its answer.Source: Training data · p. 5
  • Mismatch negatives pair each question with frames from another BPSD category and reward abstention, penalizing answers based on language priors alone.One negative is added per matched example, creating a 50/50 training split; loss is applied only to answer tokens. The training set has 1,940 examples, with 297 matched test examples from a video-level split of 31 training and 7 testing videos.Source: Training data · p. 5

Relation to prior work

  • The contribution is rejection on mismatch: the model learns to check whether frames bear on a question and abstain when visual evidence is insufficient, rather than learn more dementia-care knowledge.The passage distinguishes this training signal from standard LoRA fine-tuning and answer-only loss masking [Hu et al. 2022].Source: Relation to prior work · p. 5

Experiments

Across the evaluated models, strong text-only performance often obscures weak reliance on video; fine-tuning Qwen3-VL-32B raises benchmark accuracy and changes its video-dependence gain from negative to positive. The modality breakdown qualifies that change: much of the gain reflects reduced guessing on text-answerable questions, while temporal accuracy also improves.

Experiments

  • The evaluation covered twelve proprietary and open-weight VLMs and tested 0, 8, 16, 32, 48, 64, or 128 uniformly sampled frames per question.Zero frames meant text-only prompting; video frames were extracted at 1 fps and resized to 512 × 512.Source: Experiments · p. 5

The Language-Prior Shortcut

  • Six of twelve models scored highest with no frames, and a seventh tied; the four leading text-only models had video-dependence gains of +0.3, 0.0, −1.8, and −0.2.Gemma-3-27B-IT, Claude-Sonnet-4-6, Qwen3-VL-32B, and Qwen3-VL-235B-A22B scored 84.8, 83.7, 83.6, and 82.3 text-only, respectively.Source: The Language-Prior Shortcut · p. 5, Table 2 · p. 6
  • Qwen3-VL-32B answered clinical-safety questions correctly 100% of the time without frames, scored 94% on person attributes, and 50%—chance—on caregiver-response appropriateness.The passage attributes the first two results to clinical knowledge and cues in the question stem; judging response appropriateness requires watching the clip.Source: The Language-Prior Shortcut · p. 6
  • Keye-VL-8B and MiniCPM-V-2.6, the weakest text-only models at 62.5% and 53.2%, gained 12.5 and 6.7 points, respectively, when frames were supplied.The passage interprets their larger gains as reflecting thinner language priors and more room for visual evidence to contribute, not better clip understanding than stronger models.Source: The Language-Prior Shortcut · p. 6

DemCare-VLM Restores Video-Dependence

  • DemCare-VLM scored 90.3% with few frames and 93.1% at its best, versus base Qwen3-VL-32B’s 83.6% peak without frames; ∆v shifted from −3.3 to +4.5 points.About seven-eighths of the ∆v swing came from LLM-Answerable items (−9.9 to −1.7), while Temporal ∆v rose from +17.4 to +23.6. Temporal accuracy increased by 14.5 points at mf=8 and 13.5 at mf=32; Semantic accuracy gains of 7.9 and 15.8 points came with nearly unchanged ∆v (+53.2 to +52.8).Source: DemCare-VLM Restores Video-Dependence · p. 6, Table 3 · p. 7

Limitations

The benchmark is limited by dependence on GPT-4o for question validation, dramatized actor portrayals rather than real caregiving footage, and corpus coverage shaped by available training material rather than clinical prevalence.

Limitations

  • GPT-4o sets the benchmark’s difficulty ceiling, and its and Qwen3-VL-235B-A22B’s scores are not directly comparable with other models because both validate or generate questions.Candidates GPT-4o cannot answer correctly with ordered frames are discarded and absent from the released corpus; a reference panel outside the evaluation pool would reduce this dependence.Source: Limitations · p. 7
  • The videos portray people with dementia through actors, making behaviors more legible and narratively structured than real caregiving footage; performance is necessary but not sufficient for deployment.The source material is professionally produced training content, which makes the corpus ethically distributable.Source: Limitations · p. 7
  • The corpus comprises 56 videos and 94 clips, with BPSD-category coverage reflecting available training material rather than clinical prevalence.Disinhibition contributes two videos, three sources account for most of the corpus, and results on sparsest categories are indicative.Source: Limitations · p. 7

Conclusion

DementiaCare-Bench reveals that many questions intended to test ordered-video understanding do not require it, and that current VLM performance often reflects clinical knowledge rather than video understanding. DemCare-VLM turns this diagnosis into a training signal that improves video dependence.

Conclusion

  • Modality validation reduces the share of questions written to require ordered video from 77.7% to 34.8%.The benchmark’s methodological claim is that questions should be measured to verify what they test, rather than relying on intended modality labels.Source: Conclusion · p. 7
  • Across twelve current VLMs, accuracy falls 17 points on questions requiring the sequence; DemCare-VLM shifts video dependence from −3.3 to +4.5 points.DemCare-VLM uses a LoRA adapter trained with reject-on-mismatch negatives and adds 0.12% of the model’s parameters.Source: Conclusion · p. 7
  • Future work should validate labels with an external model panel, test transfer to consented naturalistic recordings, and assess calibration under distribution shift.The current label validation depends on one reference model, dramatized training footage limits real-care transfer evidence, and reject-on-mismatch training may extend to other VideoQA domains where language priors dominate.Source: Conclusion · p. 7
Loading 2609.12929v1…