PaperlayerView exact revision on arXiv

Source-linked AI summary

SemReward-VL: Semantic Reward-Guided Video-Language Adaptation for Developmental Behavior Assessment

De Jiang, Shuo Zhang, Kehong Yuan, Hongen Liao

arXiv:2609.33082v1cs.CVcs.AI

TL;DR

Developmental records often contain outcomes but not descriptions of the behaviors visible in screening videos. SemReward-VL uses semantic rewards to adapt item-specific descriptions, improving accuracy and item-level recall coverage across its evaluated backbones. The descriptions also expose remaining errors in temporal direction, duration, and age-specific interpretation.

  • Problem

    Clinical records commonly provide an outcome without describing the video behavior, leaving the visual evidence for item decisions implicit.

    Source: §1 · p. 1

  • Method

    SemReward-VL trains a vision-language model to generate item-specific descriptions, using a frozen language model’s semantic rewards to guide GRPO adaptation.

    Source: §1 · p. 1

  • Results

    5/41 items with recall above 0.5 under broad prompts rises to 30/41 with item-specific prompts and 38/41 after adaptation; accuracy also improves on both VLM backbones.

    Source: §4 · p. 4

  • Takeaways & Limitations

    The descriptions provide an inspectable account of behavior supporting item decisions and expose errors in temporal direction, duration, and age-specific interpretation.

    Source: §1 · p. 1, §4 · p. 4

    Descriptions may miss temporal attributes, misinterpret visible actions under the item criterion, or lack the requested evidence.

    Source: §3.5 · p. 4

Abstract

from arXiv · show

Developmental screening videos show how children perform specific behaviors, but clinical records usually contain outcomes rather than descriptions of what happened. We present SemReward-VL, which learns to describe item-specific behavior from these outcomes. A vision-language model generates a description, and a frozen language model scores its agreement with the clinical outcome, relevance to the item, abstention on unrelated video-item pairs, and clarity. Group relative policy optimization (GRPO) updates LoRA adapters using this semantic reward. On 13,379 videos covering 41 items, the method improves aggregate accuracy and the number of items with recall above 0.5. Errors remain in temporal direction, duration, and age-specific interpretations of behavior.

1. INTRODUCTION

SemReward-VL addresses the gap between clinical outcomes and descriptions of the behavior visible in developmental videos. It uses item-conditioned descriptions and semantic rewards to adapt a vision-language model, improving assessment results while leaving temporal and age-specific errors.

1. INTRODUCTION

  • Clinical records usually provide an outcome but lack a description of the video’s behavior, leaving the visual evidence for the decision implicit.Each record includes an age-specific item and pass criterion alongside the clinician-reviewed outcome and uploaded video.Source: §1 · p. 1
  • Because an item may hinge on a small detail, a broad clip description can omit the relevant direction, sustained posture, or social context.The item criterion must guide both what the model describes and how the description is interpreted.Source: §1 · p. 1
  • The same pass/fail outcome can reflect different visible behaviors, so the approach generates an item-specific description and can abstain when requested evidence is absent.A direct classifier compresses distinct cases into one label, while a single caption target cannot represent multiple valid descriptions.Source: §1 · p. 1
  • A VLM describes the video under the item criterion, while a frozen LLM interprets and scores descriptions for outcome agreement, relevance, abstention, and quality.GRPO updates rank-16 LoRA adapters while the visual backbone and judge remain frozen.Source: §1 · p. 1
  • The paper frames assessment as outcome-supervised description followed by decision, adding an inspectable evidence layer without requiring a unique expert-written caption for every video.The intermediate description lets clinicians inspect support for an item decision and distinguish observation errors from errors applying the criterion.Source: §1 · p. 1
  • Across 13,379 videos and 41 behaviors, accuracy and item-level recall coverage improve on both VLM generations, while errors remain in temporal direction, duration, and age-specific semantics.The evaluation includes videos from 4,670 children.Source: §1 · p. 1

2. METHOD

The method separates visual description from item interpretation, then trains item-aware descriptions with semantic rewards and group-relative updates. It allows varied descriptions and an unable-to-assess response when relevant evidence is unavailable.

2.1. Problem formulation and two-stage inference

  • Inference first generates a behavior description from the video and item criterion, then applies a frozen semantic decision model to interpret it.The judge can return pass, fail, or unable-to-assess.Source: §2.1 · p. 2

2.2. Item-conditioned prompting and abstention

  • Item-conditioned prompts focus descriptions on criterion-relevant evidence, while absent, occluded, or out-of-clip evidence prompts abstention rather than an observed failure.Mismatched video–item pairs also reward abstention.Source: §2.2 · p. 2

2.3. LLM-as-a-Judge semantic reward

  • Four rewards score outcome agreement with sufficient evidence, abstention on mismatched pairs, item relevance, and generation quality.Outcome agreement alone produced irrelevant or malformed responses, especially under strong pass-label imbalance.Source: §2.3 · p. 2

2.4. GRPO optimization

  • GRPO standardizes rewards across 16 sampled descriptions and applies a clipped policy objective with a KL reference anchor, using the group as baseline without a separate critic.Similar rewards across candidates provide little directional signal for an update.Source: §2.4 · p. 3

2.5. Parameter-efficient adaptation

  • Rank-16 LoRA adapts only the VLM language component; the visual backbone and judge stay frozen, after preliminary full-parameter tuning proved unstable.Training uses AdamW at 10^-5, batch size 8, and two epochs.Source: §2.5 · p. 3

3. EXPERIMENTS

Across six experiments subsections, the paper evaluates item-specific video descriptions on a 13,379-video developmental dataset, comparing prompts and GRPO-LoRA adaptation. Accuracy and item coverage improve, while temporal and interpretation errors remain.

3.1. Dataset and evaluation protocol

  • The dataset contains 13,379 naturalistic videos from 4,670 children and 41 developmental items, with clinician-reviewed outcomes but no dense behavior descriptions.Each example includes the item, filming guidance, pass criterion, outcome, and caregiver- or clinician-uploaded video; ages range from 2 to 42 months.Source: §3.1 · p. 3

3.2. Evaluation protocol

  • The evaluation uses a frozen LLM judge to assess relevance and criterion-supported behavior, with unrelated video–item pairs testing abstention.It measures whether the description supports an item decision rather than requiring a fixed sentence; aggregate accuracy reflects agreement with clinician-reviewed outcomes.Source: §3.2 · p. 3

3.3. Prompt specificity before adaptation

  • Item-specific prompts yield recall > 0.5 on 30 of 41 items, compared with 5 of 41 under broad prompts, and produce shorter, more relevant descriptions.The prompt states the exact item and pass criterion and asks the model to abstain when evidence is absent.Source: §3.3 · p. 3

3.4. GRPO-LoRA adaptation

  • Qwen2.5-VL accuracy rises from 0.669 to 0.817 (+22.12% relative), while Qwen3-VL rises from 0.808 to 0.890 (+10.15% relative).Adapted Qwen2.5-VL approaches unadapted Qwen3-VL; adapted Qwen3-VL reaches the highest accuracy.Source: Table 3 · p. 4, §3.4 · p. 4
  • Coverage rises from 5/41 items with broad prompts to 30/41 with item-specific prompts and 38/41 after adaptation, measured as items with recall > 0.5.The item-level figure compares base-VLM bars with GRPO-LoRA lines over 41 age–item codes.Source: Table 4 · p. 4, §3.4 · p. 4, Figure 2 · p. 4

3.5. Item-level behavior and failure modes

  • Three items remain below 0.5 recall: walking backward (0.000), expressing “no” (0.053), and standing on one foot for 2 s (0.471).Temporal direction and duration are difficult; when all 16 sampled descriptions repeat an error, GRPO has little signal to favor a correction. Refusal may also appear as turning away or pushing an object.Source: §3.5 · p. 4

3.6. Discussion and stratified analysis

  • Item conditioning directs descriptions toward criterion-relevant evidence, while the description layer helps distinguish visual-grounding errors from errors applying the criterion.Unrelated-pair abstention addresses the risk of apparent accuracy when most clinical outcomes are passes; sex-stratified precision is broadly similar, but three test subsets lack valid female examples.Source: §3.6 · p. 4

4. CONCLUSION

SemReward-VL separates behavior description from item decision, using frozen-LLM rewards to guide GRPO-LoRA adaptation without caption labels. Across 13,379 videos and 41 items, it improves item recall coverage and accuracy while exposing remaining interpretation errors.

4. CONCLUSION

  • Item-specific prompts raise the number of items with recall above 0.5 from 5 to 30, and adaptation raises it to 38.Accuracy also improves on both VLM backbones.Source: §4 · p. 4
  • Frozen-LLM rewards guide GRPO-LoRA adaptation without caption labels, after SemReward-VL separates behavior description from item decision.The approach covers 13,379 videos across 41 items.Source: §4 · p. 4
  • Descriptions make decisions easier to inspect but reveal errors in temporal direction, duration, and age-specific behavior interpretation.Source: §4 · p. 4

Related papers

Loading 2609.33082v1…