PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded Human-Attributed Description and Activity Recognition in Videos

Jay N. Paranjape, Jurijs Nazarovs, Rohan Sarkar, Yue Guo, Lipeng Ke, Sankar Venkataraman, Vishal M. Patel, Nataraj Jammalamadaka

arXiv:2610.22295v1cs.CV

TL;DR

The paper addresses limited coverage of open-set, per-person human descriptions and activity recognition in multi-person videos. It introduces GHADAR, AVA-Captions, CAMP, and video-based evaluation, reporting consistent gains over strong VLM baselines on two datasets.

  • Problem

    Existing datasets often simplify human-centric video understanding to group activity, single-person, or closed-set recognition, while GHADAR targets open-set descriptions for every located person.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces AVA-Captions, trains VLMs with CAMP’s grounding-informed attention masking, and proposes VLM evaluation comparing predictions with video content.

    Source: §1 · pp. 2–3, §7 · p. 14

  • Results

    The authors report consistent gains over strong VLM baselines on AVA-Captions and HC-STVG v2 across standard and proposed metrics.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The work provides a dataset, training method, and video-based evaluation framework for GHADAR.

    Source: §7 · p. 14

    Because VLM-based metrics can be biased, the authors recommend using them for relative model comparisons rather than absolute single-model performance.

    Source: §6 · p. 14

Abstract

from arXiv · show

Understanding characteristics and activities of individuals in complex multi-person environments is crucial for real-world applications. However, existing datasets often simplify this problem to grounded group activity recognition, single-person activity recognition, or close-set activity recognition, thus limiting the generalizability of models trained on these datasets. In this work, we introduce the task of Grounded Human-Attributed Description and Activity Recognition (GHADAR), which involves describing the characteristics and activity descriptions of \textbf{every} person in a video, provided the location of the person in the video, which is more practical for real-world applications. To facilitate this, we introduce a new dataset derived from AVA-Actions by generating open-set captions for the person description and activity. In addition, we propose a novel method to effectively utilize the information contained in grounding during training by constraining the cross-attention masks during training in VLMs to improve performance for this task. Our experiments show that our method outperforms SOTA VLMs on this task. Finally, we demonstrate the limitations of existing evaluation metrics, which are overly reliant on human-annotations and exact text-text matching. As an added video-based evaluation, we propose a holistic VLM-based evaluation schema that compares concepts {\em directly} between the video and the generated predictions. Thus, in this work, we develop a complete framework for GHADAR, including a dataset, a novel method and an evaluation schema, thereby establishing a strong foundation for future research in this domain.

1 Introduction

GHADAR asks for open-set descriptions of every person’s attributes and activity, given their location in a video. The paper introduces AVA-Captions, CAMP, and a video-based evaluation framework, and reports gains over strong VLM baselines on two datasets.

1 Introduction

  • GHADAR generates open-set captions describing each person’s attributes and activity, given that person’s bounding box in a video.The task calls for per-person reasoning in multi-actor scenes using local appearance and broader context.Source: §1 · p. 2
  • CAMP uses two-stage attention masking to leverage grounding for per-person reasoning.Phase 1 restricts output-token attention to visual tokens in an expanded target-person box; Phase 2 removes the constraints and fine-tunes at a reduced learning rate.Source: §1 · pp. 2–3
  • The proposed VLM-based evaluation compares video content with predictions for completeness, uniqueness, and correctness, complementing text-matching metrics.Existing n-gram and embedding-similarity metrics can penalize semantically correct predictions that differ lexically from human references.Source: §1 · p. 2
  • AVA-Captions extends AVA-Actions with open-set, per-person captions for appearance and activity across multi-person interactions.The dataset is built from AVA-Actions per-person labels, with natural-language descriptions generated using a high-capacity VLM and identity-aware deduplication.Source: §1 · p. 2
  • The paper reports consistent gains over strong VLM baselines on AVA-Captions and HC-STVG v2 across standard and proposed metrics.The reported comparison covers both datasets and both metric families.Source: §1 · p. 3

2 Related Work

Prior datasets and activity-recognition approaches cover relaxed versions of human-centric video understanding, including single-target, single-person, group-level, or closed-set settings. The related-work discussion positions open-set, multi-person descriptions as a less constrained setting.

Existing Datasets

  • Table 1 defines Multi-Activity as availability of captions for multiple activities per person and Multi-Person as annotations for multiple people.Most listed clips are 0–20 seconds, except AVA-Actions and MEVA, with 15-minute and 5-minute annotated videos, respectively.Source: Table 1 · p. 3
  • Existing datasets variously lack open-set captions, per-person grounding, or multi-person coverage, motivating AVA-Captions.HC-STVG is the closest benchmark but annotates only one target person per clip; AVA-Actions has per-person labels but no person descriptions and only 80 predetermined action classes.Source: §2 · p. 4

Human Activity Recognition Methods

  • Prior activity-recognition methods use constrained settings such as single-person processing, group activity recognition [13], or predefined action classes.The paper describes its approach as among the first to allow open-set captions and human-attributed action recognition with multiple people in view.Source: §2 · p. 4

3 AVA-Captions Dataset

AVA-Captions is constructed from AVA-Actions by segmenting videos, generating per-person appearance and activity captions, and deduplicating identities. Manual validation assesses caption quality and informs the ReID threshold.

3 AVA-Captions Dataset

  • AVA-Captions uses approximately 36.8K training and 11.1K testing clips from AVA-Actions, segmented into 10-second clips with up to 5 seconds of overlap.The source contains 235 training and 64 validation videos; its official test split is not used because AVA-Actions does not release test annotations.Source: §3 · p. 5
  • Nova Pro generates separate per-person appearance and activity captions using video, a person crop, and, for activity, the original action labels.Appearance prompts request apparel, age, and location; activity prompts ask for a complete caption grounded in the video and labels.Source: §3 · p. 5
  • A Person-ReID filter merges repeated PIDs when crop similarity exceeds 0.75, retaining the earlier PID and merging its captions.The filter addresses cases where a camera pans away and returns to the same person under a new PID.Source: §3 · pp. 5–6
  • Manual inspection found that about 95% of cases used all action labels and produced accurate person and activity descriptions.Repeated actions occurred in roughly 15% of cases; about 15% still assigned multiple PIDs, while a 0.85 threshold raised this to 20% and 0.65 caused false merges in around 30%.Source: §3 · p. 6

4 Constrained Attention Masking-based Pretraining (CAMP)

CAMP uses grounding to focus person-specific caption generation, then restores global context through a second training phase. Its formulation produces both appearance and activity captions for every grounded person in a video.

4.1 Problem Formulation

  • Given a video and per-person bounding boxes, GHADAR requires both a person-description caption and an activity caption for every subject.Source: §4.1 · p. 6

4.2 CAMP Phase 1

  • Phase 1 restricts answer-token attention to video tokens within a relaxed box around the target person, retaining surrounding context for activity descriptions.The box expands more in width than height because person boxes are generally taller than they are wide and activity cues occur nearby.Source: Figure 3 · p. 7, §4.2 · pp. 8–9

4.3 CAMP Phase 2

  • Phase 1 alone cannot capture activities requiring global context, and its person-specific masks are impractical at inference because they require knowing which person each output token describes.Dynamically updating masks during generation would increase inference complexity and latency.Source: §4.3 · p. 9

4.4 Motivation

  • Phase 2 removes the mask constraints and fine-tunes at a reduced learning rate, restoring global cues while preserving focus on the person of interest and avoiding inference-time mask changes.The paper presents the two phases as balancing localized grounding with global contextual reasoning.Source: §4.3 · p. 9, §4.4 · p. 9

5 Experiments

CAMP improves on similar-scale open-source VLM baselines across classical and embedding-based metrics for subject and activity description on HC-STVG and AVA-Captions. Ablations report benefits from both training phases and from expanding the bounding box more in width.

Comparison with Similar Scale VLMs

  • CAMP improves all classical and embedding-based metrics on HC-STVG and AVA-Captions for both subject and activity descriptions.The comparison includes InternVL3-1B and Qwen2.5VL-3B backbones; reported relative improvements are greater for InternVL3-1B than for Qwen2.5VL-3B.Source: Table 2 · p. 10, §5 · p. 10, Table 3 · p. 11

Ablation on Relaxation of Bounding Box

  • On HC-STVG, increasing bounding-box context in width is more effective than increasing it in height.Table 4 varies the one-sided width and height relaxations as percentages of the original dimensions.Source: §5 · p. 11, Table 4 · p. 11

Ablation on the phases of CAMP

  • Using only Phase 1 causes a 10% drop, while using Phase 2 without Phase 1 causes a 4% drop, supporting the use of both phases.Source: §5 · p. 11

6 VLM-based Metrics

Text-matching metrics can underrate visually valid captions when they diverge from incomplete ground truth. The paper complements them with VLM judges that compare predictions directly with video content and report correctness, completeness, and subject uniqueness.

Metric Definition

  • Text-text and embedding metrics can undervalue valid predictions that differ lexically from ground truth or describe details its annotations omit.The proposed VLM judges receive the video, a target-subject crop, and the predicted caption, and are intended to complement—not replace—classical metrics.Source: §6 · pp. 11–12
  • VLM judges score subject and activity captions for correctness (CRT), completeness (CMP), and subject uniqueness (UNQ), alongside thresholded accuracy.The paper averages each score and computes accuracy using CRT ≥ 4 with UNQ ≥ 3 for subjects or CMP ≥ 3 for activities.Source: §6 · pp. 12–13
  • CRT and CMP respectively resemble precision and recall, while UNQ measures whether a subject description distinguishes its person within that video.The paper notes that CMP is ill-defined for subject descriptions because the same person can be described in many ways.Source: §6 · pp. 12–13

Comparison with Similar Scale VLMs

  • CAMP-based predictions consistently outperform baselines in the reported VLM-based comparisons, with thresholded accuracy separating methods more strongly than modest average-score increases.The authors attribute localization of actions to constrained attention; without CAMP, InternVL3 confuses person IDs and LLAVA-OV confuses person descriptions.Source: §6 · p. 13, Table 5 · p. 14, Table 6 · p. 14

Limitations

  • VLM judges can be biased and prompt-sensitive, so the authors recommend relative model comparisons rather than treating one model’s absolute score as definitive.The cited comparison tables cover subject description and activity description, using Nova Pro v1 and Qwen2.5 VL 32B as expert models.Source: §6 · p. 14, Table 5 · p. 14, Table 6 · p. 14

7 Conclusion

The paper presents GHADAR, an AVA-Actions-derived dataset, CAMP training, and VLM-based video–prediction metrics. It also notes that the judges’ reliability depends on their ability to understand the video and instruction.

7 Conclusion

  • The paper introduces GHADAR and AVA-Captions, derived from AVA-Actions, and proposes CAMP to use grounding during training for person-focused captioning.Its VLM-based metrics compare predictions with video content rather than relying only on incomplete human annotations, while complementing classical metrics.Source: §7 · pp. 14–15
  • The VLM-based metric depends on the VQA model’s ability to understand the video and instruction.Source: §7 · p. 15

Supplementary - Grounded Human-Attributed 001 001

The supplementary describes generating per-person appearance and activity captions with Nova Pro v1 from video, a person crop, and AVA-Actions labels.

Supplementary - Grounded Human-Attributed 001 001

  • Nova Pro v1 receives the video, person crop, and that person’s AVA-Actions labels to generate appearance and activity captions.For videos with multiple people, the model runs once per person and the resulting captions are merged.Source: Supplementary - Grounded… · p. 18
  • The prompt requests appearance details and an activity caption that uses all supplied action labels in temporal order.The output format pairs a person description with an activity caption, separated by a colon.Source: Supplementary - Grounded… · p. 18

2 Experimental Setup 023 023

Experiments use 10 equidistant frames per video and evaluate subject and activity captions separately with text-matching and embedding-similarity metrics.

2 Experimental Setup 023 023

  • Experiments sample 10 equidistant 448×448 frames per video, approximately 1 FPS for the mostly 10-second clips.The setup uses three A100-SXM4 GPUs and batch size 3.Source: §2 · p. 18
  • Subject and activity captions are evaluated separately using BLEU4, ROUGE, METEOR, CLIP similarity, and SBERT similarity.CLIP and SentenceBert embeddings are compared by cosine similarity.Source: §2 · p. 18

3 More examples from AVA-Captions Dataset 038 038

AVA-Captions examples pair person descriptions with open-set activity captions that follow the original action labels. The dataset also reports how many people appear per clip.

3 More examples from AVA-Captions Dataset 038 038

  • Activity captions follow the ground-truth action labels while remaining open-set captions.The examples also show person descriptions with details absent from the original ground truth.Source: §3 · p. 19
  • Table 1 reports the number of people per video clip in AVA-Captions.Source: Table 1 · p. 20

4 Ablation on Phases of CAMP 044 044

The ablation finds lower scores on both datasets when only Phase 1 is used and its attention constraints are also applied at inference. The table compares this variant with the baseline and full CAMP.

4 Ablation on Phases of CAMP 044 044

  • Applying Phase 1 constraints during both training and inference yields significantly lower scores on HC-STVG and AVA-Captions.The authors attribute the risk to false outputs producing incorrect attention masks for subsequent tokens.Source: §4 · p. 20, Table 2 · p. 20
  • Table 2 compares the baseline without CAMP, Phase 1 alone, and CAMP with both phases.Source: Table 2 · p. 20

5 Motivating the need for VLM-metrics 053 053

Lexical mismatch and details missing from human annotations can make text-based metrics pessimistic about video-grounded captions. The section motivates video-based judging while cautioning that VLM metrics may themselves be biased.

5 Motivating the need for VLM-metrics 053 053

  • Classical metrics can score valid predictions poorly when their wording differs from ground truth or they describe details omitted by annotators.The authors motivate evaluating predictions against the video content directly.Source: §5 · p. 20, Figure 2 · p. 21
  • VLM evaluation can assign the lowest completeness and uniqueness scores even when a caption identifies the person and accurately describes some activities.Figure 3 illustrates this limiting case.Source: §5 · p. 21, Figure 3 · p. 21
  • Because VLM metrics can be biased toward pessimistic completeness and uniqueness values, the authors recommend using them for relative model comparison rather than absolute performance.Source: §5 · p. 22

Related papers

Loading 2610.22295v1…