PaperlayerView exact revision on arXiv

Source-linked AI summary

IAE-VTG: Interaction-Aligned Action-Entity Video Temporal Grounding

Shiwen Zhao, Qi Zhang, Sezer Karaoglu, Theo Gevers, Martin R. Oswald

arXiv:2609.09736v1cs.CVcs.MM

TL;DR

VTG methods may localize segments containing relevant actions and entities without verifying that they jointly form the queried event. IAE-VTG addresses this with disentangled action–entity interaction modeling and interaction-aware assignment, achieving competitive or state-of-the-art performance across several benchmarks.

  • Problem

    Existing VTG methods do not explicitly verify that queried actions and entities occur together, allowing semantically incorrect but temporally plausible segments.

  • Method

    IAE-VTG uses FDIM to align action and entity query information with motion and appearance features, then uses ISA to include interaction evidence in bipartite matching.

  • Results

    IAE-VTG achieves competitive or state-of-the-art grounding performance on QVHighlights, Charades-STA, and TACoS, with analyses supporting explicit interaction modeling.

  • Takeaways & Limitations

    Explicit action–entity interaction modeling improves grounding for compositional splits and complex-query groups while producing more reliable assignments.

Abstract

from arXiv · show

Video Temporal Grounding (VTG) localizes the video segment that matches a natural-language query. Many queries describe an action performed by a particular entity. Existing methods often encode the query as a whole or use general video-text interactions, without explicitly checking whether the action and entity occur together. They may therefore select a segment that contains both concepts but not the event described by the query. We propose Interaction Aligned Action-Entity Video Temporal Grounding (IAE-VTG), which models this rela?tionship at both the representation and training assignment levels. First, the Fine-grained Disentangled Interaction Module (FDIM) separates action and entity related query information and aligns it with complementary motion and appearance features. It then combines token-level interactions to build representations that capture the relationship between the action and entity. Second, Interaction-Sensitive Assignment (ISA) adds this interaction evidence to bipartite matching, so training targets are selected using both temporal overlap and semantic compatibility. This reduces supervision from temporally plausible but semantically incorrect proposals. Experiments on QVHighlights, Charades?STA, and TACoS show that IAE-VTG consistently improves strong baselines and achieves competitive or state-of-the-art performance on standard grounding metrics. Additional analyses show that the method is especially effective when similar actions or entities appear at multiple times and produces more reliable assignments for complex events.

I. INTRODUCTION

IAE-VTG addresses action–entity ambiguity in video temporal grounding by checking whether both components form the queried event at the same moment. It combines interaction-aware representations with interaction-sensitive supervision and reports competitive or state-of-the-art performance across standard benchmarks.

  • IAE-VTG targets cases where models detect relevant actions or entities independently without verifying their joint occurrence within the localized moment.
  • FDIM grounds action- and entity-related query tokens in complementary motion and appearance streams to construct composition-sensitive video–text representations.
  • ISA adds interaction consistency to bipartite matching, discouraging temporally plausible predictions with incomplete action–entity support from receiving positive supervision.
  • IAE-VTG achieves state-of-the-art or competitive performance across standard grounding metrics on multiple datasets.

B. Fine-Grained Cross-Modal Interaction and Assignment

Prior VTG methods improve multimodal interaction, temporal reasoning, and saliency, but holistic relevance and richer temporal representations do not ensure action–entity consistency. IAE-VTG addresses this gap jointly through FDIM representations and ISA assignments.

  • Query conditioning and saliency estimate relevance to the complete query or clip, but do not establish that action and entity evidence are compositionally consistent.
  • IAE-VTG specifically targets action–entity binding ambiguity rather than unrestricted phrase-level or graph-level relations.
  • The framework uses appearance and motion features, interaction-refined proposals, and interaction-sensitive matching to align representation learning and supervision.

B. Fine-grained Disentangled Interaction Module

FDIM separates action and entity information, grounds each role in complementary visual streams, and retains token-level responses for pairing and interaction binding. These operations produce clip-level evidence for whether the queried action and entity are jointly supported.

  • FDIM performs role-specific grounding, action–entity token pairing, and interaction binding to construct interaction evidence.
  • A lightweight parser identifies noun and verb tokens, while binary masks select role-specific token sets without replacing the complete query used by the grounding backbone.
  • Appearance features preserve subjects, objects, and scene attributes, whereas motion features respond more strongly to temporal changes.
  • Cross-attention yields action-aware and entity-aware clip representations plus token-level response matrices for subsequent noun–verb pairing.
  • The pairing mechanism favors noun–verb combinations whose appearance-related noun and motion-related verb responses are simultaneously strong at each clip.

2) Action–Entity Token Pairing:

The method learns which noun–verb combinations describe the target event by combining role-specific visual evidence with language and local-context compatibility.

  • 2) Action–Entity Token Pairing:: IAE-VTG retains noun–verb pairs whose pairing is most strongly supported by visual content and then uses them for interaction aggregation.Only the top Kpair = 8 pairs are retained in subsequent operations.
  • 2) Action–Entity Token Pairing:: The parser identifies token roles, while the model learns the correspondence between nouns and verbs.
  • 2) Action–Entity Token Pairing:: Pairing probabilities combine cross-modal, language-level, and local appearance–motion compatibility evidence.The cross-modal prior favors pairs with simultaneously strong appearance-related noun and motion-related verb responses.
  • 2) Action–Entity Token Pairing:: The learned pair distribution yields noun and verb marginal probabilities for constructing pair-aware action and entity representations.

3) Interaction Binding:

The interaction-binding representation combines action and entity evidence to measure whether both are jointly supported at each temporal location.

  • 3) Interaction Binding:: The product term captures action–entity coactivation, while the absolute difference measures disagreement between their evidence.An MLP maps these features to a clip-level binding score.
  • 3) Interaction Binding:: A high binding score indicates that the queried action and entity are jointly supported at the corresponding temporal location.

C. Interaction-guided Proposal Refinement

Interaction-guided proposal refinement uses clip-level binding evidence to adjust confidence for selected broad-context proposals while preserving the backbone’s original prediction pathways.

  • C. Interaction-guided Proposal Refinement: The refinement retains the backbone’s multi-scale proposal generation and injects FDIM binding evidence only during proposal-level refinement.
  • C. Interaction-guided Proposal Refinement: Proposal interaction support is computed by averaging binding scores across each proposal’s temporal span.
  • C. Interaction-guided Proposal Refinement: Refinement targets the last Lc temporal-pyramid levels, where proposals cover broader temporal contexts.In the experiments, interaction refinement is applied only to the coarsest level, Lc = 1.
  • C. Interaction-guided Proposal Refinement: The top 20% of proposals ranked by interaction support are selected for confidence modulation.The selection proportion is fixed across datasets.
  • C. Interaction-guided Proposal Refinement: ISA adds interaction consistency to bipartite matching so temporally plausible proposals with incomplete semantic evidence receive higher assignment costs.
  • C. Interaction-guided Proposal Refinement: The original localization, classification, and saliency losses are retained, while interaction evidence affects optimization through refinement and target assignment.Hungarian matching remains discrete and ISA adds no inference-time operation.

IV. EXPERIMENTS

Experiments evaluate IAE-VTG across three VTG benchmarks using standard moment-retrieval metrics and comparisons with recent grounding systems.

  • IV. EXPERIMENTS: Experiments cover QVHighlights, Charades-STA, and TACoS, spanning primary, daily-activity, and cooking moment-retrieval settings.
  • IV. EXPERIMENTS: Evaluation reports R1@X, mAP, mIoU, and mAP@0.5/0.75 under the FlashVTG protocol.
  • IV. EXPERIMENTS: IAE-VTG increases the parameter count from 11.81M to 13.26M and runs at 139 FPS versus 168 FPS for the baseline.ISA is training-only and introduces no additional inference-time operation.
  • IV. EXPERIMENTS: The study compares IAE-VTG with proposal-free DETR-style baselines, recent state-of-the-art systems, and classical MR baselines.

B. Quantitative Comparison

IAE-VTG achieves competitive or state-of-the-art grounding performance across QVHighlights, Charades-STA, and TACoS, while qualitative cases show improved localization under temporal ambiguity.

  • 52.91% Average mAP on QVHighlights Test exceeds FlashVTG by 0.91%, with gains on stricter metrics such as mAP@0.75.
  • 37.58% R1@0.7 (SF+C) and 50.03% R1@0.7 (IV2) on Charades-STA demonstrate generalization across visual feature settings.
  • In sequential-action and entity-persistence cases, IAE-VTG localizes the queried interaction more precisely than baselines distracted by later events or persistent entities.
  • IAE-VTG produces compact predictions for temporal over-extension and temporally separated distractor cases, aligning more closely with ground-truth intervals.
  • Additional cases show that interaction evidence suppresses appearance shortcuts and emphasizes action-consistent regions when visually plausible events compete.
  • Across six examples, interaction-aware modeling mitigates distractors from separated events, persistent evidence, and incorrect appearance-based temporal positions.

D. Ablation Study

Ablations show that FDIM and ISA contribute complementary gains, while role information and explicit interaction modeling improve precise moment selection beyond feature enrichment alone.

  • Progressively enabling FDIM and ISA yields consistent QVHighlights gains, with ISA elevating high-precision metrics through binding–saliency consistency.
  • Performance remains stable across a wide range of α values, while β sensitivity is evaluated on the QVHighlights validation set.
  • Top-1 retrieval deteriorates as noun/verb role assignments become less reliable, with the largest reduction under complete corruption.
  • IAE-VTG consistently outperforms a baseline augmented with SlowFast motion features, indicating benefits beyond richer visual evidence alone.
  • IAE-VTG improves rejection accuracy by +16.13% RA-ID and +7.22% RA-OOD while also improving R1 localization on negative queries.
  • Verb, object, and relation swaps substantially reduce overlap and confidence while increasing rejection rates, showing sensitivity to semantic inconsistencies.

E. Generalization and Query Complexity

IAE-VTG generalizes across compositional splits and query groups, but multi-action queries remain challenging; comparisons also position its interaction bias as complementary to scale and saliency-based improvements.

  • IAE-VTG improves both R1@0.7 and mIoU on all three compositional splits, with the largest gain on Novel-W.
  • The interaction branch supports compositional transfer without replacing the original full-sentence representation.
  • Comparable performance is maintained on multi-entity and mixed-complexity queries, whereas multi-action queries remain more challenging.
  • Large multimodal models achieve higher absolute recall, while IAE-VTG uses orders-of-magnitude fewer parameters and supplies an explicit lightweight action–entity consistency bias.
  • IAE-VTG improves over SG-DETR on all reported metrics, while the hybrid detector remains stronger overall, making the directions complementary.

V. CONCLUSION

IAE-VTG addresses action–entity ambiguity by modeling interaction consistency in both temporal representations and bipartite supervision. Experiments support competitive or state-of-the-art grounding performance, with gains extending to compositional and complex-query settings while leaving broader relational reasoning as future work.

  • IAE-VTG models interaction consistency at both representation and supervision levels through FDIM and ISA.FDIM grounds action- and entity-related query information in complementary motion and appearance streams, while ISA incorporates the interaction criterion into bipartite assignment.
  • IAE-VTG achieves competitive or state-of-the-art grounding performance on QVHighlights, Charades-STA, and TACoS.
  • Ablations, role-label corruption, semantic perturbations, and qualitative analyses attribute the gains to explicit interaction modeling rather than simply increased feature capacity.
  • Improvements on compositional splits and complex-query groups indicate that the interaction branch generalizes beyond the standard evaluation distribution while preserving the original full-sentence pathway.
  • Action–entity ambiguity is not universal, leaving graph-based relational reasoning, rare linguistic phenomena, and paragraph-level dense grounding as future directions.
Loading 2609.09736v1…