PaperlayerView exact revision on arXiv

Source-linked AI summary

EventLens: Event-Structure Reinforcement Learning for Video Understanding

Guowen Zhang, Boshen Xu, Zihao Yue, Ziheng Wang, Xiaokun Liu, Xin Tao, Wenyu Qin, Pengfei Wan, Qin Jin

arXiv:2610.16172v1cs.CVcs.AIcs.MM

TL;DR

Video MLLMs can perform well on video QA yet fail to recover event boundaries, progression, and relations from visual evidence, while common post-training objectives can permit language shortcuts. EventLens trains event-structure recovery through verifiable temporal tasks and specialized policies consolidated by distillation; it reports improved temporal understanding with competitive general video understanding and reduced training time versus direct mixed RL.

  • Problem

    Video MLLMs may recognize visible content without recovering event boundaries, single-event progression, or relations, and text-supervised objectives can allow language shortcuts instead of visual temporal grounding.

    Source: §1 · pp. 1–2

  • Method

    EventLens perturbs temporal structure and trains recovery across a three-level ontology with deterministic-reward task families, specialized GRPO policies, and MT-OPD consolidation.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    EventLens reports consistent gains in temporal grounding and event-sensitive reasoning while maintaining competitive general video understanding and reducing training time versus direct mixed RL.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The results suggest that recovering event structure from visual evidence is an effective training principle for advancing temporal understanding in video MLLMs.

    Source: §5 · p. 10

    EventLens can confuse adjacent events during gradual visual changes or high-level activities spanning fine-grained actions, and similar repeated actions remain difficult when context is subtle.

    Source: §4.4 · p. 10

Abstract

from arXiv · show

Recent multimodal large language models (MLLMs) achieve strong results on video question answering, yet often fail to recover the temporal structure that makes a video coherent. They may merge neighboring events, miss single-event progression, or infer implausible relations across events. We study structured temporal understanding as the ability to recover event structure from visual evidence, including event boundaries, single-event progression, and multi-event relations. We argue that this gap arises from a mismatch between current post-training objectives and temporal reasoning, where caption-style supervision allows models to rely on language shortcuts rather than visual temporal grounding. To address this, we propose EventLens, a vision-centric reinforcement learning framework that learns video understanding through event-structure recovery. Instead of reconstructing captions, EventLens trains models to recover temporal structure under controlled perturbations, such as altered temporal granularity, reversed progression, and shuffled event order. We instantiate this framework with a three-level temporal ontology and derive three verifiable task families: event segmentation, progression discrimination, and multi-event relation reasoning. These tasks admit deterministic rewards, enabling task-specialized GRPO training without human preference labels or learned reward models. We further introduce multi-teacher on-policy distillation (MT-OPD) to consolidate specialized policies into a unified model. Experiments on Qwen3-VL backbones show consistent improvements on temporal grounding and event-sensitive reasoning benchmarks, while preserving general video-QA performance. Learning curves further demonstrate that EventLens achieves comparable downstream transfer with significantly reduced training cost compared to direct mixed RL optimization.

1 Introduction

Video MLLMs can perform well on coarse video QA yet struggle to recover event boundaries, progression, and relations from visual evidence. EventLens trains recovery of this structure through verifiable temporal tasks and reports gains in temporal understanding with competitive general video understanding.

1 Introduction

  • Video MLLMs can recognize visible content yet fail to separate events, track progression, or reason about relations when explicit temporal structure is required.The paper frames structured temporal understanding as recovering boundaries, single-event progression, and multi-event relations from visual evidence.Source: §1 · p. 1, Figure 1 · p. 2
  • Caption- and answer-aligned post-training can permit language shortcuts, while existing learning signals do not jointly train recovery of boundaries, progression, and relations.The paper distinguishes prior work on grounding, event prediction, temporal QA, and representation-learning pretext tasks from a unified post-training objective.Source: §1 · p. 2
  • EventLens trains models to recover original temporal structure after perturbations to granularity, playback direction, and event order, using deterministic rewards without preference labels or learned reward models.The approach is designed to ground predictions in visual temporal evidence rather than match language descriptions.Source: §1 · p. 2
  • A three-level temporal ontology supports segmentation, progression discrimination, and multi-event reasoning, followed by task-specialized GRPO and MT-OPD policy consolidation.The tasks include next-event prediction, fill-in, and clip reordering; consolidation avoids instability from directly mixing heterogeneous reward signals.Source: §1 · p. 3
  • EventLens improves temporal grounding and event-sensitive reasoning across Qwen-VL generations while maintaining competitive general video understanding and reducing training time versus direct mixed RL.Direct recovery-task evaluations show that segmentation, progression, and relation skills are integrated into one model.Source: §1 · p. 3

2 Related work

Prior work covers temporal grounding, event prediction, temporal reasoning, and transformation-based proxy tasks, but typically addresses only parts of structured temporal recovery. EventLens combines these skills and uses multi-teacher on-policy distillation to consolidate specialized policies.

2 Related work

  • Temporal proxy methods for video MLLMs typically optimize one temporal skill, perturbation family, or downstream format, whereas EventLens targets boundaries, progression, and multi-event relations through a shared temporal abstraction.The cited proxy-based methods include Visual Jigsaw, VideoSSR, ViSS-R1, and VidBridge-R1.Source: §2 · p. 3
  • Prior temporal learning spans dense-label segmentation, event-centric evaluation, video-grounded entailment, event prediction, and classical pretext learning; EventLens adapts these ideas to MLLM RL with deterministic verifiers.Its tasks combine prompt-conditioned segmentation across temporal levels with progression and relation reasoning.Source: §2 · p. 3
  • On-policy distillation trains on student-generated trajectories, and EventLens uses recovery-family-matched teachers to distill specialized policies into one deployable student.The teachers handle heterogeneous tasks whose reward scales, sparsity, and response formats differ.Source: §2 · pp. 3–4, Figure 2 · p. 4

3 Method

EventLens recovers event structure from visual evidence using deterministic-reward tasks and task-specialized GRPO teachers consolidated into one student through multi-teacher on-policy distillation.

3 Method

  • EventLens trains task-specialized GRPO teachers with deterministic verifiers, then consolidates their segmentation, progression, and relation skills into one student via on-policy distillation.Rewards are computed directly from constructed target structures, without human preference labels or learned reward models.Source: §3 · p. 4

3.1 Hierarchical temporal ontology and corpus construction

  • A shared three-level ontology represents thematic phases, semantic events, and atomic actions; shot boundaries are refined into events and grouped into phases.The corpus uses official training partitions from TimeLens-100K and ET-Instruct-164K; recovery instances can target different ontology levels through prompts.Source: §3.1 · pp. 4–5, Figure 3 · p. 5

3.2 Event-structure recovery tasks

  • Three recovery families target prompted-level segmentation, single-event direction, and dependencies among neighboring events through next-event, fill-in, and clip-reordering tasks.Segmentation uses one-to-one matching and an F1-style reward; progression uses original or reversed event clips, while relation tasks use hard visual distractors and order-aware rewards.Source: §3.2 · pp. 5–6

3.3 Task-specialized GRPO teachers

  • Each deterministic reward family trains a separate GRPO teacher; mixed-GRPO baselines instead sample multiple families and normalize task-wise reward statistics with EMA-GRPO.The mixed objective addresses differing reward scales and variance profiles.Source: §3.3 · pp. 6–7

3.4 EventLens teacher consolidation

  • MT-OPD routes each student-generated trajectory to its task-specific teacher, which supplies top-k token distributions for sparse forward-KL distillation on that trajectory.Teacher and student probabilities are renormalized over the teacher's top-k set; training on student trajectories reduces the train-inference mismatch of offline teacher-output distillation.Source: §3.4 · p. 7

4 Experiments

EventLens is evaluated on temporal grounding, event-sensitive reasoning, and broader video understanding, with analyses of teacher consolidation, transfer, efficiency, and supervision robustness. The results show stronger performance on temporally targeted evaluations while retaining comparable broad-QA performance, alongside remaining limits in event disambiguation and long-range reasoning.

4.1 Experimental setup

  • Evaluation uses matched inference settings across pretrained, SFT, and EventLens variants, treating temporal grounding and event-sensitive reasoning as target evaluations and broad and long-video QA as retention checks.The Qwen3-VL variants use a shared 256-frame video budget and matched inference protocols.Source: §4.1 · p. 7

4.2 Main results

  • Across Qwen3-VL backbones, EventLens leads every temporal-grounding column and improves event-sensitive reasoning while keeping broad-QA performance comparable; Qwen2.5-VL-7B comparisons show the same temporal concentration.The comparison covers temporal grounding and complex reasoning in Table 1, general and long-video QA in Table 2, and other post-training methods in Table 3.Source: §4.2 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8

4.3 Ablation studies

  • Different task-specialized teachers lead on different metrics, while full MT-OPD recovers most of the teacher-oracle envelope without dominating every retention metric.Adding the relation teacher preserves temporal-grounding strength and slightly improves the event-centric reasoning average.Source: §4.3 · p. 9, Table 4 · p. 9
  • Adding relation recovery improves event-relation reasoning while preserving segmentation and progression, and the full student stays within 0.015 of the best score in each column.Omitting a recovery objective reduces its corresponding capability.Source: §4.3 · p. 9, Table 5 · p. 9
  • MT-OPD reaches comparable transfer after roughly 13 hours, while direct Mixed-GRPO needs roughly 37 hours.The proposed pathway trains task-specific teachers before student learning from dense top-k token distributions on student trajectories.Source: §4.3 · p. 9, Figure 4 · p. 10

4.4 Analysis: Transfer and Failure Modes

  • Event-structure proxy supervision transfers across output formats because its shared operation is boundary-sensitive temporal comparison, supporting stable gains on temporal grounding and event-sensitive reasoning.Transfer spans timestamp localization, event-level QA, and long-video QA.Source: §4.4 · p. 9
  • With 20% supervision perturbation, all six metrics remain above the instruction baseline and close to clean training, but gradual state changes and repeated actions remain difficult.EventLens also does not solve all long-range narrative reasoning.Source: §4.4 · p. 10, Table 6 · p. 10

5 Conclusion

The conclusion presents EventLens as a vision-centric reinforcement-learning framework that recovers event boundaries, progression, and inter-event relations from visual evidence. The paper also identifies its appendix as a source of implementation details, evaluation protocols, qualitative analyses, and discussion of current limitations.

5 Conclusion

  • EventLens combines verifiable event-structure recovery tasks, task-specialized reinforcement learning, and policy consolidation, with consistent gains on temporal grounding and structured reasoning while preserving broad video-understanding ability.The conclusion frames event-structure recovery from visual evidence as a training principle beyond caption-style supervision alone.Source: §5 · p. 10

Appendix

  • The appendix provides implementation details, evaluation protocols, qualitative analyses, additional empirical comparisons, and a discussion of current limitations and future work.Its analyses include related-method comparisons, case studies, and fuller efficiency curves.Source: Appendix · p. 15

A Proxy-task data details

The proxy-task pipeline converts videos into event-structure recovery examples across diverse domains, with hierarchical event annotations and deterministic verification. The appendix details reward parsing, multi-teacher distillation, and shared training and inference settings.

A Proxy-task data details

  • The proxy-task pool uses official training partitions of TimeLens-100K and ET-Instruct-164K, excludes held-out evaluation videos, and draws on diverse video domains rather than one narrow source.Examples store prompts, frame lists, structured targets, metadata, and a recovery-family label for verifier and teacher selection.Source: A Proxy-task data details · p. 15, Figure 5 · p. 15

A.1 Hierarchy construction and supervision assessment

  • The temporal hierarchy groups atomic actions into semantic events and related events into thematic phases, with annotation checks for coverage, ordering, containment, and non-overlap.A 100-chain model-assisted audit found 87% met all four criteria under majority voting; timestamp perturbation alters selected chains' coordinates while preserving duration, event text, and identity.Source: §A.1 · pp. 16–17, Table 9 · p. 17

B Reward and parsing details

  • The reward parser maps segmentation intervals, progression choices, and relation responses to task-specific verifier inputs, assigning segmentation and progression rewards from valid matches and relation-order scores from temporal displacement.For segmentation, valid predicted intervals are de-duplicated and matched to ground truth; relation reordering receives full credit for exact order and partial credit for near-correct order.Source: B Reward and parsing details · p. 17

C Multi-teacher OPD implementation

  • In multi-teacher OPD, the student generates an on-policy response, routes it to the matching recovery-family teacher, and learns from that teacher's top-k token distributions using a sparse forward KL loss.Teachers are routed by segmentation, progression, or relation task; the implementation avoids reward groups and advantages used in GRPO teacher training.Source: C Multi-teacher OPD implementation · pp. 17–18

D Training hyperparameters

  • The shared OPD setting uses one student rollout because teachers score student trajectories at the token level, while top-k distributions keep distillation sparse and memory-efficient.The appendix also specifies fixed video-budget and decoding settings across pretrained and post-trained variants.Source: D Training hyperparameters · p. 18, Table 10 · p. 18

E Individual-teacher recovery capabilities

Individual teachers specialize in their corresponding recovery tasks, and full MT-OPD consolidates those capabilities into one student.

E Individual-teacher recovery capabilities

  • R1 improves segmentation by 0.188, R2 improves progression by 0.135, and R3 improves relation reasoning by 0.150 relative to Base; full MT-OPD matches or exceeds the best individual teacher in every column.The recovery scores are reported for Qwen3-VL-4B.Source: E Individual-teacher recovery… · p. 18, Table 11 · p. 18

F Temporal grounding evaluation details

Temporal grounding is evaluated across three TimeLens subsets with standardized video inputs, timestamp parsing, and IoU-based scoring.

F Temporal grounding evaluation details

  • QVHighlights, ActivityNet, and Charades-STA are loaded through TimeLensBench as TSV rows containing video ID, query, ground-truth span, and video duration.Videos are supplied as structured inputs when supported; otherwise, frames are sampled at 2 FPS up to 60 seconds, 1 FPS from 60 to 256 seconds, and capped at 256 frames for longer videos.Source: F Temporal grounding evaluation details · pp. 18–19
  • All models are greedily decoded without benchmark-specific demonstrations or external judges, using the same evaluation implementation across pretrained, SFT, and EventLens variants.The main temporal-grounding prompt asks for the event’s start and end times in seconds.Source: F Temporal grounding evaluation details · pp. 18–19
  • The parser extracts the first valid timestamp pair, assigns zero IoU when none is found, excludes malformed references, and reports mIoU plus recall at IoU thresholds 0.3, 0.5, and 0.7.Pairs with end times earlier than start times are discarded.Source: F Temporal grounding evaluation details · p. 19

G Comparison with related video post-training work

EventLens-8B reports the highest scores on three temporal-grounding benchmarks and improves the mean TimeLens score over VideoSSR-8B. Table 12 compares related video post-training methods across several benchmarks.

G Comparison with related video post-training work

H Qualitative case study

The qualitative cases contrast coarse event recognition with finer temporal alignment. EventLens with MT-OPD separates dense events and grounds answers in the relevant temporal evidence.

H Qualitative case study

  • In the chips example, the base model merges salt and pepper actions and shifts the final transfer step, while EventLens separates event boundaries more faithfully and achieves perfect F1@0.3.EventLens keeps continuous oiling and leaf placement together, then separates spreading, salting, peppering, oven placement, and transfer.Source: Figure 6 · p. 20, H Qualitative case study · p. 20
  • The chips case shows that plausible recipe semantics can still miss dense event boundaries needed for precise temporal evaluation.The base model combines salt and pepper into one interval, whereas EventLens separates them.Source: H Qualitative case study · p. 20
  • Across repeated action localization, event matching, and grounded QA, the base model misaligns plausible answers with temporal evidence, while MT-OPD recovers finer boundaries and support intervals.Examples include recovering repeated folding events, selecting the later matched segment, and grounding the small-knife answer in supporting frames.Source: H Qualitative case study · p. 20, Figure 7 · p. 21

I Capability-specific transfer and trade-offs

EventLens improves several event-sensitive capabilities, while capability changes vary across benchmark subsets. Across the eight-metric suite, it reaches comparable average transfer to direct Mixed-GRPO in substantially less wall-clock time.

I Capability-specific transfer and trade-offs

  • Event localization improves by 7.61/7.87 points at 4B/8B, event grounding by 5.14/8.33, and text-event matching by 8.00/7.00.Temporal ordering, motion direction, and group-level matching also improve at both scales relative to the corresponding Qwen3-VL instruction base.Source: Table 13 · p. 21, I Capability-specific transfer and… · p. 21
  • Montage plot attributes, motion-speed estimation, and anomaly recognition decline at both scales; narrative-scale ordering also needs context beyond the finite event windows.The authors motivate complementing event-structure recovery with narrative-scale and fine-grained perceptual objectives.Source: I Capability-specific transfer and… · p. 21

J Additional EventLens efficiency curves

  • Across eight metrics, EventLens gains +3.39 after 12.7 hours, versus Mixed-GRPO’s +2.87 at 13.0 hours and +3.38 after 36.7 hours.Figure 8 supplies the four additional benchmark curves that complement Figure 4.Source: Figure 8 · p. 22, J Additional EventLens efficiency curves · p. 22
  • Video-MME and Video-Holmes show small, non-monotonic changes, unlike the larger gains on ETBench, TimeLens, and Vinoground.The full-suite curves are presented as a transparency check, not evidence that every benchmark improves monotonically.Source: J Additional EventLens efficiency curves · p. 22

K Limitations

The limitations concern the quality assessment of automatically constructed event chains, the scope of the evaluation, and the accounting of training compute.

K Limitations

  • Automatically constructed event-chain coverage, granularity, and semantic accuracy determine recovery-supervision quality; its audit uses model-assisted judgments, and timestamp perturbation is a 100-step pilot.Human validation and tests varying event semantics or hierarchy would extend the assessment.Source: K Limitations · p. 22
  • Evaluation covers Qwen3-VL and Qwen2.5-VL under reported video-input settings, with recovery tasks limited to event clips and local multi-event windows.The authors identify longer narratives, denser temporal sampling, and higher-resolution motion as next steps.Source: K Limitations · p. 22
  • Efficiency curves cover only the MT-OPD student stage, while a complete compute account also includes specialized teacher training and teacher-distribution queries.The consolidated student runs independently of its teachers at inference.Source: K Limitations · p. 22
Loading 2610.16172v1…