PaperlayerView exact revision on arXiv

Source-linked AI summary

MELON: A Large-Scale Dataset for Multi-Event Text-to-Long-Video Retrieval

Chan Hur, SeungWoo Song, Jeong-hun Hong, Won Jun Oh, Hyeyoung Park, KyungTae Lim

arXiv:2609.01654v1cs.IR

TL;DR

Existing retrieval datasets are dominated by short, single-event clips and therefore do not fully represent long-form videos with multiple events and non-contiguous matches. The paper introduces MELON with multi-event temporal annotations and proposes the MEA loss to distinguish full-event from partial-event matches. MELON provides a realistic long-form benchmark, while MEA consistently improves retrieval performance across evaluated settings.

  • Problem

    Existing text-video retrieval datasets mainly use short, single-event clips, limiting evaluation of long-form videos containing multiple semantically distinct events and non-contiguous matches.

  • Method

    The paper constructs MELON with multiple event-level descriptions and temporal intervals per long video, and introduces MEA to assign lower similarity to partial query matches than to full multi-event queries.

  • Results

    MELON demonstrates the difficulty and benchmarking value of multi-event retrieval, while MEA consistently improves performance across representative models and yields R@1 gains of up to +2.4% on ActivityNet-Cap and +3.2% on DiDeMo with TempMe.

  • Takeaways & Limitations

    MELON and MEA provide a foundation for evaluating and improving text-video retrieval in realistic long-form, multi-event scenarios.

  • Takeaways & Limitations

    MELON contains relatively few extra-long videos exceeding 30 minutes, which may cause underfitting for extreme-duration cases; very large query-event counts also require modified encoding.

Abstract

from arXiv · show

Existing text-video retrieval datasets primarily consist of short-form clips containing a single dominant event. While suitable for measuring basic vision-language alignment, they are limited in capturing real-world retrieval scenarios, where long-form videos naturally contain multiple semantically distinct events and a single text query may correspond to several non-contiguous temporal segments. To bridge this gap, we introduce MELON, the first large-scale dataset designed to extend text-video retrieval to long-form videos featuring complex, multi-event structures. MELON explicitly annotates multiple event intervals per video along with their corresponding textual descriptions, enabling both training and evaluation of multi-event understanding in long, untrimmed videos. In addition, we propose a multi-event aware loss that encourages models to differentiate between full-event and partial-event matches, yielding substantial improvements in retrieval accuracy. Together, the MELON dataset and our proposed loss establish a robust foundation for expanding text-to-video retrieval to complex long-form scenarios and provide a more realistic evaluation setting for future research in the field.

1 Introduction

Existing text-video retrieval datasets largely pair short, single-event clips with queries, limiting their ability to represent long-form videos containing multiple events and ambiguous partial matches. MELON addresses this gap with multi-event annotations for long videos and the MEA loss, which improves event-level discrimination in retrieval.

  • Motivation: Existing datasets predominantly use short videos with a single dominant event, while multi-event benchmarks remain shorter than real-world long-form content.These settings can produce annotation ambiguity and fail to capture complex retrieval scenarios.
  • Motivation: Traditional single-event retrieval creates redundant matches because queries do not capture the composite event structure of long-form videos.The paper identifies event redundancy as a central challenge requiring broader multi-event context.
  • MELON: MELON contains over 12K long-form videos with independent event descriptions, including descriptions that can correspond to multiple temporal intervals within one video.Its domains include daily life, cooking, travel, and educational content.
  • Method: The Multi-Event Aware loss supervises the embedding space so full multi-event queries receive higher similarity than query subsets.It is designed as a plug-and-play objective within the standard retrieval framework.
  • Results: The proposed MEA loss integrates smoothly with representative retrieval models and consistently improves performance in empirical evaluations.The experiments are presented as evidence of MELON’s difficulty and benchmarking value.

2 Related works

Text-video retrieval has been advanced by datasets such as MSVD, MSR-VTT, and VATEX, but most contain short clips averaging 5–15 seconds with single-event queries. ActivityNet-Cap and DiDeMo include multiple events, yet their limited temporal scale does not reflect complex long-form content.

  • Datasets for Text–Video Retrieval: Foundational datasets such as MSVD, MSR-VTT, and VATEX predominantly contain 5–15-second clips where a single-event query covers the video.These datasets established core benchmarks for text-video retrieval.
  • Datasets for Text–Video Retrieval: ActivityNet-Cap and DiDeMo incorporate multiple events but remain limited in temporal scale relative to real-world long-form videos.The passage contrasts their temporal scope with the complexity of long-form content.

3 The MELON Dataset

MELON is constructed through an automated, multi-stage pipeline that unifies annotations from long videos, filters redundant or misaligned events, and synthesizes salient multi-event summaries. The resulting dataset contains 12,000 videos with structured temporal annotations and diverse statistics for long-form retrieval evaluation.

  • Pipeline: MELON uses an automated pipeline to refine, filter, and synthesize event-level annotations from existing video sources.The pipeline is designed to scale construction of long-form multi-event retrieval data at lower annotation cost.
  • Multi-Event Summary: The multi-event summary stage generates concise summaries of major events, with each summary containing a theme, description, and timestamp.The pipeline uses an LLM for synthesis and applies additional validation and human filtering to test data.
  • Dataset Structure: The dataset targets videos at least 2 minutes long, each containing 3 to 5 non-overlapping events, with 12,000 samples split into 8,400 training and 3,600 test videos.This structure directly supports retrieval in extended temporal settings.
  • Annotation Refinement: Annotation refinement merges source datasets and converts available temporal captions or timestamped QA pairs into unified event descriptions.Each event is represented through a description associated with a time interval.
  • Quality Assurance: Quality assurance removes videos with redundant adjacent events and timestamps whose text poorly matches the visual content.T2T detects event redundancy, whereas V2T filters low-quality timestamp annotations.
  • Quality Assurance: T2T filtering removes 14.2% of videos, while V2T filtering removes approximately 16.6% of event segments.These filters address distinct video-level and annotation-level quality problems.
  • Dataset Statistics: MELON statistics cover thematic distribution across 12,000 videos, durations for 42,526 annotated events, description lengths, and source-video durations.These distributions characterize the dataset’s content and temporal scope.

4 Method

MELON formulates long-video retrieval around multi-event queries and uses a dual-encoder baseline augmented with a multi-event aware loss to distinguish complete matches from partial overlaps.

  • Problem Definition: A retrieval query is a set of K event-level descriptions, while each video contains T frames and may span multiple semantically distinct events.The model scores how well the video matches the multi-event query.
  • Problem Definition: The baseline uses video and text encoders plus a similarity score, concatenating event descriptions so existing models can process multi-event queries.Videos are uniformly sampled into frames before encoding.
  • Standard Contrastive Learning: Standard InfoNCE learning raises similarity for matched video–query pairs while lowering similarity for mismatched pairs in a batch.The standard objective combines query-to-video and video-to-query contrastive terms.
  • Multi-Event Aware Loss: Standard contrastive learning can overestimate partially overlapping videos because long-form videos often share subsets of events with a query.MEA addresses this partial-event matching problem by encouraging complete event coverage.
  • Multi-Event Aware Loss: MEA uses push and pull terms: the push term lowers partial-query similarity relative to the full query, while the pull term keeps partial matches above the hardest negative.An adaptive threshold and upper bound adjust the push constraint during training.
  • Multi-Event Aware Loss: The final objective preserves standard alignment while adding the MEA loss, with a weighting hyperparameter λ.The total loss is L_total = L_std + λ · L_mea, and MEA is described as improving discriminative capability in multi-event scenarios.

5 Experimental results

Experiments evaluate MELON’s difficulty, the effect of MEA loss, its generalization, and its ability to distinguish complete from partial multi-event matches.

  • MELON requires aligning complex multi-event queries with long, untrimmed videos; most standard one-to-one models achieve around 50% R@1.
  • Performance is strongest on videos shorter than 10 minutes and drops sharply for extra-long videos.
  • The full multi-event query reaches 57.9% R@1 for five-event queries, whereas Avg-Event R@1 falls to 7.7%.This gap indicates that partial-event matches are insufficient for retrieving the correct video.
  • The full MEA loss improves baseline R@1 by +2.4% through complementary pushing, adaptive upper-bound, and pulling components.The upper bound limits excessive penalties, while the pulling term restores overly displaced partial queries.
  • MEA improves retrieval across ActivityNet-Cap and DiDeMo, with TempMe R@1 gains of up to +2.4% and +3.2%, respectively.The results are reported for both Text-to-Video and Video-to-Text retrieval across backbones.
  • Qualitative rankings show that the top-ranked video matches all queried events, unlike a highly relevant second-ranked video matching only two.

6 Conclusion

The paper introduces MELON for realistic long-form, multi-event text-to-video retrieval and proposes MEA loss to address partial-event matching. Together, they expand evaluation and improve retrieval accuracy in complex scenarios.

  • MELON contains over 12K long-form videos with multiple event-level annotations and corresponding text queries.
  • MEA loss explicitly addresses partial-event matching and significantly improves retrieval accuracy in long-form video retrieval.
  • The dataset and loss are presented as a foundation for advancing text-video retrieval in complex multi-event scenarios.

A Limitations

The paper identifies two limitations: few extra-long videos and a query-length constraint that requires modified encoding when event counts become large.

  • MELON contains relatively few extra-long videos exceeding 30 minutes, which may cause underfitting for extreme-duration cases.
  • When the number of query events K becomes excessively large, the encoding strategy must be modified.
  • Future work could use encoder architectures capable of handling longer textual contexts.

B.2 Ablation Study

Ablations examine query encoding and MEA-loss weighting, showing that concatenating event sentences outperforms mean pooling and that MEA generally provides stable gains across λ values.

  • Concatenation-based query encoding retains comprehensive context, but is constrained by the encoder’s maximum token length.
  • Mean pooling achieves 49.7% R@1, which is 3.0% lower than concatenation-based encoding.
  • Varying λ generally produces clear and stable retrieval gains for most evaluated models, with larger improvements often appearing in V2T retrieval.
  • Fig. 8 compares T2V R@1 and V2T R@1 across λ ∈[0.1, 1.0], using dashed lines to mark λ = 0 baselines.

Impact of Individual MEA Loss Components on Training Dynamics.

The ablation shows that Lpush alone destabilizes training, while adding Lpull and all MEA components progressively improves alignment and retrieval performance. The full loss yields the most stable learning and best T2V and V2T results, while query-order changes cause only a marginal drop.

  • Individual MEA Components: Lpush alone produces erratic training loss, a persistently high θt, and the lowest T2V and V2T retrieval performance.The results indicate that pushing without regulation destabilizes the intended embedding geometry.
  • Individual MEA Components: Adding ξt to Lpush has only marginal effects on training dynamics, while V2T performance remains low.Bounding the push limits some extremes but does not adequately align partial queries.
  • Individual MEA Components: Adding Lpull to Lpush substantially improves retrieval, particularly V2T, while making training dynamics more stable though some instability persists.The pull term helps correct overly pushed embeddings and mis-evaluated events.
  • Individual MEA Components: Using all MEA components produces remarkably stable learning and the best T2V and V2T retrieval performance.The components act synergistically to align partial queries and distinguish subtle event-level differences.
  • Robustness: Reversed and shuffled event orderings cause only a marginal performance drop, indicating robustness to query-event ordering.The evaluation targets realistic multi-event queries without strict temporal constraints.
  • Partial Query Supervision: The partial-query strategy is evaluated against word replacement and event shuffling as alternative sources of partial supervision.These augmentations are considered because they retain only subsets of the original semantics.

C Dataset Filtering

MELON construction applies staged filtering to convert noisy web-sourced videos into a high-fidelity multi-event benchmark. The pipeline removes incomplete, visually misaligned, redundant, and unreliable samples through automated checks and human review.

  • Data Sources and Preprocessing: The pipeline integrates six video datasets but addresses variable annotation quality before constructing the benchmark.Web-sourced subsets may contain broken URLs, metadata misalignment, and sparse captions.
  • Data Sources and Preprocessing: Videos shorter than 2 minutes or lacking essential timestamps and valid descriptions are removed during preprocessing.These constraints ensure sufficient temporal capacity and annotation completeness for multi-event retrieval.
  • Data Sources and Preprocessing: From 2,007 youcook2 candidates, preprocessing retains 1,690 high-quality samples.The initial cleansing removes noisy or invalid examples before subsequent processing.
  • Automated Filtering: Dual filtering checks visual-text relevance and semantic distinctiveness after preprocessing and caption generation.Semantic similarity scores are used to identify descriptions that do not align with video segments.
  • Automated Filtering: Retention varies sharply by source: videochapter7m retains 14.6%, whereas curated longvale and youcook2 retain about 99%.Thresholds are determined empirically by inspecting 100 samples per dataset.
  • Automated Filtering: Adjacent captions with cosine similarity above 0.70 trigger removal to preserve distinct, non-overlapping narrative events.SBERT similarity is used to detect semantically redundant consecutive events.
  • Human Verification: Test-set verification combines an LMM judge with expert review, with 617 of 3,600 videos flagged for possible inaccuracies.Three annotators inspect flagged segments and classify issues such as hallucination or vague descriptions.
  • Human Verification: Human review reaches 86.2% inter-annotator agreement, including 532 unanimous decisions among the 617 flagged videos.Remaining disagreements are resolved through discussion, while a 100-video unflagged spot-check finds 97% error-free.

D Detailed Dataset

MELON is characterized by long event durations, balanced train-test composition, and broad thematic and lexical coverage. These properties support evaluation of long-form video understanding across varied real-world content.

  • Temporal Statistics: Gaming and Entertainment events average over 5 minutes, extending beyond the short atomic actions common in standard datasets.The longer events are intended to capture complexity in real-world videos.
  • Dataset Split: The dataset uses a balanced 70:30 split with 8,400 training and 3,600 test samples while preserving duration and theme distributions.The strategy is designed to keep the test set representative of the full dataset.
  • Semantic Coverage: Distinct vocabulary across major themes indicates broad topical coverage and helps minimize domain overfitting.Qualitative examples further demonstrate dense, contextually relevant multi-event annotations.

E.1 MELON Dataset Samples

MELON samples demonstrate lexical diversity, high-fidelity multi-event annotations, and retrieval behavior that distinguishes complete event matches from partial overlaps. The appendix also documents prompts and review procedures used to construct and verify these examples.

  • MELON Dataset Samples: Word clouds and examples from Cooking, Gaming, Travel, and Education illustrate lexical diversity and aligned multi-event descriptions.The examples present chronological events with timestamps across varied themes.
  • Qualitative Retrieval Results: The retrieval model ranks fully matched videos above visually similar partial matches by capturing fine-grained details across multiple events.In the sewing example, it recognizes that a Rank 2 video lacks measuring or hemming despite containing stitching.
  • Construction and Verification Prompts: The appendix provides prompts for QA-to-caption conversion, multi-event summarization, timestamp verification, and human review.The documented prompts convert sparse inputs into captions, retain narrative highlights in structured JSON, and identify alignment errors before annotation.
  • Construction and Verification Prompts: The multi-event summarization prompt filters noisy scenes, retains semantically meaningful highlights, and outputs Theme, Description, and Timestamp fields.Its stated objective is to focus on major narrative, emotional, or informational progression rather than filler scenes.
  • Construction and Verification Prompts: The verification prompt checks visual alignment and proposes corrections, after which annotators assess each Video, Description, Timestamp triplet.Human review covers hallucinations and vague mismatches before final quality labels are assigned.
Loading 2609.01654v1…