PaperlayerView exact revision on arXiv

Source-linked AI summary

TREND-10K: A Comprehensive Dataset for Next-Generation Video Quality Assessment Based on Preference-Driven Media

Ziheng Jia, Zicheng Zhang, Junqi Zhang, Jiaying Qian, Jiarui Wang, Yushuo Zheng, Xiongkuo Min

arXiv:2609.26187v1cs.AI

TL;DR

Existing VQA datasets often predate short dramas and AIGC videos, and do not provide unified assessment across diverse content and changing trends. TREND-10K combines trend-driven sampling and a static subset with three evaluation dimensions; the authors report it as an effective resource for VQA pretraining and finetuning.

  • Problem

    Existing VQA datasets often predate short dramas and AIGC videos, cover limited content types, or use different protocols that impede unified assessment.

    Source: Introduction · p. 1, T2V AIGC-VQA · p. 3

  • Method

    TREND-10K combines TREND-Search preference-based sampling with a static subset and evaluates technical quality, aesthetic quality, and perceptual AIGC traces.

    Source: Introduction · p. 2, Subjective Experiment · p. 4

  • Results

    The authors report that TREND-10K demonstrates intra-dataset consistency and generalization ability and serves as an effective resource for VQA pretraining and finetuning.

    Source: Introduction · p. 2, Conclusion · p. 8

  • Takeaways & Limitations

    The dataset provides a resource for VQA pretraining and finetuning, with planned updates through TREND-Search as media trends evolve.

    Source: Conclusion · p. 8

    The dataset is distributed only as offline downloadable files, without a dedicated online platform or integration into existing access systems.

    Source: §A.1 · p. 10

Abstract

from arXiv · show

The increasing prominence of short-video platforms, coupled with the advanced commercialization of AI-generated content (AIGC) videos, has led to a shift in the types of video media trend consumed by users in their daily lives. Traditional user-generated content (UGC) is gradually being replaced by professional short dramas and AIGC entertainment. Consequently, VQA for contemporary media content has become increasingly important. This requires a unified evaluation framework that can handle diverse video content and evolving media trends. In this context, we introduce TREND-10K, a next-generation comprehensive VQA dataset consisting of the trend-driven part and the static part, containing $10,000$ videos across a wide spectrum of content types. The trend-driven part is based on the TREND-Search framework, which captures user preference profiles from trending lists on online platforms and formulates sampling strategies based on these profiles. The static part, on the other hand, is composed of supplementary samples selected from publicly available datasets. To support unified evaluation for various video types, we incorporate three evaluation dimensions: technical, aesthetic, and AIGC-trace. Experiments show that our dataset ensures high annotation quality and exhibits remarkable generalization across multiple content categories. In conclusion, our work presents a robust framework for advancing VQA, addressing challenges caused by the temporal evolution of user perceptual habits and preferences.

Introduction

Media trends are changing, while existing VQA datasets often predate short dramas and AIGC videos, focus on one content type, and lack robust update mechanisms. TREND-10K combines trend-responsive sampling with a static component to support evaluation across diverse content.

Introduction

  • The paper asks whether VQA datasets should be continuously revised to align with evolving media-consumption trends.It argues that such changes reshape everyday users’ visual experiences and offer insights for VQA.Source: Introduction · p. 1
  • Existing VQA datasets often predate short dramas and AIGC videos, separate content types, and lack robust mechanisms for adapting to changing media trends.These limitations motivate a comprehensive dataset for contemporary media.Source: Introduction · p. 1
  • TREND-Search captures short-term preferences from YouTube trending lists, aggregates them into long-term profiles, and translates them into sampling plans.The dataset also includes a static part to balance its diverse content.Source: Introduction · pp. 1–2, Figure 1 · p. 2

Related Works

Prior work develops VQA methods for user-generated and text-to-video content, but the existing datasets and protocols leave gaps for evaluating contemporary media consistently. TREND-10K is motivated by the need to assess diverse emerging content under a unified approach.

UGC-VQA

  • UGC-VQA spans diverse scenarios and methods, from natural-scene-statistics features to deep learning and multimodal alignment.The cited work includes MOS-based datasets and methods using quality-aware multimodal representations.Source: UGC-VQA · p. 2

T2V AIGC-VQA

  • T2V AIGC-VQA covers preference comparisons, multidimensional MOS, attribute assessment, fine-grained supervision, and task-specific evaluation methods.Existing AIGC datasets often feature very short videos from early models with evident artifacts, while UGC and AIGC datasets use different protocols.Source: T2V AIGC-VQA · pp. 2–3

The TREND-10K

TREND-10K combines a trend-driven component with a fixed static subset and uses TREND-Search to build trend-aligned samples. Its subjective study rates technical quality, aesthetic quality, and perceived AIGC traces using a defined annotation and scoring procedure.

The TREND-10K

  • TREND-10K addresses evolving preferences, copyright and safety constraints, and the need for richer AIGC content with 7,000 trend-driven videos and 3,000 static videos.The static portion re-annotates open-source samples to improve quality, diversity, and annotation balance.Source: The TREND-10K · p. 3

Trend-driven Part Construction

  • TREND-Search builds a taxonomy from monthly trending-video metadata, aligns categories across periods, and exponentially smooths their contributions into a sampling profile.The system uses titles, descriptions, and tags to classify videos and their content origins.Source: Trend-driven Part Construction · p. 3
  • Online crawling executes sampling plans with fallback queries, screening, metadata checks, and engagement-based selection before downloading and clipping videos.The workflow also rechecks downloaded material for suitability.Source: Trend-driven Part Construction · pp. 3–4
  • TREND-Search supplements limited copyright-compliant crawled AIGC with T2V videos generated from trend-derived prompts and expert-designed challenging scenarios.Prompts are expanded with an LLM and assigned across multiple T2V model providers.Source: The TREND-10K · p. 3, Trend-driven Part Construction · p. 4

Static Part Construction

  • The static part samples open-source datasets across five quality levels using existing scores, then re-annotates selected videos in the subjective study.The samples come from VITAL-Pretrain and AIGVE-60K.Source: Static Part Construction · p. 4

Subjective Experiment

  • The study rates technical fidelity and temporal consistency, aesthetic expressiveness, and how strongly AIGC traits are perceptible in the video.Aesthetic quality is assessed beyond basic technical fidelity; perceived AIGC traces are judged from content rather than explicit cues such as watermarks or titles.Source: Subjective Experiment · p. 4
  • More than 180 college students aged 20–25 rated videos on 1–5 scales, with training, warmups, quality checks, and at least 15 annotators per video.Scores were averaged after excluding outliers more than three standard deviations from the mean, then scaled to 0–5 and grouped into five levels.Source: Subjective Experiment · pp. 4–5

Data Analysis and Experiments

The analysis finds reliable annotations with relationships among technical, aesthetic, and AIGC-trace scores, while experiments report broad cross-dataset generalization and examine joint supervision and trace-level sampling.

Data Analysis

  • 0.621, 0.586, and 0.677 are the Krippendorff’s Alpha values for technical, aesthetic, and AIGC-trace annotations, respectively; technical and aesthetic scores correlate at SRCC 0.815.The authors characterize agreement as reliable while retaining individual variability, and aesthetic judgments as more subjective.Source: Data Analysis · p. 6
  • Most videos have low AIGC-trace scores (0–1), while technical and aesthetic scores remain positively correlated across trace levels; the trend-driven component has higher quality distributions than the static component.At the highest AIGC-trace level (4–5), the technical-aesthetic SRCC is 0.907; the static component helps balance the dataset across quality levels.Source: Data Analysis · p. 6

Experiments

  • TREND-10K-trained models perform comparably to or better than LSVQ-trained models across test datasets, with the advantage particularly evident on short-video and AIGC benchmarks.The comparison covers cross-dataset performance of models pretrained from scratch on different sources.Source: Table 2 · p. 6, Experiments · p. 7
  • TREND-10K fine-tuning achieves the best generalization under LSVQ pretraining, with VITAL surpassing all compared methods.The transfer evaluation spans conventional UGC and content-specific out-of-distribution test sets.Source: Experiments · p. 7, Table 3 · p. 7
  • Joint optimization using technical and aesthetic labels slightly outperforms single-dimension supervision in the DOVER comparison.The joint setup uses DOVER++’s default coefficients: 0.6104 for technical and 0.3896 for aesthetic.Source: Experiments · p. 7, Table 4 · p. 8
  • Training subsets combine videos from different AIGC-trace levels, and higher trace levels make trained models increasingly biased toward AIGC-oriented datasets.The design examines whether perceptual-authenticity-aware subsets improve generalization across content-specific test sets.Source: Experiments · p. 8, Table 5 · p. 8

Conclusion

TREND-10K combines preference-driven trend sampling with a static component and evaluates technical, aesthetic, and perceptual AIGC-trace quality. The authors report it is effective for VQA pretraining and finetuning and plan to update it as media trends evolve.

Conclusion

  • TREND-10K combines trend-aligned sampling and generated AIGC videos with a static component, and evaluates technical, aesthetic, and perceptual AIGC-trace quality.The authors report that the dataset is an effective resource for VQA pretraining and finetuning and plan to keep updating it as trends evolve.Source: Conclusion · p. 8

A.1 Limitations and Social Impacts

The authors plan periodic dataset updates to track evolving media trends, with each release receiving complete subjective annotation. They note offline-only distribution and a young, culturally homogeneous annotator pool as limitations, while anticipating benefits from accumulated releases.

A.1 Limitations and Social Impacts

  • The dataset is currently distributed only as offline files, with no dedicated online platform or integration into existing online-access systems.The authors say this may make distribution and use more difficult.Source: §A.1 · p. 10
  • TREND-10K annotators are college students aged 20–25, so labels may not fully reflect older users’ perceptual preferences and may introduce demographic bias.The authors note that this design improves annotation consistency and reduces inter-group variability.Source: §A.1 · p. 10
  • The authors plan to update TREND-10K every three to six months, with complete subjective annotation in each release cycle.Each version is intended to serve as a benchmark during its release period.Source: §A.1 · p. 10
  • Accumulating releases are expected to expand TREND-10K into a large-scale benchmark and improve training effectiveness and cross-domain generalization.The authors also say TREND-Search’s automation makes scheduled large-scale acquisition practical with minimal human intervention.Source: §A.1 · p. 10
  • The authors plan to open-source TREND-Search, the annotation platform, and each dataset release.Source: §A.1 · p. 10

A.2 Summary of Existing Datasets

The paper points readers to tables summarizing representative datasets for UGC/PGC-VQA and AIGC-VQA.

A.2 Summary of Existing Datasets

  • Tables 6 and 7 summarize representative datasets for UGC/PGC-VQA and AIGC-VQA, respectively.Source: §A.2 · p. 10

A.3 Composition statistics of the TREND-10K dataset

Table 8 summarizes TREND-10K’s composition, including video sources and sample counts for each dataset component.

A.3 Composition statistics of the TREND-10K dataset

A.5 Details for the Subjective Experiment Online Platform

The platform gathers subjective judgments of technical quality, aesthetic quality, and perceived AI-trace intensity. It combines structured annotator training and synchronized rating sessions with timestamped records and authenticated video access.

A.5 Details for the Subjective Experiment Online Platform

  • The platform collects human perceptual judgments across technical quality, aesthetic quality, and AI-trace intensity.AI-trace intensity captures the degree to which a video is perceived as synthetically generated.Source: §A.5 · p. 10
  • Annotators complete training on 32 representative videos, then rate all three dimensions on a 1–5 Likert scale during synchronized playback.Each submission records the three scores together with a UTC timestamp; free-text remarks are optional.Source: §A.5 · p. 10
  • The platform uses cookie-based sessions and mediates video access through internal Nginx locations to prevent direct URL-based enumeration.Its backend uses FastAPI and PostgreSQL, with Nginx and X-Accel-Redirect for authenticated video streaming.Source: §A.5 · p. 10

A.6 Subjective Experiment Guideline

The appendix documents the annotation guidance and TREND-Search sampling and generation workflows, alongside supplementary dataset visualizations and annotation statistics. It also visualizes relationships between AIGC-trace levels and technical and aesthetic scores.

A.6 Subjective Experiment Guideline

  • The appendix presents the subjective-experiment guidelines and summarizes existing UGC/PGC and AIGC VQA datasets.The guidelines are shown in Figures 5 and 6; Tables 6 and 7 summarize the respective dataset groups.Source: §A.6 · p. 10, Table 6 · p. 11, Table 7 · p. 11

A.7 Details for Trending Profile and Sampling Plan Construction Process

  • Figures 7 and 8 illustrate construction of TREND-Search’s trending profile and corresponding sampling plan.Source: §A.7 · p. 11

A.8 Details for Supplementary T2V Generation Models and Prompts

  • Supplementary T2V data are characterized by generation models, prompt word patterns and lengths, and generated-video durations.The appendix also shows word-cloud and histogram visualizations for these materials.Source: §A.8 · p. 17, Figure 9 · p. 18, Figure 10 · p. 19

A.9 Details for Online Search and Video Crawling Process

  • The online-search and crawling workflow includes query generation, candidate retrieval, metadata verification, and video collection.Source: §A.9 · p. 17

A.10 Details for T2V Generation Process

  • The T2V-generation workflow comprises prompt construction and refinement, video generation, and quality verification.Source: §A.10 · p. 17

A.11 Visualization and More Statistical Analysis for TREND-10K Dataset

  • The appendix visualizes relationships between AIGC-trace levels and technical and aesthetic quality scores across the annotated videos.The supplied text describes these relationships but does not state a particular trend or outcome.Source: §A.11 · p. 17

A.12 More Details for the Annotated Labels of TREND-10K

Loading 2609.26187v1…