Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment

1 Introduction

• Existing VQA models struggle with T2V videos because their distortions differ from natural-video distortions and traditional metrics omit text-video alignment.The paper also notes that IS, FVD, and CLIPSim fail to reflect real user preferences.Source: §1 · p. 2
• T2VQA-DB contains 10,000 videos generated by nine representative T2V models, with each video's MOS collected through a subjective study involving 27 subjects.The dataset uses 1,000 text prompts.Source: §1 · p. 2
• T2VQA combines text-video alignment and video fidelity features, then uses a large language model to predict video quality.The model uses BLIP and Video Swin Transformer to extract features, which are fused through cross-attention before score regression.Source: §1 · p. 2
• T2VQA outperforms existing T2V generation metrics and state-of-the-art VQA models on T2VQA-DB and other T2V datasets.Qualitative experiments also indicate that it can benefit evaluation of T2V generation algorithms.Source: §1 · p. 2

2.1 Text-to-video Dataset

• Earlier T2V datasets vary in scale, and several have limited human annotation; the authors argue these datasets are insufficient to represent current algorithms or train deep models.Chivileva et al. has 1,005 videos and meets the cited 15-annotator standard, while ECTV and FETV use three annotators and VBench does not specify its annotator count.Source: §2.1 · p. 2

2.2 Metrics for T2V Generation

• Common video metrics have perceptual or practical limits, while alignment-only and natural-video VQA approaches do not provide a comprehensive score for generated videos.IS and FVD are criticized for poor correlation with human visual perception; CLIPSim omits temporal information and video quality, and existing VQA models do not consider text-video alignment.Source: §2.2 · p. 3

2.3 Text-to-video Generation

• Text-to-video generation conditions video synthesis on text descriptions, and prior methods extend pretrained text-to-image models with temporal modules or use decomposed diffusion.CogVideo uses hierarchical training, Make-a-video adds spatial-temporal modules, and VideoFusion presents a decomposed diffusion process.Source: §2.3 · p. 3

3 Subjective-Aligned Text-to-Video Dataset

• T2VQA-DB contains 10,000 videos from nine T2V models, generated from 1,000 prompts and scored by 27 subjects.The dataset was designed to address the relatively small video counts of prior datasets.Source: §3 · p. 4

3.1.1 Prompt Selection.

• The 1,000 prompts were selected by clustering one million WebVid-10M prompts into 100 groups and randomly sampling 10 from each group.Sentence-BERT vectors and cosine distance are used in the graph-based grouping procedure.Source: §3.1.1 · p. 4

3.1.2 Video Generation.

• The study uses nine different video-generation models, including Tune-a-video; its two pretrained-weight variants yield 10 generation models, and videos are standardized to 512 × 512, 16 frames, and 4fps.Source: §3.1.2 · p. 4

3.2 Subjective Study

• Twenty-seven subjects rate video quality on a 0–100 slider based on text-video alignment and video fidelity, and scores are normalized to reduce inter-subject differences.The reported rescaling maps Z-scores to a mean of 50 and standard deviation of 16.6.Source: §3.2 · p. 4

3.3 Dataset Analysis

• LaVie has the highest average MOS at 66.9, while the two Tune-a-Video models have the lowest scores, 39.1 and 39.9.The authors attribute Tune-a-Video's low performance mainly to low inter-frame consistency.Source: §3.3 · p. 5
• LaVie outperforms other models across all prompt categories, while human prompts have the worst performance across models.LVDM is second-worst except for “others”; the authors suggest human faces and actions may require more sophisticated modeling.Source: §3.3 · p. 5

4 Subjective-Aligned T2V Metric

• The model separates assessment into text-video alignment and video fidelity, then fuses their features before LLM-based quality regression.The architecture is presented as a transformer-based approach to subjective-aligned text-to-video quality assessment.Source: §4 · p. 5

4.1 Text-video Alignment Encoder

• BLIP encodes the prompt and individual video frames, and a cross-attention alignment encoder outputs text-frame similarity features.Features from each text-frame pair are concatenated.Source: §4.1 · p. 5

4.2 Video Fidelity Encoder

• Swin-T serves as the fidelity encoder, extracting video features for spatial and temporal distortions.The passage identifies examples including blur, noise, contrast changes, jitter, stalls, and motion blur.Source: §4.2 · pp. 5–6

4.3 Feature Fusion

• A transformer fusion module processes fidelity features with self-attention and combines them with alignment features through cross-attention.The module contains blocks with self-attention, cross-attention, and feed-forward layers, and is initialized using BERT_base.Source: §4.3 · p. 6

4.4 Quality Regression

• The quality-regression module prompts an LLM to predict one of five ITU-standard quality levels and uses their token probabilities to produce a score.The five levels are bad, poor, fair, good, and excellent, assigned weights from 1 to 5.Source: §4.4 · p. 6

5.1.1 Train-test splitting.

• Ten random 80/20 train-test splits are averaged to reduce dependence on a single split.Source: §5.1.1 · p. 6

5.1.2 Training Settings.

• Training uses frozen BLIP image and LLM components, samples eight resized frames per video, and runs for 30 epochs with Adam.The learning rate starts at 1e−5 and decays with a cosine scheduler; training uses batch size 4.Source: §5.1.2 · p. 6

5.1.3 Loss Function.

• The training loss combines differentiable PLCC with a rank loss intended to help distinguish videos’ relative quality.The balancing hyperparameter λ is set to 0.3.Source: §5.1.3 · p. 6

5.1.4 Evaluation Metrics.

• Evaluation reports SRCC, KRCC, PLCC, and RMSE, with higher correlation and lower RMSE indicating better performance.Before PLCC calculation, objective scores are mapped to subject scores using a four-parameter logistic function; Figure 6 plots predicted scores against MOSs with fitted curves and density-coded points.Source: §5.1.4 · p. 6, Figure 6 · p. 7

5.2.1 Reference Algorithms.

• The reference set includes text-video metrics, zero-shot image-quality metrics adapted to video, and general VQA models, with general VQA models fine-tuned on T2VQA-DB.Results are averaged after ten-fold splitting.Source: §5.2.1 · p. 7

5.2.2 Results Analysis.

• T2VQA performs best in SRCC on T2VQA-DB, surpassing Q-Align by 4.79% in SRCC and 3.84% in PLCC.The comparison is reported in Table 2.Source: §5.2.2 · p. 7

5.2.3 Cross-dataset Validation.

• T2VQA has the best reported generalization among the tested models, though performance drops on other datasets whose videos have higher resolution, frame rate, and length.The authors state they will include high-fidelity videos in T2VQA-DB in future work.Source: §5.2.3 · p. 7

5.2.4 Qualitative Analysis.

• In the examples, T2VQA’s predictions align with MOS across good, fair, and poor cases involving prompt match, blur, and frame consistency.The poor example has individually high-definition frames but fails to match the prompt and loses consistency between frames.Source: §5.2.4 · pp. 7–8, Table 3 · p. 8

5.3 Ablation Studies

• The ablations evaluate the alignment and fidelity encoders, fusion module, and regression module, with results averaged over 10-fold splitting.Source: §5.3 · p. 8

5.3.1 Alignment Encoder.

• Replacing BLIP with CLIP as the alignment encoder causes severe performance degradation; the authors suggest BLIP’s cross-attention interaction may explain the difference.Source: §5.3.1 · p. 8

5.3.3 Fusion and Regression Modules.

• T2VQA achieves the best performance across all evaluating metrics, while concatenation and alternative regressors are similar but inferior.The authors attribute a non-negligible improvement to cross-attention fusion and LLM regression.Source: Table 4 · p. 8, §5.3.3 · p. 8

5.3.2 Fidelity Encoder.

• Swin-T as the fidelity encoder performs better than the 3D ResNet control, supporting the transformer-based encoder choice.Source: §5.3.2 · p. 8

5.3.3 Fusion and Regression Modules.

• The fusion and regression ablations compare cross-attention with concatenation and LLM regression with linear and non-linear regression.The alternative regressors use two fully connected layers or two 1D convolution blocks, respectively.Source: §5.3.3 · p. 8

• The paper establishes T2VQA-DB, a dataset of 10,000 videos generated by 9 advanced T2V models with MOS ratings from a subjective study.Source: §6 · p. 8
• T2VQA assesses text-generated video quality using features from text-video alignment and video fidelity, fused before an LLM regresses the final prediction.Source: §6 · p. 8
• Experiments indicate that T2VQA is effective in evaluating the quality of text-generated videos.Source: §6 · p. 8