PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation

Shenghai Yuan, Xianyi He, Yufan Deng, Yang Ye, Jinfa Huang, Bin Lin, Jiebo Luo, Li Yuan

arXiv:2505.20292v4cs.CVcs.AI

TL;DR

S2V research lacks fine-grained evaluation of subject consistency, naturalness, and fidelity, as well as large-scale data designed for complex subject-focused generation. OpenS2V-Nexus introduces OpenS2V-Eval with three automatic metrics and OpenS2V-5M with enriched subject representations, while evaluations reveal model strengths and weaknesses across content.

  • Problem

    Existing S2V benchmarks are coarse-grained or limited in scope, and existing datasets rely mainly on regular triples that inadequately support complex S2V tasks.

  • Method

    The paper builds a seven-category benchmark with NexusScore, NaturalScore, and GmeScore, and constructs OpenS2V-5M using regular data, cross-video associations, and GPT-Image-1 synthesis.

  • Results

    Evaluations of representative S2V models provide insights into their strengths and weaknesses across different content.

  • Takeaways & Limitations

    OpenS2V-Nexus provides benchmark and dataset infrastructure for further S2V generation research.

  • Takeaways & Limitations

    NexusScore and NaturalScore correlate with human preferences by approximately 75%, and only about 4 million videos are publicly available from OpenS2V-5M due to licensing constraints.

Abstract

from arXiv · show

Subject-to-Video (S2V) generation aims to create videos that faithfully incorporate reference content, providing enhanced flexibility in the production of videos. To establish the infrastructure for S2V generation, we propose OpenS2V-Nexus, consisting of (i) OpenS2V-Eval, a fine-grained benchmark, and (ii) OpenS2V-5M, a million-scale dataset. In contrast to existing S2V benchmarks inherited from VBench that focus on global and coarse-grained assessment of generated videos, OpenS2V-Eval focuses on the model's ability to generate subject-consistent videos with natural subject appearance and identity fidelity. For these purposes, OpenS2V-Eval introduces 180 prompts from seven major categories of S2V, which incorporate both real and synthetic test data. Furthermore, to accurately align human preferences with S2V benchmarks, we propose three automatic metrics, NexusScore, NaturalScore and GmeScore, to separately quantify subject consistency, naturalness, and text relevance in generated videos. Building on this, we conduct a comprehensive evaluation of 18 representative S2V models, highlighting their strengths and weaknesses across different content. Moreover, we create the first open-source large-scale S2V generation dataset OpenS2V-5M, which consists of five million high-quality 720P subject-text-video triples. Specifically, we ensure subject-information diversity in our dataset by (1) segmenting subjects and building pairing information via cross-video associations and (2) prompting GPT-Image-1 on raw frames to synthesize multi-view representations. Through OpenS2V-Nexus, we deliver a robust infrastructure to accelerate future S2V generation research.

1 Introduction

OpenS2V-Nexus addresses gaps in S2V evaluation and data by introducing a comprehensive benchmark with perception-aligned metrics and a million-scale dataset enriched for subject diversity.

  • Motivation: Existing S2V benchmarks provide limited objective evidence because many focus on coarse evaluation, omit subject naturalness, or assess only facial consistency.Some inherited consistency metrics compare uncropped video frames with reference images, introducing background noise.
  • Motivation: S2V models face poor cross-category generalization, copy-paste artifacts, and weaker human identity fidelity than non-human subject fidelity.These challenges motivate benchmarks that separately examine subject consistency and natural appearance.
  • OpenS2V-Eval: OpenS2V-Eval defines seven S2V categories with 30 visually rich test samples per category to assess generalization across subjects.The categories cover single and multiple faces, bodies, and entities, plus human-entity-to-video.
  • OpenS2V-Eval: OpenS2V-Eval introduces NexusScore, NaturalScore, and GmeScore for subject consistency, subject naturalness, and text relevance, respectively.The benchmark combines qualitative and quantitative evaluation of nearly all open-source and closed-source S2V models.
  • OpenS2V-5M: OpenS2V-5M combines regular subject-text-video triples with Nexus Data built from cross-video pairing and GPT-Image-1 multi-view synthesis.The dataset contains 5.1M regular data and 0.35M Nexus Data intended to address the three core S2V challenges at the data level.

2 Related Work

Prior video benchmarks and datasets largely center on text-to-video generation, while OpenS2V-Nexus extends evaluation and data infrastructure toward complex subject-to-video tasks.

  • Automatic Metrics for Subject-to-Video Generation: Existing video-generation benchmarks evaluate multiple dimensions but predominantly target text-to-video rather than open-domain subject-to-video generation.ConsisID-Bench represents an S2V-oriented benchmark, but the supplied passage does not complete its characterization.
  • Datasets for Subject-to-Video Generation: Large-scale datasets such as Panda-70M, Koala-36M, and ChronoMagic-Pro provide millions of high-resolution video-text pairs, but downstream S2V extension remains insufficiently supported.The passage frames this as a limitation of existing video-generation datasets for complex downstream tasks.
  • OpenS2V-Nexus: OpenS2V-Eval uses real and GPT-Image-1-synthesized subject images, while its metrics assess both S2V characteristics and basic video elements.The construction pipeline is designed to broaden subject coverage and evaluation dimensions.
  • Datasets for Subject-to-Video Generation: OpenS2V-5M augments segmented-frame subject images with cross-video pairing information and GPT-Image-1 synthesis to address three core S2V issues.This construction provides the dataset's subject-information enrichment beyond regular subject-text-video triples.

3 OpenS2V-Eval

OpenS2V-Eval is designed to assess S2V models across diverse subject categories while measuring subject consistency and naturalness beyond global video quality. Its metrics use subject detection, multimodal similarity, and GPT-4o-based frame scoring to evaluate these properties.

  • Benchmark design: OpenS2V-Eval divides S2V evaluation into seven categories spanning single or multiple faces, bodies, entities, and human entities.The categories are designed to cover diverse visual concepts and testing scenarios.
  • Benchmark statistics: 180 high-quality subject-text pairs comprise 80 real and 100 synthetic samples across the seven categories.Most categories contain 30 samples, while multi-face-to-video and multi-body-to-video each contain 15.
  • Evaluation motivation: OpenS2V-Eval is motivated by the need to assess subject consistency and naturalness alongside global visual quality and motion.Existing S2V benchmarks are described as largely adapted from T2V evaluation and coarse-grained.
  • Automatic metrics: NexusScore detects target subjects, crops their regions, filters low-confidence matches, and compares detected subjects with references.The pipeline uses Mdetect for bounding boxes and Mretrieve for entity matching and image-space similarity.
  • Automatic metrics: NaturalScore evaluates whether generated subjects appear natural and conform to physical laws, addressing copy-paste outputs that can score highly on consistency.GPT-4o assigns five-point scores to uniformly sampled video frames, which are then averaged.

4 OpenS2V-5M

OpenS2V-5M is a million-scale subject-to-video dataset combining regular triples with Nexus Data designed to improve subject-information diversity and address core S2V challenges.

  • Data Construction: The dataset construction begins with raw videos, selecting human-containing content and filtering low-quality videos using aesthetic, motion, and technical criteria.
  • Nexus Data: Nexus Data addresses shortcut learning by pairing subjects across views and videos rather than relying only on reference images extracted from the ground-truth video.Regular data may share the same view as training frames or contain incomplete subjects, encouraging reference-image copying.
  • Nexus Data: GPT-Frame Pairs use GPT-Image-1 to reconstruct complete subjects and generate consistent multi-perspective representations from an initial frame and subject keywords.The resulting pair is formed from the original frame and the generated subject image.
  • Nexus Data: Cross-Frame Pairs identify similar subjects across temporally and semantically related clips using multimodal retrieval and assign aesthetic and GmeScore values to samples.Clips are aggregated from the same long video before cross-clip subject similarity is computed.
  • Dataset Overview: OpenS2V-5M contains 5.1M regular samples and 0.35M Nexus Data samples generated through GPT-Image-1 and cross-video associations.The dataset is described as the first open-source million-scale S2V dataset.

5 Experiments

The experiments evaluate 18 S2V models across open-domain, human-domain, and single-domain settings using six automated dimensions plus human preference validation. Results expose trade-offs among text relevance, subject fidelity, naturalness, and generalization.

  • Evaluation Setup: 18 S2V models are evaluated, including four closed-source and fourteen open models with either broad subject support or human-identity specialization.
  • Evaluation Setup: OpenS2V-Eval scores subject consistency, naturalness, and text relevance alongside visual quality, motion amplitude, and face similarity.Human evaluation can provide a more precise assessment.
  • Quantitative Evaluation: Pika achieves the highest GmeScore in open-domain S2V, while Kling achieves the highest NexusScore and NaturalScore.These results distinguish instruction alignment from subject fidelity and realism.
  • Quantitative Evaluation: Hailuo achieves the highest human-domain Total Score at 60.20%, while EchoVideo achieves the highest score among open-source human-domain models.
  • Qualitative Evaluation: Qualitative results identify poor generalization, copy-paste artifacts, and inadequate human fidelity as recurring weaknesses across S2V models.Models particularly struggle with profile rendering, identity preservation, and multi-subject cases.
  • Human Preference: NexusScore, NaturalScore, and GmeScore align with human perception and are comparable to existing metrics in human-preference evaluation.The validation uses 60 generated videos and votes from 173 participants.

6 Conclusion

OpenS2V-Nexus addresses limitations in S2V evaluation and data by introducing a comprehensive benchmark, human-aligned metrics, and a large-scale dataset. The paper concludes that these resources support analysis of subject consistency, naturalness, and text relevance.

  • Conclusion: OpenS2V-Eval targets subject consistency and naturalness that existing globally focused, coarse-grained benchmarks often overlook.Earlier benchmarks may also measure consistency directly on uncropped frames, introducing background noise.
  • Conclusion: OpenS2V-5M combines regular subject-text-video triples with Nexus Data generated through GPT-Image-1 and cross-video associations.The dataset is presented as the first open-source million-scale S2V dataset.
  • Conclusion: NaturalScore is intended to evaluate subject naturalness, addressing the tendency of existing anomaly-detection and multimodal models to misidentify generated content as real.
  • Conclusion: NexusScore and GmeScore assess subject consistency and text relevance more accurately than DINO-I, CLIP-I, and CLIP-T in the reported human-preference comparison.The comparison reports higher scores as better.

B.2 Comparison with Existing Metrics for Subject Naturalness

The proposed metrics are designed to evaluate video naturalness and align automated scoring with human perception. NexusScore, NaturalScore, and GmeScore are reported as especially consistent with human judgments.

  • NexusScore evaluates whether generated videos comply with physical laws and common sense, using GPT-4o-based scoring.The paper contrasts it with anomaly detectors and open-source multimodal models, which reportedly suffer from accuracy, instruction-following, or hallucination problems.
  • Existing automatic approaches are less effective because anomaly detectors lack accuracy, while multimodal models exhibit poor instruction following and hallucinations.
  • The three proposed metrics target subject consistency, subject naturalness, and text relevance, respectively.Visual examples indicate that these metrics are particularly consistent with human perception.

B.4 More Qualitative Analysis

Qualitative analyses identify recurring weaknesses in current S2V systems, including poor generalization, copy-paste artifacts, weak human identity preservation, blurry or copied initial frames, and fading subject consistency. The section also presents model-selection guidance based on OpenS2V-Eval results.

  • Poor Generalization: Poor generalization appears when models lose subject attributes or retain less reference information as the number of reference images increases.The problem is especially pronounced in open-source models, with examples including lost fur color and complete loss of reference information.
  • Copy-Paste Issue: Copy-paste behavior transfers reference lighting, pose, expression, or contours directly into videos, producing content that appears unnatural.The paper illustrates this with a face placed onto a person leaning against a pillar.
  • Inadequate Human Fidelity: Current models preserve human identity less effectively than non-human entities, motivating specialized human-domain models.
  • First Frame Blurry or Copy: Some systems either replicate the reference image or begin with blurry frames that gradually become clear.The blurry-initial-frame phenomenon is also observed in Phantom, ConsisID, and Concat-ID and is attributed as likely related to VAE control signals.
  • Consistency Fade: Subject consistency can fade over time, such as when a diamond embedded in a ring gradually disappears later in the video.
  • Guideline for Model Selection: For model selection, the paper recommends Kling for creators, Phantom or VACE for developers, Hailuo for human-centric videos, and HunyuanCustom or ConsisID for human-centric development.Pika and Vidu are presented as more cost-effective alternatives to Kling, while Hailuo is reported to preserve human identity better than open-domain models.

C.2 Additional Details of Dataset Statistics

OpenS2V-5M is characterized as a high-quality, large-scale dataset with diverse content, durations, captions, and resolutions. Additional qualitative results are presented as evidence of the dataset’s utility for S2V generation.

  • Dataset Statistics: 91% of OpenS2V-5M videos are between 0 and 10 seconds, while 65% have 720P resolution.The remaining videos exceed 10 seconds or use higher resolutions.
  • Dataset Statistics: The dataset contains diverse visual categories, caption lengths, and mostly high-quality videos after motion, technical, and aesthetic filtering.
  • Dataset Statistics: OpenS2V-5M settings target DiT-based models that favor long prompts and face input limitations such as 81 frames and 480P resolution.
  • Qualitative Results: Additional qualitative results show that Ours‡ generates high-quality videos, supporting the reported effectiveness of OpenS2V-5M.

C.4 Samples of Collected Data

OpenS2V-5M provides subject-text-video triples across diverse categories, combining segmented subjects with Nexus Data generated through cross-video association and GPT-Image-1. The section illustrates this diversity through dataset samples and documents evaluation-model settings.

  • Collected Data Samples: OpenS2V-5M samples span humans, objects, backgrounds, and other categories in subject-text-video triples.The dataset combines regular segmented data with Nexus Data produced through cross-video association and GPT-Image-1.
  • Collected Data Samples: The dataset’s Nexus Data construction is intended to provide richer subject information for S2V generation.
  • Evaluation Models: The evaluation covers closed-source and open models supporting either open-domain subjects or human identity, using official implementations or interfaces.
  • Evaluation Models: VACE unifies text, image, video, and mask inputs for multiple video-generation and editing tasks, including open-domain S2V.
  • Evaluation Models: Phantom extracts reference information with CLIP and VAE and uses windowed attention for open-domain S2V generation.
  • Evaluation Models: Human-domain models such as ConsisID, Concat-ID, FantasyID, and EchoVideo support single-face input, whereas HunyuanCustom currently has publicly released weights for single-subject support.

D.6 Additional Details of Human Evaluation

Human evaluation uses pairwise comparisons across six criteria, with questionnaire design and response-quality controls documented in the supplementary materials.

  • Questionnaire Design: Human evaluation compares video pairs across subject consistency, subject naturalness, text relevance, face similarity, visual quality, and motion amplitude.Participants choose between two video options for each criterion.
  • Response Quality Control: Questionnaire completion is controlled through one response per IP address, required login, and completion-time validity checks.The survey is expected to take 5 to 10 minutes.
  • Supplementary Visualizations: Figure 19 visualizes the questionnaire used for the user study, while Figure 20 visualizes input text prompts used in annotation-related processing.The prompt visualization covers tags obtained through Deepseek and NaturalScore annotation using GPT-4o.
  • Response Quality Control: Responses are screened for extreme-rating patterns, excluding questionnaires where over 50% of evaluations are extreme values.Extreme values are defined using the combined proportion of ratings 5 and 1.

E.1 Limitations and Future Work

The paper identifies metric alignment, licensing, environmental cost, language and cultural bias, misuse, and governance as limitations or future-work concerns surrounding OpenS2V-Nexus.

  • Evaluation Limitations: NexusScore and NaturalScore correlate with human preferences by approximately 75%, motivating future work on better automated-metric alignment.This limitation concerns the evaluation metrics’ agreement with human judgments.
  • Data Availability: Only approximately 4 million OpenS2V-5M videos can be publicly released because source videos must comply with CC BY 4.0 or be copyright-free.The dataset draws videos from multiple platforms, but licensing restricts public availability.
  • Environmental Impact: Training S2V models can consume tens of thousands of kilowatt-hours in one large-scale session, with emissions comparable to several dozen cars’ annual emissions.The paper proposes model lightweighting, more efficient distributed training, and renewable-energy data centers as responses.
  • Language and Cultural Bias: OpenS2V-Nexus currently uses English prompts, which may introduce multilingual and cultural bias for contexts such as Chinese symbols, Hanfu, and Kung fu.The paper suggests multilingual annotation as a possible solution.
  • Deepfake Misuse: Subject-consistency generation may be misused for political misinformation, forged celebrity images, or fabricated criminal evidence, prompting governance and labeling measures.Suggested responses include imperceptible watermarks, content traceability, legislation, and media-literacy campaigns.
  • Responsible Release: The released datasets and benchmarks use CC-BY-4.0 licensing, while the code uses Apache licensing and metadata supports removal of invalid videos.The project also states that the data is intended for academic research.
  • Additional Qualitative Results: Supplementary showcases cover open-domain, single-domain, and human-domain S2V generation in OpenS2V-Eval.Figures 21–24 provide additional qualitative examples across these domains.
Loading 2505.20292v4…