PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenSubject: Leveraging Video-Derived Identity and Diversity Priors for Subject-driven Image Generation and Manipulation

Yexin Liu, Manyuan Zhang, Yueze Wang, Hongyu Li, Dian Zheng, Weiming Zhang, Changsheng Lu, Xunliang Cai, Yan Feng, Peng Pei, Harry Yang

arXiv:2512.08294v2cs.CV

TL;DR

Subject-driven models struggle to preserve identities in complex multi-subject scenes and manipulations. OpenSubject addresses this with a 2.5M-sample, 4.35M-image video-derived corpus, a four-stage construction pipeline, and a benchmark spanning generation and manipulation. Training with OpenSubject improves identity fidelity and manipulation robustness, particularly in complex scenes.

  • Problem

    Subject-driven models often deviate from reference identities and struggle to preserve multiple identities and non-target content in complex scenes.

  • Method

    OpenSubject uses video-derived cross-frame supervision, VLM-guided mining and pairing, task-specific outpainting or inpainting, and verification with captioning.

  • Results

    OpenSubject training improves identity fidelity and manipulation robustness, especially for multi-subject settings and complex scenes.

  • Takeaways & Limitations

    The corpus and benchmark provide broad supervision and evaluation for subject-driven generation and manipulation across single- and multi-subject tasks.

  • Takeaways & Limitations

    The dataset is released for research-only use under an acceptable-use policy prohibiting biometric identification, re-identification, surveillance, and non-consensual impersonation.

Abstract

from arXiv · show

Despite the promising progress in subject-driven image generation, current models often deviate from the reference identities and struggle in complex scenes with multiple subjects. To address this challenge, we introduce OpenSubject, a video-derived large-scale corpus with 2.5M samples and 4.35M images for subject-driven generation and manipulation. The dataset is built with a four-stage pipeline that exploits cross-frame identity priors. (i) Video Curation. We apply resolution and aesthetic filtering to obtain high-quality clips. (ii) Cross-Frame Subject Mining and Pairing. We utilize vision-language model (VLM)-based category consensus, local grounding, and diversity-aware pairing to select image pairs. (iii) Identity-Preserving Reference Image Synthesis. We introduce segmentation map-guided outpainting to synthesize the input images for subject-driven generation and box-guided inpainting to generate input images for subject-driven manipulation, together with geometry-aware augmentations and irregular boundary erosion. (iv) Verification and Captioning. We utilize a VLM to validate synthesized samples, re-synthesize failed samples based on stage (iii), and then construct short and long captions. In addition, we introduce a benchmark covering subject-driven generation and manipulation, and then evaluate identity fidelity, prompt adherence, manipulation consistency, and background consistency with a VLM judge. Extensive experiments show that training with OpenSubject improves generation and manipulation performance, particularly in complex scenes.

1. Introduction

Multi-subject personalization must preserve each identity across novel poses and contexts while following textual specifications. OpenSubject addresses this gap with a video-derived corpus, construction pipeline, and benchmark targeting identity and manipulation quality.

  • Multi-subject personalization is harder than single-subject generation because models must preserve each identity across novel poses and contexts while following textual specifications.
  • Existing synthesis-based datasets offer controlled variation but inherit model biases and identity inconsistency, while retrieval-based datasets scale poorly to multiple subjects and favor public figures.
  • Video provides cross-frame identity-consistent supervision with variation in viewpoint, illumination, and environment, but pairing must balance diversity, identity consistency, and context variation.
  • OpenSubject constructs a 2.5M-sample, 4.35M-image corpus through video curation, VLM-guided subject mining and pairing, identity-preserving input synthesis, and verification with captioning.
  • Its benchmark covers generation and manipulation across four sub-tasks, and OpenSubject training improves identity preservation and manipulation fidelity, especially in multi-subject settings.

2. Related Works

Related work spans architectural approaches and dataset-construction strategies for subject-driven generation. Existing datasets differ in how they obtain paired examples and in their coverage of real-world subjects and contexts.

  • Subject-driven generation methods include U-Net-based and DiT-based architectures, with DiT-based approaches offering stronger representational capacity and more flexible conditioning.
  • Table 1 compares subject-driven generation and manipulation datasets by paired-input sample counts.
  • Dataset construction follows synthesis-based and retrieval-based paradigms for preserving identity across diverse poses, illumination, and scenes.
  • Synthesis-based corpora create varied identity-consistent pairs with strong base models, while retrieval-based datasets curate real-world images but provide different coverage trade-offs.

3. OpenSubject

OpenSubject builds a large video-derived corpus through quality filtering, cross-frame identity mining, task-specific input synthesis, and automated verification. The resulting dataset supports both single- and multi-reference generation and manipulation across broad subjects and scenes.

  • 3.1. Video Curation: Video curation uses publicly available OpenVid, OpenHumanVid, and OpenS2V clips filtered by resolution and aesthetic quality.Videos below 720p or with aesthetic scores below 5.8 are discarded.
  • 3.2. Cross-Frame Subject Mining and Pairing: Cross-frame pairing combines clip-level subject consensus, instance-level grounding and VLM verification, and maximal DINOv2 embedding distance to increase contextual diversity.The procedure samples four mid-range frames and discards clips with fewer than two valid frames.
  • 3.3. Identity-Preserving Reference Image Synthesis: Generation inputs use mask-guided outpainting across frames, whereas manipulation inputs use box-guided inpainting within a frame; geometry-aware augmentation and boundary erosion reduce synthesis artifacts.The two tasks use different input-target relationships while preserving the referenced subject identity or non-target content.
  • 3.4. Verification and Captioning: A VLM verifies artifacts and physical plausibility, resynthesizes failed samples, and produces short and long captions for accepted samples.
  • 3.5. Statistical Analysis: OpenSubject contains 2.5M samples and 4.35M images spanning single-subject generation, single-subject manipulation, multi-subject generation, and multi-subject manipulation.Generation accounts for roughly two-thirds of the data, and the corpus includes people, objects, and diverse environments.

4. Benchmark

OSBench evaluates subject-driven generation and manipulation in both single- and multi-subject settings, addressing the limited complex-scene coverage of prior benchmarks. Its rubricized VLM evaluation measures prompt, identity, manipulation, and background criteria.

  • Tasks: OSBench contains four sub-tasks: single-subject generation, multi-subject generation, single-subject manipulation, and multi-subject manipulation.Each sub-task contains 60 samples, including complex-scene replacement while preserving non-target content.
  • Motivation: Prior benchmarks focus mainly on clean single-subject portraits and commonly omit subject-driven manipulation, motivating OSBench’s broader task coverage.
  • Quantitative evaluation: Table 2 reports quantitative results for single-subject and multi-subject generation and manipulation on OSBench.
  • Ablation study: Table 3 isolates the effects of synthetic T2I data and additional sampled OpenSubject data during fine-tuning.
  • Evaluation dimensions: The benchmark uses GPT-4.1 with rubricized prompts and independent 0–10 criteria for prompt adherence, identity fidelity, manipulation fidelity, and background consistency.

5. Experiment

Experiments evaluate closed- and open-source models across generation and editing benchmarks, showing that OpenSubject fine-tuning improves identity preservation, manipulation fidelity, and multi-subject compositional control.

  • Quantitative evaluation: Existing methods perform poorly on multi-subject generation and manipulation, with open-source baselines reaching at most 0.78 single-subject and 0.73 multi-subject manipulation Overall.Gemini 2.5 Flash Image Preview also falls to 5.12 on multi-subject manipulation, while Qwen-Image-Edit-2509 reaches 5.10.
  • Quantitative evaluation: Fine-tuning OmniGen2 with OpenSubject and T2I data raises the overall score from 6.43 to 7.22, with the largest gains in manipulation.Manipulation improves by +0.81 for single-subject and +1.91 for multi-subject cases, while generation identity fidelity improves without sacrificing overall performance.
  • Other benchmarks: OpenSubject fine-tuning improves OmniContext performance from 7.18 to 7.34, especially for multiple-reference and character–object composition settings.MULTIPLE/Char.+Obj. increases from 7.45 to 7.87, while SINGLE/Object remains essentially unchanged.
  • Other benchmarks: On ImgEdit, OpenSubject fine-tuning raises the overall score from 3.44 to 3.72, with strongest gains on Extract, Hybrid, Add, and Background subsets.Style and Action show minor reductions despite improvements in Replace, Remove, and Adjust.
  • Qualitative results: Qualitative comparisons show that OpenSubject fine-tuning better preserves identity, follows attributes, composes multiple subjects, and confines edits to intended regions.Other methods more often exhibit identity drift or unintended changes to non-target content.

6. Conclusion

OpenSubject is a large-scale video-derived resource and benchmark for subject-driven generation and manipulation. Experiments show improved identity fidelity and manipulation robustness, while release is restricted by an acceptable-use policy.

  • Conclusion: Fine-tuning on OpenSubject improves identity fidelity and manipulation robustness across the authors’ benchmark and external evaluation suites.The paper reports that prevailing methods degrade on multi-subject generation and subject-conditioned editing in complex scenes.
  • Ethical considerations: The dataset uses publicly available open-licensed data and is released for research-only use under restrictions against biometric identification, re-identification, surveillance, and non-consensual impersonation.These restrictions define the permitted-use boundary for the released resource.

7. Implementation Details about OpenSubject Data Construction Pipeline

OpenSubject constructs reliable training pairs from curated videos through cross-frame subject verification, local quality filtering, and subsequent VLM-based validation and captioning.

  • Video curation and subject verification: Four-stage filtering retains high-quality, identity-consistent clips and subject pairs through video screening, cross-frame verification, local checks, and VLM validation.The pipeline combines role-based rules with semantic quality assessment to remove unreliable detections and preserve usable training targets.
  • Cross-frame subject mining: Four uniformly sampled middle frames provide efficient candidates while avoiding transitional or unstable video segments.The procedure retains objects appearing consistently across at least two sampled frames.
  • Local verification: Local verification removes irrelevant semantic roles and rejects subjects affected by occlusion, truncation, blur, missing parts, or insufficient visibility.Qwen2.5-VL-7B assesses whether cropped regions depict complete, clearly visible subjects after role-based filtering.
  • Rule-based cleaning: Human-only and mixed-object clips use separate cleaning procedures with resolution, area, confidence, object-count, overlap, border, and category constraints.The mixed-object procedure additionally removes semantically unreliable detections using VLM checks, while the human procedure applies dedicated single- and multi-person rules.
  • Filtering implementation: Algorithmic filtering rewrites retained records into the original JSONL files and permanently removes discarded samples.The implementation includes dedicated procedures for human clips and mixed-object clips, with blacklisted-category filtering for the latter.

8. Implementation Details about OpenSubject Benchmark

The benchmark evaluates subject-driven generation and manipulation with independent rubricized VLM scores, while the construction pipeline synthesizes and assesses identity-preserving image inputs.

  • Evaluation protocol: GPT-4.1 assigns independent 0–10 scores using rubricized prompts for controllable, fine-grained benchmark assessment.Generation and manipulation use separate dimensions covering prompt adherence, identity fidelity, manipulation fidelity, and background consistency.
  • Generation evaluation: Generation Overall is the geometric mean of Prompt Adherence and Identity Fidelity, balancing textual compliance with identity preservation.Identity Fidelity considers facial features, hairstyle, clothing cues, and global appearance.
  • Manipulation evaluation: Manipulation Overall is the geometric mean of Manipulation Fidelity and Background Consistency, balancing edit accuracy with scene stability.Background Consistency evaluates whether non-edited regions retain their structure, lighting, layout, and scene components.
  • Artifact assessment: Artifact assessment prompts inspect geometry errors, texture artifacts, lighting violations, and conflicts between subjects and backgrounds.These criteria target visual defects and physical implausibility in synthesized samples.
  • Reference image synthesis: The synthesis implementation cleans overlapping masks, rescales and places foreground crops, creates hole masks, and optionally applies tear-border erosion.Placement uses a Gaussian center prior, while erosion is intended to mimic rough hand-drawn edges before synthesis.

9. Visualization

The visualization section presents additional qualitative examples and the prompt templates used for captions, artifact assessment, and benchmark scoring.

  • Additional visual results: Figures 21–28 provide additional visual results spanning a wide range of scenarios.The passages identify these as supplementary qualitative examples rather than quantitative comparisons.
  • Caption prompt templates: Caption prompts cover subject replacement and multi-subject combination, each with short and long generation-style variants.The manipulation prompts describe replacing a subject while preserving the surrounding scene, whereas combination prompts transplant target objects into a new scene.
  • Visual comparison setup: Reference-versus-output examples frame the first image as ground truth and the second as the model output for scene-similarity assessment.This comparison format is repeated across the visualization examples.
  • Scoring prompts: The visualization materials include prompts for Prompt Adherence, Identity Fidelity, Manipulation Fidelity, and Background Consistency scoring.These prompts correspond to the benchmark’s independent evaluation dimensions.
Loading 2512.08294v2…