Source-linked AI summary
Identity-Preserving Text-to-Video Generation via Agentic Enhancement and Semantic Repair
Jiayi Gao, Changcheng Hua, Jiaqi Tang, Yuxin Peng, Yang Liu
TL;DR
Identity-preserving video generation must follow complex instructions without losing subject identity, yet closed-source models still exhibit identity drift and missing or incomplete visual content. AESR addresses this through agentic prompt enhancement, sample-level visual semantic repair, and lightweight candidate selection, and its system ranked first in Track 1 under the official challenge evaluation.
Problem
Closed-source video models can struggle with identity preservation and complex instruction following, while direct parameter optimization is often infeasible.
Method
AESR learns model-specific and human-centered prompting knowledge into a reusable playbook, repairs local errors with VLM-guided edited references, and selects among candidate outputs.
Results
AESR ranked first in the Facial Identity-Preserving Video Generation Track under the official ACM MM 2026 challenge evaluation protocol.
Takeaways & Limitations
AESR provides a practical interface-level enhancement framework for improving controllability of closed-source identity-preserving video models.
Takeaways & Limitations
Automatic face-recognition metrics can fluctuate even when human observers judge identity preservation similarly, motivating human scoring alongside automatic metrics.
Abstract
from arXiv · showhide
Identity-preserving video generation aims to synthesize videos that follow natural-language instructions while maintaining the visual identity of a given subject. Recent commercial video generation models have achieved strong visual quality and motion realism, but they still suffer from identity drift, incomplete instruction following, and missing visual details under complex prompts. Since these models are usually closed-source black boxes, directly improving them through parameter optimization is often infeasible. We therefore propose Agentic Enhancement and Semantic Repair (AESR), a lightweight enhancement framework for identity-preserving video generation. To improve prompt construction before generation and mitigate the above failures, AESR introduces a global agentic prompt enhancement module. This module learns model-specific prompting formats from official documentation, acquires human-centered video generation priors from human-interaction data, and accumulates test-domain identity-preserving generation experience into a reusable playbook through an agentic loop. To further repair errors in videos generated with enhanced prompts, AESR introduces a sample-level visual semantic repair module, which uses a VLM to locate erroneous video segments and design repair instructions, edits selected frames into explicit visual references, and guides a video editing model to fix local semantic or identity-related errors. We also adopt a lightweight Mixture-of-Experts selection strategy to choose reliable outputs from different generation and refinement paths. Under the official evaluation protocol of the ACM MM 2026 Identity-Preserving Video Generation Challenge, our system MIPL\_Video ranked first in Track 1, demonstrating the effectiveness of AESR for practical identity-preserving video generation. The code is available at https://github.com/oceanflowlab/AESR.
1 Introduction
Identity-preserving video generation must follow complex instructions while retaining a subject’s identity, but black-box models often omit elements, drift in identity, or underspecify actions. AESR addresses these interface-level failures with agentic prompt enhancement, visual semantic repair, and candidate selection.
- Motivation: Closed-source video models can produce strong visual quality and motion realism yet struggle with identity-sensitive prompts containing multiple visual elements or motion events.Reported failures include identity drift, missing objects, incomplete actions, and underspecified target states.
- Motivation: Manual rewriting and repeated sampling are costly and brittle because effective prompts depend on model-specific preferences and test-domain experience.Different models may prefer different arrangements of action, identity, and camera-motion descriptions.
- Motivation: Explicit visual references can repair local omissions more concretely than adding text when models must coordinate appearance, actions, and camera motion across frames.The paper motivates edited references for errors such as missing visual elements.
- Approach: AESR combines global agentic prompt enhancement with sample-level visual semantic repair to improve identity-preserving video generation through interface-level interventions.The framework converts prompting preferences and test-domain experience into reusable knowledge, then repairs selected video segments using edited frames and a video editing model.
- Outcome: The system uses lightweight Mixture-of-Experts selection to choose reliable videos from different generation or refinement paths and ranked first in the challenge’s Facial Identity-Preserving Video Generation Track.The result follows the official ACM MM 2026 challenge evaluation protocol.
- Contributions: The paper presents AESR as a lightweight framework with complementary global prompt enhancement and sample-level visual semantic repair modules.Its stated contributions also include first place in the Facial Identity-Preserving Video Generation Track.
2 Related Work
Identity-preserving video generation has progressed from per-identity fine-tuning toward tuning-free strategies, reflecting a shift toward more scalable deployment. The related-work discussion situates these approaches within diffusion-model advances and the paper’s 2026 publication context.
- Background: Diffusion models have propelled progress across downstream tasks, including identity-preserving generation.
- Prior methods: Early identity-preserving methods relied on per-identity fine-tuning or additional modules, requiring retraining for each new identity and limiting scalability.MotionBooth and DreamVideo are cited as examples.
- Prior methods: The paper identifies tuning-free strategies such as ACE++ and PhotoMaker as later approaches to identity-preserving generation.
3 Method
AESR enhances black-box video models through a global playbook that learns reusable prompting knowledge and a sample-level repair process that diagnoses and corrects local errors. The pipeline combines agentic feedback, edited keyframes, and video editing inputs.
- Overview: AESR uses two interface-level interventions: Global-Level Agentic Playbook Enhancement for instruction improvement and Sample-Level Visual Semantic Repair for local correction.
- Global-Level Agentic Playbook Enhancement: The playbook targets transferable failure patterns by encoding strategies, templates, and common failure modes for prompt enhancement.Examples include staged action descriptions, strengthened identity details, separated subject and camera motion, and remedies for missing objects or unclear final states.
- Global-Level Agentic Playbook Enhancement: AESR initializes the playbook from the target model’s official prompting documentation and an external HOI-Edit human-object interaction playbook.These sources provide model-specific organization preferences and reusable human-object interaction experience.
- Global-Level Agentic Playbook Enhancement: An agentic loop performs generation, failure analysis, reflection, and consolidation so test-domain feedback can update the playbook for later prompts.The updated playbook supplies relevant strategies, templates, and pitfalls when enhancing prompts for challenging samples.
- Sample-Level Visual Semantic Repair: Sample-level repair addresses errors that textual expansion cannot reliably fix, including missing elements, identity drift, and misunderstood camera motion.
- Sample-Level Visual Semantic Repair: A vision-language critic identifies mismatches, temporal segments, error types, descriptions, and repair suggestions for four classes of semantic failure.The classes are missing visual elements, incorrect motion end states, incorrect motion processes, and other semantic inconsistencies.
- Pipeline: Figure 2 presents the pipeline as Stage I global playbook enhancement followed by Stage II visual semantic repair using pencil-style reference images.
- Sample-Level Visual Semantic Repair: AESR edits representative keyframes into explicit visual targets and supplies them with structured instructions and the original identity image to a video editing model.Edited frames provide guidance for spatial relations, object appearance, human pose, and identity-related details.
4 Experiments
Experiments evaluate AESR for facial identity-preserving video generation through challenge metrics, qualitative comparisons, component analysis, and an MoE-based selection strategy. The results show complementary strengths across enhancement paths and strong performance under the official Track 1 protocol.
- Evaluation Protocol: Track 1 evaluates facial identity preservation throughout videos alongside perceptual quality, motion smoothness, and text alignment using automatic metrics and human evaluation.
- Evaluation Protocol: AESR evaluates text alignment, identity consistency, and video quality to support comprehensive assessment and candidate selection.CLIPScore is not used as the primary text-alignment metric because 89% of prompts are truncated at CLIP’s 77-token limit.
- MoE Selection Strategy: Five candidate-generation methods are scored with a weighted overall metric, and the highest-scoring video is selected for each test sample.The MoE strategy is designed to exploit complementary strengths among text alignment, identity preservation, and video quality.
- Qualitative Results: Qualitative comparisons show that playbook enhancement improves textual and motion alignment, reduces artifacts, and benefits from test-domain experience beyond official prompting formats.Examples include steadier gaze, better complex actions, more accurate scene relations, and more reliable bunk-bed generation.
- Qualitative Results: Sample-wise visual repair corrects local gaze, camera-state, and identity-detail errors that remain after generation, complementing instruction-level prompt enhancement.The repair process uses VLM localization, edited reference frames, and video editing to guide local corrections.
- Quantitative Results: Human evaluation found full AESR best among compared variants, and MIPL_Video ranked first in Track 1 with a final score of 2.5.The authors note that face-recognition scores can differ substantially even when human observers judge identities similarly preserved.
5 Conclusion
The paper presents AESR as a lightweight interface-level framework for improving identity-preserving video generation with closed-source models. It combines agentic prompt enhancement, visual semantic repair, and MoE selection, and reports first place in the official ACM MM 2026 Track 1 challenge.
- AESR improves controllability without model-parameter access by combining global agentic prompt enhancement with sample-level visual semantic repair.
- Global enhancement accumulates reusable knowledge from prompting priors and test-sample feedback, while visual repair uses diagnosis and edited reference frames for local corrections.
- A lightweight MoE strategy selects reliable outputs across different generation and refinement paths.
- AESR consistently improves identity preservation, semantic fidelity, and visual quality across diverse editing scenarios.
- MIPL_Video achieved first place in Track 1 of the official ACM MM 2026 Identity-Preserving Video Generation Challenge.