PaperlayerView exact revision on arXiv

Source-linked AI summary

PISCES: Annotation-free Text-to-Video Post-Training via Optimal Transport-Aligned Rewards

Minh-Quan Le, Gaurav Mittal, Cheng Zhao, David Gu, Dimitris Samaras, Mei Chen

arXiv:2602.01624v2cs.CV

TL;DR

T2V reward post-training is constrained by costly human preference annotations and misaligned VLM embeddings, motivating scalable, better-aligned supervision. PISCES uses Dual OT-aligned Quality and Semantic Rewards at distributional and token levels, outperforming existing methods across short- and long-video VBench evaluations and human studies.

  • Problem

    Existing T2V reward methods either require large-scale human preference annotations or derive supervision from pre-trained VLM embeddings that may be misaligned with real-video distributions.

  • Method

    PISCES uses a Dual OT-aligned Rewards module with distributional quality alignment and discrete token-level semantic alignment between text and video.

  • Results

    PISCES outperforms annotation-based and annotation-free methods on VBench Quality and Semantic scores for short- and long-video generation, with human evaluations validating the result.

  • Takeaways & Limitations

    The Dual OT-aligned Rewards module supports scalable annotation-free T2V post-training across direct backpropagation and reinforcement learning fine-tuning.

  • Takeaways & Limitations

    Responsible deployment still requires established safeguards such as content moderation, provenance, disclosure mechanisms, and careful dataset and evaluation practices.

Abstract

from arXiv · show

Text-to-video (T2V) generation aims to synthesize videos with high visual quality and temporal consistency that are semantically aligned with input text. Reward-based post-training has emerged as a promising direction to improve the quality and semantic alignment of generated videos. However, recent methods either rely on large-scale human preference annotations or operate on misaligned embeddings from pre-trained vision-language models, leading to limited scalability or suboptimal supervision. We present $\texttt{PISCES}$, an annotation-free post-training algorithm that addresses these limitations via a novel Dual Optimal Transport (OT)-aligned Rewards module. To align reward signals with human judgment, $\texttt{PISCES}$ uses OT to bridge text and video embeddings at both distributional and discrete token levels, enabling reward supervision to fulfill two objectives: (i) a Distributional OT-aligned Quality Reward that captures overall visual quality and temporal coherence; and (ii) a Discrete Token-level OT-aligned Semantic Reward that enforces semantic, spatio-temporal correspondence between text and video tokens. To our knowledge, $\texttt{PISCES}$ is the first to improve annotation-free reward supervision in generative post-training through the lens of OT. Experiments on both short- and long-video generation show that $\texttt{PISCES}$ outperforms both annotation-based and annotation-free methods on VBench across Quality and Semantic scores, with human preference studies further validating its effectiveness. We show that the Dual OT-aligned Rewards module is compatible with multiple optimization paradigms, including direct backpropagation and reinforcement learning fine-tuning. Project page: https://roar-ai.github.io/pisces

1. Introduction

PISCES addresses scalability and supervision problems in T2V reward-based post-training by aligning text and video embeddings with Optimal Transport. Its Dual OT-aligned Rewards improve quality and semantic alignment across short- and long-video generation.

  • Motivation: T2V generation targets realistic, temporally consistent videos that accurately reflect textual prompts, with evaluation organized around Quality and Semantic scores.Quality measures visual quality and temporal consistency, while Semantic measures correspondence between generated videos and text.
  • Motivation: Annotation-based rewards require large-scale human preference annotations, limiting their scalability, while annotation-free methods rely on potentially misaligned pre-trained VLM embeddings.These limitations motivate reward supervision that is both scalable and better aligned with human judgments.
  • Approach: PISCES uses Optimal Transport to align text and video embeddings at distributional and token levels without annotations.The approach is designed to preserve video semantic structure while improving alignment between text instructions and real-video representations.
  • Approach: Its Dual OT-aligned Rewards module combines a Distributional OT-aligned Quality Reward with a Discrete Token-level OT-aligned Semantic Reward.The quality reward targets overall visual quality and temporal consistency; the semantic reward targets localized text-video alignment.
  • Results: PISCES outperforms annotation-based and annotation-free methods on Quality and Semantic scores for both short- and long-video generation.Automatic metrics and human evaluations validate the method, and the rewards work with direct backpropagation and reinforcement learning fine-tuning.

2. Related Work

Prior T2V reward methods either use costly human preference annotations or annotation-free signals from pre-trained VLMs. PISCES applies Optimal Transport to reward modeling for generative T2V post-training, aligning distributional and token-level text-video representations.

  • Reward-based Post-Training for T2V: Annotation-based T2V methods train rewards or apply preference optimization using large-scale human preference datasets, but their annotation costs limit scalability.These methods collect judgments of generated video pairs on quality and semantics.
  • Optimal Transport: Optimal Transport provides a framework for aligning probability distributions and has been applied to domain adaptation, generative modeling, and cross-modal retrieval.Neural Optimal Transport offers a scalable alternative based on learned transport maps.
  • Optimal Transport: Before PISCES, prior work had not explored Optimal Transport for reward modeling in generative T2V post-training.PISCES addresses misaligned pre-trained VLM text-video embeddings through distributional and discrete token-level alignment.

3. Method

PISCES post-trains text-to-video models with Dual OT-aligned rewards that separately assess global quality and fine-grained semantic correspondence. The method integrates distributional and token-level OT with differentiable reward optimization and supports both direct backpropagation and GRPO.

  • 3.1. Distributional OT-aligned Quality Reward: PISCES uses a distributional OT map to align text embeddings with real-video embeddings before computing a global [CLS]-based Quality Reward.The aligned text embedding serves as a proxy for a real-video embedding, enabling cosine similarity to measure visual quality and temporal coherence.
  • 3.3. Post-Training: The Dual OT-aligned Rewards module supervises denoiser refinement through direct backpropagation or GRPO reinforcement learning.The procedure uses LoRA while freezing parameters other than the denoiser, and computes rewards before optimization.
  • 3.1. Distributional OT-aligned Quality Reward: The Quality Reward compares OT-aligned text and generated-video [CLS] representations through cosine similarity.This geometry-respecting comparison is intended to capture whether generated videos share the quality direction of real videos.
  • 3.2. Discrete Token-level OT-aligned Semantic Reward: PISCES augments VLM cross-attention with a partial OT plan that selectively matches important text tokens to relevant video patches.Unlike diffuse vanilla attention, the transport plan provides structured grounding while allowing unmatched token mass.
  • 3.2. Discrete Token-level OT-aligned Semantic Reward: The semantic transport cost combines token-patch meaning, temporal alignment, and spatial alignment, with γ and η balancing temporal and spatial penalties.An entropic Sinkhorn solver produces a partial transport plan with fraction-of-mass m = 0.9.
  • 3.2. Discrete Token-level OT-aligned Semantic Reward: POT-refined features are passed to InternVideo2’s VTM classifier, whose positive softmax logit supplies the Semantic Reward.The transport plan is treated as a structural prior while preserving differentiability through cross-attention.

4. Experiments

Experiments evaluate PISCES on short- and long-video generators using VBench, human preferences, qualitative comparisons, and OT-focused analyses. PISCES achieves strong automatic and human-evaluated quality and semantic alignment, while ablations and controlled studies support complementary rewards and stable OT-based supervision.

  • Evaluation Setup: PISCES is evaluated on VideoCrafter2 and HunyuanVideo using VBench, which summarizes 16 dimensions into Quality, Semantic, and Total Scores.VideoCrafter2 represents short-video generation, while HunyuanVideo represents long-video generation.
  • Automatic Evaluation: PISCES achieves the highest Total, Quality, and Semantic scores among annotation-based and annotation-free reward-based post-training methods.The comparison covers both short-video and long-video generation settings.
  • Human Evaluation: PISCES is consistently preferred over HunyuanVideo, T2V-Turbo-v2, and VideoReward-DPO for visual quality, motion quality, and semantic alignment.The human study uses 400 prompts and 85 participants.
  • Ablation Study: Using only the Quality Reward raises VBench Quality Score from 82.20 to 83.77, while only the Semantic Reward raises Semantic Score from 73.42 to 76.99.Combining both rewards gives the best overall performance, supporting their complementary roles.
  • OT Alignment Analysis: Spatio-temporal constraints in the discrete OT cost matrix produce accurate text-token and video-region correspondences instead of diffuse or misaligned attention.The designed discrete POT also improves InternVideo2 video-text matching performance by 8.11%.
  • OT Alignment Analysis: OT achieves the strongest Mutual KNN and Spearman correlation while preserving structure, and OT-aligned rewards produce the best Quality and Semantic Scores after HunyuanVideo post-training.OT-transformed text embeddings also move closer to video embeddings in t-SNE and pairwise-distance analyses.
  • Sampling Stability: With the same random seed, OT-aligned rewards preserve the robot, background, and motion while changing only the apple color, whereas L2 mapping causes unstable outputs and artifacts.The L2 comparison includes varying object placement, changing robot appearance, and disappearing objects.

5. Conclusion

PISCES is presented as an annotation-free T2V post-training algorithm using Dual OT-aligned Rewards to address distributional and token-level embedding misalignment. It outperforms existing annotation-based and annotation-free methods on VBench and human evaluation across short- and long-video generation.

  • Conclusion: PISCES combines Distributional OT-aligned Quality Reward with Discrete token-level Semantic Reward to improve visual quality and semantic consistency.The method targets both distributional and token-level misalignment in VLM embeddings.
  • Conclusion: PISCES outperforms existing annotation-based and annotation-free methods on VBench and human evaluation across short- and long-video generation.The Dual OT-aligned Rewards module is compatible with direct backpropagation and reinforcement learning fine-tuning.

Impact Statement

PISCES combines annotation-free post-training with OT-aligned rewards and supports multiple optimization routes. Its design targets scalable alignment while preserving semantic structure and improving quality and text-video correspondence.

  • Impact Statement: PISCES reduces reliance on costly human-labeling pipelines, supporting more scalable reward-based post-training experimentation.The paper connects annotation-free training with reduced dependence on extensive preference datasets and associated practical challenges.
  • Impact Statement: PISCES relies on existing model signals and does not introduce risk categories beyond those already associated with modern T2V systems.The paper still recommends established safeguards for responsible deployment.
  • Impact Statement: PISCES is an annotation-free T2V post-training framework that uses OT to align text and video embeddings.The alignment addresses misaligned VLM embeddings while preserving embedding structure.
  • Impact Statement: The framework is compatible with direct backpropagation and reinforcement-learning fine-tuning, including GRPO.Both optimization routes produce consistent gains over vanilla baselines on VBench across VideoCrafter2 and HunyuanVideo.

M 1M Map partial mass to unbalanced strength:

The reported ablations show that OT alignment and the combined reward design improve both global quality and fine-grained semantic correspondence. Full PISCES achieves the strongest aggregate outcome among the compared reward configurations.

  • M 1M Map partial mass to unbalanced strength:: 83.44 → 83.73 Quality Score and 75.82 → 77.63 Semantic Score result when comparing PISCES without OT against full PISCES.The improvement supports aligning text-video distributions before post-training.
  • M 1M Map partial mass to unbalanced strength:: 83.77 Quality Score, 66.92 Aesthetic Quality, and 97.07 Subject Consistency are reported for the OT-aligned Quality Reward alone.The quality reward primarily improves global coherence.
  • M 1M Map partial mass to unbalanced strength:: 76.99 Semantic Score, 96.60 Human Action, and improved Spatial Relation are reported for the OT-aligned Semantic Reward alone.The semantic reward enhances fine-grained text-video alignment.
  • M 1M Map partial mass to unbalanced strength:: 82.51 Total Score is achieved by full PISCES, the highest result among the compared reward configurations.Overall Consistency reaches 29.10 and Temporal Style reaches 26.97 with both rewards integrated.

F. Optimal Transport Plan Analysis

Partial OT and spatio-temporal penalties improve token-to-region matching by selectively transporting informative tokens and encoding spatial and temporal structure.

  • F. Optimal Transport Plan Analysis: 89.36% matching accuracy is achieved with Partial OT at m = 0.9, improving 8.11% over vanilla cross-attention.The evaluation uses 10,000 WebVid10M video-text pairs.
  • F. Optimal Transport Plan Analysis: Partial OT filters uninformative words while preserving key semantic correspondences between text tokens and visual regions.The authors note that articles and stopwords need not be explicitly grounded in the visual domain.
  • F. Optimal Transport Plan Analysis: m = 0.5 removes too many tokens, causing essential words to be ignored and degrading alignment.The result shows that the transported fraction must retain informative semantic content.
  • F. Optimal Transport Plan Analysis: Spatial and temporal penalties with η = 0.2 and γ = 0.2 raise matching from 81.25% to 89.36%.The structured cost produces sharper token-level correspondences.

G. Motion Guidance in T2V Post-Training

PISCES remains effective without motion guidance and across alternative encoders and out-of-distribution prompts. These evaluations support robustness beyond the primary training and evaluation setting.

  • G. Motion Guidance in T2V Post-Training: Without motion guidance, PISCES-direct reaches 83.73 Quality and 77.63 Semantic on VideoCrafter2, exceeding T2V-Turbo-v2’s 83.26 and 76.30.Adding motion guidance makes PISCES-direct outperform T2V-Turbo-v2 across all metrics on VideoCrafter2.
  • G. Motion Guidance in T2V Post-Training: 85.05 versus 84.50 Total Score favors PISCES-direct without motion over T2V-Turbo-v2 with motion on HunyuanVideo.The advantage is also reflected in the human preference study.
  • G. Motion Guidance in T2V Post-Training: ViCLIP-based rewards and evaluation confirm comparable PISCES performance beyond the InternVideo2 representation space.The controlled experiment changes the video-text encoder and training corpus representation space.
  • G. Motion Guidance in T2V Post-Training: PISCES shows stronger alignment on 100 diverse out-of-distribution prompts evaluated with ViCLIP cosine similarity.The prompts include robotics, embodied tasks, procedural instructions, abstract concepts, and rare object-event compositions.

J. Hyperparameters Sensitivity Analysis

The sensitivity analysis finds stable VTM performance across structured-penalty weights, while ablations identify Partial OT, spatio-temporal constraints, and both OT-aligned rewards as important for alignment and post-training performance.

  • Robustness to penalty weights: VTM accuracy varies smoothly across γ and η, with only 1.48% standard deviation over the tested range.The sweep covers γ, η ∈ [0.0, 0.5] in 0.1 increments on 10,000 WebVid10M video-text pairs.
  • Reward ablations: Full PISCES, combining OT alignment with both OT-aligned rewards, achieves the best Total, Quality, and Semantic Scores in the ablation.The comparison includes variants without OT and variants using individual OT-aligned rewards.
  • Partial OT and constraints: Partial OT with m = 0.9 achieves the best reported alignment, while spatio-temporal constraints further improve VTM accuracy.The ablation uses γ = 0.2 and η = 0.2 for the spatio-temporal constraints.
  • Motion guidance: PISCES significantly outperforms T2V-Turbo-v2 on VideoCrafter2 and HunyuanVideo with and without motion guidance.The comparison covers both short- and long-video generators under both motion-guidance settings.
  • Robustness to penalty weights: The minimal variation across penalty weights indicates that Partial OT remains stable with respect to its structured penalty terms.The reported variation has negligible impact on performance.

K. Failure Example of Discrete Partial OT

The failure analysis shows that Partial OT suppresses irrelevant matches but can also discard valid fine-grained tokens, while the underlying video-text representation limits precise localization.

  • Sensitivity analysis: The VTM accuracy remains stable across the tested γ, η configurations, with ±1.48% standard deviation.This indicates limited sensitivity to the structured penalty weights in the reported sweep.
  • Partial OT failure: With m = 0.5, Partial OT removes the valid “glasses” token, reproducing semantic misalignment.At m = 0.9, the token is retained and aligned better, but only the right lens is activated.
  • Representation limitation: The OT module’s performance depends on InternVideo2’s representational precision, which was not optimized for fine-grained localization such as segmentation.Insufficient spatial detail can leave detailed grounding errors unresolved even with a structured transport plan.
  • Dual-reward interaction: The dual rewards improve steadily while their final gradient cosine similarity is 0.0074, indicating largely orthogonal supervision signals.The final semantic and quality rewards are 0.9268 and 0.7814, respectively, with no reported reward conflict or mode collapse.

P. Training Cost and Efficiency Analysis

PISCES adds a modest one-time post-training cost, but avoids reward-model inference overhead and can substantially reduce denoising steps for GRPO-tuned models.

  • OT-map training: Training the OT map uses 24 A100 GPU-hours, compared with 72 A800 GPU-hours for VideoReward-DPO reward-model training.The OT map is trained on 8-frame video clips using frozen InternVideo2.
  • Post-training cost: PISCES post-training costs 29.78 GPU-hours on 8×A100s versus 26.52 GPU-hours for T2V-Turbo-v2 training.The reported comparison excludes evaluation time.
  • Relative cost: PISCES incurs only a marginal increase over T2V-Turbo-v2 training and remains small relative to T2V pre-training costs.The paper compares this one-time cost with 665,000 GPU-hours for Seaweed-7B pre-training and higher costs for HunyuanVideo-13B.
  • Inference efficiency: Reward models add no inference cost, while consistency distillation reduces GRPO-tuned denoising steps from 50 to 16.This yields approximately 3× faster inference.
Loading 2602.01624v2…