Token-Level Video Reinforcement Learning

1 INTRODUCTION

• Video GRPO applies one group-relative advantage uniformly, while TVRL uses score-gradient magnitudes to route updates toward video tokens relevant to each check.The gradient magnitudes are detached and weight denoising log-probabilities, preserving the clipped GRPO structure without backpropagating reward loss through the generator.Source: §1 · p. 2, Figure 2 · p. 2
• A frozen VLM’s teacher-forced answer likelihood both contributes to the video-level reward and provides a question-conditioned sensitivity map through its video-input gradient.Video-grounded questions can check prompt following, physical plausibility, object interaction, and temporal consistency.Source: §1 · p. 2
• TVRL averages question rewards before group-relative normalization, then uses each question’s detached credit map to reweight dense denoising-transition log-probabilities in the clipped surrogate.The shared advantage determines update direction and strength; credit maps determine its spatio-temporal allocation.Source: §1 · pp. 2–3
• With SAGE, TVRL improves VBench-2.0 Overall over matched baselines by +1.33 to +3.15 points across four reward models.With Qwen3.5-9B fixed, it improves Overall by +2.68 to +3.15 points across SAGE, Flow, and Dance samplers.Source: §1 · p. 3

2 RELATED WORK

• Direct differentiable reward optimization backpropagates through the decoder and denoising chain, while dense-feedback alternatives require added annotation or separately trained localized reward models.TVRL instead stops the VLM gradient at a detached credit map and needs neither dense labels nor a localized reward model.Source: §2 · p. 3
• Video GRPO methods use scalar terminal rewards and apply each group-relative advantage uniformly across denoising steps and latent video tokens, indicating rollout rank but not where updates should concentrate.TVRL retains the clipped group-relative update and replaces uniform assignment with VLM-gradient token credit.Source: §2 · pp. 3–4
• TVRL averages teacher-forced likelihoods for prompt-derived checks into one rollout advantage, then uses detached gradients of those likelihoods to route dense denoising log-probabilities.The overview depicts shared rollout-level scoring alongside question-conditioned routing weights.Source: Figure 3 · p. 4, §2 · p. 4

3 METHOD

• TVRL uses a frozen VLM to score sampled videos with teacher-forced question answering, sharing one group-relative advantage across checks while their detached gradients route it over latent units.The VLM gradient does not reach the generator.Source: §3 · p. 4

3.1 POLICY AND VIDEO-GROUNDED CHECKS

• The diffusion policy samples stochastic denoising transitions, and prompt-only decomposition produces atomic yes/no checks for requirements verifiable in the video.Checks may target subjects, attributes, actions, relations, scene elements, or camera cues; a structured schema scores them in one teacher-forced VLM pass.Source: §3.1 · p. 4

3.2 TEACHER-FORCED REWARD AND TOKEN CREDIT

• Each check’s answer likelihood is differentiated with respect to video-frame input, and its gradient magnitude is aggregated into normalized, question-conditioned token-credit weights.The reward-sensitivity map measures local, model-dependent response to small video-token perturbations, not causal attribution; granularity trades localization against noise.Source: §3.2 · p. 5

3.3 SHARED ROLLOUT ADVANTAGE

• Question rewards are averaged within each rollout before normalization across the prompt group, preserving video-level ranking and giving every check the same advantage.The maps allocate that video-level reward rather than supervise each question independently.Source: §3.3 · p. 5

3.4 TOKEN-LEVEL POLICY UPDATE

• The policy update reduces dense transition log-probabilities by credit unit, applies question-specific weights, and forms one weighted ratio before exponentiation and clipping.Uniform final latent-grid weights with identical check maps recover the scalar baseline reduction; the weighted ratio is a local credit-assignment surrogate, not the exact full-trajectory likelihood ratio.Source: §3.4 · pp. 5–6
• In matched validation examples, only TVRL satisfies all three specified prompt requirements; baselines miss details including the instrument, raised hand, or framing and dress.The figure compares prompt- and seed-matched outputs.Source: Figure 4 · p. 6

3.5 KL REGULARIZATION

• Reference and stepwise old-policy KL regularizers remain global over transition states; token credit changes only the clipped policy-ratio term.An adaptive controller updates the KL coefficient.Source: §3.5 · p. 6

4.1 EXPERIMENTAL SETUP

• The setup fixes SAGE for reward-model comparisons, varies samplers, evaluates VBench-2.0 dimensions, and uses a blind human study with 600 videos per model.Evaluated checkpoints use 100 optimizer steps; table entries and optimization curves average three random seeds unless stated otherwise.Source: §4.1 · p. 7

4.2 MAIN RESULTS

• With Qwen3.5-9B, TVRL raises Overall by +3.15 with SAGE, +2.92 with Flow, and +2.68 with Dance.With SAGE, it also improves Overall for all four tested reward models: VideoAlign 54.18 to 55.54 (+1.36), VideoScore2 54.66 to 55.99 (+1.33), UnifiedReward2 54.82 to 56.67 (+1.85), and Qwen3.5-9B 54.54 to 57.69 (+3.15).Source: Figure 5 · p. 7, §4.2 · p. 7, Table 1 · p. 7
• Prompt- and seed-matched examples show TVRL meeting specified visual requirements that baselines miss, including an upright erhu, finger-counting gesture, and coherent garden scene.The examples illustrate, but do not substitute for, the quantitative and human studies.Source: §4.2 · p. 8
• In blind comparisons, TVRL is preferred over matched GRPO 36.3% versus 23.3% and over the base model 31.3% versus 19.2%.Ties account for 40.4% and 49.5%, respectively; excluding ties, TVRL wins 60.9% and 62.0% of decisive comparisons.Source: Table 2 · p. 8, §4.2 · p. 8

4.3 ABLATION STUDY

• On the tested range, later training reward improves with progressively finer credit through 3×3, but unsmoothed 1×1 credit falls below frame-level credit.Held-out Overall rises from 54.54 with uniform routing to 56.47 with frame credit, 57.08 with 7×7, and 57.69 with 3×3; shuffled 3×3 weights score 53.70, though shuffling does not isolate semantic alignment.Source: Table 3 · p. 9, §4.3 · p. 9, Figure 7 · p. 9
• With SAGE, the tested TVRL critics rank from Qwen3.5-0.8B at 51.85 Overall to Qwen3.5-9B at 57.69, with similar ordering across other families.Changing the critic changes both its scalar reward and gradient map, so this comparison cannot attribute gains to either component alone or establish a general scaling law.Source: Figure 6 · p. 8, Table 4 · p. 9, §4.3 · p. 9

5 CONCLUSION

• TVRL improves over matched GRPO across all four SAGE reward models and all three Qwen3.5-9B sampler settings, while 3×3 credit outperforms coarser and unsmoothed 1×1 credit.The framework uses averaged teacher-forced question rewards for a shared group-relative advantage and detached VLM gradients to allocate updates within the clipped policy ratio.Source: §5 · p. 9

A.1 HUMAN EVALUATION PROTOCOL

• The blind study compares TVRL with matched SAGE-GRPO and the base model on the first 200 VideoGen-Eval prompts, using three seeds and eight annotators.Annotators judge text–video alignment as a win, loss, or tie; decisive preference excludes ties.Source: §A.1 · p. 13

A.2 PROMPTS FOR THE QUALITATIVE COMPARISON

• Figure 4 examples use validation prompts 005, 015, and 016; Figure 6 examples use VideoGen-Eval prompts 002, 003, and 007, with methods matched by prompt and seed.The full prompt text follows in the appendix.Source: §A.2 · p. 13

A.3 SEED STATISTICS

• The reported seed statistics average independent training and generation seeds, evaluated at step 100 on the same prompt set.Tables A1 and A2 provide means and standard deviations behind the main-text numbers.Source: §A.3 · p. 15, Table A1 · p. 15, Table A2 · p. 15

A.4 ADDITIONAL ABLATIONS

• Free-generation scoring, target log-probability scoring, and token credit yield similar Overall scores, ranging from 54.28 to 54.54; teacher-forced reward alone does not explain the main-result gain.The stronger result combines VLM answer likelihoods with shared rollout-level advantage and weighted-log-probability token credit.Source: §A.4 · p. 15, Table A3 · p. 16
• Increasing sampled reward frames from 5 to 30 raises Overall from 53.49 to 55.71, while increasing decomposed questions from 1 to 3 raises it from 51.34 to 55.69.The rating-threshold-5 variant reaches 55.41 Overall, but only one threshold is reported, so it does not establish an optimum.Source: §A.4 · pp. 15–16, Table A4 · p. 16
• Per-question routing reaches 56.47 Overall, while averaged-map single-ratio and generic-check variants each reach 54.96 and per-frame ratios reach 52.95.The comparisons support a single weighted ratio per check; at the old policy all ratios equal one, so first-order updates coincide before exponentiation and clipping distinguish the ratio forms.Source: §A.4 · p. 16

A.5 TRAINING OVERHEAD

• Mean observed time rises from 666.05 to 786.04 seconds per update, a difference of 119.99 seconds (18.02%) in the compared historical runs.The comparison uses updates 11–100 and includes work outside rollout and optimizer-loop timers, as well as logging or checkpoint activity within those intervals.Source: §A.5 · p. 16, Table A6 · p. 17
• The 18.02% is not a universal overhead estimate or isolated VLM-backward cost; the runs also differ in reward weights, prompt files, and frame-count handling.The credit implementation includes recomputing old-policy log-probabilities for weighted routing.Source: §A.5 · pp. 16–17

A.6 QUESTION-CONDITIONED SPATIAL SENSITIVITY

• The shuffled 3×3 control scored 53.70 Overall, below unshuffled routing at 57.69 and above uniform routing at 54.54.Shuffling preserves the routing-grid weight distribution before smoothing, but does not isolate semantic correspondence from spatial and temporal coherence or match final-map entropy.Source: §A.6 · p. 17
• The diagnostic used 12 prompt-pair groups and two seeds per pair, with 20 sampled frames per source and panel layouts that place paired videos in AB and BA order.The critic’s 20×32×64 input yields a 10×2×4 gradient grid, reconstructed spatially and smoothed to 20×16×16; smoothing leaks credit across panel boundaries.Source: §A.6 · p. 17, Table A7 · p. 18
• +5.39 pp was the qualified-cohort question–position interaction (95% interval [2.83, 7.77], sign-flip p = 0.0034), supporting question-conditioned regional sensitivity in this constructed setting.Mean queried-panel mass was 52.70%; the all-pairs interaction was +6.41 pp and the letterbox result was +3.79 pp.Source: §A.6 · p. 18, Table A7 · p. 18
• Question-conditioned maps differed by 30.5 pp total variation, mainly across time (28.8 pp) rather than space (8.3 pp); frame credit accounts for most of the training gain.In Table 3, frame credit raises Overall from 54.54 to 56.47, while 3×3 spatial credit adds a smaller increment to 57.69.Source: §A.6 · p. 18
• The split-screen diagnostic does not establish precise object segmentation or causal credit for generator updates, and its selected examples and low critic resolution limit generalization.Its bootstrap assumes exchangeable groups and its sign-flip test assumes sign symmetry under the null; it also does not compare gradients with attention or establish 3×3 superiority over 7×7.Source: §A.6 · p. 18

B.1 DIFFUSION POLICY DETAILS

• Different SDE samplers change the transition mean and noise schedule, while TVRL retains token-credit construction and aggregates unreduced elementwise log-probabilities over credit units.The training objective is KL-regularized.Source: §B.1 · pp. 18–19

• The main training and reward setting is the SAGE/Qwen3.5-9B token-credit video-adv run; Flow and Dance variants retain its token-credit reward and GRPO configuration while changing the stochastic denoising sampler.The settings are summarized in Table B1.Source: §B.2 · p. 19, Table B1 · p. 19
• The training prompt file contains 10,000 prompts, each decomposed into an overall question, five subquestions, a structured VLM question prompt, and a structured reference answer.No supervised target videos are used for post-training; the frozen VLM scores generated videos by teacher-forced answer likelihood, whose gradients provide token-level credit.Source: §B.2 · p. 19

• All runs use VBench-2.0 automatic metrics on the same held-out prompt split for matched baselines and TVRL variants.Tables list Overall first, followed by creativity, common sense, controllability, human action, and physics.Source: §B.3 · p. 20

B.4 SUPPLEMENTARY VIDEOALIGN SAMPLER RESULTS

• TVRL improves VBench-2.0 Overall over matched GRPO with VideoAlign under all three samplers: SAGE 54.18 to 55.54 (+1.36), Flow 55.07 to 55.82 (+0.75), and Dance 48.02 to 50.30 (+2.28).The table reports VBench-2.0 Overall with VideoAlign; the prose gives the matched-run values and gains.Source: Table B2 · p. 20, §B.4 · p. 20

B.5 PROMPT DECOMPOSITION TEMPLATE

• The offline decomposition model uses only the text prompt to produce five atomic yes/no checks grounded in video-verifiable prompt claims.The template targets concrete observable content such as the subject, scene, action, objects, and camera or composition when present.Source: §B.5 · p. 20

B.6 VLM REWARD PROMPT AND REFERENCE ANSWER

• Each check’s reward is its mean target-answer log-probability, while its absolute input-gradient map supplies normalized, question-conditioned credit and all slots share one rollout advantage.All slots are teacher-forced in one sequence, so later slots condition on preceding reference tokens rather than independent answers.Source: §B.6 · p. 23

B.7 MATHEMATICAL INTERPRETATION OF TOKEN-CREDIT ROUTING

• A frame’s routed contribution scales with its question-conditioned weight, while the shared advantage sets update sign and magnitude; the weighted surrogate is biased relative to exact trajectory likelihood.PPO clipping can bound or remove a sample-check gradient while leaving the frame weights to route an active signal.Source: §B.7 · pp. 23–25
• The analysis is conditional: absolute VLM gradients are local, model-dependent sensitivities, not causal semantic explanations or signed credit, and idealized allocation does not guarantee improved shared-generator updates.The analysis does not establish that positive alignment alone beats uniform allocation.Source: §B.7 · pp. 24–25

B.8 UNSMOOTHED CELL-LEVEL CREDIT

• Raw gradient maps can preserve isolated high-frequency responses, concentrating credit on a few latent cells and increasing update variance; clipping cannot correct persistently mislocalized routing.The maps estimate local sensitivity, not noise-free causal attribution.Source: §B.8 · p. 25
• Removing spatial aggregation does not extend the improvement from frame-level to spatially aggregated credit, and 1×1 is particularly unstable with VideoScore2.The VideoScore2 reward drops sharply for the 1×1 control.Source: §B.8 · p. 25, Figure B1 · p. 25
• 55.10 Overall for 1×1 falls below frame credit (56.47), 7×7 (57.08), and 3×3 (57.69) in held-out Qwen3.5-9B evaluation.Figure B1 reports the unsmoothed 1×1 control alongside the spatially aggregated variants.Source: Figure B1 · p. 25, §B.8 · p. 25
• Figure B2's strongest residuals lie near the pen and gold-rimmed barrel and near the white headband, but these normalized maps are not causal masks.The examples show five sampled frames with queried evidence marked in red.Source: Figure B2 · p. 26