PaperlayerView exact revision on arXiv

Source-linked AI summary

Behavior Pack Optimization for Video MLLM Post-Training

Zhaolu Kang, Shiyu Liu, Tailong Luo, Wei Zhang, Yingjie He, Lei Wei, Guansu Wang, Liang He, Siheng Wang, Guangyuan Dong, Jiaqi Su, Shuang Chen, Haoyu Ji, Qishi Zhan, Kaiyue Zhou

arXiv:2610.03141v1cs.CV

TL;DR

Video MLLM post-training often scores a response to the original clip without specifying behavior across counterfactual views. BPO jointly scores question-conditioned packs with cross-view contracts and an original-view anchor, improving benchmark accuracy and abstention over budget-matched vanilla GRPO.

  • Problem

    Video post-training often grades individual responses or same-input groups, leaving behavior across related counterfactual views unspecified.

    Source: §1 · pp. 1–2

  • Method

    BPO jointly scores question-type-conditioned packs of original and counterfactual views under stability, sensitivity, and honest-abstention contracts, using the original-view rollout as an anchor.

    Source: §1 · p. 2

  • Results

    BPO improves macro accuracy by 4.7 pp, temporal-hard accuracy by 7.8 pp, and abstention F1 by 20.0 pp over budget-matched vanilla GRPO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Under matched compute, the pack objective, anchor-relative advantage, and type-aware view selection improve accuracy, temporal-hard examples, and behavior diagnostics.

    Source: §5 · p. 9

    BPO depends on useful interventions and reliable verifier signals; open-ended generation, dense videos, and larger backbones may require new view families and stronger audits.

    Source: §5 · p. 9

Abstract

from arXiv · show

Video multimodal large language models (MLLMs) keep climbing video question answering benchmarks, yet shuffling the frames, masking the segment that supports the answer, or occluding the target object barely changes their predictions. The accuracy rests on appearance and language priors, not on the temporal evidence the question asks for. We trace this to the unit of post-training: rewards are computed on a single response to the original clip, so the model is never asked to behave consistently across views. We propose Behavior Pack Optimization (BPO), which replaces the single response with a behavior pack of outputs across counterfactual views chosen by question type, scored jointly. The pack reward asks for stability when the intervention is irrelevant, sensitivity when key evidence is removed, and abstention when no evidence remains. To keep this objective stable at small pack sizes, BPO uses an anchor-relative advantage: the response on the original view serves as a per-prompt reference instead of a group mean over mixed views. On TempCompass, MVBench, and NExT-QA, BPO improves the macro accuracy of Qwen2.5-VL-7B-Instruct by 4.7 pp, the temporal-hard subset by 7.8 pp, and abstention F1 by 20.0 pp over a budget-matched vanilla GRPO baseline from the same SFT checkpoint. The gains transfer to Video-MME, LongVideoBench, and to LLaVA-Video-7B; ablations confirm they follow the view sets, not the rollout count. We hope this pack-level perspective offers a useful starting point for the video MLLM and multimodal post-training community as the field moves toward evidence-grounded video reasoning.

1 Introduction

Video QA post-training often scores responses locally, leaving behavior across counterfactual views unspecified. BPO jointly optimizes question-conditioned view packs, and improves accuracy and abstention over budget-matched GRPO.

1 Introduction

  • Local response grading leaves the policy without a direct signal to change behavior appropriately across related counterfactual views.Adding manipulated clips alone does not specify behavior across interventions when each response is still graded against a local target.Source: §1 · pp. 1–2
  • BPO jointly scores outputs across question-type-conditioned counterfactual views using contracts for stability, sensitivity, and honest abstention.Evidence-removal views target event, binding, and count questions; temporal interventions target order and direction questions.Source: §1 · p. 2
  • BPO uses the original clip’s rollout as a per-prompt anchor instead of centering rewards by the sampled group mean.The anchor is graded against the original clip’s ground-truth answer; the estimator can reduce variance when anchor and intervened-view rewards are correlated.Source: §1 · p. 2
  • BPO improves macro accuracy by 4.7 pp, temporal-hard accuracy by 7.8 pp, and abstention F1 by 20.0 pp over budget-matched vanilla GRPO.The evaluation uses Qwen2.5-VL-7B-Instruct on TempCompass, MVBench, and NExT-QA.Source: §1 · p. 2
  • The paper identifies optimization granularity, rather than data coverage alone, as a bottleneck linking shortcut behavior to the post-training objective.Its contribution combines question-conditioned view sets, cross-view contracts, and anchor-relative advantages.Source: §1 · pp. 1–2

2 Related Work

Prior video post-training adds temporal rewards and counterfactual supervision, but generally optimizes individual responses, same-input groups, or pairwise comparisons. BPO instead rewards joint behavior across structured views.

Faithful temporal grounding

  • Video MLLMs can score well without faithful temporal grounding, preserving answers after supporting evidence is reordered, masked, or removed.Counterfactual diagnostics complement static answer matching by checking stability under label-preserving edits and change or abstention when evidence is altered or removed.Source: §2 · p. 3

Video RL and counterfactual supervision

  • Video-R1 and related methods add video-specific rewards or signals [4] [15] [16] [17] [18], while edited-video preferences provide another counterfactual supervision line [5] [14].These signals are typically applied to an individual response, a same-input group, or a pairwise comparison tied to one manipulated instance.Source: §2 · p. 3

Behavior-pack optimization

  • BPO jointly scores a question-type-conditioned pack of original and diagnostic views under stability, sensitivity, and honest-abstention contracts.It keeps the GRPO-style clipped surrogate while changing the optimized unit and using an anchor-relative reference from the original-view rollout.Source: §2 · p. 3

3 Method

BPO treats video post-training as cross-view behavior optimization: it builds question-typed counterfactual packs, scores outputs jointly, and updates non-anchor views relative to the original-view rollout.

3.1 Setup and view records

  • A sample pairs a view set, question, and target answer; the policy also produces an Entity-State-Temporal trace installed during Stage A SFT.The trace is parsed by a deterministic grammar with success rate ≥0.99.Source: §3.1 · p. 3

3.2 Behavior packs

  • BPO samples one rollout per view in a behavior pack, exposing whether answers stay stable, change, or abstain as evidence is intervened on.The views are scored jointly rather than treated as ordinary data augmentation; the ablation baseline uses mean per-view reward as its scalar pack reward.Source: §3.2 · p. 4

3.3 Anchor-relative optimization

  • Because small-pack batch centering mixes prompt difficulty with intervention behavior, BPO uses the original-view rollout as a prompt-specific reference for non-anchor views.The anchor has zero self-relative advantage and contributes its reward as the baseline; under the stated exchangeable reward model, the anchor-relative signal has lower unnormalized variance above the specified correlation threshold.Source: §3.2 · p. 5, §3.3 · p. 5

3.4 Question-typed counterfactual views

  • BPO assigns evidence-presence probes to event, binding, and count questions, and temporal probes to order, direction, and trend questions.A diagonal experiment supports matching question types to view families; swapping view sets degrades temporal-hard accuracy.Source: §3.4 · p. 5

3.5 Verifier-grounded contract reward

  • The verifier combines view-level correctness and evidence checks with cross-view contracts for stability, sensitivity, and honest abstention.It also scores EST consistency and output format; the anchor has no contract reward, and the verifier reaches 0.93 accuracy on a 200-sample human-labeled audit.Source: §3.5 · p. 6

3.6 Training pipeline

  • Stage A installs the EST interface by SFT; Stage B freezes the vision encoder and updates the LLM using question-typed views and BPO.The training pipeline uses Equation (13) for the view set and Equation (9) for the update.Source: §3.6 · p. 6

4 Experiments

Experiments test whether BPO improves video QA under matched Stage-B compute and whether gains reflect cross-view mechanisms rather than extra rollouts or verifier artifacts. The controlled comparisons report gains on primary and extended benchmarks and behavioral measures.

4.1 Setup

  • B2–B5 use matched data, backbone, optimizer family, update and hyperparameter-search budgets, with Stage-B FLOPs within 5%.The suite includes TempCompass, MVBench, and NExT-QA; Video-MME and LongVideoBench are extended transfer checks.Source: §4.1 · p. 6

4.2 Main results

  • 4.7 pp macro accuracy, 7.8 pp temporal-hard accuracy, and 20.0 pp abstention F1: B5 beats budget-aligned vanilla GRPO (B2).The primary comparison uses the same SFT initialization and matched per-prompt rollout budget.Source: §4.2 · p. 7, Table 1 · p. 7
  • B5 exceeds B4 by 1.6 pp macro and 2.3 pp temporal-hard accuracy; the B3.m=1 control lands within 0.2 pp of B2.The controlled ladder attributes gains to pack optimization, anchor-relative advantages, and question-typed view selection rather than rollout budget alone.Source: §4.2 · p. 7
  • B5 improves over B2 on Video-MME and LongVideoBench by 3.8/3.6 pp macro accuracy and 6.1/5.8 pp temporal-hard accuracy.The extended benchmarks use the same matched budget.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 Anchor-relative advantage in training

  • B4 reduces per-step advantage variance from 0.341 to 0.142 relative to B3 and avoids B3’s early KL spikes.B3 and B4 use the same packs and verifier rewards; B4 compares intervened views with each prompt’s original-view anchor.Source: §4.3 · pp. 7–8, Figure 3 · p. 8

4.4 Type-Aware View Selection

  • Matched question-type/view-family cells lead on their corresponding evaluation regimes; the aggregate margin is 12.8 pp, 3.2× the maximum off-diagonal gain.Off-diagonal entries are single-seed diagnostics, and mismatched interventions provide less informative contrasts for the corresponding question types.Source: §4.4 · p. 8, Table 3 · p. 9

4.5 Reward and view-set ablations

  • Removing the sensitivity contract drops key-sensitivity from 54.6 to 38.4 and reverse-sensitivity from 48.9 to 35.2.Removing type conditioning gives back 2.3 pp on temporal-hard accuracy; removing honest abstention lowers abstain-F1 by 0.24.Source: §4.5 · p. 8, Table 4 · p. 9

4.6 Compute frontier and verifier audit

  • B5 remains above B2 across rollout budgets of 4 to 64 per prompt, while the GRPO slope does not close the gap.A 200-sample audit reports 0.93 overall verifier accuracy; reward-hacking stays below 2.3%, and 10% label-flip noise reduces B4 macro accuracy by 0.7 pp versus 3.2 pp for B3.Source: §4.6 · p. 9

5 Conclusion

BPO improves accuracy, temporal-hard examples, and behavior diagnostics under matched compute, while its current view families and verifier signals constrain its scope. The appendix provides formal analyses and a single-view sanity reduction for the pack-baseline ablation.

5 Conclusion

  • Under matched compute, BPO’s pack objective, anchor-relative advantage, and type-aware view selection improve accuracy, temporal-hard examples, and behavior diagnostics.The conclusion also identifies evidence-masking, reordering, and removal as failure cases BPO addresses.Source: §5 · p. 9
  • BPO depends on useful interventions and reliable verifier signals; open-ended generation, dense videos, and larger backbones may need new view families and stronger audits.Source: §5 · p. 9
  • BPO remains ahead across rollout budgets.Source: Figure 4 · p. 10

A Theoretical results

  • The appendix analyzes a single-view sanity reduction, anchor-relative estimator variance, reward-estimate concentration, and evidence favoring type-matched interventions over uniform gains from adding views.Source: A Theoretical results · p. 12

A.1 Single-view reduction of the pack-baseline objective

  • With one view and zero contract weight, the pack-baseline objective reduces to a batch-normalized single-view clipped policy-gradient update; this sanity reduction does not give anchor-relative BPO a meaningful single-view instance.At one view, there are no non-anchor views, so the anchor-relative RL term is empty.Source: §A.1 · p. 12

A.2 Variance reduction of the anchor-relative update signal

This appendix analyzes when anchor-relative centering lowers update-signal variance and separately checks concentration of bounded reward estimates under matched rollout budgets.

A.2 Variance reduction of the anchor-relative update signal

  • Under the exchangeable reward model, anchor-relative centering has lower unnormalized update-signal variance when cross-view correlation exceeds the threshold ρ⋆(m).The comparison concerns different signals: absolute pack reward relative to a baseline versus an intervened view relative to its original-view control.Source: §A.2 · pp. 12–14
  • For pack size m = 4, ρ⋆(4) = 0.636; measured correlation reaches at least 0.7 after the first ∼200 training steps.The observed variance gap is reported to be on the same order as the corollary's prediction and to match Figure 3.Source: §A.2 · pp. 13–14
  • The analysis assumes bounded rewards and exchangeable view structure; with non-exchangeable correlations, the threshold depends on the full covariance structure.The authors frame the proposition as a sufficient-regime analysis for the measured setting, not a universal dominance statement.Source: §A.2 · pp. 12, 14

A.3 Matched-budget concentration of bounded reward estimates

  • Under matched rollout budgets, the bounded-reward concentration result is a sanity check, not a claim that the methods' normalized advantages estimate the same quantity.It supports interpreting the Table 1 comparison as not giving BPO more sampled responses per update.Source: §A.3 · p. 14

A.4 Diagonal pattern: alternative explanations

The diagnostic matrix supports question-type-matched view families as mechanism evidence, not formal causal identification. The appendix also documents the implementation components used for Stage B.

A.4 Diagonal pattern: alternative explanations

  • The matched diagonal advantage totals 12.8 pp, inconsistent with uniform-view-gain, type-only, and null explanations; single-seed off-diagonals make it mechanism evidence, not causal identification.At fixed train type, matched-direction contrasts are 6.5 pp and 6.3 pp.Source: §A.4 · p. 15

B Implementation details

  • Appendix B documents the SFT setup, output schema, counterfactual views, verifier, contract rewards, classifier, and hardware/framework stack needed to reproduce Stage B.Baseline reimplementations, cross-backbone transfer, and additional ablations are documented in other appendices.Source: B Implementation details · p. 15

B.1 Stage A EST-SFT setup and the SFT-augmentation gap

  • Adding shuffled and reversed clips to SFT raises validation loss while structural parsing remains saturated; the authors attribute this to single-target supervision conflicting across views.The augmented inputs are still supervised against original-clip answers.Source: §B.1 · p. 15, Table 5 · p. 16

B.2 Entity–State–Temporal (EST) output schema

  • The EST output contains entity, state, and temporal fields, and all five structural pass-rate indicators reach approximately 1.00 after Stage A.The indicators cover parsing, entity references, temporal order, state vocabulary, and evidence completeness.Source: §B.2 · p. 16

B.3 Counterfactual view construction

  • Six view types are built and cached offline; cached records store view-specific targets, including insufficient-evidence targets when required support is absent.The types include original, object-mask variants, event-span removal, reversal, and frame shuffling.Source: §B.3 · p. 16

B.4 Verifier specification

  • The verifier returns four scores for answer correctness, evidence support, EST topology consistency, and format validity.Correctness uses exact match or an LLM judge, while evidence support checks frame-index overlap with key-evidence ranges.Source: §B.4 · p. 16

B.5 Per-view contract-reward forms

  • Per-view contracts compare each view output with the anchor output; default contract weights are 1/3 each, and the anchor-view weight is set to zero.The anchor reward therefore remains tied to the original-clip ground-truth answer.Source: §B.5 · pp. 16–17

B.6 Question-type rule classifier

  • The rule classifier routes order, direction, and trend triggers to Type 2 and defaults other questions to Type 1; it reaches 0.94 agreement and 97.3% benchmark coverage.Uncertain questions default to V1.Source: §B.6 · p. 17

B.7 Stage B training pseudocode

Stage B classifies each prompt, gathers a rollout pack over its assigned views, and updates the policy using anchor-relative advantages. The appendix reports budget alignment and run settings.

B.7 Stage B training pseudocode

  • Each prompt is mapped to a question type and fixed view set, then receives one rollout per view; rewards use the original clip as anchor and non-anchor views receive anchor-relative advantages.The optimization unit is the behavior pack rather than an isolated response.Source: §B.7 · p. 17

B.8 Compute alignment

  • Per-step FLOPs and wall-clock time stay within about four percent of single-video GRPO while sampled responses per prompt remain matched.Pack variants distribute responses across four cached views rather than drawing four independent original-video responses.Source: Table 6 · p. 18, §B.8 · p. 18

B.9 Hardware and frameworks

  • Experiments use 8× NVIDIA H20; headline RL results use five seeds, while ablation and diagonal rows use a three-seed subset.The headline seed set is {42, 1337, 2024, 2025, 7}.Source: §B.9 · p. 18

C Baseline re-implementation: Video-R1 and TPO

The appendix reimplements Video-R1 and TPO under aligned training conditions, checks their replication against published results, and lists further supporting analyses.

C Baseline re-implementation: Video-R1 and TPO

  • Video-R1 uses shuffled-frame GRPO with answer-only rewards, whereas TPO uses verifier-ranked manipulated-input DPO; matched reimplementations reproduce headline metrics within 1.5 pp.The comparisons align backbone, Stage A checkpoint, training data, per-prompt rollout count, and total FLOP budget.Source: C Baseline re-implementation: Video-R1… · p. 18

D Additional ablations and sweeps

D.1 View-set ablation

The ablations compare view-set choices, model scales, pack sizes, and predicted versus oracle question types, with the reported scope and compute controls varying by analysis.

D.1 View-set ablation

  • Four-view type-conditioned sets approach the full oracle on their target type under matched compute, supporting conditioning over simply enlarging the pack.The table reports macro and temporal-hard accuracy; conditioned rows use three seeds, while minimum and full-oracle rows use one seed.Source: §D.1 · p. 19

D.2 3B–7B scaling reference

  • Macro accuracy improves 4.1 pp at 3B and 4.7 pp at 7B, while temporal-hard accuracy improves 4.9 pp and 7.8 pp, respectively.Diagonal advantage is approximately 8.4 pp at 3B and 12.8 pp at 7B; positive claims are restricted to the 3B–7B regime, with 13B+ left for future work.Source: §D.2 · p. 19

D.3 Pack-size sweep protocol (m-sweep)

  • The TempCompass pack-size sweep tests m∈{2, 3, 4, 6, 8} with one seed and 3,000 Stage B steps, measuring estimator variance and cross-view reward correlation.Per-view rollout counts are reduced as pack size grows to keep total FLOPs within ±5% of the m=4 reference.Source: §D.3 · p. 19, Figure 5 · p. 20

D.4 Question-type classifier: oracle vs. predicted

  • Oracle-versus-predicted macro gaps are 1.4 pp on 7B and 1.7 pp on 3B; Table 9 reports gaps below 2 pp on both backbones, and Table 1 uses predicted types.Source: Table 9 · p. 20, §D.4 · p. 20

E Cross-backbone transfer: LLaVA-Video-7B

On LLaVA-Video-7B, BPO improves cross-backbone benchmark accuracy over B2, while the appendix defines behavior indicators and audits verifier quality.

E Cross-backbone transfer: LLaVA-Video-7B

  • On LLaVA-Video-7B, using the same data, verifier, and budget, B5-Llava exceeds B2-Llava by 4.6 pp in macro accuracy and 6.6 pp in temporal-hard accuracy; gains were consistently positive across benchmarks [9].Source: E Cross-backbone transfer:… · p. 20

F.1 Behavior-indicator definitions (master table)

  • Key-sensitivity and abstention F1 are reported in the main table, while non-key-stability and reverse-sensitivity appear in the reward ablation.The indicators map to stability, sensitivity, and honest-abstention contracts; they are computed on the original-correct subset unless otherwise stated.Source: §F.1 · pp. 20–21

F.2 Verifier audit

  • The verifier reaches 0.93 overall accuracy, exceeds 0.85 in every structure category, and has a reward-hacking ratio below 2.3%.The audit uses 200 validation-pack samples and the hacking ratio remains nearly flat during Stage B training.Source: §F.2 · p. 21

F.3 Reproducibility statement

The appendix documents reproducibility artifacts and seeds, states the scope of positive claims, and describes remaining failure modes and broader-impact considerations. Its extended related-work discussion compares BPO with prior approaches across optimization unit, intervention conditioning, and advantage baseline.

F.3 Reproducibility statement

  • The release package provides code, configurations, data-generation scripts, conversation pools, verifier annotations, and audit files for reproducing reported experiments.Headline RL results use five seeds; ablation and diagonal rows use three, with single-seed rows identified in captions.Source: §F.3 · p. 21
  • Positive claims are limited to 3B–7B backbones and the stated evaluation suite, with verifier-schema coverage, ambiguous anchors, and synthetic-view artifacts among the scope limits..Source: §F.3 · pp. 21–22

F.4 Broader impacts

  • The authors frame BPO as a reliability mechanism for audited settings, not a deployment guarantee, and note that high-stakes use requires consent, privacy safeguards, fairness audits, and human oversight.Source: §F.4 · p. 22

F.5 Qualitative failure cases

  • Manual validation cases expose failures on out-of-schema questions, ambiguous anchors, and key-mask views with visible synthesis seams.These cases respectively produce a default-to-anchor answer, seed-level oscillation, and apparent key-sensitivity for the wrong reason.Source: §F.5 · p. 22

G Extended related work

  • BPO differs from single-response SFT, uniformly shuffled-frame Video-R1 [4], and pairwise preference approaches such as TPO [5] by conditioning views on question type and scoring multi-view contracts jointly.The discussion presents this as aligning the optimization unit with the pack-level behavior being assessed.Source: G Extended related work · p. 23
  • The anchor-relative baseline uses the same prompt’s identity-view rollout and guarantees strict variance reduction when cross-view reward correlation exceeds a pack-size-dependent threshold.For the m = 4 packs used, the stated threshold is about 0.64.Source: G Extended related work · p. 23
  • Table 12 positions BPO as combining three design axes that prior work covers individually.Source: Table 12 · p. 22, G Extended related work · p. 23

Related papers

Loading 2610.03141v1…