PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable Scores, Unstable Answers: Frame Phase and Option Order in Video Multiple-Choice Evaluation

Lichen Zhu, Yiheng Wang, Yueqian Lin, Hai "Helen" Li, Yiran Chen

arXiv:2610.08649v1cs.CV

TL;DR

Uniform-grid benchmarks report frame rate but not phase, although changing phase can change answers without materially changing accuracy. The paper measures this sensitivity and averages option posteriors across offset grids. This fusion matches a 32-frame pass in accuracy within a prespecified margin and reduces answer changes under a joint half-step shift.

  • Problem

    Benchmarks report a uniform grid’s frame budget but omit its phase, leaving unanswered how much this unreported sampling choice affects answers.

    Source: §1 · p. 1

  • Method

    PHASEFUSION averages option posteriors across offset grids, which are the polyphase components of a denser uniform grid.

    Source: §1 · p. 1, §3.1 · p. 2

  • Results

    10.1% of fusion answers change under a half-step shift of all three grids, versus 18.2% for a single 8-frame pass, while fusion matches the 32-frame pass in accuracy within a prespecified margin.

    Source: §4.1 · p. 3, §4.2 · p. 4

  • Takeaways & Limitations

    Report the phase convention with the frame budget or marginalize over phase; option order requires a separate treatment because pooling phases leaves its sensitivity unchanged.

    Source: §5 · p. 4

Abstract

from arXiv · show

Video-language models are ranked by multiple-choice accuracy on frames from a uniform grid. The grid has two parameters, a rate and a phase, and benchmarks report only the rate. The phase moves answers: two deployed samplers differing only by a half-step phase offset answer 23.6% of questions differently while scoring within a point, and across four releases from two families shifting only the phase changes roughly one answer in five after controlling option order. PHASEFUSION decodes three offset grids and averages the option posteriors. The grids are the polyphase components of the dense grid. Fusion matches a 32-frame single pass in accuracy within a prespecified margin (logit-scored) and cuts the answers a half-step shift of all three grids changes from 18.2% to 10.1%. Option order, which changes only the presentation, is flagged instead by a one-pass answer margin. Report the phase convention with the budget, or marginalize it.

1. INTRODUCTION

Uniform-grid benchmarks report a frame budget but leave the grid’s phase to the implementation, even though phase can change answers without changing accuracy. The paper measures this answer-level sensitivity and proposes averaging over offset grids, while treating option order as a separate nuisance.

1. INTRODUCTION

  • Benchmarks report the uniform grid’s rate but omit its phase, the location of its first sample.A re-run holds phase fixed, so it does not test sensitivity to that unreported parameter.Source: §1 · p. 1
  • Prior work varies frame inputs or option presentation, while this study holds rate fixed and measures answer changes caused by phase.The paper distinguishes sampling phase from PAS’s phase offsets inside rotary temporal encoding: here phase belongs to the sampling strategy, not the model.Source: §1 · p. 1
  • The paper averages option posteriors over offset grids to remove phase choice, but flags option-order sensitivity with a separate one-pass answer margin.The offset grids are polyphase components of a dense grid; exhaustive averaging over option order is factorial.Source: §1 · p. 1

2. PHASE SENSITIVITY OF UNIFORM FRAME SAMPLING

After controlling for option order, changing only the sampling phase changes roughly one answer in five while accuracy stays nearly flat. Deployed samplers expose the same issue, and larger frame budgets do not eliminate it.

2.1. Protocol

  • 39.2% of a single ordering’s gold-correct answers are wrong under at least one other ordering, although reported scores move by at most 5.8 points.Across nine cells, single-ordering answers flip at rates of 40.8–66.4%; on MVBench and Video-MME, one ordering reverses the InternVL2.5–Qwen3-VL ranking reported by the other nine.Source: §2.1 · pp. 1–2

2.2. The sampling phase

  • 18.7% of Qwen3-VL and 20.1% of Qwen2.5-VL consensus answers change between two phases, while per-phase accuracy moves by less than two points.The phase-flip rates are 8.0× and 5.5× their respective re-run floors; the comparison is across the same ten option orderings.Source: §2.2 · p. 2
  • At 32 frames, phase disagreement remains 3.4× and 6.0× the re-run floor on EgoSchema and Video-MME, respectively.As Qwen2.5-VL’s budget rises from 4 to 32 frames, pairwise disagreement falls from 24.0% to 12.4% on EgoSchema and from 34.7% to 22.1% on Video-MME.Source: §2.2 · p. 2

2.3. Samplers in the wild

  • 23.6% of consensus answers differ between deployed left-endpoint and midpoint samplers that differ only by a half-step phase, while all four samplers’ scores stay within 1.2 points.The figure compares four uniform samplers taken from deployed harnesses.Source: §2.3 · p. 2
  • Depending on the phase, Qwen3-VL’s measured lead over Qwen2.5-VL is 3.97, 6.08, or 6.74 points, a 2.77-point swing.Standardizing phase picks one value; marginalization removes the free parameter from the protocol.Source: §2.3 · p. 2

3. POLYPHASE MARGINALIZATION

The method treats offset grids as complementary views of one denser uniform grid, then averages their option posteriors. This removes dependence on the selected phase set, but leaves option-order sensitivity unchanged.

3.1. The offset grids are polyphase components

  • The S offset grids are exactly the polyphase components of a dense Sb-frame grid, with disjoint samples when the clip has at least Sb frames, up to one frame of rounding.Each phase-s sample is dense-grid sample k=iS+s, so the decomposition fixes the views deterministically.Source: §3.1 · p. 2

3.2. PHASEFUSION

  • PHASEFUSION averages the content-space option distributions from the S phase-specific passes and selects the highest-scoring option.All phases enter symmetrically, so the output no longer depends on which prespecified phase an evaluator selects.Source: §3.2 · p. 3
  • Pooling phases leaves option-order sensitivity unchanged, so the ordering axis requires its own instrument.Order flips were 53.1% before pooling and 53.4% after pooling (p=1.0, n=446); the three passes cost 3× one 8-frame pass.Source: §3.2 · p. 3

4. EXPERIMENTS

PHASEFUSION matches a 32-frame pass in accuracy with three short passes, while reducing answer changes under phase shifts. A one-pass answer margin flags order fragility; fusion’s gains are concentrated on long Video-MME clips.

4.1. Averaging costs no accuracy

  • At 3×8 frames, PHASEFUSION matches 32-frame accuracy at 3/16 of its visual-prefix attention.At 192 attention units, it outscores the 16-frame pass at 256 units by 2.9 points and majority vote, 59.3% to 57.8%.Source: §4.1 · p. 3, Table 1 · p. 4

4.2. Averaging buys reproducibility

  • With a half-spacing shift, fusion changes 10.1% of answers, versus 13.3% for the same 24-frame dense pass and 18.2% for one 8-frame pass.Across independent random draws, the change rates are 12.0% for fusion and 19.6% for one 8-frame pass.Source: §4.2 · p. 4, Table 1 · p. 4

4.3. Flagging order fragility in one pass

  • A plain-pass answer margin ranks order flips at 0.831 AUROC, above correctness prediction (0.695), and transfers zero-shot to InternVL2.5 and MiMo-VL (0.867/0.876).Flagging the lowest-margin fifth marks items that flip 89.6% of the time, versus 48.5% outside it, without another pass.Source: §4.3 · p. 4

4.4. When it helps

  • Fusion’s significant gains occur on long Video-MME clips: +4.4 and +5.0 points for Qwen3-VL and MiMo-VL; other listed gains are small and mostly nonsignificant.The exploratory 1 s split separates gains of −0.3 points below it from +2.7 points above it.Source: §4.4 · p. 4
  • The 24-frame equivalence to a 32-frame pass is supported on the logit-scored path, while the generation-path margin is not certifiable at the reported sample sizes.The equivalence margin is ±2.5 points and was fixed before fusion was decoded.Source: §4.4 · p. 4

5. CONCLUSION

The conclusion recommends reporting the sampler’s phase convention or marginalizing over phase with PHASEFUSION. Its evidence is limited to open 7–8B releases and requires option log-probabilities.

  • Report the sampler’s phase convention with its frame budget, or marginalize phase rather than leave it to implementation details.Source: §5 · p. 4
  • PHASEFUSION averages posteriors from offset grids that are polyphase components of the dense grid, damping phase within a prespecified accuracy margin.The conclusion describes marginalization as three short passes whose union is the dense grid.Source: §5 · p. 4
  • The evidence covers five open 7–8B releases from three families and requires option log-probabilities.The authors conjecture, rather than establish, that the frame-offset finding also holds for audio-language evaluation.Source: §5 · p. 4

6. COMPLIANCE WITH ETHICAL STANDARDS

The study uses publicly released models and benchmarks under their published terms, with no human or animal subjects. The authors report no conflicts of interest.

  • The computational study uses publicly released models and benchmarks under their published terms.Source: §6 · p. 5
  • No new data were collected from human or animal subjects, and no ethical approval was required.Source: §6 · p. 5
  • The authors declare no conflict of interest.Source: §6 · p. 5

Supplementary Material

The supplement supplies per-cell breakdowns, the full protocol, an intervention ladder, and auxiliary analyses. It also describes an executable audit planned for release with the corpus.

  • The supplement provides per-cell breakdowns behind pooled main-paper numbers and the full protocol.Source: Supplementary Material · p. 6
  • It includes the intervention ladder motivating flagging rather than repairing option-order sensitivity, plus auxiliary analyses referenced in the main paper.Source: Supplementary Material · p. 6
  • An executable audit that checks load-bearing numbers against result files will be released with the corpus.Source: Supplementary Material · p. 6

A. PROTOCOL AND DATASET STATISTICS

The evaluation pairs shared option orderings across sampled benchmark subsets, then probes phase sensitivity, deployed sampler conventions, and ordering controls across model releases.

Option-order evaluation

  • 63.5% of three-option items flip across ten option orderings, versus 50.2% of four-option and 51.7% of five-option items.Three-option items exhaust their six possible permutations; for four- and five-option items, the ten-ordering rate remains a lower bound.Source: A. PROTOCOL AND DATASET STATISTICS · p. 6

Protocol and populations

  • The core ordering sweep uses nine 500-question model-benchmark cells, while phase sweeps cover 593–599 items per model and share ten orderings.MVBench and Video-MME subsets are random samples; the 500-item EgoSchema subset is its complete public subset.Source: A. PROTOCOL AND DATASET STATISTICS · p. 6

Dense phase sweep

  • 20.4% of Qwen2.5-VL consensus answers disagree pairwise across 13 phases, close to 20.1% across the three-phase grid; 49.3% change somewhere across all 13.The three-phase figures are conservative lower bounds on answer changes across the denser phase sweep.Source: A. PROTOCOL AND DATASET STATISTICS · p. 6

Deployed samplers

  • 11.7–23.6% of Qwen2.5-VL consensus answers differ between deployed sampler conventions, although their accuracies range only from 51.1–52.3%.At least one pair changes consensus on 33.3% of items.Source: A. PROTOCOL AND DATASET STATISTICS · p. 7

Matched-vote control

  • With 50 random splits per item, matched five-ordering halves show disagreement rising from 19.4/18.5/19.3% within phase to 26.1/27.9/25.1% across phases for Q3/Q2.5/IV2.5, respectively.Source: A. PROTOCOL AND DATASET STATISTICS · p. 7

Phase results

  • 18.7% and 20.1% of item pairs change consensus across phases for Qwen3-VL and Qwen2.5-VL, while InternVL2.5 and InternVL3.5 change on 22.9% and 22.2% of questions.For InternVL2.5, per-phase accuracy is 50.7/51.4/51.4%; for InternVL3.5 it is 52.9/53.4/53.0%.Source: A. PROTOCOL AND DATASET STATISTICS · p. 7

Multiplicity

B. INTERVENTIONS THAT DO NOT REMOVE ORDER SENSITIVITY

The tested interventions do not eliminate answer instability, which includes both item-shared and model-specific components. Content fixation exceeds letter fixation and position-mass fixation across cells.

B. INTERVENTIONS THAT DO NOT REMOVE ORDER SENSITIVITY

  • 18.2% of items still split between two disjoint four-ordering votes, and oracle allocation, margin-guided reallocation, and routing do not recover accuracy.The oracle allocator has no headroom over the best uniform budget; repetition without new evidence leaves a residual ordering choice.Source: B. INTERVENTIONS THAT DO NOT REMOVE… · p. 7
  • Removing option labels reduces flips only from 53.0% to 49.0%, so letter priors do not explain most presentation sensitivity.Content fixation exceeds letter fixation, 0.81 versus 0.46.Source: B. INTERVENTIONS THAT DO NOT REMOVE… · p. 7
  • At λ=10, distribution matching raises flips from 53.3% to 80.0%, while high-confidence pseudo-label consistency also worsens flips.The seven frozen-backbone LoRA arms were trained on 4,996 NExT-QA MCQs; the only reduction was not significant and cost accuracy.Source: B. INTERVENTIONS THAT DO NOT REMOVE… · p. 8
  • Removing video raises flips from 50.3% to 79.9% as accuracy falls from 57.7% to 38.5%, locating the studied instability after visual evidence is available.The authors describe visual evidence as removing roughly thirty points of blind-text instability.Source: B. INTERVENTIONS THAT DO NOT REMOVE… · p. 8

C. PER-CELL INSTABILITY

  • Shared-unstable mass exceeds its independence null across every model pair and benchmark, while substantial model-specific instability remains.The pattern holds across architectures, indicating that fragility is not reducible to item weakness.Source: C. PER-CELL INSTABILITY · p. 8

D. MECHANISM: PER-CELL FIXATION

  • Content fixation exceeds letter fixation, which exceeds position-mass fixation in every cell; pooled values are 0.81, 0.46, and 0.38, respectively.A significant letter prior exists, but content fixation exceeds letter fixation in every cell; PriDe changes pooled accuracy from 56.2% to 56.4%, while full permutation marginalization reaches 58.0%, statistically indistinguishable from the 57.8% ordering ensemble.Source: D. MECHANISM: PER-CELL FIXATION · p. 8

E. THE UNION CONTROL

The union control separates the benefit of seeing more frames from the reproducibility benefit of averaging phase offsets. Fusion matches the 32-frame pass within the prespecified accuracy margin while reducing sensitivity to origin shifts.

E. THE UNION CONTROL

  • The three S=3 phase grids partition the uniform 24-frame grid, making a single pass over those same frames the control for complementary evidence versus frame dose.On the generation-scored path, fusion differs from the 24-frame union pass by +1.12 points for Q3 and −1.15 points for Q2.5; McNemar tests do not establish equivalence.Source: E. THE UNION CONTROL · p. 8, Table S7 · p. 10
  • On one NVIDIA L40S at batch 1, for 100 Video-MME questions, three serial short passes took 3.18 s versus 2.86 s for the 24-frame union pass; fusion’s cost case is attention compute and phase-draw removal.Source: E. THE UNION CONTROL · p. 10
  • PHASEFUSION leaves option-order flips unchanged at 53.1→53.4% (p=1.0), while its order-averaged gain over one 8-frame pass is +2.6 points for Q3 and +1.5 for Q2.5.The gain is positive under all ten orderings for Qwen3-VL and under eight of ten for Qwen2.5-VL.Source: E. THE UNION CONTROL · p. 10
  • Under a matched half-step origin shift, answers change on 18.2% of items for an 8-frame pass, 13.3% for the 24-frame union pass, and 10.1% for PHASEFUSION.The comparisons are paired on 587 items carrying every arm.Source: E. THE UNION CONTROL · p. 10
  • Visual distinctness tracks phase offset (ρ=0.77) but predicts phase-disagreement rate weakly (r=0.13, p=10−7); across distinctness quartiles, fusion gains were −0.2, +2.7, +2.4, and +3.6 points.In an exploratory one-second phase-offset split, PHASEFUSION changed performance by −0.27 points below (n=370, p=1.0) and gained +2.73 above (n=1,427, p=9 × 10−4).Source: E. THE UNION CONTROL · p. 10
  • −0.27 points is the paired fusion-versus-32-frame difference, with a 90% CI of [−2.09, +1.56]; the logit-path test establishes equivalence within ±2.5 points.The comparison covers 1,132 observations on 581 distinct questions; the same margin is cleared per model on the union control.Source: E. THE UNION CONTROL · pp. 9–10

F. ORIGIN ROBUSTNESS OF PHASEFUSION

A half-step origin shift still changes some fused answers, but PHASEFUSION damps that sensitivity. Three phases outperform two in the diagnostic, while separate margin-based signals support selective prediction and transfer across models.

F. ORIGIN ROBUSTNESS OF PHASEFUSION

  • 17.5% of single-pass answers change between origins, versus 11.0% for PHASEFUSION, a 1.6× damping; accuracy remains flat across origins.The S=6 fusion scores 60.28%, differing from S=3 on 4% of items (+0.13 points, p=.89), consistent with origin-invariance being approached as S grows.Source: F. ORIGIN ROBUSTNESS OF PHASEFUSION · p. 11

G. WHY THREE PHASES? THE PHASE-COUNT DIAGNOSTIC

  • The S=3 mixture reaches 61.38%, beating the S=2 mixture by +3.0 points (p=.016); with two views there is no majority to form.On items where phases disagree, the S=3 mixture scores 47.7% versus 36.2 / 38.9 / 43.6% for its views, while the S=2 mixture is below both of its own.Source: G. WHY THREE PHASES? THE PHASE-COUNT… · p. 11

H. RISK–COVERAGE

  • The continuous single-pass margin keeps ranking items where the ten-pass vote-share saturates, pulling ahead by up to 13.1 points where abstention matters most.At α=.20, every signal certifies zero coverage because the fixed-sequence walk fails at its first threshold under δ=10%.Source: H. RISK–COVERAGE · p. 11, Table S10 · p. 11

I. FRAGILITY-RANKING TRANSFER

  • The training-free margin retains its ranking on untouched models, including a reasoning/RL model with a different elicitation protocol.The margin needs no fit; its scale does not transfer, so thresholds are calibrated per model-benchmark cell.Source: §I · p. 11, H. RISK–COVERAGE · p. 11

J. PER-CELL FRAGILITY RANKING

  • Pooling nine cells with different margin scales and base rates costs roughly four AUROC points relative to the mean within-cell value.The score ranks questions within a model-benchmark pair; risk control additionally needs per-cell calibration.Source: J. PER-CELL FRAGILITY RANKING · p. 11

K. RELATION TO CIRCULAREVAL AND FLUCTUATION RATE

  • CircularEval’s unanimity rule scores 34.0% against 55.99% naive per-ordering accuracy, measuring robustness rather than capability and returning no answer for items that flip.On the same 1,298 items admitting an exact reverse, the two-ordering forward/reverse rate is 32.2%, undercounting the ten-ordering flip rate of 58.3%.Source: K. RELATION TO CIRCULAREVAL AND… · p. 12

L. VIDMCQ-FLIP DATASHEET

The datasheet describes a paired-record corpus of model passes and derived analyses, intended to support evaluation and reliability research without GPU inference. Benchmark content remains referenced by question id rather than redistributed.

L. VIDMCQ-FLIP DATASHEET

  • The corpus contains paired records for the ordering, frame-budget, phase, fusion, control, and timing captures, totaling 289k passes.Derived JSON files hold the analyses behind the tables and figures.Source: L. VIDMCQ-FLIP DATASHEET · p. 12
  • Each record carries a question id, benchmark, option count, and gold content index, with additional fields varying by capture.Permutation records store orderings, while phase captures store the phase index; benchmark content is referenced by question id under original licenses.Source: L. VIDMCQ-FLIP DATASHEET · p. 12, Table S14 · p. 13
  • The paired corpus supports flip prediction, confidence calibration, selective-prediction, and evaluation-protocol research without GPU inference.The planned public release includes an audit script; benchmark questions and videos are not redistributed.Source: L. VIDMCQ-FLIP DATASHEET · p. 13
Loading 2610.08649v1…