Confident First, Reflective Second: Reinforcement Learning of Confidence-Preserving Reflection for Video Understanding

1 Introduction

• Long-form video understanding is difficult because models process only a subset of frames and may miss decisive moments, fine details, or temporal relationships.Video agents therefore often answer first, retrieve or re-examine segments, then verify or revise their answer.Source: §1 · p. 1
• Existing reflective objectives reward final correctness or revision improvement, leaving open how reflection optimization affects initial-answer accuracy.Answering from available observations and evidence-conditioned revision are distinct capabilities, and reflection can misinterpret new evidence or overturn a correct answer.Source: §1 · p. 1
• Outcome RL raises Video-MME reflection gain from −0.3 to +0.5 points while initial-answer accuracy stays nearly unchanged, from 60.4% to 60.3%.The paper calls this imbalance confidence erosion when accompanied by initial-answer degradation; confidence means label-verified initial-answer reliability, not self-reported certainty.Source: §1 · p. 2
• Outcome-based rewards can give equal credit to an initially incorrect answer later repaired and a correct answer left unchanged, while shared parameters permit drift and gradient interference.Stage-specific credit alone does not resolve the shared-parameter problem.Source: §1 · p. 2
• COFIRE protects initial-answer accuracy while encouraging beneficial revision, formalizing a confidence floor relative to a frozen reference policy.Its mechanisms separately credit initial correctness and revision outcomes, anchor verified-correct decisions, and prioritize the protected objective when gradients conflict.Source: §1 · p. 2
• 67.0% average accuracy places COFIRE 1.7 points above the strongest compared 7B method, with initial-answer and final accuracy gains of 4.1 and 3.7 points over Outcome RL on average.Reflection still helps on all three benchmarks.Source: §1 · p. 2

Answering and re-examining video

• Most trained variants reward only the final answer, which cannot distinguish an initially correct answer from a repaired one; stage-decoupled credit still trains the initial answer as input to revision.The paper instead makes initial-answer accuracy an explicit, protected training target.Source: §2 · pp. 2–3
• Video-R1 and VideoChat-R1 train temporal reasoning or spatio-temporal perception but answer in a single pass, unlike agents that seek or re-examine evidence.The latter include training-free evidence-seeking agents, TimeSearch-R, and Reflect-R1.Source: §2 · p. 2

Learning to revise

• Self-Refine and Reflexion use generated feedback, while SCoRe, S2R, and vision-language reflection methods train self-correction, but intrinsic correction can fail without reliable feedback.The paper argues per-revision penalties and fixed first-attempt regularizers do not correct initial-answer erosion.Source: §2 · p. 3

3.1 Answering and Re-examining a Video

• The initial stage uses a uniform 2 fps preview and up to eight higher-resolution frames to commit to an initial answer.The gate can TRUST, REVIEW, SEEK, or MULTI-HOP; retrieval ranks cached frames by query similarity and option-discriminating scores, and revision produces a candidate answer.Source: §3.1 · p. 3
• At inference, TRUST returns the initial answer and other modes return the revised answer; training generates a revised answer in every mode.Because the initial answer is explicit, the task label identifies whether it was correct and whether reflection repaired or preserved it, without step-level annotation.Source: §3.1 · p. 3

3.2 Why Reflective Training Erodes Confidence

• For fixed reflective behavior, outcome reward values a correct initial answer at only 1 − ρ − δ, so stronger reflection weakens pressure to answer correctly.Outcome RL turns negative reflection gain positive while initial-answer accuracy stays flat; the paper says selecting by reflection gain can therefore select for erosion.Source: §3.2 · p. 3
• Stage-routed credit does not prevent erosion when answering and reflecting share parameters: reflective updates can oppose the initial-answer gradient or shift answers on unsampled questions.Neither a fixed reward weight nor self-reported confidence detects these effects.Source: §3.2 · pp. 3–4
• In the illustrated Video-MME example, the first look shows two mugshots and yields C; retrieved frames reveal a third, and revision changes the answer to D.The gate chooses SEEK after reporting low confidence, and the revision is a FIX.Source: Figure 2 · p. 4

4.1 Confident First, Reflective Second

• COFIRE constrains initial-answer reliability to remain above a floor relative to a frozen reference while optimizing reflective utility.The objective treats initial-answer reliability as a separate term and omits reflection gain, which could reward erosion.Source: §4.1 · p. 4

4.2 Initial-Answer Credit under a Confidence Floor

• Adaptive initial-answer credit penalizes an incorrect first answer even when reflection repairs it, while crediting the repair at reflective stages.The initial-answer weight rises below the reliability floor and relaxes above it; a fix earns +1, a break costs −1.5, and an incorrect-option change costs −0.2.Source: §4.2 · pp. 4–5

4.3 Verified-Decision Anchoring

• Selective anchoring consolidates reference decisions only when they are verified correct, while KL regularization bounds drift at the answer-decision token.Later gate and reflection tokens receive looser regularization than the answer decision.Source: §4.3 · pp. 5–6

4.4 Asymmetric Gradient Protection

• Asymmetric gradient protection removes reflection’s component that conflicts with the protected initial-answer update while keeping that protected update intact.The protected objective learns correct initial answers from current rollouts and consolidates verified reference decisions; protection also preserves updates away from incorrect answers.Source: §4.4 · p. 6

5.2 Main Results

• 67.0% average accuracy makes COFIRE the best compared 7B model on every benchmark, 1.0–2.7 points above Reflect-R1, with the largest margins on long Video-MME videos and LongVideoBench.Source: §5.2 · p. 7

5.3 Does COFIRE Prevent Confidence Erosion?

• On Video-MME, COFIRE raises initial-answer accuracy by 3.9 points and final accuracy by 3.7, with a smaller reflection gain (+0.3); reflection still helps on every benchmark, most on LongVideoBench (+2.9).Source: §5.3 · p. 7, Table 2 · p. 7
• Routed initial-answer credit raises initial and final accuracy by 0.7 and 0.8 points; anchoring raises initial accuracy by 1.3 while reflection gain falls to +0.1; gradient protection adds 1.2 initial, 1.1 final, and 2.0 long-video points.Source: §5.3 · pp. 7–8, Table 3 · p. 7

5.4 Reflection on a Confident Initial Answer

• On SEEK questions, COFIRE’s initial-answer accuracy is 49.8% versus 37.2%, while error-repair rates are similar at 26.3% versus 26.6%, compared with 6.2% for REVIEW.Source: §5.4 · p. 8, Table 4 · p. 8
• Raising β from 1.5 to 2.0 cuts SEEK rate by a third and lowers final accuracy without strengthening the initial answer; gate confidence predicts correctness with AUC 0.68, while option belief separates fixes from breaks at 0.53.Source: §5.4 · p. 8, Figure 4 · p. 8

• RL on answer-and-re-examine video loops can raise reflection gain while initial-answer accuracy stalls, a pattern the paper terms confidence erosion.Source: §6 · p. 8
• COFIRE applies dedicated mechanisms to the incentive, drift, and interference channels under the principle confident first, reflective second.Source: §6 · p. 8
• COFIRE achieves the best compared 7B results on three video benchmarks and improves initial-answer and final accuracy over Outcome RL by 4.1 and 3.7 points on average.Source: §6 · p. 8

Limitations

• The evaluation covers one backbone and one answer-and-re-examine loop on long-video multiple-choice benchmarks, with one run per configuration.Extending the analysis to other loops trained with final-outcome rewards, such as agentic temporal search, is future work.Source: Limitations · p. 9

A Extended Related Work

• COFIRE combines a confidence floor on initial-answer accuracy, consolidation of label-verified reference decisions, and asymmetric gradient surgery that prioritizes answering.The paper adapts constrained RL, continual learning, and gradient-conflict ideas to answering and reflecting as coupled capabilities.Source: A Extended Related Work · p. 11

B.1 RL Question Pool

• The pool totals 42,602 questions: 30,974 short questions cover 15,147 videos lasting 4–180 seconds (median 32 seconds), while 11,628 long questions cover 3,527 Panda-70M videos lasting 10 seconds to one hour (median 17 minutes).Source: §B.1 · p. 11
• Cached direct-answer predictions are 73.9% accurate on short questions, 15.3% on long questions, and 57.9% overall.The cache includes 498 long-video questions recomputed from 12 uniformly sampled frames.Source: §B.1 · p. 11
• Training-video YouTube identifiers did not overlap with the 900 Video-MME videos, while exact question-string matches covered 12 generic templates.Source: §B.1 · p. 11

B.2 Supervised Initialization

• Supervised trajectories teach an overview request, initial answer, gate decision, and final answer, with gate confidence targets built from correctness labels.All RL runs start from the same supervised checkpoint.Source: §B.2 · p. 11

C.1 Training Settings

• COFIRE trains on previews, first-look frames, and retrieved frames through separate stages; Outcome RL uses final correctness plus a correction bonus across the whole trajectory.The preview shares a pixel budget across frames, so it is coarse for long videos; evaluation samples up to 768 frames.Source: §C.1 · p. 11

C.2 Reference Policies and the Confidence Floor

• The confidence floor is fixed at 0.10 below a frozen reference policy, and its multiplier is updated after each rollout batch.Reference and policy accuracy estimates use the same training sampler; the cached reference correctness avoids another reference-model generation.Source: §C.2 · p. 12, §C.3 · pp. 12–13

C.3 Estimating the Accuracy Signals

• Accuracy estimates use sampled initial answers and cached reference correctness, with moving averages updated after each policy step.The first estimate excludes the multiplier, group centering, transition utility, and auxiliary shaping.Source: §C.3 · pp. 12–13

C.4 Stage Surrogates and Candidate Rewards

• Stage routing omits downstream transition utility from initial-answer score terms, so the separate surrogates are not an unbiased joint-gradient estimator.Under TRUST, the gate is trained on candidate utility even though inference returns the initial answer.Source: §C.4 · p. 13

C.5 Reporting and Reproducibility Notes

• Table 1 compares complete systems using their original frame budgets and inference protocols, while Table 2 provides a controlled comparison under one protocol.The official Reflect-R1 reproduction reaches 62.6% on Video-MME, 0.9 points below its published 63.5%.Source: §C.5 · p. 14

D.1 Video-MME Duration Breakdowns

• Table 7 reports Video-MME duration breakdowns for the training configurations and the official-pipeline Reflect-R1 reproduction.Video-MME duration groups are equal in size, so the overall score is their mean.Source: §D.1 · p. 14, §C.5 · p. 14

D.2 Complete Training Progression

• Table 8 lists nine progressive configurations, adding routed initial-answer credit, anchoring, a floor multiplier, decision-local KL, transition utility, and asymmetric gradient protection in sequence.The final pair adds later-stage KL and reference-error reweighting together.Source: §D.2 · p. 14

D.3 Self-Reported Confidence as a Selection Signal

• Neither the model's self-reported confidence nor its option beliefs reliably separates outcomes relevant to deciding whether to keep or overturn the initial answer.This motivates protection based on label-verified correctness.Source: §D.3 · p. 14

E Training Algorithm

• Algorithm 1 estimates the projection coefficient on a tensor subset, detaches it before forming the full-model loss, and updates moving averages and the multiplier after each policy step.Source: E Training Algorithm · p. 14

F Analysis of Asymmetric Gradient Protection

• Under Proposition 1’s fixed-batch assumptions, the second first-order term is non-positive, a result attributed to A-GEM’s asymmetric projection geometry.The proposition assumes a locally Lipschitz gradient, fixed batch and advantages, exact gradients, and εproj = 0.Source: F Analysis of Asymmetric Gradient… · p. 14
• The coefficient bω is estimated using the last 16 trainable tensors, while the final update uses one backward pass through all parameters.The selected tensors are top-layer normalization weights and biases, with at most 106 entries each.Source: F Analysis of Asymmetric Gradient… · p. 14
• Tables 7–9 cover Video-MME duration breakdowns, progressive training configurations, and AUCs of self-reported signals, respectively.Table 7 reports final accuracy without subtitles; Table 8 adds components cumulatively except for its explicitly marked no-projection row; Table 9 uses 0.5 as chance.Source: Table 7 · p. 15, Table 8 · p. 15, Table 9 · p. 15

G Qualitative Examples

• The examples include three selected fixes from 56 FIX cases among 424 SEEK questions and one of 47 BREAK cases.Fixes were retained only when the revision cited newly retrieved frames that supported its claim; fixes correct only by guessing were excluded.Source: G Qualitative Examples · p. 15
• Three fixes use retrieved evidence to correct a scoreboard misread, identify a briefly visible bib, or replace a language prior with an on-screen headline.The bib is visible in 12 retrieved frames from the treatment segment; the headline is retrieved from a 42-minute video.Source: G Qualitative Examples · p. 15, Figure 5 · p. 17, Figure 6 · p. 17
• A correct but ungrounded route answer is broken when a retrieved 1916 rail statistic matches an option but is attributed to the traveler.The example notes that the BREAK cost does not replace checking whether retrieved evidence answers the question asked.Source: G Qualitative Examples · pp. 15–16, Figure 8 · p. 18