PaperlayerView exact revision on arXiv

Source-linked AI summary

Timbre: Reliability-aware harmonic consensus for label-free multimodal test-time rexinforcement learning

Haotian Lyu, Haoyu Wang, Ru Wang, Feng Ji, Ziwen Li, Jiaxian Guo, Mingming Gong

arXiv:2610.39439v1cs.AI

TL;DR

In label-free multimodal test-time reinforcement learning, a consistent but incorrect consensus can be reinforced, while image and text transformations can alter access to relevant evidence. TIMBRE uses reliability-aware weighted harmonic consensus, and under matched rollout budgets it outperforms TTRV across most benchmark–model settings on both adaptation and held-out instances.

  • Problem

    Visual misinterpretation and sensitivity to textual formulation can produce a consistent but incorrect consensus, while transformations may change access to task-relevant evidence.

    Source: §1 · pp. 1–2

  • Method

    TIMBRE combines four views through weighted harmonic pseudo-label selection and online modality-level reliability weighting, without requiring correct-answer invariance.

    Source: §1 · p. 2

  • Results

    Under matched rollout budgets, TIMBRE outperforms TTRV in 16 of 18 benchmark–model settings on adaptation instances and 17 of 18 on held-out instances.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Held-out results support transfer to unseen instances within benchmarks without additional updates.

    Source: §1 · p. 2

    The method is designed to mitigate, but does not eliminate, reinforcement of incorrect consensus, so adapted models should be validated before high-stakes deployment.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Label-free test-time reinforcement learning adapts vision--language models using rewards derived from their own predictions. However, misinterpreted visual evidence and sensitivity to textual formulation can produce a consistent but incorrect consensus that majority voting reinforces. Cross-view aggregation can expose such errors, but image and text transformations may themselves degrade task-relevant evidence to different extents. We introduce \method, a reliability-aware harmonic consensus framework for multimodal test-time reinforcement learning. \method constructs a $2\times2$ grid combining original and perturbed images with direct-solving and reframing modes, selects pseudo-labels through weighted harmonic aggregation, and adapts modality-level weights using batch-averaged, temporally smoothed shifts in pseudo-label confidence as a heuristic reliability signal. We formulate consensus as a weighted $\chi^2$-barycenter problem whose unique minimizer is the normalized weighted harmonic aggregate, without requiring correct-answer invariance. An exact support--dispersion decomposition characterizes when harmonic aggregation preserves or corrects arithmetic rankings, while a sensitivity analysis establishes how modality reweighting changes pairwise ranking margins, informing interpretation of the heuristic updates. Across six benchmarks and three Qwen3-VL model scales, \method outperforms TTRV in 16 of 18 benchmark--model settings on adaptation instances and 17 of 18 settings on held-out instances under matched estimation and optimization rollout counts. These results demonstrate improved target-task adaptation and transfer to unseen instances without further updates.

1 INTRODUCTION

Multimodal test-time reinforcement learning can reinforce a consistent but incorrect consensus, and transformed views can change access to task-relevant evidence. TIMBRE uses reliability-weighted harmonic aggregation, with theoretical analysis and benchmark results supporting its approach.

1 INTRODUCTION

  • A model’s dominant answer can be wrong, and visual or reasoning errors can produce a consistent but incorrect consensus that label-free optimization may reinforce.Intrinsic reward optimization sharpens existing answer preferences, making effectiveness dependent on alignment between confidence and correctness.Source: §1 · p. 1
  • Self-Harmony aggregates frequencies from original and reframed questions harmonically, but its rationale assumes correct-answer probability invariance that can fail in multimodal settings.Image perturbations and textual reframing can change how easily relevant evidence is used, motivating explicit control of each view’s influence.Source: §1 · p. 2
  • A weighted Pearson χ2-barycenter has the normalized weighted harmonic aggregate as its unique minimizer, while support–dispersion and sensitivity analyses characterize ranking changes.The formulation does not require correct-answer invariance; the analyses characterize when harmonic aggregation preserves or corrects arithmetic rankings and how modality reweighting changes pairwise margins.Source: §1 · p. 2
  • TIMBRE combines four image–text views with weighted harmonic pseudo-label selection and online modality-level reliability weighting.The views pair original and perturbed images with original and reframed queries; batch-averaged confidence shifts serve as a heuristic reliability signal.Source: §1 · p. 2
  • TIMBRE outperforms TTRV in 16 of 18 benchmark–model settings on adaptation instances and 17 of 18 on held-out instances under matched rollout budgets.Held-out results show within-benchmark transfer to unseen instances without additional updates.Source: §1 · p. 2

2 RELATED WORK

Prior label-free test-time RL and pseudo-label methods derive learning signals from model responses, while cross-view harmonic aggregation offers an alternative. TIMBRE formulates weighted aggregation to account for view influence without requiring correct-answer invariance.

2 RELATED WORK

  • TTRL [Zuo et al., 2025] and TTRV [Singh et al., 2026] derive rewards from model responses to a single input presentation, with TTRV using frequency-based self-rewards for VLMs.ETTRL [Liu et al., 2025a] and SPINE [Wu et al., 2025] retain majority-vote pseudo-labels while changing sampling or optimization.Source: §2 · p. 3
  • Self-consistency [Wang et al., 2023], SCRL [Yan et al., 2026], SCOPE, and Co-Rewarding-I [Zhang et al., 2025] use voting, selective supervision, confidence, subgroup consensus, or cross-supervision.These approaches address pseudo-label selection beyond majority voting or single-input aggregation.Source: §2 · p. 3
  • Self-Harmony [Wang et al., 2026a] harmonically aggregates answer frequencies from original and reframed questions, but its analysis assumes correct-answer invariance across views.That assumption is fragile when multimodal transformations alter access to task-relevant evidence.Source: §2 · p. 3
  • TIMBRE instead weights view influence in a Pearson χ2-barycenter objective whose unique minimizer is the normalized weighted harmonic aggregate.Its online view weights use a label-free confidence-shift heuristic.Source: §2 · p. 3

3 METHOD

TIMBRE adapts an unlabeled vision–language model through four image-and-prompt views, weighted harmonic pseudo-label selection, online modality reliability weighting, and pooled cross-view rewards for GRPO.

3 METHOD

  • TIMBRE adapts an unlabeled vision–language model by selecting pseudo-labels from four views and optimizing with jointly normalized cross-view rewards.The views pair original or perturbed images with direct-solving or reframing prompts.Source: §3 · p. 3, Figure 1 · p. 4

3.1 MULTI-VIEW GENERATION

  • Original and perturbed images are paired with direct-solving and reframe-and-solve prompts; M rollouts estimate distributions and reliability, while only K < M are optimized.Image transformations vary low-level appearance while preserving coordinates; cropping and large-angle rotation are excluded to limit disruption of spatial relations and diagram content, though appearance changes can obscure relevant details.Source: §3.1 · pp. 3–4

3.2 WEIGHTED HARMONIC CONSENSUS

  • For each candidate answer, weighted harmonic aggregation penalizes weak support in individual views and selects the pseudo-label; normalized scores uniquely minimize a weighted Pearson χ2 consensus objective.Additive smoothing prevents a zero count in one view from vetoing a candidate; empty candidate sets receive zero rewards and are excluded from reliability estimation.Source: §3.2 · p. 4

3.3 ADAPTIVE RELIABILITY WEIGHTING

  • Batch-averaged shifts in pseudo-label confidence across views act as a heuristic reliability signal, with target factors exponentially smoothed to set subsequent view weights.Lower reliability reduces the relative influence of views using that transformation; confidence shifts alone cannot distinguish degraded evidence from beneficial corrections.Source: §3.3 · pp. 4–5

3.4 CONFIDENCE-GATED REFRAMING REWARDS AND POOLED OPTIMIZATION

  • Direct-solving rollouts are rewarded for matching the pseudo-label, while reframed rollouts also receive a format-sensitive, confidence-gated diversity bonus; rewards are jointly normalized across 4K rollouts for GRPO.Advantages are applied to response tokens, including rewritten-query and solution tokens for reframed responses.Source: §3.4 · p. 5

4 THEORETICAL ANALYSIS

The section characterizes weighted harmonic consensus as a distributional optimization solution and analyzes how cross-view support, dispersion, and modality weights shape candidate rankings. Its ranking guarantees and sensitivity results specify conditions under which consensus preserves or corrects arithmetic rankings.

4.1 HARMONIC CONSENSUS AS A WEIGHTED χ2 BARYCENTER

  • The weighted Pearson χ² objective has a unique minimizer, and maximizing its harmonic score selects a mode without requiring correct-answer invariance.Whether the selector favors the correct answer depends on the relative support and dispersion of competing candidates.Source: §4.1 · p. 6

4.2 EXACT SUPPORT–DISPERSION AND RANKING ANALYSIS

  • Harmonic consensus discounts weighted arithmetic support according to cross-view dispersion, with low-support views contributing more strongly to the discount.The discount is a χ² divergence between assigned view weights and the candidate’s shares of weighted support.Source: §4.2 · p. 6
  • Exact correct-answer support invariance preserves every strict arithmetic win, while sufficient competitor dispersion can correct an arithmetic error.A correct arithmetic ranking is preserved whenever mC > (1 + Γw(C))mA; correction depends on the competitor’s dispersion threshold.Source: §4.2 · p. 7

4.3 EXACT SENSITIVITY TO MODALITY REWEIGHTING

  • For fixed view distributions and text weights, downweighting an image transformation increases the reciprocal ranking margin exactly when D0(A) > D1(A).The analysis characterizes instance-level margin changes; batch-averaged, temporally smoothed pseudo-label support shifts serve as an observable stability heuristic for shared reliability factors.Source: §4.3 · p. 7

5 EXPERIMENTS

Across six multimodal reasoning benchmarks and three Qwen3-VL model scales, TIMBRE generally exceeds TTRV under matched rollout budgets. The experiments evaluate adaptation and held-out transfer, and compare aggregation strategies in an ablation.

5.1 EXPERIMENTAL SETUP

  • The evaluation covers five visual mathematics benchmarks and the MMMU-Pro Science subset, with separate unlabeled adaptation and held-out sets.The study uses Qwen3-VL-Instruct 2B, 4B, and 8B models and matches methods at 64 estimation and 32 optimization rollouts per instance.Source: §5.1 · p. 8

5.2 MAIN RESULTS

  • TIMBRE achieves higher VAL-TTT performance in 16 of 18 settings and higher held-out VAL performance in 17 of 18 settings than TTRV under matched rollout budgets.Held-out instances are evaluated without further parameter updates.Source: §5.2 · p. 8, Table 1 · p. 8
  • On DynaMath with Qwen3-VL-8B-Instruct, TIMBRE maintains higher pseudo-label accuracy than TTRV over most later adaptation steps.Ground-truth answers are used only for post-hoc evaluation, not during adaptation; Figure 2 curves are smoothed for visualization.Source: §5.2 · p. 9, Figure 2 · p. 9

5.3 ABLATION OF CONSENSUS AGGREGATION

  • The Qwen3-VL-4B ablation compares weighted harmonic, weighted arithmetic, and uniform harmonic aggregation on MathVista and DynaMath.All variants use the same four-view generation procedure and optimization settings, separating reliability weighting from harmonic dispersion discounting.Source: §5.3 · p. 9, Table 2 · p. 9

6 CONCLUSION

The conclusion presents TIMBRE’s method and benchmark results, alongside statements on AI use, ethics, reproducibility, and the scope of additional theory.

6 CONCLUSION

  • TIMBRE outperforms TTRV in 16 of 18 benchmark–model settings on adaptation instances and 17 of 18 on held-out instances under matched rollout budgets.Held-out results show transfer to unseen instances without additional updates.Source: §6 · p. 10

AI USE STATEMENT

  • The authors used generative AI for implementation and debugging, writing edits, and related-work discovery, and report reviewing AI-assisted work.They state AI tools were not used to develop theory, formulate claims, design experiments, or interpret results.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

  • The work uses public benchmarks and Qwen3-VL models; because TIMBRE does not eliminate incorrect-consensus reinforcement, adapted models should be validated before high-stakes deployment.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

  • The reproducibility statement locates procedure and proofs, benchmark data and splits, hyperparameters, and rollout budgets across the paper and appendices.Source: REPRODUCIBILITY STATEMENT · p. 11

A ADDITIONAL THEORY FOR WEIGHTED HARMONIC CONSENSUS

A.1 PROOF OF THE WEIGHTED χ2 BARYCENTER CHARACTERIZATION

The proof establishes that the weighted Pearson χ2-barycenter has a unique normalized weighted harmonic minimizer and explains how disagreement affects that characterization.

A.1 PROOF OF THE WEIGHTED χ2 BARYCENTER CHARACTERIZATION

  • Strict convexity of the objective and a Cauchy–Schwarz lower bound establish the normalized distribution as its unique minimizer.The minimizer is strictly positive and its shared positive normalizer preserves rankings and ties.Source: §A.1 · p. 14
  • The minimum objective value is characterized in terms of the weighted harmonic score and the candidate’s arithmetic support.Source: §A.1 · p. 14
  • The barycenter balances disagreement across views without requiring identical view distributions; reversing the divergence arguments generally does not yield normalized harmonic scores.Source: §A.1 · p. 14

A.2 PROOF OF THE SUPPORT–DISPERSION DECOMPOSITION

The support–dispersion proof interprets harmonic consensus as arithmetic support discounted by disagreement between view weights and candidate-specific support shares.

A.2 PROOF OF THE SUPPORT–DISPERSION DECOMPOSITION

  • The proof obtains the decomposition by expressing dispersion through the harmonic score and the candidate’s arithmetic support.Source: §A.2 · p. 14
  • For each candidate, a view’s support share is its weighted answer probability divided by total weighted arithmetic support.Source: §A.2 · p. 14
  • The harmonic discount measures the discrepancy between assigned view weights and candidate-specific support shares.Source: §A.2 · p. 14

A.3 PAIRWISE RANKINGS AND INVARIANCE

The appendix gives conditions for harmonic consensus to preserve or correct arithmetic rankings, analyzes modality reweighting, proves a reward-gating guarantee, and specifies the reframing instructions.

A.3 PAIRWISE RANKINGS AND INVARIANCE

  • Exact invariance of the correct answer’s support across views preserves every strict arithmetic win; approximate relative support stability also gives a sufficient preservation threshold.The threshold is O(ε^2) as ε approaches zero.Source: §A.3 · p. 15
  • When the correct answer has lower arithmetic support, harmonic aggregation corrects the ranking if its dispersion sufficiently exceeds the competitor’s.Under exact correct-answer invariance, the correction condition reduces to Γw(A) > mA/mC − 1.Source: §A.3 · p. 15

A.4 MODALITY MARGINALS AND EXACT WEIGHT SENSITIVITY

  • Decreasing the image weight strictly increases the pairwise margin exactly when D0(A) > D1(A); text reweighting has the analogous result.Under the stated probability inequalities, decreasing that weight weakly increases the margin.Source: §A.4 · p. 15

A.5 REWARD CONSTRUCTION AND CONFIDENCE-GATING GUARANTEE

  • With the stated bounds, confidence gating keeps the pooled reward mean at most 1, giving nonnegative pooled advantages to matching direct-solving and format-compliant reframed responses.Source: §A.5 · pp. 16–17

B.1 REFRAMING INSTRUCTION

  • The reframing instructions require carrying over the original text’s constraints, avoiding image-derived facts in the rewritten problem, and preserving the original answer.Multiple-choice rewrites retain the original answer choices verbatim and in order.Source: §B.1 · pp. 17–18

B.2 DATA CONSTRUCTION

The evaluation uses separate unlabeled adaptation and held-out sets, with ground-truth answers reserved for evaluation. The appendix also reports the experimental hyperparameter schedule and per-view rollout allocation.

B.2 DATA CONSTRUCTION

  • Table 3 summarizes benchmark data sources and split sizes; ground-truth answers are used only for evaluation.Source: §B.2 · p. 19, Table 3 · p. 19
  • MathVista, MathVerse, MathVision, DynaMath, and WeMath use benchmark-specific adaptation and held-out splits, with no identical instances across the sets.MathVision uses test instances for adaptation and testmini for held-out evaluation; DynaMath uses distinct variants of shared seed problems.Source: §B.2 · p. 19
  • MMMU-Pro uses its Science subset in the standard four-option setting, assigning test-prefixed instances to adaptation and validation-prefixed instances to held-out evaluation.This yields 198 adaptation and 93 held-out instances.Source: §B.2 · p. 19

B.3 HYPERPARAMETERS

  • The four-view method divides estimation and optimization budgets equally across views, and the learning rate stays constant after a linear warmup over the first 3% of training steps.Source: §B.3 · p. 20, Table 4 · p. 20

B.4 ANALYSIS OF ADAPTIVE RELIABILITY WEIGHTING

The reliability update uses shifts in support for the current pseudo-label as a label-free stability heuristic because correctness cannot be measured during adaptation. The reported view weights generally align with empirical pseudo-label accuracy, while remaining smooth and allowing stable transformed views to contribute.

B.4 ANALYSIS OF ADAPTIVE RELIABILITY WEIGHTING

  • Because the correct answer is unavailable during adaptation, reliability updates estimate view reliability from shifts in support for the current pseudo-label rather than directly measuring correctness.Source: §B.4 · p. 20
  • Views with higher pseudo-label accuracy generally retain larger consensus weights, while lower-accuracy views receive less influence.Figure 3 pairs each view’s weight trajectory with its pseudo-label accuracy during adaptation on DynaMath with Qwen3-VL-8B-Instruct.Source: §B.4 · p. 20, Figure 3 · p. 21
  • Exponential reliability mapping and temporal smoothing keep weighting dynamics smooth, and transformed views continue contributing when pseudo-label support remains stable.The heuristic does not guarantee that every update improves the instance-level ranking margin.Source: §B.4 · p. 21
Loading 2610.39439v1…