PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Multimodal Reward Modeling

Yueying Li, Jingxuan Fan, Kianté Brantley, Sham M. Kakade, Hanlin Zhang

arXiv:2610.12707v1cs.CLcs.AIcs.CV

TL;DR

Vision-language reward models commonly rely on preference annotations that are costly and can disagree, and benchmark accuracy is used to compare them. This paper trains a linear head on frozen intermediate-layer features against same-image perturbations, then reports a 32B probe at or above GPT-4o on three benchmarks and 6.6 points ahead on VL-RewardBench.

  • Problem

    Vision-language reward models use annotated preference pairs that are costly to collect and can disagree, while benchmark accuracy is used to compare and trust models.

    Source: §1 · p. 2

  • Method

    A linear head on a frozen intermediate-layer backbone learns from same-image perturbations, with labeled data used only to select the probe layer.

    Source: §1 · p. 2, §2.1 · p. 4

  • Results

    At 32B, the probe is at or above GPT-4o on all three benchmarks and 6.6 points higher on VL-RewardBench.

    Source: §3.2 · p. 7

  • Takeaways & Limitations

    Reward-model selection should test how well a model ranks its own policy’s samples and report which pairs trained it.

    Source: §5 · p. 11

    The safety/bias benchmark column is a length artifact, so the authors compare heads on the five trainable categories instead.

    Source: §3.2 · p. 8

Abstract

from arXiv · show

Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark accuracy. We build one with no human preference label: a linear head on an intermediate layer of a frozen Qwen3-VL backbone, trained to score a web caption above a perturbed version of the same caption; labeled data enters only to select the layer. The layer is the decisive choice, worth 5.8 points over the last layer, and one recipe transferred without re-tuning beats or at least matches GPT-4o on three benchmarks at 32B, 6.6 points ahead on VL-RewardBench, from a few thousand parameters at about a thousandth of fine-tuning compute. At an equal pair budget, perturbations teach a frozen head 3 to 5 points more than curated RLAIF-V pairs. Because the backbone is frozen, the objective is convex, and the perturbation sampler specifies the reward, which the benchmark does not identify: two heads trained on disjoint halves of the same perturbations tie on the benchmark and differ by 6.4 points in hallucination judging. Retraining the head on the policy's own verifier-labeled candidates cuts caption hallucination by 5.1 CHAIR points and lifts image-generation pass rate by 5.4, while scoring lower on every reward benchmark. Together, an intermediate-layer probe, free same-image perturbations, and a verifier-driven bootstrap turn give a recipe for multimodal reward modeling that needs no annotator, costs a fraction of fine-tuning, and is specified by the comparisons it is trained on rather than by a benchmark rank.

1 Introduction

Annotated preference pairs for vision-language reward models are costly and can disagree, while benchmark accuracy is used to compare and trust the resulting models. This paper studies how representation depth and training comparisons shape a label-free reward probe, and whether benchmark rank predicts deployment utility.

1 Introduction

  • The recipe uses a few-thousand-parameter linear head on a frozen intermediate-layer backbone, trained against same-image perturbations without human preference labels.The paper also reports that, at an equal pair count, mined pairs teach a frozen head more than curated RLAIF-V pairs.Source: §1 · p. 2, Figure 2 · p. 3
  • The frozen-backbone probe's representation and comparison distribution determine what reward it learns; labeled data selects the layer, not the training pairs.The paper reports that training-side changes move benchmark accuracy by no more than .014, while the best layer lies well below the top across tested backbones.Source: §1 · p. 2
  • At 32B, the probe is at or above GPT-4o on all three benchmarks and 6.6 points higher on VL-RewardBench.It is also level with GPT-4o on Multimodal RewardBench and MMRB2.Source: §1 · pp. 2–3
  • Two probes trained on disjoint perturbation subsets tie on benchmark accuracy yet differ by 6.4 points in hallucination judging.The authors report that benchmark rank orders Best-of-N utility in neither direction.Source: §1 · pp. 2–3

2 Methodology

The method scores responses from frozen intermediate-layer features with a linear head, using perturbations to create rejected responses. Its objective makes the negative sampler consequential: the pair distribution shapes the learned reward, with qualifications from the objective’s assumptions and experimental setup.

2.1 Algorithm

  • A frozen Qwen3-VL backbone supplies mean-pooled response-token states from layer N−k to a bias-free linear reward head.At 2B, the head has 2,048 parameters; the backbone is not trained.Source: §2.1 · p. 4
  • Same-image caption perturbations edit facts in the response, while image-response pairs use same-seed renders of a perturbed prompt.Caption edits include span substitutions, tail replacement, clause shuffling, and captions from other images; the caption bank of 22k pairs builds in 10 seconds on one CPU core.Source: §2.1 · p. 4

2.2 What the Objective Identifies

  • At B=8, coupled same-entry terms carry 12.5% of the objective and move the unrestricted optimum away from the cross-entry density-ratio solution.The sampler also violates the absolute-continuity assumption, and its bank holds length fixed on two thirds of pairs but shortens negatives by about 5% on the rest.Source: §2.2 · p. 5
  • Same-context edits provide a quality comparison, whereas independent negatives from the caption marginal can reward topic association without distinguishing accuracy.In the paper’s stylized model, the edited-pair ordering has margin log η/(1−η), even when some originals are wrong.Source: §2.2 · p. 5
  • The fixed-penalty linear objective’s minimizer is determined by the sampler’s pair law, which weights pair-difference vectors by how often the head misorders them.Different pair laws can therefore produce different heads even with the same perturbations.Source: §2.2 · p. 5

2.3 Experimental Setup

  • The experiments use 70k PixMo captions and frozen-vision-tower Qwen3-VL Instruct backbones at 2B, 4B, 8B, and 32B.The headline 2B recipe uses layer N−4 and 200 steps of 256 pairs; evaluation includes three reward benchmarks and Best-of-N selection over 16 candidates.Source: §2.3 · p. 6

3 Experiments

Across experiments, probe depth and the negative sampler mattered more than optimizer changes, and the untuned 32B probe matched or exceeded GPT-4o on three reward benchmarks. Verifier-trained heads improved deployment outcomes while scoring lower on reward benchmarks.

3.1 Two Choices Set the Accuracy: The Probe Layer and the Sampler

  • The best probe is below the last layer, with a 5.8-point gain over it; at 15k pairs, mined negatives lead curated RLAIF-V by 3–4 points on Multimodal RewardBench and 2–5 on VL-RewardBench.The preferred depth varies by backbone: N−4 for 2B, N−6 for 4B, and N−12 for 8B and 32B.Source: §3.1 · pp. 6–7

3.2 One Untuned Recipe Reaches GPT-4o Across Backbones and Benchmarks

  • The untuned 32B probe is at or above GPT-4o on all three benchmarks and 6.6 points higher on VL-RewardBench.The comparison is unresolved on Multimodal RewardBench and MMRB2 reasoning; VL-RewardBench separates the probe by 3.6 standard errors.Source: §3.2 · p. 7, Table 1 · p. 8

3.3 Two Probes, One Score, Two Rewards

  • Similar benchmark scores do not ensure similar rewards: probes diverge on hallucination and reasoning, while Best-of-N gains depend on whether the candidate pool makes majority vote reliable.The CLIP-visible head scores 72.0 versus 78.4 on VL-RewardBench hallucination and 63.4 versus 56.8 on reasoning; external scorers beat majority vote when the generator is confidently wrong.Source: §3.3 · pp. 8–9

3.4 Learning from the Policy’s Own Errors: Better Deployment, Lower Benchmarks

  • Verifier-trained heads reduce caption hallucination and improve image selection, but score lower on reward benchmarks than perturbation probes.The captioning gain is 5.1 CHAIR points below the base generators; on held-out image prompts, the 8B verifier-trained head gains 5.4 points over random selection.Source: §3.4 · pp. 9–10, Figure 5 · p. 10
  • One DPO turn lowers COCO hallucination from .446 to .390 CHAIRs with probe-picked pairs, .414 with verified pairs, and .412 with random pairs, at no cost on MathVista or MMMU.The label adds little over random pairs, so the selector’s effect on the actor remains unshown.Source: §3.4 · p. 10

4 Related Work

Prior multimodal reward models commonly fit scorers to annotated preference pairs and rank them against human labels; this paper instead asks what those benchmark scores measure. Related work connects probe depth, negative sampling, and the use of rewards for selecting policy outputs.

Multimodal reward models and the benchmarks that rank them

  • Multimodal reward models commonly fit scorers to annotated preference pairs, while Multimodal RewardBench, VL-RewardBench, and MMRB2 rank models by accuracy against human labels.The cited examples include LLaVA-RLHF, RLHF-V, MM-RLHF, IXC-2.5-Reward, and UnifiedReward.Source: §4 · pp. 10–11

Probing and probe depth

  • Linear probes are established encoder-analysis tools, and intermediate layers can yield better features than the last layer; LLaVA-Reward mixes depths with a preference-trained cross-attention module.The paper contrasts that approach with its 2,048-parameter head, trained without preference labels.Source: §4 · p. 11

Density-ratio estimation and negative sampling

  • Negative-sampler choices shape noise-contrastive objectives, including caption perturbations such as word-order and attribute swaps; with frozen features, they act through the objective.Fan et al. (2026) is described as the closest objective, using web continuations as text negatives.Source: §4 · p. 11

Best-of-N and learning from self-generated data

  • Reward optimization can degrade proxy rewards, while this work locates an earlier mismatch: benchmark score may not predict Best-of-N value against self-consistency.The paper frames its reward-side analysis as analogous to policy-side work on expert iteration, STaR, and GKD.Source: §4 · p. 11

5 Concluding Remarks

The conclusion presents frozen-representation probes trained on caption perturbations as a low-cost reward-modeling approach, while emphasizing that benchmark rank may not predict deployment utility. It recommends evaluating reward models on their policy’s samples and reporting the comparisons used for training.

5 Concluding Remarks

  • Higher preference-benchmark accuracy need not yield better selection of a policy’s outputs, and retraining on verified policy outputs improves caption selection while lowering benchmark accuracy.The conclusion says deployment evaluation should test selection among the policy’s own samples and report the training comparison distribution.Source: §5 · p. 11
  • Frozen vision-language representations support reward modeling with a linear head and caption perturbations, but probe depth and training comparisons determine which signals the head extracts.The conclusion characterizes the approach as using only a small head over frozen representations.Source: §5 · p. 11

A Limitations and Broader Impacts

The paper reports reduced additional training and preference-annotation requirements from fitting a small head over a frozen backbone, while limiting its claims to the settings and metrics evaluated. It identifies further evaluation needed before broader deployment.

A Limitations and Broader Impacts

  • The approach lowers additional training and preference-annotation requirements, but retains the inference cost of the underlying backbone.The authors say this may make task-specific reward models more accessible to researchers and practitioners with limited annotation budgets.Source: A Limitations and Broader Impacts · p. 19
  • Experiments cover two backbone families, and verifier-based adaptation is evaluated for only one round.Source: A Limitations and Broader Impacts · p. 19
  • Caption gains measured with CHAIR establish fewer absent COCO-object mentions, not general factuality or preserved informativeness; the authors identify POPE as a needed independent metric.Source: A Limitations and Broader Impacts · p. 19
  • Best-of-N cells use a single seed, the hard-QA set comes from one LLM family, and Multimodal RewardBench omits its withheld toxicity split.Source: A Limitations and Broader Impacts · p. 19
  • The method establishes no safety or fairness capabilities; broader deployment requires independent evaluation of output quality, coverage, and task-specific failure modes.The authors also warn that benchmark-only deployment justification risks overlooking systematic errors and shortcuts, including response-length biases.Source: A Limitations and Broader Impacts · p. 19

B Reproducibility

The reproducibility recipe specifies a frozen Qwen3-VL checkpoint, probe layer, response-token pooling, and a bias-free linear head trained for 200 steps. Its public inputs and pair-construction procedure support rebuilding the perturbation bank, while the paper audits overlap with evaluation data.

B Reproducibility

  • The recipe fixes a frozen Qwen3-VL checkpoint, probe layer, response-token mean pooling, and a bias-free linear head trained with AdamW for 200 steps.Source: B Reproducibility · p. 19
  • Public PixMo-Cap captions and string-edited negatives form a 22k-pair bank that rebuilds in 10 seconds on one CPU core; public RLAIF-V supplies the curated comparison set.Source: B Reproducibility · p. 19
  • The image arm renders 1,212 PixMo-derived prompts with Qwen-Image, and its generated training images and prompts are not those of MMRB2.Source: B Reproducibility · p. 19
  • The perturbation set shares 5 near-duplicate images with Multimodal RewardBench and none with VL-RewardBench or the listed MMRB2 and COCO Best-of-N images; RLAIF-V overlaps 326 VL-RewardBench rows and 18 Multimodal RewardBench rows.The authors report decontaminated subsets for RLAIF-V-trained rows and say pair construction, training, selectors, and plotting scripts will be released.Source: B Reproducibility · p. 19

C Evaluation Protocol

The evaluation combines preference benchmarks with Best-of-N captioning and QA pools, and specifies how overlap, scoring format, and uncertainty affect interpretation.

C Evaluation Protocol

  • VL-RewardBench has 1,247 pairs, including 326 that overlap RLAIF-V training; RLAIF-V’s .80 score on overlapping rows falls to no lead on the 921 clean rows.MMRB1 releases 4,711 of 5,211 pairs, with toxicity withheld; its reported overall number is recomputed for comparability.Source: C Evaluation Protocol · p. 20
  • MMRB2 averages four tasks; under the trained heads’ prompt-and-response format, the 32B probe scores 59.9, versus 64.6 for its generative judge and 59.7 for GPT-4o.Scoring prompt and response in one user turn costs text heads 2–3 reasoning points and 5–9 points on the text-only subset.Source: C Evaluation Protocol · p. 20
  • Best-of-N evaluation uses 500 prompts per listed captioning and QA set, including 500 hard COCO questions, with candidates sampled at temperature 1.0 and top-p 0.95 plus greedy decoding.The mid-training runs use 3,462 verifier-labeled candidates; head training uses AdamW on precomputed vision-tower outputs.Source: C Evaluation Protocol · p. 20

D Negative Mining and Probe Depth

This section specifies perturbation and probe-depth choices, then compares their benchmark performance across modalities and training settings. Same-image perturbations outperform curated pairs at matched pair counts, while the selected intermediate layer is a decisive probe choice.

D Negative Mining and Probe Depth

  • For text responses with a 2B backbone, the June full-fine-tune negative-mining rows report safety/bias scores of .224 and .451, alongside a length shortcut.Source: Table 3 · p. 21
  • Perturbing one, three, or all editable spans leaves both benchmarks within .008, despite validation accuracy rising from .82 to .996.The density variants use token substitutions on the same 21,521 images and captions.Source: D Negative Mining and Probe Depth · p. 20, Table 4 · p. 21
  • 3.5 points: at matched pair budgets, mined negatives beat curated RLAIF-V pairs on trainable Multimodal RewardBench accuracy (.574–.584 vs .544).They score .679–.709 vs .660 on VL-RewardBench and .769–.781 vs .737 on hallucination; the comparison matches pairs, not tokens.Source: D Negative Mining and Probe Depth · pp. 21–22, Table 5 · p. 23
  • The 15k-pair comparison is matched by pairs but not response tokens: perturbation captions provide 6.1M tokens versus 2.6M for RLAIF-V.The pair banks also differ across backbone rows; the rebuilt 15k bank costs the 2B probe 2–3 Multimodal RewardBench points relative to the original 60k bank.Source: D Negative Mining and Probe Depth · p. 20, Table 5 · p. 23
  • 5.8 trainable points separate the chosen probe layer from the last layer, but the depth sweep is only a relative instrument and can mis-rank pooling choices.Last-token pooling ranks first in several sweep models yet loses 7.5 points to mean pooling in the real run.Source: D Negative Mining and Probe Depth · p. 23
  • On GLM-4.1V-9B, the same recipe scores .692 on Multimodal RewardBench and .602 on VL-RewardBench at N−12, versus .628 and .598 at the last layer.The untrained judge scores .565 and .570, respectively.Source: D Negative Mining and Probe Depth · p. 23

E Extended Results for Reward-Model Training

Figure 7 compares reward-benchmark performance across backbone sizes and model components. The intermediate-layer linear probe consistently exceeds the last-layer probe, while full fine-tuning and RLAIF-V probes show benchmark-specific patterns.

E Extended Results for Reward-Model Training

  • The N−k linear probe improves with backbone size on all three benchmarks and beats the last-layer probe at every size.Figure 7 plots the three reward benchmarks against backbone size, with confidence bands.Source: E Extended Results for Reward-Model… · p. 23, Figure 7 · p. 24
  • Full fine-tuning matches the linear probe on MMRB1 only from 8B upward and trails it on VL-RewardBench at 32B.The 32B fine-tune was stopped at step 150 of 400.Source: Figure 7 · p. 24
  • The RLAIF-V probe trails on MMRB1 at every size and leads on VL-RewardBench only at 32B, where it shares source images with the comparison.The figure reports component-wise results against backbone size.Source: E Extended Results for Reward-Model… · p. 23, Figure 7 · p. 24

F Extended Results for Best-of-N

Extended Best-of-N results show that benchmark scores do not resolve which candidates are most truthful or useful in deployment. Verifier-mined training improves caption selection in one loop turn, while model-written negative constructions can produce unhelpful selection behavior.

F Extended Results for Best-of-N

  • The probe detects one-edit decoys but cannot identify the most truthful of 16 real captions; it does not prefer the source over truth-preserving edits.Every scorer ranks a source above its one-edit decoy at least 93% of the time, yet the pick’s CHAIRs does not move.Source: F Extended Results for Best-of-N · p. 24
  • Continuing the mined-negative head helps only when the verifier-pair budget is too small to reach the optimum: at 4B it leads a fresh head by 7 points, while 2B results differ by at most 0.3.The comparison is across verifier-trained heads.Source: F Extended Results for Best-of-N · p. 24
  • At 2B, self-corruption ties greedy and random across four captioning cells, while human-versus-policy training scores .360 on Multimodal RewardBench and selects more hallucinated captions than random.The latter head distinguishes human from model authorship, a direction that anti-correlates with labels when both responses are model-written.Source: F Extended Results for Best-of-N · p. 24
  • On hard QA where the 8B generator is confidently wrong, every external scorer beats majority vote; on the 2B pool, probe and log-likelihood are within noise of the vote.Table 6 evaluates selected-candidate accuracy on 500 questions and a 241-question counting/spatial subset.Source: Table 6 · p. 25, Figure 8 · p. 25
  • Verifier training cuts pooled caption hallucination by 5.1 CHAIR points against greedy, but the resulting head ranks worse than the mined-negative probe on both reward benchmarks.The held-out COCO-2 result is a 4.1-point CHAIR reduction; the table describes this as the best ranker being the worst recent judge.Source: F Extended Results for Best-of-N · p. 24, Table 7 · p. 25

G Actor Training

The actor and CLIP studies test how reward-model comparisons transfer to downstream selection, while the accompanying analysis explains how the comparison law shapes the reward and what its idealized targets do not guarantee.

G Actor Training

  • DPO on self-generated pairs mildly reduced COCO hallucination across all corpora, but showed no selector effect on QA or image generation.Verified, probe-picked, and random pairs yielded CHAIRi .083 to .073, .079 and .082; image-generation variants stayed within ±2 points of their random control.Source: G Actor Training · p. 26, Table 8 · p. 26

H CLIP Studies

  • The CLIP-visible and CLIP-blind quartiles had different reward-benchmark patterns, while 43% of pairs were CLIP-blind because edits fell beyond its 77-token window.Multimodal RewardBench scores were .685 / .698 / .678 for top, random, and bottom quartiles; VL-RewardBench scores were .670 / .686 / .682.Source: H CLIP Studies · p. 27, Figure 9 · p. 27

I Reading the Two Statements

  • For a fixed-penalty objective, convexity fixes the minimizer, but the analysis does not derive where AdamW’s experimentally trained heads land.Changes to the negative pool, hardness, or temperature change the population objective rather than merely the path to its minimizer.Source: I Reading the Two Statements · p. 27
  • Length is not generally removed from the target: conditional matching by content is required, while the perturbation bank neither holds length fixed for every edit nor matches it exactly.The bank’s splice and shuffle edits shorten negatives; pairwise length preservation alone does not make the target length-free.Source: I Reading the Two Statements · p. 28

I.1 What Constructed Comparisons Supervise

  • Constructed comparisons are specified by their joint winner–loser law, so judged candidates’ marginals alone do not determine the objective.For real captions μ and a negative-construction kernel K, J_x(y, y′) = μ(y | x)K_x(y′ | y).Source: §I.1 · p. 28

I.2 Independent Negatives: Density Ratios and PMI

  • With independent negatives from the caption marginal and common support, the unrestricted score is image–caption PMI up to a context offset, not correctness or preference agreement.The result does not apply to edited copies or judge-ordered candidates, and finite corpora need not satisfy the shared-support assumption.Source: §I.2 · pp. 28–29

I.3 When Perturbations Expose an Additional Quality Signal

  • Under the stylized factorization and edit assumptions, perturbation training ranks correct versions above incorrect ones, unlike independent negatives in that example.The edit preserves topic while switching versions; the result is one additional quality distinction, not a general superiority claim.Source: §I.3 · p. 29

J Proofs

The proofs characterize the fixed-penalty head’s unique minimizer and clarify where idealized ratio results stop applying. The compute analysis finds head fitting inexpensive relative to the rest of the study.

J Proofs

  • A fixed quadratic penalty makes the linear-head objective strongly convex with exactly one minimizer, and gradient descent converges under the stated step-size condition.The convergence claim assumes finite expected squared feature-difference norm and a constant step below 2/L_s.Source: J Proofs · p. 30
  • The unrestricted density-ratio optimum requires mutually absolutely continuous positive and negative laws; the sampler violates that condition, and coupled same-entry terms move the optimum.At B=8, cross-entry terms carry 87.5% of the objective, while coupling changes the two-response example’s optimum from log 16 to log 11.Source: J Proofs · pp. 30–31
  • The regularized minimizer is an error-weighted combination of difference vectors, so different samplers can produce different heads even when benchmark pair orderings agree.A benchmark measures only the sign pattern on its pairs and may miss divergence elsewhere.Source: J Proofs · pp. 31–32

K Compute Resources

  • A 540-configuration 2B head grid took about nine GPU-minutes, while the caption negative bank took 10 CPU-seconds for 22,000 pairs.The quoted thousandth-of-fine-tuning compute comparison counts head training against full fine-tuning on the same pairs and excludes shared backbone forward passes.Source: K Compute Resources · p. 32
Loading 2610.12707v1…