PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Improving VLM Judges Without Human Annotations

Inna Wanyin Lin, Yushi Hu, Shuyue Stella Li, Scott Geng, Pang Wei Koh, Luke Zettlemoyer, Tim Althoff, Marjan Ghazvininejad

arXiv:2512.05145v1cs.CV

TL;DR

Human preference annotations for VLM judges are costly and can become obsolete as models advance. This paper trains a judge iteratively from self-generated preference data and reasoning traces, reporting improved VL-RewardBench accuracy for an 11B model. Its results also identify task-specific gains and a safety-data boundary.

  • Problem

    Training VLM judges relies on human preference annotations that are costly, labor-intensive, and can become obsolete as models advance and new tasks arise.

    Source: §1 · p. 1

  • Method

    The framework generates synthetic preference pairs, filters the judge’s self-generated judgments and reasoning traces against known preferences, and iteratively trains on retained examples.

    Source: §1 · pp. 1–2, §3.1.2 · p. 4

  • Results

    0.38 to 0.51 overall accuracy on VL-RewardBench: the iteratively trained Llama-3.2-11B judge often outperforms Llama-3.2-90B and Claude 3.5 Sonnet.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework can create preference data without ground-truth answers, making it applicable to emerging visual tasks where annotations are scarce or unavailable.

    Source: §1 · p. 2, §7 · p. 10

    The framework does not generate synthetic training data containing biased, toxic, or otherwise harmful content, limiting its general-purpose approach to safety evaluation.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Effective judges of Vision-Language Models (VLMs) are crucial for model development. Current methods for training VLM judges mainly rely on large-scale human preference annotations. However, such an approach is costly, and the annotations easily become obsolete as models rapidly improve. In this work, we present a framework to self-train a VLM judge model without any human preference annotations, using only self-synthesized data. Our method is iterative and has three stages: (1) generate diverse multimodal instruction-response pairs at varying quality levels, (2) generate reasoning traces and judgments for each pair, removing the ones that do not match our expected quality levels, and (3) training on correct judge answers and their reasoning traces. We evaluate the resulting judge on Multimodal RewardBench and VL-RewardBench across domains: correctness, preference, reasoning, safety, and visual question-answering. Our method improves a Llama-3.2-11B multimodal judge from 0.38 to 0.51 in overall accuracy on VL-RewardBench, often outperforming much larger models including Llama-3.2-90B, GPT-4o, and Claude 3.5 Sonnet, with particularly strong gains in general, hallucination, and reasoning dimensions. The overall strength of these human-annotation-free results suggest the potential for a future self-judge that evolves alongside rapidly improving VLM capabilities.

1 Introduction

The paper proposes training VLM judges from model-generated preference data rather than human preference annotations. It uses an iterative pipeline and reports improved benchmark accuracy for a compact judge.

1 Introduction

  • Human preference annotation is costly, labor-intensive, and can become obsolete as models advance and new tasks arise.The proposed method requires no human preference annotations.Source: §1 · p. 1
  • The three-stage loop constructs preference pairs, retains the previous judge’s judgments that match their synthetic preferences, then fine-tunes it on the filtered reasoning traces and answers.Pair construction uses majority answers against random alternatives for closed-ended questions and injected errors for open-ended responses.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • 0.38 to 0.51 overall accuracy on VL-RewardBench: the iteratively trained Llama-3.2-11B judge often outperforms Llama-3.2-90B and Claude 3.5 Sonnet.The framework is evaluated on VL-RewardBench and Multimodal RewardBench.Source: §1 · p. 2
  • The authors describe synthetic preference data and iterative self-training as applicable to visual tasks where ground-truth annotations are scarce or unavailable.They report analysis of synthetic data design and iterative refinement, including conditions for effective self-improvement.Source: §1 · p. 2

2 Related work

Prior VLM-judge training relies heavily on human preferences or knowledge distilled from large closed-source models. Related work also studies synthetic-data self-improvement, while highlighting multimodal grounding challenges.

2.1 LLM and VLM as a judge

  • VLM judges are trained mainly through scaled human preference collection or distillation from large closed-source models, which also depend on human annotation.Specialized synthetic-data judges have addressed dimensions such as hallucination, while general-purpose judges typically require substantial preference data.Source: §2.1 · p. 3

2.2 Synthetic Data for Model Self-Improvement

  • Self-improvement research uses model-generated data for task expansion, iterative refinement, and self-rewarding, but uncontrolled synthesis can cause degenerative feedback loops and preference leakage.In multimodal settings, visual grounding, factual accuracy, hallucination, and text-image misalignment pose distinct challenges; much prior work relies on larger models to generate data.Source: §2.2 · p. 3

3 Method

The method creates preference pairs from a VLM’s own outputs, filters its judgments against constructed preferences, and iteratively trains on the retained reasoning traces and answers.

3 Method

  • The framework has three stages: synthetic preference-pair generation, filtered judge-training-data construction, and iterative judge training.The same model generates and perturbs responses, creating meaningful contrasts whose preference direction is known by construction.Source: §3 · p. 3, §3.1 · p. 3

3.1.1 Detail Alteration for Open-Ended Task

  • Open-ended tasks use original responses paired with altered versions containing targeted inaccuracies, preserving plausible style and coherence while introducing errors.The approach assumes the original is preferred even when absolute accuracy cannot be verified; caption prompts also vary requested length and style.Source: §3.1.1 · p. 4

3.1.2 Majority Voting for Closed-Ended Tasks

  • For closed-ended tasks, majority voting selects a likely correct response and pairs it with a random alternative; examples require at least five identical generations.The method uses this construction because changing short answers directly can create trivially easy pairs and descriptive edits do not simulate reasoning errors well.Source: §3.1.2 · p. 4

3.1.3 Correct Answer Filtering

  • An alternative closed-ended strategy pairs the response matching the dataset’s gold answer with one that does not, using external ground truth rather than model consensus.The passage cautions that gold labels may not align with what the model can reliably distinguish.Source: §3.1.3 · pp. 4–5

3.2 Training Data Generation

  • At each iteration, the current judge generates judgments and reasoning traces for synthetic pairs, and training retains judgments that correctly identify the constructed preference.Pairs are checked in both response orders to reduce positional-bias cases; the authors describe learning from reasoning only when the answer is correct.Source: §3.2 · p. 5

3.3 Training

  • Supervised fine-tuning on the curated set maximizes the likelihood of generating correct reasoning traces and judgments from the image, question, and candidate responses.The output contains both the reasoning trace and final judgment.Source: §3.3 · p. 5

4 Experimental Setups

Experiments use capped single-image subsets of LLaVA-OneVision and evaluate judges on two multimodal reward benchmarks. Training runs iteratively until performance plateaus.

4.1 Dataset

  • The synthetic training set draws 100k prompts from single-image LLaVA-OneVision subsets, capped at 10k examples per sub-dataset for balanced category coverage.The source spans reasoning, math and coding, and captioning tasks.Source: §4.1 · p. 5

4.2 Evaluation

  • Evaluation uses Multimodal RewardBench and VL-RewardBench, covering general instructions, hallucinations, correctness, reasoning, and visual question answering.Multimodal RewardBench contains 5,211 preference pairs; VL-RewardBench combines human-verified and AI-annotated labels.Source: §4.2 · pp. 5–6

4.3 Implementation details

  • The Llama-3.2-11B judge is fine-tuned for five epochs per iteration, and training stops after less than 1% relative improvement over three consecutive iterations on either benchmark.The experiments observe diminishing returns after iteration 4, though dimensions can peak at different iterations.Source: §4.3 · p. 6

5 Results

Iterative training improves the 11B judge on both reward benchmarks, with the larger gain on VL-RewardBench. Improvements vary by dimension: general instruction following and hallucination improve strongly, while some MMRB dimensions change little.

5.1 Iterative Training Improvements

  • VLRB accuracy rises from 0.383 at base to 0.538 at iteration 4 (40.5% relative gain), while MMRB accuracy rises from 0.499 to 0.539 (7.5% relative gain).Improvements continue across iterations, although trajectories vary by dimension.Source: §5.1 · p. 6
  • On general instruction following, the model scores 0.503, ahead of Llama-3.2-90B (0.426), Claude-3.5-Sonnet (0.434), and GPT-4o (0.491).On VLRB hallucination and MMRB VQA, it scores 0.514 and 0.689, respectively, approaching larger models despite weaker base scores of 0.304 and 0.565.Source: §5.1 · p. 6

5.2 Performance Gains Across Dimensions

  • VLRB general improves 69% (0.297 →0.503) and hallucination improves 40.9% (0.304 →0.514); reasoning gains 8.6% (0.549 →0.596 at iteration 4).Reasoning peaks at iteration 3 (0.612), then declines, suggesting possible overfitting in later iterations.Source: §5.2 · pp. 6–7
  • MMRB VQA improves 18.0% (0.565 →0.689), while reasoning rises 2.7% (0.517 to 0.531), general evaluation stays nearly flat (0.590 →0.591), and safety changes from 0.317 to 0.329.The authors suggest distributional mismatch between synthetic data and supervision may explain the marginal gains in these dimensions.Source: §5.2 · p. 7

6 Analysis and Discussion

The discussion reports increasing retained training data and reasoning quality across iterations, while noting that gains differ by task and data-filtering strategy. It also identifies strong VLRB general and hallucination results, weaker MMRB general and safety outcomes, and limitations tied to image diversity and safety-focused data.

6 Analysis and Discussion

  • The authors frame self-improvement as a promising route to multimodal judges without costly human annotations or stronger teacher models.The analysis examines iteration scaling, majority-vote filtering, and dimensions with the strongest and weakest gains.Source: §6 · p. 7

6.1 Scaling number of iterations

  • Retained training data rises from 19% at baseline to 43% by iteration 4, with the largest gains early; later iterations also produced superior reasoning in 55% of sampled comparisons versus 10% for early iterations.Average improvement is 10% across VLRB and MMRB in the first iteration and 3% per iteration thereafter, with task-specific rates and variability.Source: §6.1 · p. 7

6.2 Analysis on reasoning / VQA: filtering with majority votes v.s.correct answers

  • For closed-ended reasoning and VQA, majority-vote pairs yield 8.6% and 9.5% average improvement over correctness-filtered pairs on VLRB reasoning and MMRB VQA.The authors report that matching the majority set's data size to the correct-answer set reduces its performance, but a gap remains; correct preferred answers do not necessarily ensure sound judgment reasoning.Source: §6.2 · p. 8
  • Majority voting avoids ground-truth labels and, according to the authors, can scale to image datasets without pre-existing answers.Although ground-truth filtering improves performance in this setup, majority voting filters samples more efficiently and achieves greater gains.Source: §6.2 · p. 8

6.3 Analysis on other task dimensions

  • The 11B judge outperforms Llama-3.2-90B and Claude-3.5-Sonnet on VLRB general, with strong hallucination results; the authors say these gains reflect signals in real-world preference datasets.MMRB general remains relatively flat: after four iterations, VisitBench correctness rises from 0.54 to 0.64, while NoCaps correctness declines or plateaus from 0.58 to 0.52; the authors suggest image-domain mismatch as a possible explanation.Source: §6.3 · pp. 8–9
  • MMRB safety improvement is inconsistent; the authors attribute this likely to not explicitly generating biased or toxic training pairs and leave comprehensive safety-focused synthesis for future work.They note that safety evaluation requires specialized guardrails and red-teaming beyond this method's scope.Source: §6.3 · p. 9

7 Conclusion

The framework trains VLM judges without human preference annotations through synthetic data generation and iterative refinement. Its compact 11B model is competitive with larger proprietary systems across evaluation dimensions, with strong performance in general instruction following, hallucination detection, and VQA.

7 Conclusion

  • The 11B model is competitive with significantly larger proprietary systems across multiple evaluation dimensions, particularly general instruction following and hallucination detection.The conclusion also identifies VQA as an area of strength.Source: §7 · pp. 9–10
  • Targeted data augmentation in specialized domains such as safety evaluation may provide pathways for further improvement.The framework is described as applicable to emerging visual tasks where ground-truth annotations are scarce or unavailable.Source: §7 · p. 10

Limitations

The general-purpose self-improvement framework has limits in safety evaluation and does not improve all judgment dimensions equally.

Limitations

  • The framework excludes synthetic training data containing biased, toxic, or otherwise harmful content, leaving specialized safety infrastructure outside its scope.The text names red-teaming, adversarial examples, and nuanced policy-violation understanding as components needed for robust safety evaluation.Source: Limitations · p. 10
  • Improvements are substantial in general instruction following, hallucination detection, and VQA, but more limited in complex reasoning and nuanced preference judgments.Mixture-of-experts with domain-specific synthetic data is proposed as a future direction.Source: Limitations · p. 10

A Training example from different iterations

The examples illustrate how judge reasoning changes across iterations and how prompts support comparison and synthetic caption generation. The appendix also specifies judging criteria and detail-alternation instructions.

A Training example from different iterations

  • By iteration 3, the judge identifies the blue-versus-red bus factual error, shifting from iteration 1’s focus on detail and engagement.The example compares assistant responses and judge reasoning across training iterations.Source: Table 3 · p. 14

B Judge Prompt

  • The judge prompt asks for an impartial comparison based on helpfulness, relevance, accuracy, depth, creativity, and detail, followed by a strictly formatted A-or-B verdict.It instructs the judge to avoid position, response-length, and assistant-name biases.Source: B Judge Prompt · pp. 14–15

C Detail Alternation Prompts

  • Detail-alternation prompts request one, two, or one-to-two changed details while preserving style and identity, with some variants specifying where changes should appear.The variants place required changes at the beginning, middle, or near the end of the text.Source: C Detail Alternation Prompts · pp. 15–16

D Generation Meta Prompt for Captions

E Hyperparameters

The appendix reports generation and training settings, majority-vote sampling choices, and approximate compute and cost comparisons.

E Hyperparameters

  • Data generation uses temperature 0.6, top_k=50, and top_p=0.9.Source: E Hyperparameters · p. 18
  • The majority-vote setup samples 16 candidate responses per closed-ended question and retains cases with at least 5 identical responses.The threshold was selected for its balance of data quality and quantity; compared with a threshold of 3, it achieved similar final performance while requiring less training data.Source: E Hyperparameters · p. 18
  • Training uses the llama-cookbook default setup, a learning rate of 1e-5, and 5 epochs.Source: E Hyperparameters · p. 18
  • Data generation and training used approximately 400 H100 GPU hours, compared with estimated costs of $500K-1M for equivalent human annotation or $5K-10K for GPT-4 distillation.Source: E Hyperparameters · p. 18

F Correctness Filter Negative Example

The example shows that correctness-based filtering can retain a judge response with the right choice but faulty reasoning. It motivates consistency-based filtering across varied synthetic pairs, though the authors do not claim this eliminates all imperfect reasoning.

F Correctness Filter Negative Example

  • The judge selects the correct answer but misses the extra zero in Response A, so its reasoning does not identify the decisive factual error.Table 4 illustrates a correct verdict paired with incorrect reasoning.Source: Table 4 · p. 19, F Correctness Filter Negative Example · p. 19
  • Instead, its rationale emphasizes Response B's extra descriptive details and contextual observations rather than the transcription discrepancy.The authors describe this as a case where correctness-based filtering retains problematic reasoning.Source: F Correctness Filter Negative Example · p. 19
  • The authors propose consistency across varied synthetic contrasts as potentially stronger evidence of genuine understanding than a single correct answer.They report empirical advantages for consistency-based filtering, but say they cannot verify that it eliminates all imperfect reasoning; they frame the example as illustrative, not definitive.Source: F Correctness Filter Negative Example · p. 19
Loading 2512.05145v1…