Visual Preference Optimization with Rubric Rewards

1 Introduction

• Earlier preference-pair pipelines often produce off-policy data that can drift from the target model’s generation behavior.They use response perturbations or visual-input perturbations such as diffusion noise and image editing.Source: §1 · p. 1
• On-policy methods reduce distribution mismatch, but many still use limited-granularity rewards that miss grounding, completeness, and reasoning quality.Source: §1 · p. 1
• Large proprietary judges are expensive at scale, while open-source judges often rely on fixed prompts and coarse scores or rankings.The resulting feedback can lack transparency and fine-grained signals for reasoning errors or hallucinations.Source: §1 · pp. 1–2
• rDPO creates instance-specific essential and additional criteria for each image-instruction pair and reuses an offline instruction-rubric pool during on-policy data construction.The contribution list specifies a 300K instruction-rubric pool for generating target-policy-specific preference data.Source: §1 · p. 2
• Rubric-based prompting improves the judgment quality of a 30B-A3B open-source judge on public reward-modeling benchmarks without additional training.Source: §1 · p. 2
• In method validation, rDPO raises the macro average to 82.69 from 81.14; in scaling validation, it scores 61.01, above the style-constrained baseline (52.36) and base model (59.48).Outcome-based filtering reduces the method-validation macro average from 81.14 to 75.82.Source: §1 · p. 2

2.1 Reward Models for Multimodal Alignment

• Multimodal reward models have advanced, but often produce scalar scores or use rigid prompts that lack transparent, multidimensional feedback for nuanced visual reasoning.Examples include Prometheus-Vision [29], SIMA [30], and later multimodal reward models such as MM-RLHF-Reward [28] and Skywork-VL Reward [34].Source: §2.1 · p. 3

2.2 Rubric-Grounded Evaluation and Alignment

• Rubric-based evaluation and alignment are established in LLM settings, with work on customized criteria, structured rewards, contrastive rubric generation, and dynamic refinement.Examples include Rubricsas-Rewards [47], Rubicon [48], CARMO [37], and OnlineRubrics [51].Source: §2.2 · p. 3
• Multimodal rubric alignment remains early: JudgeAnything [54] and Multi-Crit [55] use instance-specific checklists, while Omni-RRM [40] uses fixed rubric templates.rDPO instead generates instance-specific criteria for preference-pair mining.Source: §2.2 · p. 3

3.1 DPO

• DPO uses preference triplets and a gradient that scales updates by the reward margin, increasing the chosen response’s likelihood while penalizing the rejected response.The reference model initializes the policy, and β is the KL penalty coefficient.Source: §3.1 · p. 3

3.2 MPO

• MPO augments the DPO objective with an auxiliary SFT loss that reinforces the chosen response and increases its gradient weight to help prevent policy collapse.The SFT term omits length normalization so its magnitude remains consistent with the DPO term.Source: §3.2 · p. 4

3.3 Rubric

• rDPO scores on-policy responses against an instance-specific rubric and returns a multidimensional feedback vector for transparent, fine-grained filtering and preference-pair construction.The rubric can include criteria such as factual accuracy and instruction following.Source: §3.3 · p. 4

4 Methodology

• The instruction-rubric pool is built through model-agnostic seed mining and rubric drafting, then reused across different target models.Preference-data curation samples on-policy responses for a specific target policy.Source: §4 · p. 4

4.1 Seed Mining

• Seed mining retains instructions where an ensemble of models disagrees, then samples uniformly across original data sources to maintain domain balance.The ensemble spans dense and Mixture-of-Experts architectures.Source: §4.1 · p. 4

4.2 Rubric Drafting

• Rubric criteria are designed to be atomic, comprehensive, precise, and objectively verifiable, and are divided into essential and additional criteria.Each check item pairs an observable criterion with an image-grounded reference and a weight from 1 (Auxiliary) to 3 (Key).Source: §4.2 · p. 5
• Frontier reasoning models generate instance-specific checklist rubrics, using trusted annotations when available and excluding noisy or missing answers otherwise.Multiple models’ candidate rubrics are synthesized into a unified checklist to mitigate single-model bias and perception errors.Source: §4.2 · p. 5

4.3 Preference Data Curation

• For each instruction, the target policy supplies multiple on-policy responses, which form a diverse candidate set for preference-pair construction.Source: §4.3 · p. 5

4.3.1 Reward Modeling

• A zero-shot rubric-grounded judge assigns each criterion a score of 0, 0.5, or 1; the criterion scores are aggregated using their weights into an overall reward.The judge scores each response three times independently, and the median score per criterion is used.Source: §4.3.1 · p. 5

4.3.2 Pair Mining

• Preference pairs require a qualified chosen response, a flawed rejected response, and a reward margin of at least δ; up to four largest-margin pairs are selected per instruction, with each response used at most twice.Chosen responses must receive full credit on most essential criteria, allowing at most one partial score; rejected responses must fail an essential criterion or receive partial credit on at least two.Source: §4.3.2 · p. 6

5.1 Evaluation Results of Reward Modeling

• 74.83 macro average: rubric prompting raises the Qwen3-VL-30B-A3B judge from 71.73 and brings it close to GPT-5.4 (74.91).Rubric prompting also raises Qwen3-VL-32B from 73.90 to 75.05; chain-of-thought prompting does not consistently help.Source: §5.1 · pp. 6–7

5.2 Evaluation Results of Preference Optimization

• 82.69 macro average: rubric filtering improves on the base model’s 81.14, while outcome filtering falls to 75.82.Rubric filtering also improves all three reported tasks over the base model.Source: §5.2 · p. 7, Table 2 · p. 7
• 61.01 average: rDPO exceeds the style-constrained baseline (52.36) and base model (59.48) in scaling validation.The comparison evaluates the comprehensive benchmark under concise-response constraints.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Ablation Study

• On MMPR distribution-shift ablations, SFT-to-DPO converges most stably, with the lowest final loss and highest reward accuracy among three paradigms.The authors attribute standard DPO’s slow convergence and unstable gradients to a mismatch between its reference policy and off-policy data.Source: §5.3 · p. 8
• β = 0.5 gives the reported optimal KL-penalty trade-off, with a substantial penalty anchoring the policy to the reference distribution.The authors say this mitigates policy collapse and implicit-reward overfitting.Source: §5.3 · p. 8
• SFT-loss effects vary by task: it often worsens results and may overfit low-complexity samples, but can help tasks where the base model has undesirable priors.Counting is the example given; there, SFT loss constrains the model toward a high-quality rejection-sampled distribution.Source: §5.3 · p. 8
• Iterative DPO on three sequential data splits scores 61.01 versus 59.27 for single-stage DPO.The authors attribute the improvement to progressively updating the reference model to calibrate the implicit reward.Source: §5.3 · p. 9, Table 4 · p. 9
• Essential/additional qualification raises average performance from 58.36 to 61.01; maximum reward-margin pairs score 61.01 versus 59.96 and 60.76, while removing diversity control yields 59.34.These ablations respectively examine criterion qualification, reward-margin selection, and response-frequency limits.Source: §5.3 · p. 9

Conclusion

• rDPO separates offline instruction-rubric pool construction from rollout, scoring, and on-policy data curation.The framework is built on instance-specific rubric-based reward modeling.Source: §6 · p. 9
• On the reported benchmarks, rubric prompting improves Qwen3-VL judges, rubric filtering raises the method-validation macro average to 82.69, and rDPO scores 61.01 in scaling validation.The scaling result exceeds the 52.36 style-constrained baseline and 59.48 base model; the 30B-A3B judge reaches 74.83 versus GPT-5.4’s 74.91.Source: §6 · p. 9

Limitations

• The current pipeline uses criteria to construct preference pairs, leaving criterion-level feedback outside iterative DPO’s optimization loop.The authors identify tighter integration with online optimization, including GRPO, as future work.Source: §6 · p. 9

A.1 Output JSON Schemas

• Rubric generation and response scoring return structured JSON, enabling reliable parsing and programmatic aggregation and filtering.The rubric schema has essential and additional arrays; each criterion includes a criterion, reference, and weight.Source: §A.1 · p. 17
• Rubric criteria use a three-level weight for importance, while score outputs preserve rubric order and include criterion, rationale, and credit.Credit is 0, 0.5, or 1, and is independent of weight; weighted aggregation is performed externally.Source: §A.1 · p. 17

A.2 Rubric Generation

• Generation prompts create an instance-specific checklist for an image-question pair, with atomic, comprehensive, precise, and objective criteria.The criteria serve as the reward model’s sole evaluation standard.Source: §A.2 · p. 18
• When trustworthy ground truth is available, expert-grounded generation verifies that it satisfies every essential criterion before finalizing the rubric.For noisy or absent annotations, answer-agnostic generation omits this verification to avoid anchoring on potentially unreliable references.Source: §A.2 · p. 18
• Rubrics distinguish essential criteria, prerequisites for conceptually sound answers, from optional additional criteria covering relevant image facts, supplementary knowledge, or reasoning steps.Criteria are concrete assertions with references and importance weights; the prompt cautions against double-counting and over-decomposition.Source: §A.2 · p. 18

A.3 Rubric Aggregation

• Aggregation merges independently generated candidate checklists, retaining majority-supported items, removing duplicates or irrelevant checks, and verifying references.It also applies the same four construction principles used during generation.Source: §A.3 · p. 19

A.4 Response Scoring

• Response scoring evaluates each candidate against the finalized rubric and assigns criterion-level credit on a three-point scale.The prompt directs the judge to use the checklist as its sole evaluation standard; each score includes a rationale.Source: §A.4 · p. 20

Plane-count case

• For the plane-count question, both responses earn full credit for identifying three visible planes; additional criteria separately check aircraft identities.The checklist also distinguishes top-left, middle, and bottom aircraft.Source: B Case Study of Reward Modeling · p. 20

Door-status case

• For the door question, both responses correctly identify the open white gate, but the rubric separately checks the closed black door and left-building door.The examples assign different credit for an omitted additional door criterion: one response receives 0 and the other 0.5.Source: B Case Study of Reward Modeling · p. 21

Architect case

• For the architect question, the essential criteria separately require identifying Blenheim Palace and Sir John Vanbrugh; additional criteria cover Hawksmoor and English Baroque style.The examples distinguish the building’s name from its primary architect.Source: B Case Study of Reward Modeling · p. 21

Allicin case

• The allicin example’s essential criterion asks for C6H10OS2, while additional criteria distinguish the pictured molecule’s formula, name, and structure.The checklist describes the pictured molecule as distinct from allicin.Source: B Case Study of Reward Modeling · p. 22

• The molecule examples assess whether responses distinguish allicin from the pictured structure and provide the corresponding formula, name, or structural difference.One reference gives allicin’s formula as C6H10OS2; another identifies the pictured molecule as C5H8OS2 and distinguishes its vinyl group from allicin’s second allyl group.Source: TabMWP · p. 22
• The rubric checks both the non-red marble probability and its calculation; the correct result is 4/11, while a red-marble miscount yields 8/11.The reference identifies 11 total marbles and 4 green, non-red marbles.Source: TabMWP · p. 22
• Tinder at 48% is the largest chart share, but the Spanish flag may indicate Spain rather than the Mexico asked about.The rubric also calls for listing Bumble (29%), Badoo (9%), Happn (8%), and Grindr (6%) in descending order.Source: TabMWP · p. 23
• For the fusion diagram, U is the neutron; the rubric also checks why U is a neutron and whether the response identifies nuclear fusion.The reference describes U as an uncharged purple sphere and the reaction as deuterium and tritium fusing to form helium and a neutron.Source: TabMWP · p. 23

• Other revisions identify a cobblestone path absent from the earlier description and replace two incorrect movie identifications with “Catch Me If You Can.”The revised book-cover response instead says the image is likely a blush from the FREEDOM series.Source: C Case Study of Preference Optimization · p. 24
• The bottle-count answer changes from 10 to 14 after the image count is revised from five bottles to seven.The revised response counts three bottles on the left, one in the middle, two on the right, and one partly behind the fruit basket.Source: C Case Study of Preference Optimization · p. 24
• The sign translation changes from “Do not trample on the flowers” to “Please care for the flowers, grass, and trees.”Source: C Case Study of Preference Optimization · p. 24
• The parking-space answer changes from 006 to 007; the revised response says space 006 is next to 007 and that the driver is parked within the green marked area.Source: C Case Study of Preference Optimization · p. 25
• The bus identification changes from Liuzhou to Guilin, with the revised answer citing the Guilin Bus Group and the displayed plate.Both answers describe a Route 16 bus.Source: C Case Study of Preference Optimization · p. 25
• The crawfish judgment changes from no obvious back-cut marks to visible cuts exposing the vein and meat.The revised response concludes that the crawfish have been back-cut.Source: C Case Study of Preference Optimization · p. 25