PaperlayerView exact revision on arXiv

Source-linked AI summary

PhyCritic: Multimodal Critic Models for Physical AI

Tianyi Xiong, Shihao Wang, Guilin Liu, Yi Dong, Ming Li, Heng Huang, Jan Kautz, Zhiding Yu

arXiv:2602.11124v1cs.CV

TL;DR

Existing multimodal critics focus mainly on general visual tasks, leaving physical evaluation of perception, causal reasoning, and planning underexplored. PhyCritic uses a two-stage RLVR pipeline with physical warmup and self-referential critic finetuning, achieving strong performance across physical and general multimodal judge benchmarks. Its main scope limitation is the need for ground-truth answers for multimodal prompts.

  • Problem

    Existing critics primarily target general visual domains, while physical AI requires judgments grounded in perception, causal reasoning, and action or plan validity.

  • Method

    PhyCritic combines physical-skill warmup with self-referential critic finetuning, generating an internal prediction before judging candidate responses.

  • Results

    PhyCritic achieves strong gains across physical reasoning and general reward benchmarks, outperforming open-source 7B/8B baselines and reaching 68.0 accuracy on PhyCritic-Bench.

  • Takeaways & Limitations

    Self-referential physical grounding supports more consistent, interpretable judgments and stronger physical reasoning across physical and broader multimodal tasks.

  • Takeaways & Limitations

    Self-referential critic finetuning requires ground-truth answers for multimodal prompts, limiting applicability to fully open-ended scenarios.

Abstract

from arXiv · show

With the rapid development of large multimodal models, reliable judge and critic models have become essential for open-ended evaluation and preference alignment, providing pairwise preferences, numerical scores, and explanatory justifications for assessing model-generated responses. However, existing critics are primarily trained in general visual domains such as captioning or image question answering, leaving physical AI tasks involving perception, causal reasoning, and planning largely underexplored. We introduce PhyCritic, a multimodal critic model optimized for physical AI through a two-stage RLVR pipeline: a physical skill warmup stage that enhances physically oriented perception and reasoning, followed by self-referential critic finetuning, where the critic generates its own prediction as an internal reference before judging candidate responses, improving judgment stability and physical correctness. Across both physical and general-purpose multimodal judge benchmarks, PhyCritic achieves strong performance gains over open-source baselines and, when applied as a policy model, further improves perception and reasoning in physically grounded tasks.

1. Introduction

Physical AI critics must evaluate perception, causal reasoning, and plans against physical constraints, but existing critics largely target general visual domains. PhyCritic addresses this gap with physical grounding, self-referential judging, a dedicated benchmark, and strong results across physical and general multimodal evaluation.

  • Physical AI requires critics to assess causal validity, physical configurations, and temporal, spatial, and dynamical constraints.
  • Its two-stage training first strengthens physical perception and reasoning with GRPO, then evaluates candidate responses against self-generated reasoning and predictions.
  • PhyCritic-Bench evaluates physical judgment using embodied-domain data, paired candidate responses, and verifiable preference labels for reasoning, grounding, and causal validity.
  • PhyCritic grounds evaluation in its own physical perception and reasoning through a self-referential critic learning framework.
  • PhyCritic contributes a physical critic dataset, a dedicated benchmark, and strong gains across physical reasoning and general reward benchmarks.

2. Related Works

Related work spans physical-AI VLMs, multimodal reward models, and reinforcement finetuning. PhyCritic extends these directions by applying RLVR-based critic training to physical perception, action, and planning evaluation.

  • Physical-AI VLM research extends beyond passive perception toward spatial grounding, 3D understanding, action, and interaction.
  • Multimodal reward and critic models evaluate generated responses, commonly using BT-style reward modeling or related critic-training paradigms.
  • Reinforcement finetuning has been applied to multimodal reasoning, perception, video reasoning, medical imaging, GUI agents, and robotic reasoning.
  • PhyCritic extends multimodal RLVR to critics that evaluate physical perception, action, and planning while also improving physical reasoning.

3. PhyCritic: Physical Critic for VLMs

PhyCritic develops physical-critic capabilities through a two-stage RLVR pipeline: physical skill warmup followed by self-referential critic finetuning. The model first solves physical questions, then uses its self-prediction to ground pairwise judgments.

  • Training Data: Training tuples contain a multimodal prompt, two candidate responses, a ground-truth answer, and a binary preference label.These components support both self-prediction and pairwise preference judgment.
  • Two-Stage Training: The two-stage pipeline first builds physical perception and reasoning, then trains judgment over paired responses using ground-truth preferences.Stage 1 uses verifiable question-answer pairs; Stage 2 uses complete prompt, candidate-response, answer, and preference tuples.
  • Self-Referential Critic: Self-referential critic finetuning requires the model to generate an internal prediction before judging two candidate responses.The critic explicitly grounds its evaluation in the previously generated self-prediction.
  • Reward Design: The total reward combines accuracy and format rewards, while accuracy reward includes self-prediction and critic components.The self-prediction reward checks agreement with the ground-truth answer, encouraging reliable problem solving before criticism.
  • Training Data: The critic dataset spans diverse robotics videos, physical scenarios, and candidate responses of varying quality.Sources include RoboVQA, BridgeData V2, HoloAssist, and AgiBot World, with manipulation behaviors and physical contexts represented.

4. PhyCritic-Bench

PhyCritic-Bench evaluates multimodal judges on physical-AI reasoning that existing general visual reward benchmarks do not cover. It combines robotics and autonomous-driving scenarios with paired responses and preference labels.

  • Motivation: PhyCritic-Bench targets physical-AI judgments involving perception, actions, and planning, beyond general visual domains such as captioning and image question answering.The benchmark addresses the limited physical coverage of existing multimodal reward benchmarks.
  • Dataset Distribution: Figure 3 describes the distributions of prompt sources and model responses in PhyCritic-Bench.The left side represents prompt sources and the right side represents model responses.
  • Benchmark Composition: The benchmark contains 225 evaluation samples covering robotics and autonomous-driving scenarios.Robotics data comes from five embodied datasets, while driving questions are drawn from LingoQA.
  • Evaluation Protocol: Each evaluation instance is a pairwise preference tuple containing a multimodal prompt, two candidate responses, and a ground-truth preference label.Response-pair construction follows the JudgeBench pipeline and uses verified correct and incorrect responses.

5. Experiments

Experiments evaluate PhyCritic as a critic and policy across physical and general multimodal benchmarks, then analyze its two-stage RL pipeline and self-referential finetuning. PhyCritic achieves strong physical judgment, transfers to general-domain judging, improves physical reasoning, and benefits from both training stages.

  • Main Results: PhyCritic-Bench accuracy reaches 68.0, outperforming Qwen2.5-VL-7B at 51.6 and RoboBrain2.0-7B at 54.7 among open-source 7B/8B critics.It performs best or tied-best on AgiBot, HoloAssist, and RoboVQA, and generalizes to RoboFail and LingoQA.
  • Main Results: PhyCritic exceeds the Qwen2.5-VL base model by +4.1 on VL-RewardBench and +1.9 on Multimodal-RewardBench.Gains appear across general, hallucination-related, and reasoning-related judgments despite physical-only critic tuning.
  • Main Results: PhyCritic achieves 63.9 on CosmosReason1-Bench, 79.7 average on CV-Bench with the best 3D score of 83.9, and 42.3 on EgoPlanBench2.These results cover physical reasoning, spatial grounding, cross-view consistency, and egocentric planning.
  • Main Results: The two-stage pipeline uses 80 warmup steps and 300 critic-finetuning steps with 4,058 total training samples.The reported setup is presented as a data-efficient alternative to approaches using millions of supervised traces.
  • Ablation Studies: Stage 1 improves CosmosReason1-Bench by +7.5, while Stage 2 adds +14.4 on PhyCritic-Bench and +2.1 on CosmosReason1-Bench.The full pipeline achieves the best results across physical critic judgment, physical reasoning, and general-domain evaluation.
  • Ablation Studies: Removing self-reference lowers PhyCritic-Bench performance from 68.0 to 64.4, while removing the self-prediction reward lowers it to 65.8.The final model’s self-prediction correctness also has a stronger association with downstream judgment correctness than the Stage 1 model.

6. Conclusion

PhyCritic is a physical-AI multimodal critic built with two-stage RLVR and self-referential critic finetuning. It achieves strong open-source performance on physical judgment, transfers to general reward tasks, and improves physical reasoning.

  • Conclusion: PhyCritic targets multimodal evaluation of physical perception, causal reasoning, and action or plan assessment.The model grounds judgments in physically informed self-predictions through a two-stage RLVR pipeline.
  • Conclusion: PhyCritic-Bench evaluates multimodal judging in physical-domain settings alongside experiments on broader multimodal reward tasks and physical reasoning.The paper presents the benchmark as a challenging test of physical-domain critic capability.
  • Conclusion: Experiments report the best performance among open-source 7B/8B models on physical judgment, generalization to broader reward tasks, and enhanced physical reasoning.The conclusion attributes these outcomes to the proposed physical grounding and self-referential critic training.

A. Additional Results and Analysis

Additional analyses test prompt design, reward weighting, test-time best-of-N selection, downstream DPO training, and comparison with a general-domain critic. They show that structured guidance and physical critic training support physical and transferable performance.

  • Prompt and Reward Analysis: The final configuration sets the self-prediction reward weight to α_sp=0.2, which provides the best overall balance across judge and reasoning benchmarks.Performance remains relatively robust across different α_sp choices while the critic reward weight is set to 0.9 − α_sp.
  • Prompt and Reward Analysis: PhyCritic uses explicit criteria such as truthfulness, visual groundedness, logical validity, and efficiency in critic prompts.Removing these criteria causes clear drops in judgment accuracy and physical reasoning, indicating that physical-AI critique requires structured guidance.
  • Test-Time Scaling: +6.5 points at N=32 lifts the base model from 54.3 to 60.8 on CosmosReason1-Bench when PhyCritic selects among reasoning trajectories.The pairwise knockout procedure produces the most consistent gains as N increases.
  • Downstream Policy Training: PhyCritic scores ordered response pairs to create best–worst preferences for DPO training of Qwen2.5-VL-7B-Instruct on Cosmos-RL data.The resulting policy is evaluated on CosmosReason1-Bench as a downstream use of the critic reward.
  • General-Domain Comparison: PhyCritic achieves robust physical-domain gains and transfers critic capacity to general-domain judging despite no general-domain critic training.UnifiedReward-Think performs strongly on general reward benchmarks but shows only marginal physical judgment gains and reduced physical reasoning.

B. Qualitative Examples

Qualitative examples illustrate PhyCritic’s self-referential judgments, physical reasoning, and transfer to general image evaluation. The examples emphasize grounded reasoning, consistency, and reduced reliance on superficial cues.

  • Judging Physical Responses: PhyCritic distinguishes response quality by solving the physical problem first and grounding its critique in that reasoning.A truck-status example contrasts this behavior with Qwen2.5-VL-7B’s focus on superficial structures.
  • Physical Policy Reasoning: As a policy model, PhyCritic produces visually grounded and logically consistent reasoning for selecting the most plausible next physical subtask.The example reports no stepwise contradictions in its reasoning.
  • General Image Judging: PhyCritic transfers its physical critic capability to general image judgments, identifying the tissue box as green and avoiding hallucinated colors.The example states that neither image-domain critic data nor the VL-RewardBench critic prompts were used during training.
Loading 2602.11124v1…