Source-linked AI summary
Improving Vision-language Models with Perception-centric Process Reward Models
Yingqian Min, Kun Zhou, Yifan Li, Yuhuan Wu, Han Peng, Yifan Du, Wayne Xin Zhao, Min Yang, Ji-Rong Wen
TL;DR
RLVR’s sequence-level rewards are too coarse to localize perceptual errors in multi-step VLM reasoning. PERCEVAL grounds image-related claims token by token and improves reasoning performance across domains during training and test-time scaling.
Problem
RLVR’s sequence-level rewards make it difficult to identify and correct localized perception or reasoning errors in multi-step VLM reasoning.
Method
PERCEVAL compares image-related claims with visual evidence, then assigns token-level penalties to hallucinated spans during RL and supports iterative inference refinement.
Results
PERCEVAL consistently outperforms GRPO across model scales and domains, with approximately 4% average improvement in 3B Visual Search and gains over major voting at test time.
Takeaways & Limitations
Perception-centric supervision improves perceptual capabilities and generalizes to broader reasoning tasks, while PRM-based refinement provides effective test-time scaling.
Abstract
from arXiv · showhide
Recent advancements in reinforcement learning with verifiable rewards (RLVR) have significantly improved the complex reasoning ability of vision-language models (VLMs). However, its outcome-level supervision is too coarse to diagnose and correct errors within the reasoning chain. To this end, we propose Perceval, a process reward model (PRM) that enables token-level error grounding, which can extract image-related claims from the response and compare them one by one with the visual evidence in the image, ultimately returning claims that contain perceptual errors. Perceval is trained with perception-intensive supervised training data. We then integrate Perceval into the RL training process to train the policy models. Specifically, compared to traditional GRPO, which applies sequence-level advantages, we apply token-level advantages by targeting penalties on hallucinated spans identified by Perceval, thus enabling fine-grained supervision signals. In addition to augmenting the training process, Perceval can also assist VLMs during the inference stage. Using Perceval, we can truncate the erroneous portions of the model's response, and then either have the model regenerate the response directly or induce the model to reflect on its previous output. This process can be repeated multiple times to achieve test-time scaling. Experiments show significant improvements on benchmarks from various domains across multiple reasoning VLMs trained with RL, highlighting the promise of perception-centric supervision as a general-purpose strategy. For test-time scaling, it also demonstrates consistent performance gains over other strategies, such as major voting. Our code and data will be publicly released at https://github.com/RUCAIBox/Perceval.
1. Introduction
The paper introduces Perceval, a perception-centric process reward model that grounds image-related claims in visual evidence to identify reasoning errors and provide fine-grained supervision. Integrated with token-level advantage reallocation and iterative test-time refinement, it improves perception and general reasoning capabilities.
- Motivation and method: Perceval identifies image–text misalignments by grounding intermediate perceptual claims, such as objects, attributes, and spatial relations, directly in the image.This perception-centric PRM provides fine-grained feedback for hallucinated spans and their visual counter-evidence.
- Method: The method trains Perceval on perception-intensive queries with structured annotations marking hallucinated spans and corresponding visual counter-evidence.Supervised fine-tuning enables the PRM to flag hallucinations arising within multi-step reasoning.
- Results: Compared with direct GRPO, the training method significantly improves perceptual capabilities and generalizes to comprehensive gains in overall reasoning ability.The reported generalization occurs even when PRM supervision is not applied during training for complex reasoning tasks.
- Training: Token-level advantage reallocation integrates Perceval with GRPO to target penalties on perceptually erroneous spans, addressing sparse rewards and credit assignment.The framework supervises intermediate reasoning tokens instead of relying only on sequence-level rewards.
- Inference: An iterative test-time refinement strategy uses Perceval to detect and correct perceptual errors through repeated response truncation, regeneration, or reflection.This enables active correction during inference and supports test-time scaling.
2. Preliminary
This section defines VLMs and RLVR foundations, then identifies sequence-level reward sparsity as a limitation for correcting localized perceptual and reasoning errors. It motivates finer-grained supervision using token-level process rewards and hallucination penalties.
- Vision-Language Models: VLMs generate textual reasoning outputs from image and text inputs using a visual encoder, connection layer, and language model.The output is denoted πθ(o|q, v), with visual features embedded before language decoding.
- Reinforcement Learning with Verifiable Rewards: RLVR trains VLMs to produce plausible outputs that also satisfy measurable criteria such as correctness and spatial consistency.GRPO samples multiple responses for a prompt-image input and uses their rewards to optimize the policy.
- Reinforcement Learning with Verifiable Rewards: GRPO’s sequence-level advantage is constant across all tokens in a response, limiting credit assignment during policy optimization.Its clipped surrogate objective updates the policy using token importance-sampling ratios and the shared response-level advantage.
- Problem Statement: RLVR’s single end-of-chain reward credits tokens equally, making localized perceptual or reasoning errors difficult to correct and weakening robust generalization.The framework addresses this limitation with fine-grained intermediate perceptual supervision and token-level penalties for hallucinated tokens.
3. Methodology
PERCEVAL is a perception-centric process reward model that verifies image-related claims and provides fine-grained supervision for VLM training and inference. It supports token-level correction during RLVR and iterative error repair at test time.
- PERCEVAL Design: PERCEVAL acts as an external, fine-grained, interpretable critic that evaluates response claims against visual evidence.It analyzes each claim in a think-then-answer format and reports whether perceptual errors are present.
- PRM Training: PERCEVAL is trained through query selection, VLM rollout generation, automated hallucination-focused verification, and supervised fine-tuning.The data emphasizes visual-search images while retaining a small proportion of mathematical reasoning and general-understanding examples.
- Training-Time Integration: The RLVR integration replaces sequence-level advantages with token-level advantages that downweight spans realizing perception-induced hallucinations.Correct tokens retain the original signal, while identified hallucination tokens receive reduced or more negative learning signals before insertion into the GRPO objective.
- Test-Time Scaling: At inference, Truncate–then–Regenerate preserves the verified prefix and resamples the response from the first detected erroneous span.The cycle repeats until no errors are flagged or a maximum of k iterations is reached, bounding latency while enabling refinement.
- Test-Time Scaling: Truncate–Thinking–then–Regenerate adds a brief reflection prompt after truncation to encourage correction of the identified object, attribute, or spatial mismatch.It follows the same iterative stopping rule as Truncate–then–Regenerate.
4. Experiment
Experiments evaluate Perceval-trained models across diverse visual reasoning benchmarks and show consistent gains over GRPO in visual search, perception, and math/chart reasoning. Perceval also supports effective test-time scaling and fine-grained hallucination suppression through token-level penalties.
- Experimental setup: Experiments cover visual search, perception-intensive reasoning, mathematical reasoning, and chart-based reasoning across multiple multimodal benchmarks.The benchmark suite includes V*, RealWorldQA, MathVista, MATH-Vision, and ChartQA.
- Experimental setup: Qwen2.5-VL serves as the backbone for matched 3B and 7B Perceval reward and policy models.Perceval is trained at both 3B and 7B scales before training same-sized policy models.
- RL training with PRM: Our method outperforms GRPO consistently, with 3B average gains of approximately 4% in Visual Search, 3% in Math and Chart reasoning, and 1% in Perception-intensive Reasoning.The strongest Visual Search improvement is on V* Positional Perception, rising from 86.95 to 90.43 at 3B.
- RL training with PRM: Perception improvements transfer to broader reasoning tasks because Math and Chart benchmarks require fine-grained abilities such as localizing chart data points and reading text.The results also show gains across general perception and math reasoning domains.
- Test-time scaling with PRM: Perceval-based test-time scaling is evaluated against major voting on the 3B policy model using Truncate and Feedback strategies.The comparison includes the proposed Truncate–then–Regenerate and Truncate–Thinking–then–Regenerate methods.
- Hyperparameter tuning: The α penalty strength must balance hallucination suppression against response quality, and experiments across α ∈ {0.03, 0.1, 0.3} show a non-monotonic trend against GRPO.The study identifies α = 0.03 as insufficient, motivating selection of an optimal intermediate penalty strength.
5. Related Work
Related work spans the evolution of vision-language models from representation alignment to multimodal reasoning, the adoption of reinforcement learning for VLM capability incentives, and multimodal reward models for preference alignment and RLVR.
- Vision-language Models: VLMs progressed from contrastive pre-training on web-scale image-text pairs to LVLMs combining visual encoders with language models.CLIP and ALIGN established transferable representations, enabling later large vision-language models.
- Reinforcement Learning for VLMs: Reinforcement learning research increasingly targets complex multimodal reasoning by combining visual perception with logical deliberation.Emergent slow-thinking behaviors in LLMs helped motivate this direction in VLMs.
- Multimodal Reward Models: Multimodal reward models align model outputs with human preferences and increasingly support RLVR for complex reasoning tasks.They are especially useful when verifiable ground truth is inaccessible, including open-ended settings.
6. Conclusion
PERCEVAL is a perception-centric process reward model that addresses sparse rewards in RLVR through token-level grounding of image–text errors. Trained on perception-intensive data, it is integrated into both VLM training and inference.
- Contribution: PERCEVAL addresses sparse rewards in RLVR by grounding errors at the token level.It provides grounded, step-aware feedback rather than relying only on outcome-level supervision.
- Contribution: PERCEVAL detects image–text misalignments within a model’s reasoning process.This enables feedback tied to specific reasoning steps.
- Contribution: PERCEVAL is trained with perception-intensive data and integrated into VLM training and inference.The paper applies the model at both stages of the VLM pipeline.