PaperlayerView exact revision on arXiv

Source-linked AI summary

CPPO: Contrastive Perception Policy Optimization for VLM Agents

Ahmad Rezaei, Mohsen Gholami, Saeed Ranjbar Alvar, Kevin Cannons, Mohammad Asiful Hossain, Zhou Weimin, Yong Zhang, Mohammad Akbari

arXiv:2601.00501v2cs.CV

TL;DR

VLM reinforcement learning must improve perception as well as reasoning, but existing approaches may require extra models, supervision, or forced separation of the two. CPPO adds a self-supervised contrastive loss to RL and selects perception tokens through entropy shifts under perturbed images. It outperforms recent RL methods across math and visual reasoning benchmarks, though it has not yet been evaluated in actual agent harnesses.

  • Problem

    Final-answer RL rewards do not separate perception errors from reasoning errors, while prior perception-reward approaches can require extra models, annotations, or forced token separation.

    Source: §1 · p. 2

  • Method

    CPPO adds an unsupervised token-level contrastive loss to RL and uses entropy increases under information-removing image perturbations to identify perception tokens.

    Source: §1 · p. 2

  • Results

    CPPO outperforms recent VLM RL methods and achieves state-of-the-art performance across multiple math and visual reasoning benchmarks.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors present self-supervised, perception-aware RL as a promising building block for VLM agents, whose unsafe behavior can stem from perception failures.

    Source: §1 · p. 1, §5 · p. 9

    CPPO was not evaluated inside actual agent harnesses such as web or computer-use environments.

    Source: G. Limitations · p. 14

Abstract

from arXiv · show

We introduce CPPO, a Contrastive Perception Policy Optimization method for finetuning vision--language models (VLMs). Reliable perception is a core requirement for VLM-based agents that must reason and act in open-ended environments: faulty visual grounding cascades directly into faulty actions, hallucinated tool calls, and unsafe decisions. While reinforcement learning (RL) has significantly improved reasoning in language models, extending these advances to multimodal agents requires improving both perception and reasoning. Prior works address this challenge mainly through explicit perception rewards, which often require extra LLM judges, ground-truth annotations, or forced separation of perception from reasoning. CPPO addresses this limitation in a self-supervised manner by extending the RL objective with a Contrastive Perception Loss (CPL) that provides a direct learning signal for visual grounding. The contrastive objective encourages the model to become more sensitive to input visual information. To apply this signal effectively, CPPO identifies perception tokens using an entropy-shift mechanism in the model's output distributions under perturbed images and applies the contrastive loss selectively to those tokens during training. Experiments show that CPPO surpasses prior methods while avoiding extra models, making training more efficient and scalable, and yielding policies that are better suited to perception-critical agentic tasks.

1. Introduction

Reliable VLM agents need visual grounding, but final-answer RL rewards do not distinguish perception errors from reasoning errors. CPPO adds self-supervised contrastive feedback to RL and selects perception-dependent tokens using the policy’s output distributions.

1. Introduction

  • Final-answer rewards conflate perception and reasoning errors, so inaccurate visual tokens can spoil answers despite correct reasoning.Source: §1 · p. 1
  • Prior approaches use forced perception/reasoning tags or LLM and CoT-based perception rewards, bringing task-flexibility, overhead, or scalability concerns.Source: §1 · p. 2
  • CPPO detects perception tokens by selecting response tokens whose entropy increases most under an information-removing image perturbation.Source: §1 · p. 2
  • CPL uses an unsupervised contrastive loss on perception tokens from correct rollouts, without extra CoT supervision or proprietary models.Source: §1 · p. 2
  • CPPO outperforms prior perception-specific RL methods across math and visual reasoning benchmarks.Source: §1 · p. 3

2. Related Work

Related VLM RL work includes rollout augmentation, combinations with SFT or off-policy data, and perception-aware approaches. CPPO is positioned in the perception-aware line, using token-level contrastive learning and image perturbations during RL training.

2. Related Work

  • Sampling and rollout-augmented methods mix clean and distorted-image trajectories or improve rollout selection; this line is orthogonal to CPPO.Source: §2 · p. 3
  • Other orthogonal approaches combine on-policy RL with off-policy CoT or SFT data to support rethinking, iterative pipelines, or training stability.Source: §2 · p. 3
  • Perception-aware methods use decoupled architectures, tagged perception/reasoning outputs, or unsupervised KL losses, with trade-offs involving extra models, supervision, or instability.Source: §2 · p. 3
  • CPPO applies token-level contrastive loss during RL to improve perception, a VLM policy-optimization use of contrastive learning described as unexplored.Source: §2 · p. 3
  • Unlike training-free, post-hoc, or vision-encoder-level perturbation methods, CPPO uses image perturbations in RL and applies loss to entropy-identified perception tokens.Source: §2 · p. 4

3. Method

GRPO provides a verifiable-reward baseline, but correctness rewards alone give no explicit signal for perceptual sensitivity. CPPO adds a contrastive perception loss and selectively applies it to image-dependent tokens.

3.1. Preliminaries

  • GRPO samples response trajectories for query-image inputs, assigns rewards, and computes relative advantages; its correctness reward alone does not explicitly improve perceptual sensitivity.The objective also uses a clipped policy update and a KL penalty against a frozen reference policy.Source: §3.1 · p. 4

3.2. CPPO: Contrastive Perception Policy Optimization

  • CPPO contrasts token distributions across the original image, an information-preserving perturbation, and an information-removing perturbation using an InfoNCE loss.Minimizing the loss keeps the original-image distribution close to the preserving view and pushes it away from the removing view, without CoT annotations.Source: §3.2 · pp. 4–5
  • Tokens with the largest entropy increases under information removal are treated as most perception-dependent, and CPL is applied selectively to them.The entropy increase serves as a proxy for dependence on query-relevant visual content; the method retains the top-k tokens.Source: §3.2 · p. 5
  • CPL is gated to trajectories with positive advantage, so it reinforces successful perceptions while avoiding incorrect trajectories.The contrastive-loss weight λ controls its strength.Source: §3.2 · p. 6

4. Experiments

Experiments compare CPPO with GRPO and perception-aware baselines, and analyze perturbations, training cost, and ablations. CPPO improves on the reported baselines, while perturbation and token-selection analyses identify training choices associated with stronger performance.

4.1. Experimental Setup

  • Training uses ViRL39K's 38.8K multimodal question–answer pairs, with evaluation on avg@8 benchmarks spanning math, general multimodal reasoning, and logical reasoning.ViRL39K covers domains from grade-school problems to STEM, social topics, charts, diagrams, tables, documents, and spatial relationships.Source: §4.1 · p. 6

4.2. Main Results

  • Against GRPO, CPPO gains 11.2% and 5.9% in average absolute performance for 3B/7B, and 6.0% and 3.7% in average relative gains, respectively.The authors report higher accuracy across all benchmarks for both model sizes.Source: §4.2 · p. 7
  • CPPO beats prior methods across all 3B benchmarks and all but WeMath at 7B; versus PAPO, avg@8 is 40.0% vs. 38.1% (3B) and 48.2% vs. 46.8% (7B).The comparison uses the same ViRL39K data and training-step count for CPPO and PAPO.Source: §4.2 · p. 7, Table 2 · p. 7
  • On Geometry3K-trained Qwen2.5-VL-3B, CPPO shows faster learning and stronger out-of-distribution generalization from early training; paired benchmark tests yield p-values ≤0.02.Figure 3 compares in-domain reward and four out-of-domain visual reasoning benchmarks across RL training steps.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3. Analysis of Selected Perturbations

  • Information-preserving augmentations reduce average performance by less than 1.5%, while information-removing perturbations lower average accuracy by over 14%; a more diverse perturbation set provides a richer learning signal and better performance.The perturbation validation tests modified images across four benchmarks; the training-impact analysis uses Geometry3K.Source: §4.3 · pp. 8–9

4.4. Complexity Analysis

  • CPPO takes 39% more time than GRPO for two epochs, yet CPPO@2 epochs averages 40.5% versus GRPO@4 epochs at 39.5%, despite GRPO's 100% longer training time.The comparison uses identical resources for the two-epoch runs, then adds two GRPO epochs on ViRL39K.Source: §4.4 · p. 9, Table 5 · p. 9

4.5. Ablations

  • With λ = 0.02, CPPO achieves 38.6% average accuracy; CPPO across tested λ values generally outperforms GRPO's 34.7% average accuracy.λ controls the strength of perceptual grounding.Source: §4.5 · p. 9, Table 6 · p. 9
  • Average accuracy rises as top-k detection expands from 5% to 50%, then declines toward 100%; the authors hypothesize larger k includes less-informative, already-confident tokens and slows training.The reported comparison holds training epochs constant.Source: §4.5 · p. 9, Table 7 · p. 9

5. Conclusion

CPPO finetunes VLMs with entropy-based perception-token identification and a self-supervised Contrastive Perception Loss. The paper reports state-of-the-art performance across multiple math and visual reasoning benchmarks.

5. Conclusion

  • CPPO is a perception-aware reinforcement-learning method for finetuning VLMs.It uses entropy shifts to distinguish perception tokens from reasoning tokens and applies an unsupervised, model-free Contrastive Perception Loss to penalize perception errors.Source: §5 · p. 9
  • CPPO outperforms recent VLM reinforcement-learning methods across multiple math and visual reasoning benchmarks.The paper characterizes its results as state of the art across these benchmarks.Source: §5 · p. 9
  • The authors view self-supervised, perception-aware reinforcement learning as a promising building block for agents operating reliably in open-ended environments.They connect this motivation to perception failures as a primary driver of unsafe behavior in vision–language agents.Source: §5 · p. 9

Impact Statement

The impact statement frames perceptual grounding as a safety lever because visual misreads can propagate into wrong actions and unsafe outcomes. The proposition treats increased token entropy under an information-removing image perturbation as a proxy for vision dependence, under an explicit mutual-information assumption.

Impact Statement

  • Improving perceptual grounding is presented as a safety lever because scene misreads can propagate into wrong actions and unsafe outcomes.Source: Impact Statement · p. 10
  • The impact statement reports no specific risks beyond those associated with the broader development of capable multimodal agents.Source: Impact Statement · p. 10

A. Proof for Proposition 1

  • An increase in a token’s predictive entropy under a perturbed image is treated as a proxy for that token’s dependence on query-relevant visual content.The perturbation removes query-relevant visual information; the proposition’s assumption is that perception tokens have greater conditional mutual information with the original image than with the perturbed image.Source: A. Proof for Proposition 1 · p. 13

B. Analysis on Performance of Perception Token Detection

The entropy-based detector’s ROUGE-1 F1 score improves as more detected perception tokens are included, with a significant gap over random selection. The evaluation uses GPT5-mini-separated outputs as a proof-of-concept reference rather than a definitive benchmark.

B. Analysis on Performance of Perception Token Detection

  • The evaluation uses Qwen2.5-VL-3B and -7B outputs on MathVista-MINI, LogicVista, MathVision-MINI, and WeMath, with GPT5-mini separating perception information.GPT5-mini’s extracted perception information serves as the reference for measuring detection accuracy with ROUGE-1 F1.Source: B. Analysis on Performance of… · p. 13
  • Because GPT5-mini’s separation is imperfect, the detection evaluation is a proof-of-concept rather than a definitive benchmark.Source: B. Analysis on Performance of… · p. 13
  • The entropy-based detector has a significant ROUGE-1 F1 advantage over random token selection, with scores improving as top-k coverage rises to all positive-∆H tokens.Figure 5 presents the quantitative perception-token detection evaluation; “100%” means selecting all tokens with positive ∆H, not all output tokens.Source: Figure 5 · p. 13, B. Analysis on Performance of… · p. 13

C. Extra Training Details

The appendix points to a table summarizing the training hyperparameter configurations for the 3B and 7B models.

C. Extra Training Details

D. Ablation on Main Components of CPPO

The ablation reports progressively higher accuracy with perception-token selection and advantage gating; the perturbation details distinguish information-removing from information-preserving image changes.

D. Ablation on Main Components of CPPO

E. Image Perturbation Details

  • Information-removing perturbations, including random occlusion and zoom crop, eliminate key visual details; preserving perturbations modify images without discarding critical information.Color jitter, random perspective, random rotation, and Gaussian blur are listed as information-preserving perturbations.Source: E. Image Perturbation Details · p. 14
  • Figure 6 shows samples of the information-removing and information-preserving perturbations, and Table 10 lists selected perturbation parameters.The prose identifies the perturbation types and states that Table 10 reports the selected parameters for each.Source: Figure 6 · p. 14, E. Image Perturbation Details · p. 14, Table 10 · p. 14

F. Qualitative Results

Across three qualitative examples, CPPO corrects perception mistakes observed in models trained with GRPO.

F. Qualitative Results

  • CPPO corrects perception errors in three examples; Figure 7 replaces GRPO’s misread 70-degree angle with the claim that the two angles sum to 180 degrees.The passage notes that wrong perception tokens can yield a wrong final answer even with a correct reasoning trajectory.Source: F. Qualitative Results · p. 14

G. Limitations

The evaluation is bounded by model scale, baseline choice, training-set size, and standard multimodal reasoning benchmarks, without tests in agent harnesses.

G. Limitations

  • Experiments covered 3B and 7B models; exploring larger VLMs such as 72B models is left for future work.The authors attribute the model-size boundary to computational constraints.Source: G. Limitations · p. 14
  • Evaluation used Qwen2.5-VL baselines, with comparisons against other baselines such as InternVL left for future work.Source: G. Limitations · p. 14
  • The study demonstrated CPPO using 40K training samples; larger-scale training with substantially larger datasets remains for future investigation.Source: G. Limitations · p. 14
  • Evaluation covered standard multimodal reasoning benchmarks, not actual web, computer-use, or robotic-style agent harnesses.Reduced hallucinated actions in such settings is a hypothesis awaiting empirical evaluation.Source: G. Limitations · p. 14
Loading 2601.00501v2…