PaperlayerView exact revision on arXiv

Source-linked AI summary

Perceptual Flow Network for Visually Grounded Reasoning

Yangfu Li, Yuning Gong, Hongjian Zhan, Teng Li, Yuanhuiyi Lyu, Tianyi Chen, Qi Liu, Ziyuan Huang, Zhihang Zhong, Dandan Zheng, Yue Lu

arXiv:2605.02730v1cs.CVcs.AI

TL;DR

LVLMs can hallucinate because standard optimization does not adequately constrain visual trajectories, while expert geometric priors may be too precision-focused for reasoning. PFlowNet uses self-conditioned perceptual flows with variational reinforcement fine-tuning, combining multi-dimensional rewards and vicinal geometric shaping. It provides a provable performance guarantee and reports strong gains across visual reasoning benchmarks, including 13.1%, 10.4%, and 21% improvements over Qwen3-VL 8B on V* Bench, TreeBench, and MME-RealWorld-lite.

  • Problem

    Expert-derived geometric priors can emphasize geometric precision over reasoning utility, while unconstrained visual trajectories contribute to LVLM hallucination.

  • Method

    PFlowNet decouples perception from reasoning through self-conditioned perceptual-flow generation and variational reinforcement fine-tuning with multi-dimensional rewards and vicinal geometric shaping.

  • Results

    13.1%, 10.4%, and 21% improvements over Qwen3-VL 8B are reported on V* Bench, TreeBench, and MME-RealWorld-lite, respectively, alongside a provable performance guarantee.

  • Takeaways & Limitations

    PFlowNet supports reasoning-oriented yet visually reliable perceptual behaviors across general-purpose and fine-grained visual tasks.

  • Takeaways & Limitations

    Theoretical guarantees rely on idealized assumptions and regularity conditions that may not strictly hold in practice, while hyperparameter optima can vary across models and domains.

Abstract

from arXiv · show

Despite the success of Large-Vision Language Models (LVLMs), general optimization objectives (e.g., standard MLE) fail to constrain visual trajectories, leading to language bias and hallucination. To mitigate this, current methods introduce geometric priors from visual experts as additional supervision. However, we observe that such supervision is typically suboptimal: it is biased toward geometric precision and offers limited reasoning utility. To bridge this gap, we propose Perceptual Flow Network (PFlowNet), which eschews rigid alignment with the expert priors and achieves interpretable yet more effective visual reasoning. Specifically, PFlowNet decouples perception from reasoning to establish a self-conditioned generation process. Based on this, it integrates multi-dimensional rewards with vicinal geometric shaping via variational reinforcement learning, thereby facilitating reasoning-oriented perceptual behaviors while preserving visual reliability. PFlowNet delivers a provable performance guarantee and competitive empirical results, particularly setting new SOTA records on V* Bench (90.6%) and MME-RealWorld-lite (67.0%).

1 Introduction

LVLMs remain vulnerable to interpretability and hallucination problems, while expert-derived geometric supervision can favor precision over reasoning utility. PFlowNet addresses this mismatch with structured perceptual flows, decoupled reasoning, and variational reinforcement fine-tuning.

  • LVLMs still face interpretability and hallucination challenges, especially in complex fine-grained visual understanding.
  • Expert-derived geometric priors anchor reasoning in visual evidence but may be biased toward geometric precision rather than downstream reasoning utility.
  • Preliminary Study: The most precise expert annotation is not the most helpful reasoning prior because it can exclude context needed for comprehensive understanding.
  • Heuristic transformations of expert priors are difficult because the optimal geometric prior is highly instance-specific.
  • PFlowNet approximates idealized perceptual behaviors with a self-parameterized variational distribution and self-conditions subsequent reasoning on sampled flows.
  • PFlowNet combines structured perceptual-flow trajectories, decoupled perception and reasoning, and multi-dimensional rewards with vicinal geometric shaping.
  • 13.1%, 10.4%, and 21% improvements over Qwen3-VL 8B are reported on V* Bench, TreeBench, and MME-RealWorld-lite, respectively.

2 Background and Motivation

Visually grounded reasoning treats visual trajectories as latent variables that should remain valid while supporting correct answers. PFlowNet replaces rigid expert imitation with variational perceptual flows and reasoning-aware optimization.

  • Problem Formulation: MLE-based LVLM optimization leaves visual reasoning trajectories insufficiently constrained, allowing invalid latent trajectories to contribute to hallucination.
  • Problem Formulation: RLVR-based methods use geometric priors from visual experts as verifiable rewards to constrain visual trajectories.
  • Problem Formulation: Visually grounded reasoning requires correct answers while anchoring latent visual rationales to a golden trajectory within the valid support S_v.
  • Problem Formulation: Because visual experts target grounding rather than downstream reasoning, their synthetic trajectories can emphasize geometric precision and yield suboptimal reasoning policies.
  • Revisit VGR as Reasoning over Perceptual Flow: PFlowNet uses a self-parameterized variational distribution to approximate the target posterior and introduces Perceptual Flow to characterize LVLM visual behavior.
  • Revisit VGR as Reasoning over Perceptual Flow: A Perceptual Flow is a structured latent trajectory containing a planning state and grounded perceptual states with regions of interest and captions.
  • PFlowNet: PFlowNet samples a perceptual flow from its intrinsic distribution and generates a grounded answer through self-conditioned generation.
  • Revisit VGR as Reasoning over Perceptual Flow: Sub-Trajectory Balance supplies dense intermediate supervision for optimizing diverse perceptual behaviors across flow sub-trajectories.

3 Perceptual Flow Network

PFlowNet learns perceptual flows and uses them to self-condition grounded output generation, combining synthetic trajectory training with variational reinforcement fine-tuning. Its reward and vicinal geometric shaping encourage useful exploration while retaining visual reliability, with theoretical guarantees under stated assumptions.

  • Architecture: PFlowNet factorizes generation into perceptual-flow sampling followed by grounded output generation conditioned on the flow and its image region.The joint distribution is pθ(Y, Z | X) = pθ(Z | X) pθ(Y | Z, ⟨X, IRoI⟩).
  • Training Data Curation & Cold Start: The training pipeline synthesizes fine-grained trajectories, filters them with verifiers, and splits samples into cold-start and reinforcement-fine-tuning sets.The split uses the performance gain induced by synthetic grounding behaviors.
  • Training Data Curation & Cold Start: Cold-start supervised fine-tuning teaches the policy to generate synthetic perceptual flows that benefit downstream reasoning.The policy minimizes cross-entropy between pθ(Z | X) and each synthetic flow Zs.
  • Variational Reinforcement Fine-tuning: Variational reinforcement fine-tuning combines a multi-dimensional reward with vicinal geometric shaping to optimize perceptual behaviors without strict expert-prior alignment.The reward evaluates flow quality and reasoning efficacy, while shaping penalizes samples outside a vicinity around the expert prior.
  • Theoretical Analysis: Under the paper’s assumptions, PFlowNet has a total-variation performance bound relative to the target posterior and can strictly tighten bounds for standard MLE and expert-guided RLVR.The guarantee requires calibration of intensity λ and radius ε, with Bε ⊆ SV.
  • Theoretical Analysis: The theoretical limits show that λ→0 removes geometric constraints, whereas ε exceeding tolerance σ includes invalid regions and degrades performance.Increasing ε within the valid region improves coverage and tightens the bound.

4 Experiment

Experiments evaluate PFlowNet across general-purpose and fine-grained tasks, efficiency, scaling, qualitative reasoning, and ablations. Results show strong accuracy, reasoning-oriented perceptual behavior, favorable efficiency, and gains across backbone scales.

  • General-purpose Tasks: 10.4% on TreeBench and 18.4% on MME-RealWorld-Lite improve over Qwen3-VL-8B, while PFlowNet surpasses nearest competitors by 5.3% and 12.6%, respectively.It achieves SOTA on 17 of 19 sub-tasks.
  • Fine-grained Visual Understanding: 13% on V, 8%/8.8% on HR-Bench 4K/8K, and 2.5%–7% on ScreenSpot exceed the Qwen3-VL series on fine-grained tasks.PFlowNet matches Qwen3-VL-32B on several challenging results, including 90.6 vs. 87.4 on V*.
  • Performance-Efficiency Trade-off: PFlowNet balances performance and efficiency by replacing complex agentic tool or code executions with structured perceptual flows.The reported comparison uses latency and averaged token-level context length as efficiency measures.
  • Qualitative Comparison: PFlowNet first localizes evidence and then expands its visual scope, supporting precise yet comprehensive reasoning through geometric shaping.The shaping energy relaxes as the sequence lengthens, facilitating broader reasoning.
  • Character-level Output Length: Generated flow lengths remain consistent for the same number of RoIs across benchmarks and are generally shorter than synthetic flows.The passage attributes this partly to grounded captions becoming more concise and discriminative.
  • Framework & Reward Design: Removing the perceptual flow causes severe degradation, whereas external fine-grained features provide only marginal improvements.The flow acts as an explicit semantic anchor conditioning autoregressive generation.
  • Cross-Scale Evaluations: 2.2% and 1.3% average SFT gains occur for Qwen3-VL 4B and 32B, while RFT adds improvements exceeding 2.3%.These gains span general-purpose and fine-grained visual understanding tasks.

5 Related Work

Related methods support visual reasoning through either multi-turn tool execution or geometric alignment with external expert priors. These paradigms motivate comparison with PFlowNet’s structured perceptual flow.

  • Agentic Frameworks: Agentic frameworks use multi-turn tool or code executions to provide dynamic image manipulation for visual reasoning.Thyme enables code-based “thinking with images,” while other systems use external sandbox tools or sketching workspaces.
  • Grounded RLVR: Grounded RLVR regularizes reasoning by maximizing geometric consistency between intermediate visual rationales and external expert priors.Representative methods interleave normalized bounding-box coordinates or points with stepwise reasoning and optimize IoU-related objectives.

6 Conclusion

The paper concludes that structured perceptual flows, reinforcement fine-tuning, and vicinal geometric shaping support high-quality, interpretable visual reasoning. Theory and experiments indicate strong task performance, efficiency, and test-time scaling.

  • Conclusion: PFlowNet uses structured perceptual flows and a tailored reward function with vicinal geometric shaping to explore valid, reasoning-oriented perceptual behaviors.The framework is presented as enabling high-quality and interpretable visual reasoning.
  • Conclusion: Formal analysis establishes a provable performance guarantee, while experiments show superiority across general-purpose and fine-grained tasks.The empirical analysis also highlights performance-efficiency balance and robust test-time scaling.

Limitations and Future Work

The paper’s theoretical guarantees rely on idealized assumptions and regularity conditions that may not strictly hold in practice. Its analysis formalizes perceptual flows, model distributions, vicinal support, and reward-induced posteriors.

  • Limitations: Theoretical bounds depend on idealized assumptions and regularity conditions on valid support that may not strictly hold in practice.The authors present the bounds as qualitative guidance supported by empirical validation.
  • Limitations and Future Work: PFlowNet lacks adaptive perception because it uses a fixed structured reasoning format across questions with different types and difficulty.The paper identifies adaptive visual reasoning as future work.
  • Flow Formalization: A perceptual flow is a finite trajectory beginning at a planning state and containing perceptual states composed of RoIs and captions.The notation represents each state as z_k = ⟨r_k, c_k⟩ for k ≥ 1.
  • Model Factorization: PFlowNet parameterizes a variational distribution over flows and a conditional generator for responses, using forward and backward sub-trajectory probabilities.The model does not assume a specific architecture beyond inducing a forward transition kernel over states.
  • Structural Hypotheses: The analysis assumes the planning state is deterministically determined by the input and uses deterministic crop operators for RoIs.These structural hypotheses support the subsequent theoretical analysis.
  • Vicinal Support: The framework defines valid flows through vicinal support around a conceptual golden RoI trajectory and uses ε-vicinity constraints on terminal RoI sets.The reference RoI set E determines the vicinity B_ε(E).
  • Reward-Induced Posterior: Geometric shaping induces an exponentially tilted posterior P_λ, while the valid-support posterior P_V serves as the target distribution for total-variation analysis.The shaping weight is one inside B_ε(E) and e^-λ outside it.

A.2 Derivation of Variational Objective

The derivation instantiates the SubTB objective for tree-structured perceptual flows and extends it to empirical data with expert-annotated regions. Under stated assumptions, minimizing the variational loss aligns the optimal policy with a reward-defined posterior.

  • Flow construction: The scalar flow at each state is defined as terminal reward divided by the termination probability.This connects the flow formulation to the shaped reward used by the policy.
  • Variational objective: The SubTB objective sums squared log-ratios over valid perceptual-flow sub-trajectories.Forward transitions are products of conditional policy probabilities, while backward transitions collapse to unity under deterministic traversal.
  • Empirical objective: The empirical objective is formed over training tuples containing inputs, answers, and expert-annotated regions of interest.The derivation incorporates the data distribution and RoI annotations into the variational reinforcement fine-tuning objective.
  • Posterior factorization: Under uniform support and ordering priors, the posterior factors into caption likelihood and perceptual efficacy terms.The support prior, deterministic initial state, ordering prior, caption sequence, and efficacy components are explicitly identified.
  • Posterior matching: Under the stated assumptions and an expressive policy, globally minimizing the variational loss yields an optimal policy proportional to the shaped reward and target posterior.The result follows from posterior matching induced by the variational objective.

A.4 Proofs

The proofs derive a total-variation bound for PFlowNet and analyze how reward intensity and vicinity radius affect it. With valid vicinities and calibrated intensity, the bound is strictly tighter than the limiting MLE and expert-guided RLVR baselines.

  • Variation-distance bound: The policy-to-posterior total-variation distance is bounded under an expressive model and a valid support vicinity.The bound is derived under the stated regularity and support assumptions.
  • Limiting regimes: As λ→∞, the bound is bottlenecked by expert bias and degenerates toward expert-guided RLVR.The limiting expression is governed by the coverage term associated with the expert-supported vicinity.
  • Limiting regimes: As reward information becomes uninformative, geometric guidance vanishes and the bound reduces toward the MLE regime.The MLE limit corresponds to λ→0.
  • Guaranteed improvement: A calibrated intensity λ⋆ strictly improves the bound over both limiting baselines in the non-degenerate interior regime.The strict inequalities require q∈(0,1) and sV∈(0,1).
  • Radius effect: Within the valid regime, increasing vicinity radius tightens the bound through a monotonic increase in the coverage ratio q.If the radius exceeds the valid scale, the vicinity includes invalid regions and can dilute guidance and degrade performance.

B.1 Dataset

The dataset section describes a multi-stage training and inference pipeline for PFlowNet, including corpus curation, supervised bootstrapping, variational reinforcement fine-tuning, and self-conditioned reasoning. Efficient reward and transition computation uses teacher forcing and parallelization with explicit position indices and attention masks.

  • Dataset: The training corpus aggregates multimodal VQA sources and is filtered to produce 95k visual-centric question-answer pairs.A subsequent quality-control pipeline processes generated perceptual trajectories into 45k high-quality SFT samples.
  • Training: PFlowNet is initialized from Qwen3-VL-8B-Instruct and trained first on SFT samples before variational reinforcement fine-tuning.The RFT stage uses 42k samples for 5 epochs.
  • Inference: Self-conditioned reasoning feeds the generated perceptual flow and zoomed-in visual evidence back into the model before final response generation.The same system prompt is used across training and self-conditioned reasoning stages.
  • Reward calculation: Teacher forcing provides reward-model outputs while transition probabilities are computed from autoregressive token log-probabilities.The computation uses sampled perceptual relations and includes transition, termination, and quality-reward components.
  • Efficient computation: Customized position indices and attention masks compute shared-prefix sub-flow terms in a single forward pass.Figures 12 and 13 illustrate parallel strategies for terminal probability and efficacy reward computation.

C.1 Benchmarks and metrics

The evaluation covers general-purpose VQA and fine-grained grounded reasoning, emphasizing perception, robustness, high-resolution understanding, and explicit evidence localization. PFlowNet is compared with general LVLMs, agentic systems, GUI grounding models, and grounded RLVR methods under standardized evaluation conditions.

  • Evaluation scope: The evaluation reports results on 15 benchmarks spanning general-purpose VQA and fine-grained grounded reasoning.The two evaluation angles separate broad multimodal capability from high-resolution and region-level evidence use.
  • Fine-grained benchmarks: V* Bench tests fine-grained visual search involving small targets, attributes, and spatial relations.Its queries typically require identifying the correct evidence region.
  • Fine-grained benchmarks: TreeBench jointly evaluates answer correctness and evidence localization quality with perception and reasoning taxonomies.The taxonomy supports analysis of whether methods improve perception behaviors, reasoning behaviors, or both.
  • Baselines: The comparison includes general-purpose LVLMs across scales and proprietary frontier models when benchmark protocols permit.These models provide backbone-strength controls and upper-bound references for general VQA and robustness.
  • Baselines: Grounded RLVR baselines optimize explicit spatial tokens toward evidence localization and answer correctness.Representative methods include TreeVGR, Pixel-Reasoner, ZoomRefine, and DyFo.
  • Evaluation protocol: Baseline results are reproduced with official pipelines and standardized infrastructure for consistent latency and memory comparisons.Standard evaluations use greedy decoding, while test-time scaling uses stochastic sampling.

D.1 Analysis of Test-Time Scaling Behaviors

Under test-time scaling, TreeVGR exhibits severe mode collapse, while PFlowNet explores more diverse and reliable perceptual regions. This diversity helps explain PFlowNet’s stronger scaling behavior.

  • Figure 14 presents qualitative grounding comparisons linking TreeVGR’s lack of latent-variable diversity to limited effective likelihood gains as computational budget increases.
  • TreeVGR’s bounding boxes overlap almost entirely across reasoning paths as computational budget increases, indicating severe perceptual mode collapse.
  • PFlowNet produces significantly more diverse Regions of Interest across multiple samples under test-time scaling.
  • TreeVGR can hallucinate and attend to featureless background regions, while its collapsed policy limits self-correction.

D.2 Analysis of Failure Case

The failure-case analysis identifies a trade-off between geometric reliability and fine-grained counting, and a lack of explicit supervision for planning states. Additional qualitative results highlight important visual-reasoning steps.

  • Figures 15–17 present additional qualitative visual-reasoning results with important reasoning steps highlighted.
  • PFlowNet’s reliable, diverse bounding boxes may merge adjacent regions, causing counting errors through bias from the number of boxes in the perceptual flow.
  • The perceptual flow strongly primes subsequent reasoning, so adding fine-grained visual features alone cannot fully mitigate counting errors.
  • The planning state lacks explicit supervision and relies on passive optimization through the sub-flow level Efficacy term in reward Equation (3).
  • In challenging out-of-distribution scenarios, planning decomposition may fail and propagate confusing perceptual behaviors and incorrect reasoning processes.
Loading 2605.02730v1…