PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Visual Contrastive Reasoning with Multi-Branch Exploration

Kegeng Tang, Jingbo Wang, Yangming Zhang, Huanshu Zhang, Shaogang Ren, Zihao Wang

arXiv:2610.36809v1cs.CVcs.CL

TL;DR

Complex visual question answering requires locating and comparing multiple pieces of relevant evidence, but latent visual reasoning often follows a single path that may blur competing hypotheses. LVCR generates multiple question-conditioned latent branches and trains them with bbox-guided contrastive alignment. On perception-intensive benchmarks, it improves over corresponding backbone and single-path latent reasoning baselines at both 3B and 7B scales.

  • Problem

    Complex visual question answering requires locating and comparing question-relevant evidence, while single-path latent visual reasoning can be limiting when multiple hypotheses or competing evidence regions matter.

    Source: §I · p. 1

  • Method

    LVCR proposes multiple question-conditioned visual branches and uses bounding-box evidence supervision with a soft contrastive objective to align relevant branches against negative regions.

    Source: §I · pp. 1–2

  • Results

    At both 3B and 7B scales, LVCR improves over backbone models and single-path latent reasoning baselines on perception-intensive visual reasoning benchmarks.

    Source: §I · p. 2, §V · p. 9

  • Takeaways & Limitations

    The results suggest structured multi-branch latent reasoning is a promising direction for visually grounded latent-space reasoning.

    Source: §V · p. 9

Abstract

from arXiv · show

Multimodal large language models have made substantial progress in visual question answering, but complex visual reasoning often requires locating and comparing multiple pieces of question-relevant visual evidence. Existing methods that move intermediate reasoning into the visual latent space usually follow a single linear reasoning trajectory, which can be limiting when multiple visual hypotheses or competing evidence regions are relevant. We propose Latent Visual Contrastive Reasoning, a branch-aware latent reasoning framework that generates multiple question-conditioned latent visual branches as soft hypotheses over image tokens. To train these branches, our method uses bounding-box evidence annotations to build target evidence representations and introduces a soft contrastive branch-evidence alignment objective. This objective encourages evidence-relevant branches to align with annotated visual regions while contrasting them against negative evidence. We instantiate our method on Qwen2.5-VL backbones and train it with standard supervised fine-tuning. Experiments on perception-intensive visual reasoning benchmarks show that the proposed method achieves overall improvements over the corresponding backbone and single-path latent reasoning baselines at both 3B and 7B scales.

I. INTRODUCTION

Complex visual question answering requires locating and comparing fine-grained, question-relevant evidence, while existing latent visual reasoning often follows one path that can blur competing hypotheses. LVCR uses multiple question-conditioned branches, box-guided contrastive alignment, and ranked branch injection to address this limitation.

I. INTRODUCTION

  • Language-space chain-of-thought represents intermediate reasoning as text while the image is typically encoded once as a static condition.Other visual-operation approaches can use external modules, hand-designed operations, or repeated image re-encoding, while still expressing reasoning largely in language.Source: §I · p. 1
  • A single latent path can be limiting when questions involve multiple plausible objects, visually similar candidates, or evidence across regions.Forcing alternatives into one path may blur distinct hypotheses and make representations less discriminative.Source: §I · p. 1
  • LVCR represents multiple question-conditioned visual hypotheses in parallel rather than following a single latent reasoning trajectory.Each branch encodes a candidate evidence state that can be injected as a latent visual reasoning token.Source: §I · pp. 1–2
  • Bounding-box supervision guides a soft contrastive objective that aligns branches with answer-relevant evidence and contrasts them against negative regions.The annotations are used as weak supervision to make the branch space evidence-aware.Source: §I · pp. 1–2
  • LVCR ranks branches by question relevance and injects their tokens into the decoder for branch-aware answer generation.This lightweight mechanism orders the branches before they are combined with the original multimodal context.Source: §I · pp. 1–2
  • At both 3B and 7B scales, LVCR improves over backbone models and single-path latent reasoning baselines on perception-intensive visual reasoning benchmarks.The reported evaluation focuses on perception-intensive benchmarks.Source: §I · p. 2

II. RELATED WORK

Related work spans multimodal visual reasoning, language- and region-grounded reasoning, and latent-space methods, while LVCR combines these directions in a branch-aware pipeline. Its branches are proposed by question-conditioned attention over visual tokens before evidence supervision and decoder injection.

A. Multimodal LLM for Visual Reasoning

  • Multimodal LLMs connect visual encoders with language models, and progress in instruction tuning and general-purpose modeling has strengthened visual question answering and compositional reasoning.Fine-grained questions may still require locating relevant evidence, comparing similar candidates, and reasoning over attributes or spatial relations.Source: A. Multimodal LLM for Visual Reasoning · p. 2

B. Chain-of-Thought and Region-Grounded Visual Reasoning

  • Chain-of-thought, region-grounded, and tool-augmented approaches expose visual evidence, but often rely on textual rationales, external tools, hand-designed operations, or repeated visual processing.These methods include box-grounded Visual CoT annotations and region-level or spatially grounded reasoning.Source: B. Chain-of-Thought and… · p. 2

C. Latent-Space Reasoning and Visual Alignment

  • Latent visual reasoning moves intermediate reasoning into visual embeddings, while contrastive alignment pulls positive pairs together and separates negative samples; structuring multiple supervised hypotheses remains underexplored.Existing latent visual reasoning work includes generating query-relevant visual states, while continuous-thought methods also explore latent states.Source: C. Latent-Space Reasoning and Visual… · p. 2

III. METHOD

  • LVCR encodes an image-question pair, proposes multiple latent branches, aligns them with answer-relevant evidence against irrelevant regions, then ranks and injects selected branches into the decoder.The decoder combines branch tokens with the original multimodal context to generate the answer.Source: Figure 1 · p. 3, §III · p. 3
  • During training, answer-relevant bounding-box annotations form positive evidence representations, and a soft contrastive loss aligns matched branches while contrasting negative regions.The model is optimized with an objective combining answer prediction and bbox-guided contrastive evidence alignment.Source: §III · p. 3

A. Question-Conditioned Branch Proposal

  • Each branch uses a learnable query to attend over visual tokens with question-conditioned scores, producing an attention-weighted candidate evidence state.Different branches can specialize in different question-conditioned evidence patterns while remaining differentiable.Source: A. Question-Conditioned Branch Proposal · p. 3

B. BBox-Guided Evidence Representation

Bounding-box annotations provide localized evidence supervision for constructing a target representation. Pooling visual tokens inside the annotated box yields the positive evidence that a suitable branch should align with.

B. BBox-Guided Evidence Representation

  • Answer-relevant bounding boxes identify the visual evidence used to construct a target representation for contrastive supervision.The formulation can use localized evidence regions from any dataset or supervision source, rather than requiring a specific dataset.Source: B. BBox-Guided Evidence Representation · p. 3
  • The target positive evidence representation pools visual tokens that fall inside the annotated bounding box.A good branch proposal module should produce at least one branch strongly aligned with this representation.Source: B. BBox-Guided Evidence Representation · p. 4

C. Contrastive Branch-Evidence Alignment

The alignment objective softly distributes annotated-evidence supervision across branches, then contrasts the resulting positive representation with negative evidence. This preserves differentiable credit assignment without requiring a hard branch selector.

C. Contrastive Branch-Evidence Alignment

D. Differentiable Branch Credit Assignment

Soft evidence compatibility lets multiple branches share credit during training, while answer prediction injects all branches as latent tokens for decoding. At inference, question similarity orders the branches without removing alternatives.

D. Differentiable Branch Credit Assignment

  • Evidence compatibility gives more weight to better-matched branches while keeping all branches in the positive representation and avoiding discrete labels or selection.The negative term also discourages branches from indiscriminately matching irrelevant visual regions.Source: D. Differentiable Branch Credit… · p. 4

E. Answer Prediction

  • All K branches enter reserved <lvcr> token positions, and decoder self-attention integrates them without a separate fusion module.The decoder predicts the answer autoregressively.Source: E. Answer Prediction · p. 4
  • At inference, branches are ordered by descending cosine similarity to the question representation, prioritizing the most question-relevant branch while preserving all branches for complementary context.Source: E. Answer Prediction · p. 4

F. Ranked Latent Sequence Construction

LVCR converts its unordered branch set into a deterministic sequence by ranking branches for the question. The sequence retains all hypotheses, allowing the decoder to use the strongest and complementary evidence together.

F. Ranked Latent Sequence Construction

G. Training Objective

LVCR trains branches to answer questions while aligning them with bounding-box evidence. Its combined objective keeps answer prediction primary and uses contrastive alignment to regularize localized evidence representations.

G. Training Objective

  • The final objective combines standard answer-prediction loss with bbox-guided contrastive branch-evidence alignment.The answer loss remains primary, while the NCE loss regularizes branches toward localized answer-related visual evidence.Source: G. Training Objective · p. 5
  • The training procedure encodes image and question tokens, constructs branches, maps the bounding box to a visual-token mask, and forms positive and negative evidence.Source: G. Training Objective · p. 5
  • Training inserts all branches for answer prediction and updates parameters using answer loss plus λ_nce-weighted contrastive loss; inference ranks branches by question similarity before decoding.Source: G. Training Objective · p. 5

H. Implementation Details

LVCR adds question-conditioned latent branches to the existing visual-token sequence, using bounding-box supervision during training without requiring annotations at inference. Its design keeps branch evidence soft, orders branches before decoder injection, and adds bounded structural computation.

H. Implementation Details

  • Bounding-box masks pool positive evidence from selected visual tokens, while negatives come from low-overlap regions or other images; annotations and contrastive loss are training-only.At inference, branches are constructed, ranked, and injected from the image–question pair alone.Source: H. Implementation Details · pp. 5–6
  • The decoder processes the K reserved branch positions alongside the original multimodal context, without an additional branch-fusion network.Training updates the language model and LVCR parameters while keeping the vision encoder and multimodal merger frozen.Source: H. Implementation Details · p. 5

I. Computational Complexity and Efficiency

  • Branch proposal and scoring require O(KNd) computation and O(KN) attention weights; branch-evidence assignment and contrastive similarities add O(Kd), and ranking costs O(K log K).Adding K latent tokens changes the decoder attention term by O((2LK+K^2)d); the analysis describes structural overhead, not a wall-clock speedup.Source: §I · p. 6

J. Design Discussion

  • Multiple learnable, question-conditioned branches preserve candidate evidence configurations in parallel instead of compressing them immediately into one latent state.Soft assignment avoids arbitrary branch identities while retaining gradients for partially aligned branches; question-similarity ranking puts the most relevant branch first while retaining complementary branches.Source: J. Design Discussion · p. 6

IV. EXPERIMENTS

Experiments use Qwen2.5-VL-Instruct backbones at 3B and 7B scales, trained with answer prediction and contrastive branch-evidence alignment. Evaluation covers perception-intensive multiple-choice benchmarks under a shared deterministic protocol and compares against backbone and reasoning baselines.

A. Experimental Setup

  • Training uses 417,453 VISUAL COT image-question-answer examples, each with one bounding-box annotation used to construct positive evidence for the alignment loss.The dataset is formatted and filtered from VISUAL COT.Source: A. Experimental Setup · p. 6
  • LVCR is built on Qwen2.5-VL-Instruct at 3B with K = 4 branches and 7B with K = 8, jointly optimizing answer prediction and contrastive alignment.The vision encoder and multimodal merger remain frozen during fine-tuning.Source: A. Experimental Setup · p. 6
  • At inference, all K branches are ranked by cosine similarity to the question representation and inserted before answer generation; the ablations test unranked and top-1 injection.Top-1 uses only the highest-ranked branch under the same similarity criterion.Source: A. Experimental Setup · p. 6
  • Evaluation covers V ∗Bench, MMVP, and BLINK, using exact-match accuracy after answer extraction and direct option-letter responses.V ∗Bench includes fine-grained detail search and relative spatial reasoning; MMVP tests subtle visual distinctions, while BLINK includes counting, IQ Test, Jigsaw, Relative Reflectance, and Spatial Relation.Source: A. Experimental Setup · p. 7
  • Comparisons include the corresponding Qwen2.5-VL backbone, Vanilla SFT, and LVR at both scales, with additional multimodal reasoning baselines at 3B and 7B.All methods use the same prompting, deterministic decoding, and answer-extraction protocol.Source: A. Experimental Setup · p. 7

B. Main Results and Analysis

Across perception-intensive benchmarks, LVCR improves over the corresponding backbone and single-path latent reasoning baselines at both model scales, although results vary by benchmark and competing method. Ablations also examine the contrastive alignment objective and branch-injection strategies.

B. Main Results and Analysis

  • At 3B, LVCR improves the overall V ∗ score and both V ∗ subsets over Qwen2.5-VL-3B, and improves most metrics over single-path LVR.The reported LVR gains are especially noted on V ∗, V ∗ R.P., Jigsaw, and Spatial Relation.Source: B. Main Results and Analysis · p. 7
  • At 7B, LVCR is best or tied-best on V ∗ D.A., MMVP, Counting, IQ-Test, and Jigsaw, and improves over LVR on MMVP, Counting, IQ-Test, Jigsaw, and Spatial Relation.The paper reports second-best performance on several other subsets.Source: B. Main Results and Analysis · p. 7
  • Removing the NCE loss causes consistent performance drops on V ∗, MMVP, and BLINK in the LVCR-3B, K = 4 ablation.The comparison is between the default λNCE = 0.1 setting and training without NCE loss.Source: Figure 2 · p. 7
  • The authors interpret the results as evidence that preserving multiple latent visual hypotheses before answer generation can improve perception-intensive reasoning.They describe branch-aware reasoning as complementary rather than uniformly superior on every benchmark.Source: B. Main Results and Analysis · p. 7
  • LVCR is not uniformly best: Vision-R1 performs strongly on MMVP and V ∗ R.P. at 3B, while other methods lead on some reported metrics.The paper also notes that Vanilla SFT improves some individual metrics but is less stable across benchmarks and scales.Source: B. Main Results and Analysis · p. 7
  • LVCR reports broad improvements over backbone and single-path latent reasoning baselines across perception-intensive subsets at both 3B and 7B scales.The main results are reported in the 3B and 7B comparison tables.Source: B. Main Results and Analysis · p. 7, Table I · p. 8, Table II · p. 8

C. Ablation Studies

Ablations show that contrastive branch-evidence alignment and relevance-ranked multi-branch injection both support LVCR’s visual reasoning. The conclusion reports consistent gains over competitive baselines at 3B and 7B scales.

C. Ablation Studies

  • Removing NCE drops LVCR-3B from 75.4 to 72.3 on V* and from 54.0 to 52.9 on BLINK, with a smaller decrease on MMVP.The ablation sets the NCE-loss coefficient to zero while keeping the remaining training configuration unchanged.Source: C. Ablation Studies · p. 8
  • Without contrastive alignment, branches receive weaker guidance about which visual regions to specialize in and may be less discriminative and less consistently evidence-aligned.The gains are especially visible on V* and BLINK, which require fine-grained visual grounding and compositional perception.Source: C. Ablation Studies · p. 8
  • The default inference strategy ranks all K branches by cosine similarity to the question representation and injects them in descending relevance order, rather than using only Top-1 or all branches unranked.Ranked performs best across V*, MMVP, and BLINK.Source: C. Ablation Studies · p. 8
  • On V*, Ranked reaches 75.4% accuracy, beating Top-1 by 1.1 points and Unranked by 3.2; on BLINK, the gaps are 1.2 and 3.8 points.The comparison suggests that using one branch discards complementary cues, while injecting all branches without ranking is weaker than relevance-based ordering.Source: C. Ablation Studies · p. 8

V. CONCLUSION

  • At both 3B and 7B scales, LVCR consistently improves over competitive baselines, using bounding-box supervision and a soft contrastive objective to align branches with answer-relevant evidence.At inference, it ranks branches by question relevance and injects selected latent tokens into the decoder for answer prediction.Source: §V · pp. 8–9

Related papers

Loading 2610.36809v1…