Source-linked AI summary

ReSight-SMC: Two-Stage Power Sampling via Island SMC with Visual Scouts

Yaowen Zhang, Xiangyu Qiu, Junyi Hu, Zhi Lu, Wenwen Tian, Aoqin Wang, Junhai Luo, Zhenming Peng

arXiv:2609.34905v1cs.CV

TL;DR

Power sampling is underexplored for open-ended LVLM decoding, where finite-particle exploration and answer-level trajectory competition remain challenges. ReSight-SMC combines isolated SMC islands, visual scouts, and answer-marginal sharpening; across reasoning and perception benchmarks it improves aggregate performance over Power-SMC.

  • Problem

    In LVLM decoding, global resampling can collapse genealogies, particle-based sampling lacks visual-cue diversification, and sequence-level weights make trajectories supporting the same answer compete.

    Source: §1 · p. 2
  • Method

    ReSight-SMC uses ancestry-isolated SMC islands and prefix-conditioned visual scouts with exact importance correction, then powers terminal mass aggregated by canonical answer and samples a supporting trajectory.

    Source: §1 · p. 2
  • Results

    Across four LVLM backbones and five benchmarks, ReSight-SMC achieves stronger aggregate performance than Power-SMC on both reasoning and perception tasks.

    Source: §1 · p. 2, §6 · p. 10
  • Takeaways & Limitations

    Without post-training, ReSight-SMC remains competitive in aggregate with backbone-matched models trained using reinforcement learning.

    Source: §1 · p. 2

    The answer readout depends on a task-appropriate canonicalizer; semantically equivalent free-form strings remain separate without an external equivalence model or judge.

    Source: §5.4 · p. 10

Abstract

from arXiv · show

Power sampling has emerged as a training-free approach to LLM reasoning, eliciting capabilities comparable to reinforcement learning by sharpening the model distribution over complete responses. Despite this success, power sampling remains underexplored in large vision-language models (LVLMs). We transfer Power-SMC to LVLM decoding by defining a sequence-power target conditioned on both the image and the prompt. This direct transfer provides a strong training-free baseline, but leaves two aspects of finite-particle multimodal inference unaddressed. At the particle level, global resampling can collapse genealogies, while particle-based power sampling does not diversify trajectories through distinct visual cues in multimodal decoding, limiting exploration under a finite particle budget. At the answer level, sequence-level sharpening makes distinct reasoning trajectories compete even when they support the same answer. We introduce ReSight-SMC, a verifier-free two-stage power sampler for LVLM inference. Its first stage uses ancestry-isolated SMC islands to preserve independent trajectory families and routes a bounded set of prefix-conditioned visual scouts to prefix-relevant image regions while discouraging redundant overlap. Each scout temporarily increases attention to the image tokens and emphasizes its routed region. Exact importance correction preserves the base LVLM sequence-power target. The second stage aggregates terminal importance mass by canonical answer, powers the answer marginal, and samples an answer together with a supporting trajectory. Across four LVLM backbones and five benchmarks, ReSight-SMC achieves stronger aggregate performance than Power-SMC over both the reasoning and perception benchmark groups. Without post-training, it remains competitive in aggregate with backbone-matched models trained using reinforcement learning.

1 INTRODUCTION

Sequence-level power sampling has been underexplored in open-ended LVLM decoding, where finite-particle exploration and answer-level competition remain challenges. ReSight-SMC combines ancestry-isolated sampling, visual scouting, and answer-marginal sharpening, with stronger aggregate performance than Power-SMC across reasoning and perception tasks.

1 INTRODUCTION

  • Global resampling can duplicate high-weight trajectories, direct Power-SMC lacks visual-cue diversification, and sequence-level weighting makes trajectories supporting the same answer compete.These issues constrain genealogical coverage and collective same-answer support under finite-particle inference.Source: §1 · p. 2
  • ReSight-SMC uses independently resampling islands and prefix-conditioned visual scouts routed to image regions, while exact importance correction preserves the base LVLM sequence-power target.Scouts temporarily reactivate image-token attention and emphasize their assigned regions.Source: §1 · p. 2
  • Its second stage powers answer masses after aggregating terminal trajectory mass by canonical answer, then samples an answer with a supporting trajectory.This lets multiple trajectories supporting one answer contribute collectively.Source: §1 · p. 2
  • Across four LVLM backbones and five benchmarks, ReSight-SMC outperforms Power-SMC in aggregate on reasoning and perception tasks, and remains competitive with backbone-matched reinforcement-learning models without post-training.The direct Power-SMC transfer also provides a strong training-free LVLM baseline.Source: §1 · p. 2

2 RELATED WORK

Related work covers test-time scaling for LVLMs, sequence-level power sampling and island SMC, answer-marginal methods, and visual conditioning. ReSight-SMC extends these lines with verifier-free weighted sampling, isolated islands, prefix-conditioned scouts, and powered answer-mass aggregation.

LVLM test-time scaling

  • LVLM test-time scaling includes visual-token retention, inference-time parameter updates, and sampling or refinement strategies; ReSight-SMC keeps parameters fixed and uses verifier-free weighted sampling.Longer reasoning traces can weaken attention to image tokens.Source: §2 · p. 3

Sequence-level power sampling and island SMC

  • Power-SMC advances weighted particles in parallel, while ReSight-SMC confines resampling ancestry to non-migrating islands for token-level LVLM sequence-power sampling.Global resampling can duplicate high-weight trajectories; prior multimodal power sampling focused on embodied tasks or MCMC refinement.Source: §2 · p. 3

Answer aggregation and marginal sharpening

  • Unlike self-consistency’s unweighted majority decision, ReSight-SMC pools corrected SMC mass by canonical answer and applies a finite second power before sampling.This composes sequence- and answer-level sharpening without unweighted voting.Source: §2 · p. 3

Visual conditioning during inference

  • Prior training-free visual-conditioning methods modulate image influence in several ways; ReSight-SMC uses prefix-conditioned scouts that emphasize routed regions while importance correction preserves the sequence-power target.Each scout reactivates attention to all image tokens and gives its routed region an additional boost.Source: §2 · p. 3

3 METHOD

ReSight-SMC separates trajectory construction from answer readout. Island SMC and corrected, prefix-conditioned visual proposals build a weighted population for the base LVLM sequence-power target; the readout pools terminal mass by answer before applying a finite answer power.

3.1 OVERVIEW

  • The overview separates a trajectory stage that builds a weighted sequence-power population from a readout that pools terminal mass by canonical answer and applies a finite power.Pooling allows an answer to draw support from multiple moderate-weight trajectories rather than only dominant paths.Source: §3.1 · pp. 3–4

3.2 LVLM SEQUENCE-POWER TARGET AND ISLAND SMC

  • The fixed LVLM distribution is conditioned on the image and prompt, and Power-SMC approaches its sequence-power target through intermediate exponents with importance correction.The locally powered next-token proposals alone do not induce the sequence-level target; the importance weight supplies the required correction.Source: §3.2 · p. 4
  • Islands resample independently at ESS-triggered checkpoints, preventing population-wide genealogical collapse while preserving within-island ancestry.Each island uses stratified resampling when its ESS falls below the threshold.Source: §3.2 · pp. 4–5, Figure 1 · p. 5

3.3 PREFIX-CONDITIONED VISUAL SCOUTS

  • At a prescribed checkpoint, a bounded scout set is routed from current prefixes to image regions, while unfinished base-proposal anchors remain active.Region routing combines prefix relevance and pooled particle mass, and discourages redundant assignments through a soft-overlap penalty.Source: §3.3 · pp. 5–6
  • Scouts temporarily increase attention to image tokens and further emphasize their routed region; exact importance correction keeps their proposals weighted for the same sequence target.The scout intervention is bounded, and decoding resumes from the persistent base-target state afterward.Source: §3.3 · p. 6

3.4 ANSWER-MARGINAL POWER READOUT

  • The readout pools terminal particle mass by canonical answer, applies a finite answer-level power, and samples an answer with a supporting trajectory.For population-level powers of two at both stages, the second power follows pooling, preserving support across trajectories rather than reverting to majority voting.Source: §3.4 · p. 7

4 EXPERIMENTS

Experiments evaluate four LVLM backbones on five benchmarks, comparing sampling methods and backbone-matched post-training references. ReSight-SMC has the strongest training-free aggregate results and each of its three components improves the all-data average in ablations.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers four LVLM backbones and five benchmarks spanning visual mathematics, logic, broad perception, and real-world spatial understanding, with matched sampling settings and RL references at selected scales.Population methods use 32 particles; scouting occurs once at token 40 and is skipped for earlier completions.Source: §4.1 · p. 7

4.2 RESULTS ACROSS BACKBONES

  • ReSight-SMC achieves the best training-free average on every backbone, beats Power-SMC on 18 of 20 benchmark results, surpasses Game-RL across reported benchmarks, and remains competitive with TRACE-RL.Table 1 reports Pass@1 accuracy across backbones and benchmarks as four-seed mean ± standard deviation, with question-weighted all-data averages.Source: §4.2 · pp. 7–8, Table 1 · p. 8

4.3 COMPONENT ABLATION

  • Island isolation, visual scouts, and answer-marginal power each improve the all-data average; island isolation improves every benchmark and provides the largest aggregate gain.Scouting improves the four benchmarks where it activates; RealWorldQA is unchanged because no evaluated run reaches the scout phase.Source: §4.3 · p. 8, Table 2 · p. 8

5 DISCUSSION

The first stage generally broadens particle coverage, while answer-level sharpening raises pass@1 but can reduce pass@4. The discussion compares sharpening rules, reports added decoding cost, and identifies limits from answer canonicalization and model access.

5.1 FROM ISLAND COVERAGE TO PASS@1

  • Across intervention tasks, the first stage generally raises Coverage@32 and pass@4, while answer-level sharpening raises pass@1 but lowers pass@4 on four of five benchmarks.Across all three reasoning benchmarks, the first stage raises Coverage@32 and pass@4, and answer-level sharpening further improves pass@1.Source: §5.1 · p. 9
  • On RealWorldQA, all responses finish before the first checkpoint, so first-stage ReSight-SMC and Power-SMC induce the same output distribution and the pass@1 gain mainly comes from the second-stage readout.Observed first-stage differences in pass@1, pass@4, and Coverage@32 reflect finite-sample variation.Source: §5.1 · p. 9

5.2 TRAJECTORY POWER AND ANSWER-LEVEL SUPPORT

  • Answer-marginal second power improves pass@1 over one-stage α = 2 on all five benchmarks and over one-stage α = 4 on all reasoning benchmarks.On MMStar-P and RealWorldQA, (2, 2) has slightly lower pass@1 than one-stage (4, 1).Source: §5.2 · p. 9

5.3 EFFICIENCY

  • Per-example latency and peak VRAM are 10.37 seconds and 17.35 GB for ReSight-SMC, versus 7.85 seconds and 17.27 GB for Power-SMC.The principal additional cost is bounded decoding on temporary scout branches; answer aggregation needs no additional model forward pass.Source: §5.3 · p. 10

5.4 LIMITATIONS

  • The answer readout cannot repair a wrong or missing first-stage answer mode and depends on a task-appropriate canonicalizer.Without deterministic merging, semantically equivalent free-form strings remain separate answers and fragment their marginal mass; model logits and decoder states also restrict the method to open-weight LVLMs.Source: §5.4 · p. 10

6 CONCLUSION

ReSight-SMC combines island-local trajectory sampling and prefix-conditioned visual scouts with answer-level mass aggregation and sharpening. The conclusion reports aggregate gains over Power-SMC, while appendix analyses describe genealogy, benchmark-level coverage and accuracy patterns, and when scouts activate.

6 CONCLUSION

  • ReSight-SMC delivers aggregate improvements over Power-SMC on reasoning and perception tasks and remains competitive with post-trained systems without parameter updates or a learned verifier.Its first stage uses island-local resampling and visual scouts, followed by answer-mass aggregation and a finite answer-level power.Source: §6 · p. 10

A.1 ISLAND GENEALOGY

  • Across all three reasoning benchmarks, island isolation increases the effective number of terminal roots and reduces the largest ancestral family’s mass at the same total particle budget and with the same visual and power parameters.The result supports the intended mechanism: a high-weight family can expand within its island without erasing promising families in other islands.Source: §A.1 · p. 13, Table 5 · p. 13

A.2 COVERAGE, PASS@1, AND PASS@4 ACROSS BENCHMARKS

  • Before answer-level sharpening, ReSight-SMC has higher empirical pass@4 than Power-SMC on four of five benchmarks while increasing Coverage@32; applying γ = 2 improves pass@1 on every benchmark but lowers pass@4 on four.LogicVista is the exception, improving both pass@1 and pass@4 with γ = 2.Source: §A.2 · p. 13, Table 6 · p. 13

A.3 GENERATION HORIZON AND SCOUT ACTIVATION

  • Nearly every reasoning particle remains active through both checkpoints, whereas direct-answer responses are usually complete before resampling and visual scouting.Completed particles continue through the exponent bridge and checkpoint calculation, but resampling them cannot create new continuations.Source: §A.3 · pp. 13–14, Table 7 · p. 14

A.4 ADDITIONAL VISUAL-STAGE ABLATIONS

The visual-stage ablations find benefits from routed-region attention and identify settings that perform best among those tested for overlap penalty, region-size normalization, and scout fraction.

A.4 ADDITIONAL VISUAL-STAGE ABLATIONS

  • Pass@1 improves over global attention only on every benchmark where scouts activate, isolating the added value of the region-specific bias.The global-attention-only variant removes the region-specific second bias.Source: §A.4 · p. 14, Table 8 · p. 14
  • The global overlap penalty raises the question-weighted average from 64.88% to 65.05% versus the island-local penalty.With μ = 1, it also gives higher means on three of four scout-active benchmarks; RealWorldQA is unchanged because no scout activates.Source: §A.4 · p. 14, Table 9 · p. 14
  • Among evaluated overlap coefficients, μ = 1 gives the highest question-weighted average, while μ = 0.5 leads on MathVista and MMStar-P.The comparison varies the global overlap coefficient while holding the biased route and other settings fixed.Source: §A.4 · p. 14, Table 9 · p. 14
  • Among tested area exponents, ζ = 0.75 gives the highest question-weighted average and is used by default.The exponent balances total attention mass against average per-token relevance in region scoring.Source: §A.4 · p. 14, Table 10 · p. 15
  • The default scout fraction ρV = 0.25 gives the highest all-data average and strongest LogicVista, MathVista, and MMStar-R results.At ρV = 0.5, those three scores decline slightly but remain above the visual-off variant.Source: §A.4 · p. 14, Table 11 · p. 15

B IMPORTANCE-WEIGHT DERIVATION

The derivation shows how importance ratios correct base and scout proposals to target the same powered sequence distribution, then describes island-local resampling and terminal mass aggregation.

B IMPORTANCE-WEIGHT DERIVATION

  • The incremental weight combines the change in intermediate target with the realized proposal probability, so proposal sampling does not add excess target mass.For any proposal based on available population history, token-level cancellation holds when it assigns positive probability wherever the target increment is positive.Source: B IMPORTANCE-WEIGHT DERIVATION · pp. 15–16
  • For an active scout, the routed attention proposal changes continuation frequency, while importance weighting maps mass back to the base LVLM intermediate target.The target remains defined by the base LVLM, and the proposal probability enters the joint proposal along the particle's ancestral path.Source: B IMPORTANCE-WEIGHT DERIVATION · pp. 15–17
  • After EOS, the completed response receives no new proposal probability, and subsequent bridge increments bring its terminal likelihood exponent to α regardless of stopping time.Response length affects model likelihood but not the terminal target exponent.Source: B IMPORTANCE-WEIGHT DERIVATION · p. 17

B.1 ISLAND RESAMPLING AND TERMINAL MASS

  • Resampling is triggered within an island by its ESS threshold, and ancestry remains confined there, preserving K separate genealogies.Triggered resampling resets weights to 1/M; otherwise particles and weights pass through unchanged.Source: §B.1 · p. 17
  • The pooled terminal mass is obtained by multiplying the probability of drawing an island by the probability of drawing a particle within it.Island normalizer estimates compare represented target mass without cross-island resampling.Source: §B.1 · p. 18

C VISUAL-SCOUT ROUTING SPECIFICATION

Visual-scout routing assigns a bounded number of unfinished particles to image regions using prefix-dependent relevance and overlap-aware selection, subject to per-island quotas.

C VISUAL-SCOUT ROUTING SPECIFICATION

  • Routing is skipped unless an island has at least two unfinished particles, and smaller active islands retain at least one base-proposal anchor.The scout quota is independent of island ESS.Source: C VISUAL-SCOUT ROUTING SPECIFICATION · p. 18
  • Region relevance aggregates visual-only attention across heads and tokens, with ζ = 0.75 partially normalizing region area.ζ = 0 favors total relevance mass and larger regions; ζ = 1 favors average relevance per visual token.Source: C VISUAL-SCOUT ROUTING SPECIFICATION · p. 18
  • Candidate utilities are rescaled using their eligible-pair minimum and maximum, with ε = 10^-12 preventing division by zero when utilities are equal.The normalized utility supports the sequential region-assignment procedure.Source: C VISUAL-SCOUT ROUTING SPECIFICATION · p. 18
  • The candidate set allows at most one view per particle and caps each island at Bk scouts.The greedy selection tracks each region’s maximum overlap with previously selected regions and has O(BKMG) complexity.Source: C VISUAL-SCOUT ROUTING SPECIFICATION · p. 18

D UNBIASEDNESS AND EXACT TARGET CORRECTION

The analysis establishes unbiasedness of the pooled unnormalized estimator under adaptive proposals and island-local resampling, and consistency as particle count grows under stated conditions. The answer readout then powers canonical-answer mass while retaining supporting trajectories.

D UNBIASEDNESS AND EXACT TARGET CORRECTION

  • Conditioned on population history, proposal probabilities cancel in importance updates, yielding an unbiased pooled unnormalized estimator when combined with conditionally unbiased resampling and the island normalizing-mass recursion.The proposition also states that the sum of island estimates is unbiased for the normalizing constant.Source: D UNBIASEDNESS AND EXACT TARGET… · pp. 19–20
  • As M grows with fixed K, finite horizon, and input, the analysis establishes consistency under normalized proposals with support wherever target increments are positive.The proof assumes conditionally independent token draws and a uniform second-moment bound; proposal routing need not itself converge as M grows.Source: D UNBIASEDNESS AND EXACT TARGET… · pp. 19, 21, 23
  • Stratified resampling preserves the induction statements, and its island-local ancestors cannot introduce cross-island lineages.The ESS trigger may depend on pre-resampling history without needing to stabilize as M grows.Source: D UNBIASEDNESS AND EXACT TARGET… · pp. 19, 25

E ANSWER-MARGINAL POWER READOUT

  • For γ > 1, the answer readout multiplies trajectories supporting the same canonical answer by the same answer-mass factor, preserving their relative probabilities within that answer.At γ = 1, the empirical readout recovers the first-stage terminal measure and does not alter the unbiased normalizing-constant statement.Source: E ANSWER-MARGINAL POWER READOUT · pp. 25–26

E.1 SPLIT SHARPENING AND CROSS-TRAJECTORY SUPPORT

Split sharpening first powers trajectory mass and then powers the mass pooled by answer, allowing multiple trajectories supporting one answer to contribute together. In the two-token example, this can select a different answer mode from base sampling or direct fourth-power sequence sampling.

E.1 SPLIT SHARPENING AND CROSS-TRAJECTORY SUPPORT

  • The excess over direct sequence power is the nonnegative sum of same-answer cross terms, which can change answer rankings.It is zero only when at most one positive-mass trajectory supports an answer; with (α, γ) = (2, 2), the readout retains the direct fourth-power term and credits collective support.Source: §E.1 · p. 27
  • In the two-token example, base sampling favors A, direct fourth-power sampling selects C, and split sharpening selects B.Split sharpening discounts weak paths while preserving the collective support of two moderate paths before applying the answer-level square.Source: §E.1 · p. 27, Table 12 · p. 27
  • The split remains fourth-order in base trajectory probabilities while avoiding unweighted majority behavior and direct fourth-power pathwise concentration.The example frames the comparison among base sampling, split sharpening, and direct fourth-power sequence sampling.Source: §E.1 · p. 27, Table 12 · p. 27

F FINITE-PARTICLE BEHAVIOR ON DIRECT-ANSWER TASKS

In the idealized direct-answer setting, each admissible answer corresponds to one trajectory, so answer aggregation adds no cross-trajectory support; finite-particle variation can make the (2, 2) readout worse than direct (4, 1). The section also outlines the inference algorithm and describes computational-cost measurements and aggregation complexity.

F FINITE-PARTICLE BEHAVIOR ON DIRECT-ANSWER TASKS

  • In the direct-answer example, direct (4, 1) can have higher expected pass@1 than (2, 2) when each answer has one trajectory and the second-stage power squares noisy empirical answer masses.The analysis assumes independent particle draws without scouting or resampling; as particle count grows, both readouts converge to 72.32% in this example.Source: F FINITE-PARTICLE BEHAVIOR ON… · pp. 27–28

G INFERENCE ALGORITHM

  • Algorithm 1 runs island-local SMC and visual scouting, then aggregates terminal mass by canonical answer and samples an answer with a supporting trajectory.The described procedure includes per-island resampling and releases temporary attention states after the scout phase.Source: G INFERENCE ALGORITHM · p. 29

H COMPUTATIONAL COST

  • Table 14 reports end-to-end latency and peak memory for base sampling, Power-SMC, and ReSight-SMC under a common setup.For terminal aggregation and sampling, the stated cost is O(KM) time and O(U) memory, with no additional model forward.Source: H COMPUTATIONAL COST · p. 29, Table 14 · p. 29

I EXPERIMENTAL DETAILS

The experimental details specify the scout schedule, sampler configurations, backbones, prompts, answer handling, and evaluation procedures. The protocols use shared prompts across compared methods and report pass@1, pass@4, and terminal-particle Coverage@32 with distinct execution procedures.

I EXPERIMENTAL DETAILS

  • The default scout phase starts after token 40, uses tokens 41–56 for scout proposals, and returns to base proposals for tokens 57–64 before resampling.The setup allows at most eight scouts, doubles attention odds for image tokens, and gives routed-region tokens a further factor of four.Source: I EXPERIMENTAL DETAILS · p. 31
  • The main ReSight-SMC configuration uses answer exponent γ = 2, while Power-SMC, the island-only variant, and the trajectory-stage ablation use γ = 1.The suite includes four Qwen LVLM backbones and backbone-matched post-training references for Qwen2.5-VL-3B and 7B.Source: I EXPERIMENTAL DETAILS · p. 31

I.1 PROMPTS AND ANSWER EVALUATION

  • Reasoning benchmarks use chain-of-thought prompts, perception benchmarks use direct-answer prompts, and the compared methods share dataset-specific prompts.Answer extraction uses a deterministic parser without an LLM judge or LLM-based extractor; answer grouping uses reference-free canonicalization.Source: §I.1 · pp. 31–32
  • Pass@1 averages four seeded executions, pass@4 uses those same executions, and Coverage@32 uses one execution’s terminal particle set because its candidates share ancestry.Timing includes image processing, visual encoding, prefill, generation, resampling, routing, and output selection.Source: §I.1 · p. 32
Loading 2609.34905v1…