Architectural Sampling: Test-Time Scaling via Computational Diversity in Frozen Vision-Language Models

• Candidate-based scaling depends on solution-space coverage, but temperature sampling changes only the final output draw after candidates share the same computation path.Source: §1 · p. 2
• Architectural sampling reuses a contiguous decoder-layer block a chosen number of times, creating distinct computation paths without changing frozen weights or adding parameters.Source: §1 · pp. 2–3
• Architectural sampling scales more favorably across candidate counts, works best with early-layer recursion, and retains coverage gains under greedy decoding.Source: §1 · p. 3
• Replacing temperature-based TTRV rollouts with architectural rollouts consistently improves accuracy, extending the method's reported benefits to test-time learning.Source: §1 · p. 3
• 6.58 percentage points: architectural sampling improves pass@9 over temperature sampling on average across five Qwen checkpoints and twelve multimodal benchmarks at the same candidate budget.Source: §1 · p. 3

• Test-time scaling uses repeated generation, aggregation, search, or online adaptation to improve outputs from a fixed checkpoint.Source: §2 · p. 3
• Existing candidate-pool methods aggregate sampled reasoning paths or paraphrases, search reasoning trees, vary inputs, or use responses for online adaptation.Source: §2 · p. 3
• Recursive architectures typically deepen or refine a forward pass, including repeated shared blocks and learned token-specific recursion depths.Source: §2 · p. 4
• Multimodal recursive systems repeatedly update visual features, answers, explanations, or search intervals, often using task-specific learned computation (Xu et al., 2026).Source: §2 · p. 4
• Prior frozen-model layer-reuse work rearranges, skips, or repeats layers at inference, while this work uses each reuse configuration as a distinct candidate-generation path.Source: §2 · p. 4

3 ARCHITECTURAL SAMPLING

• Each candidate uses the same frozen checkpoint; an architectural path revisits one contiguous layer window before resuming the original layer order.Source: §3 · p. 4, Figure 3 · p. 5

3.1 STANDARD FORWARD PASS

• The VLM combines image tokens, prompt embeddings, and previously generated token embeddings as decoder input, then applies its decoder layers once in original order on the standard path.Source: §3.1 · p. 4

3.2 REUSING A LAYER WINDOW

• A contiguous window is repeated in place while the frozen prefix and suffix remain unchanged; setting r=1 recovers the standard pass, while changing window or depth yields distinct feature trajectories.Source: §3.2 · p. 5

3.3 BUILDING THE CANDIDATE POOL

• One forward pass per window-and-depth pair forms the candidate pool, with k=|B|·|R|; the same checkpoint and decoding temperature make architectural variation the source of added diversity.Source: §3.3 · pp. 5–6

4.1 EVALUATION SETTINGS

• The evaluation covers five Qwen checkpoints and twelve multimodal benchmarks spanning visual perception, counting, knowledge, and multimodal reasoning.The main comparison uses nine candidates per method, with matching prompts, decoding settings, and answer evaluation.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• Architectural sampling also improves average performance by 1.54 points on InternVL3 and 17.63 points on Gemma3, while gains remain robust across repeated runs.The non-Qwen evaluation uses a five-benchmark subset.Source: §4.2 · pp. 6–7
• 6.58 percentage points: architectural sampling improves average pass@9 over standard-path temperature sampling across five Qwen checkpoints and twelve benchmarks.Gains occur in 55 of 60 checkpoint–benchmark comparisons.Source: §4.2 · p. 6, Table 1 · p. 7

Layer-window placement

• Early-layer windows achieve the highest pass@9 on all five benchmarks; gains decrease as the repeated window moves deeper.Every tested placement improves over temperature sampling.Source: §5 · p. 7, Table 2 · p. 8

Token reprocessing

• Jointly reprocessing visual and language tokens achieves the highest pass@9 on all five benchmarks; language-only updates retain most of the improvement.Visual-only updates and updating an averaged visual representation yield smaller gains.Source: §5 · p. 7, Table 3 · p. 8

Temperature and diversity

• Architectural candidates have higher mean 1 −Self-BLEU than temperature samples (0.51 vs. 0.22), while pass@9 remains stable across decoding temperatures, including T = 0.Higher 1 −Self-BLEU indicates lower lexical overlap among candidates.Source: §5 · pp. 7–8, Figure 4 · p. 8

Standard decoding comparisons

• At equal candidate budgets, architectural sampling achieves the highest pass@9 across benchmarks compared with temperature sampling, beam search, top-k, and top-p.These standard decoding strategies vary the output distribution while retaining the same underlying computation.Source: Table 4 · p. 8, §5 · p. 8

Test-time learning

• Architectural rollouts consistently improve accuracy over temperature-sampled rollouts in label-free TTRV across two backbones and six datasets.The comparison uses standard accuracy or pass@1, and the rollouts provide a correctness-correlated signal that optimization can exploit.Source: Table 5 · p. 9, §5 · p. 9

6 LIMITATIONS AND CONCLUSION

• Recursion adds roughly 1–2 seconds per candidate at depths up to 4, and decoder-layer access excludes closed-weight models served only through an API.The authors note that the added cost may matter in resource-constrained settings.Source: §6 · p. 9
• At a nine-candidate budget, pass@9 improves by 6.58 percentage points on average; early multimodal reprocessing works best, coverage persists under greedy decoding, and rollouts improve test-time learning.The resulting candidates also have less lexical overlap.Source: §6 · p. 9
• A frozen VLM’s computation graph provides a sampling space complementary to token-level randomness, without retraining the checkpoint.Source: §6 · p. 9
• The authors state that generative AI assisted with language editing and manuscript organization, not experimental data generation or result determination.Source: AI USE STATEMENT · p. 9
• Implementation and evaluation scripts are planned for release with the final paper.Source: REPRODUCIBILITY STATEMENT · p. 10

A APPENDIX

• The appendix is organized around implementation details, additional ablations, scaling curves, and computational-cost analysis.Source: A APPENDIX · p. 13
• Experiments were conducted on a machine equipped with an NVIDIA A100 GPU.Source: A APPENDIX · p. 13
• The complete codebase was provided during review and is planned for public release upon acceptance.The appendix also gives detailed execution instructions in README.md.Source: A APPENDIX · p. 13

B ADDITIONAL IMPLEMENTATION DETAILS

• Inference uses nine candidates: four inclusive decoder windows at recursion depths 2 and 4 plus a standard path, all with temperature 0.6.All checkpoint parameters remain frozen; generation limits are 32 new tokens for multiple-choice and 256 for open-ended questions, matching the temperature baseline.Source: B ADDITIONAL IMPLEMENTATION DETAILS · p. 13
• The label-free GRPO-based test-time learning experiment replaces temperature-sampled rollouts with architectural candidates and assigns scalar rewards by majority vote.It trains on 20 samples without ground-truth labels and reports greedy pass@1.Source: B ADDITIONAL IMPLEMENTATION DETAILS · p. 13
• The benchmark suite spans visual perception and structured reasoning, counting, broader multimodal reasoning, mathematical reasoning, and real-world understanding.The listed tasks include AI2D, ScienceQA, CV-Bench, MMStar, BLINK, CountBench, CountQA, GQA, A-OKVQA, MMMU, MathVista, and RealWorldQA.Source: B ADDITIONAL IMPLEMENTATION DETAILS · pp. 13–14, Table 6 · p. 14

C ADDITIONAL EXPERIMENTS AND ABLATIONS

• Across five benchmarks, pass@9 standard deviations remain below 0.32 points for both candidate-generation strategies in five Qwen3-VL-8B runs.Source: C ADDITIONAL EXPERIMENTS AND ABLATIONS · p. 14, Table 7 · p. 14
• Architectural sampling raises the five-benchmark pass@9 average by 1.54 points for InternVL3-2B and 17.63 points for Gemma3-4B.Source: C ADDITIONAL EXPERIMENTS AND ABLATIONS · p. 14, Table 8 · p. 15
• The results provide an initial test beyond Qwen, while broader model-family coverage remains to be studied.Source: C ADDITIONAL EXPERIMENTS AND ABLATIONS · p. 14

D SCALING CURVES

• Figures 2, 5, and 6 show pass@k against candidate budget for three representative checkpoints, extending the fixed nine-candidate comparison.Figures 5 and 6 cover Qwen2.5-VL-7B and Qwen3.5-VL-4B, respectively.Source: D SCALING CURVES · p. 15, Figure 5 · p. 15, Figure 6 · p. 15
• Across most checkpoint–benchmark pairs, architectural sampling continues gaining pass@k as candidates are added, while temperature sampling plateaus earlier.The curves are descriptive because the pattern is not uniform across panels.Source: D SCALING CURVES · p. 15
• Figure 5 plots pass@k across six benchmarks as the candidate budget grows for Qwen2.5-VL-7B.Source: Figure 5 · p. 15
• Figure 6 plots pass@k across six benchmarks as the candidate budget grows for Qwen3.5-VL-4B.Source: Figure 6 · p. 15

E COMPUTATION COST AND LATENCY

• Architectural sampling leaves parameter count unchanged while traversing some decoder layers more than once; layer applications serve as an analytic cost proxy before latency is reported.Source: E COMPUTATION COST AND LATENCY · p. 16
• The cost model counts the prefix and suffix once and a contiguous window B r times, normalizing one standard decoder traversal to unit cost.Each additional window traversal adds |B| layer applications.Source: E COMPUTATION COST AND LATENCY · p. 16
• Overhead grows with window size and recursion depth, forms a smaller fraction of a deeper decoder, and is tabulated for L = 32.Source: E COMPUTATION COST AND LATENCY · p. 16, Table 9 · p. 16
• The nine-candidate grid combines one standard path at r = 1 with recursion depths R = {2, 4} and windows of sizes 3 and 4.Source: E COMPUTATION COST AND LATENCY · p. 16
• For Qwen3-VL-8B, nine architectural candidates require 10.5 standard decoder traversals versus 9.0 for nine temperature samples; the architectural pool takes 53.8 s versus 46.1 s for that baseline.Source: Table 10 · p. 16, E COMPUTATION COST AND LATENCY · p. 16