Latent Implicit Visual Reasoning

• LMMs struggle with fine-grained visual reasoning partly because text-only outputs impose language bias and may lack the expressivity needed for sophisticated visual abstractions.Much recent progress has instead focused on tasks where most reasoning occurs in text after initial visual extraction.Source: §1 · p. 1
• Explicit intermediate supervision can be costly, encode human assumptions about useful representations, and scale or generalize poorly across diverse vision-centric tasks.For complex or abstract visual structure, even humans may not know which intermediate representation to supervise.Source: §1 · p. 1
• LIVR uses latent tokens and visual bottlenecking to make task-relevant visual information pass through those tokens before answer prediction, without human-specified intermediate steps.This lets the model learn useful visual abstractions without additional data or explicit intermediate supervision.Source: §1 · pp. 1–2
• LIVR consistently outperforms direct supervised fine-tuning across nine perception-heavy tasks, three LMM backbones, and multi-task training.These are controlled data-matched experiments.Source: §1 · p. 2
• On external reasoning benchmarks, LIVR remains competitive with or outperforms text-based and explicit-visual-intermediate reasoning methods while avoiding costly intermediate supervision.The paper lists helper images, bounding boxes, image crops, and chain-of-thought annotations as examples of supervision it avoids.Source: §1 · p. 2

Text-Based Visual Reasoning

• Text-based visual reasoning methods generate intermediate textual explanations, but text-only reasoning can struggle to express rich, spatially structured abstractions.Examples include LLaVA-CoT’s structured rationales and RL-based methods that encourage step-by-step explanations [38].Source: §2 · p. 2

Interleaved Multimodal Reasoning

• Interleaved multimodal reasoning adds visual representations to reasoning, with methods grouped into region-based visual recycling and generated visual intermediates.This direction is motivated by the difficulty of text-only reasoning on visual tasks.Source: §2 · pp. 2–3

Region-Based Visual Recycling

• Region-based methods [22] [27] [29] [33] reuse selected image regions, but this local focus is less natural for global or abstract structure and may require costly annotations.These approaches commonly predict coordinates, boxes, or regions, then crop, zoom, or resample the selected evidence.Source: §2 · p. 3

Generated Visual Intermediates

• MVoT, CoT-VLA [46], Mirage, LVR, and Aurora [5] generate visual intermediates but prescribe forms such as images, crops, depth maps, or bounding boxes.Such supervision can be costly or unavailable for ambiguous or abstract intermediates, limiting generalization and scale across vision-centric tasks.Source: §2 · p. 3

Latent Reasoning

• Coconut and Think Before You Speak explore latent computation, but LMM latent methods still use intermediate supervision, unlike LIVR’s end-to-end task-trained visual tokens.LIVR’s dedicated latent tokens operate on joint visual–text states and are trained from task objectives.Source: §2 · p. 3

3.1. Preliminaries

• LMMs combine a visual encoder, projector, and language decoder, and the paper formulates tasks as visual question answering from image and prompt inputs.The model predicts an answer, evaluated by whether it matches the ground truth.Source: §3.1 · pp. 3–4

3.2. Latent Implicit Visual Reasoning

• LIVR adds K trainable latent tokens and masks attention so answers receive visual information only through those tokens, preventing the model from ignoring them.The latent tokens are appended to the prompt, and only their embedding rows are unfrozen; answer tokens cannot attend to image tokens, and prompt tokens cannot attend to them either.Source: §3.2 · p. 4
• Stage 1 trains latent tokens with answer-only NLL loss; Stage 2 restores attention to image tokens so the model can use both image and enriched latent tokens.Stage 1 optimizes the latents to capture useful visual information, while Stage 2 keeps the loss on answer tokens and enables joint use of image and latent tokens.Source: §3.2 · p. 4

3.3. Implementation Details

• Implementation fine-tunes language-backbone attention and MLP blocks with LoRA, freezes the vision encoder and projector, and unfreezes only latent-token embedding rows.Full optimization and schedule details are provided in the Appendix.Source: §3.3 · p. 4

4.1. Data-matched Experiments

• LIVR beats Direct SFT by 6.24% on Qwen2.5-VL, 3.43% on Qwen3-VL, and 5.60% on LLaVA-OneVision-1.5 on average, and improves all six multi-task tasks.On Qwen2.5-VL, reported gains include 12% on Jigsaw and 13.02% on Functional Correspondence; multi-task results use Qwen3-VL-4B-Instruct.Source: Table 1 · p. 5, §4.1 · pp. 5–6, Table 2 · p. 6

4.2. Comparison with Prior Reasoning Methods

• On VSP, LIVR scores 66.00 (+20.00) on Qwen2.5-VL-3B and 77.50 versus Mirage’s 76.00 on 7B; it is also competitive or better across MMVP, V*, and five BLINK tasks.The broader comparison uses Qwen2.5-VL-7B and removes intermediate images and text-CoT from LIVR.Source: §4.2 · pp. 6–7, Table 4 · p. 6
• On SAT Val, three BLINK subsets, and RoboSpatial splits, LIVR achieves strong results without text-CoT, explicit grounding, or RL, and demonstrates out-of-distribution generalization.The methods are initialized from Qwen-2.5-VL-3B and trained on the same synthetic SAT-32k dataset.Source: Table 3 · p. 6, §4.2 · p. 6

4.3.1. Usefulness of Latent Tokens

• On Localization, removing LIVR’s latents drops accuracy from 83.61 to 76.23, while the latents-only control stays at 79.51 and performs at 43.44 under the bottleneck mask.LIVR’s answer-to-latent attention is 0.076 versus 0.028 for the latents-only control, and its bottleneck-mask accuracy is 70.49.Source: §4.3.1 · p. 7

4.3.2. Design Ablations for LIVR

• Ablations show that adding latent tokens without bottlenecking and bottlenecking without latent tokens both underperform LIVR, indicating both components are needed for its full gains.The mask-only variant leaves prompt tokens able to see the image, while dedicated latents can adapt to form visual abstractions.Source: Table 5 · p. 7, §4.3.2 · p. 7

4.3.3. Architectural and Training Choices

• Placing latents after the prompt beats placing them before it on all three tasks, and blocking both answer-to-vision and prompt-to-vision attention yields the best masking performance.After-versus-before scores are 83.61 vs. 80.33 for Localization, 64.75 vs. 61.87 for Semantic Correspondence, and 67.81 vs. 63.70 for Functional Correspondence.Source: §4.3.3 · pp. 7–8, Table 6 · p. 8
• The 4/6 Stage-1/Stage-2 schedule and K=16 perform best, while unshared latent embeddings outperform shared embeddings across all three tasks.Unshared-versus-shared scores are 83.61 vs. 81.15 on Localization, 64.75 vs. 61.87 on Semantic Correspondence, and 67.81 vs. 63.70 on Functional Correspondence; performance drops at K=32.Source: §4.3.3 · p. 8

4.4. Visualizations

• Latent-to-image attention visualizations show tokens matching a motorcycle handle, identifying object boxes, and focusing on objects needed for counting.These examples span Semantic Correspondence, Localization, and Counting, respectively.Source: §4.4 · p. 8

5. Conclusion

• Across nine perception-heavy tasks, LIVR outperforms direct SFT in single-task training on three LMMs and improves joint multi-task training on Qwen3-VL.The conclusion reports these results across extensive experiments.Source: §5 · p. 8
• LIVR is competitive with or outperforms text-based and explicit-visual-intermediate reasoning methods without costly helper images, bounding boxes, image crops, or chain-of-thought annotations.Source: §5 · p. 8
• Latent tokens trained with visual bottlenecking let LMMs learn useful visual representations implicitly, providing a simple, effective, task-agnostic approach to richer visual reasoning.Source: §5 · p. 8
• Across semantic correspondence, localization, and counting, latent-to-image attention concentrates on task-relevant regions, though some attention sinks persist.The attention patterns align with correct answers or visual evidence needed to resolve each task.Source: Figure 3 · p. 9

6. Limitations and Future Work

• LIVR’s latent tokens are less directly interpretable than text-based reasoning or hand-designed visual intermediates, and the precise information encoded by each remains challenging to understand.Attention visualizations suggest that latents attend to task-relevant regions, but do not resolve what each latent encodes.Source: §6 · p. 9
• Future work could scale LIVR to larger models and dynamically allocate latent tokens according to task difficulty.Source: §6 · p. 9
• Future work could study whether latent visual reasoning benefits more complex multi-step or interactive vision-language tasks.Source: §6 · p. 9

A. Datasets

• The nine-task testbed spans low-level correspondence and reflectance, mid-level jigsaw and art-style classification, and higher-level localization, counting, similarity, and correspondence tasks.The splits are designed to cover heterogeneous task types and difficulty levels.Source: A. Datasets · p. 12
• Evaluation splits are derived from BLINK and PixMo-Count to create a broad testbed for the data-matched experiments.Source: A. Datasets · p. 12

A.1. Counting

• For counting, the paper describes the data sources, train/validation/test splits, and VQA-style prompts used during training and evaluation.Source: A. Datasets · p. 12

A. Datasets

• The dataset descriptions cover nine tasks: counting, jigsaw, object localization, visual correspondence, art style, semantic correspondence, functional correspondence, relative reflectance, and visual similarity.Source: A. Datasets · p. 12
• VQA-style prompt templates are specified for both training and evaluation across the nine tasks.Source: A. Datasets · p. 12

A.1.1. Data Sources and Splits

• PixMo-Count examples provide an image URL, object label, and integer count, with official train, validation, and test splits.The object label can identify categories such as people or cars.Source: §A.1.1 · p. 12
• Counting training uses 1,000 PixMo-Count examples with counts from 2 to 10, approximately uniformly sampled after URL filtering and test-image near-duplicate removal.Near duplicates are flagged using CLIP embeddings, perceptual hashing, and SSIM-based image similarity.Source: §A.1.1 · p. 12
• The official PixMo-Count validation and test splits provide 534 and 528 examples, respectively, after removing examples with unresolved image URLs.The authors additionally de-duplicate train and validation images against test images because the official splits may overlap.Source: §A.1.1 · p. 12

A.1.2. Prompt Template

• Counting is phrased as an open-ended task, while Jigsaw is phrased as a two-way multiple-choice question over candidate patches.The supplied passages identify prompt templates for both tasks but do not include their contents.Source: §A.1.2 · p. 12, §A.2.2 · p. 13

A.2.1. Data Sources and Splits

• Jigsaw training and validation use 1,000 and 250 COCO images, respectively, to create four-quadrant missing-patch questions with a same-image distractor.The canvas is 400 px wide with random height from 170 to 230 px; the bottom-right quadrant is hidden, and the ground-truth patch is randomly assigned to A or B.Source: §A.2.1 · p. 12
• Jigsaw testing uses the official BLINK Jigsaw validation split of 150 TARA examples, disjoint by source dataset from the COCO-based generated train and validation sets.The authors therefore perform no additional train–test de-duplication across these datasets.Source: §A.2.1 · pp. 12–13

A.3.1. Data Sources and Splits

• For object localization, 1,000 training and 250 validation questions compare a COCO gold box with a jittered distractor; the 122-example BLINK validation split serves as test.Examples use non-crowd objects with specified box and mask-area filters, and the COCO pool is deduplicated against BLINK images and across generated splits.Source: §A.3.1 · p. 13, §A.3.2 · p. 13

A.4.1. Data Sources and Splits

• HPatches sequences are split 80/10/10 by viewpoint and illumination, then used to generate four-choice correspondence questions with 1,000 training, 500 validation, and 700 test examples.Each question marks a source reference point and presents its true target correspondence among three distant distractors; the task uses a four-way multiple-choice prompt.Source: §A.4.1 · p. 14, §A.4.2 · p. 14
• Because strict deduplication against BLINK would leave too little data, visual correspondence uses the constructed HPatches test split rather than BLINK’s official split.The BLINK format is retained, and the source/target sequence pairs are kept in a single split.Source: §A.4.1 · p. 14

A.5.1. Data Sources and Splits

• Art-style questions ask which candidate painting matches a reference; the ArtBench-10 data provide 1,000 training and 250 validation examples, with 117 BLINK examples for test.Positive and negative candidates come from the same and different styles, respectively; images are split-disjoint and deduplicated against BLINK, and the task uses a two-way multiple-choice prompt.Source: §A.5.1 · pp. 14–15, §A.5.2 · p. 15

A.6.1. Data Sources and Splits

• Semantic correspondence uses SPair-71k keypoint pairs to form four-choice questions, yielding 1,000 training and 250 validation examples, with 139 BLINK examples as the held-out test set.Pairs require at least four valid correspondences; each question marks one source keypoint and offers its true target match among three annotated distractors, with deduplication against BLINK and a four-way prompt.Source: §A.6.1 · p. 15, §A.6.2 · p. 15

A.7.1. Data Sources and Splits

• Functional correspondence divides FunKPoint images into action-balanced, disjoint 80/10/10 train, validation, and test splits, pairing same-action images without reusing an image within a split and action.The task is phrased as a four-way multiple-choice question over candidate correspondence points.Source: §A.7.1 · p. 16, §A.7.2 · p. 16
• Functional correspondence questions match a source functional keypoint to the same-index target among three distractors, producing 1,000 training, 144 validation, and 146 test examples.Because strict deduplication with BLINK would leave too little data, the constructed FunKPoint test split is used instead of BLINK’s functional-correspondence split.Source: §A.7.1 · p. 16

A.8.1. Data Sources and Splits

• Relative Reflectance examples pair MID scene images under 25 illumination directions with corresponding diffuse-albedo images, then sample two spatially separated points and compare their local luminance.The albedo images are converted to linear RGB before luminance is computed.Source: §A.8.1 · p. 17
• The Relative Reflectance label marks pairs within 10% luminance difference as “About the same”; otherwise it identifies the darker point, with that class sampled to roughly one quarter.The resulting pool is randomly subsampled to 1,000 training and 250 validation examples.Source: §A.8.1 · p. 17
• Relative Reflectance is tested on the 134-example BLINK Relative Reflectance validation split, built on IIW, while training and validation data come from MID.The distinct sources mean no additional cross-dataset deduplication is required.Source: §A.8.1 · p. 17

A.8.2. Prompt Template

• Relative Reflectance is phrased as a three-way multiple-choice question about relative surface brightness.The passage introduces a prompt template but does not provide its wording.Source: §A.8.2 · p. 17

A.9.1. Data Sources and Splits

• Visual Similarity uses human-tested NIGHTS triplets [7], randomized into reference, candidate, and candidate images, with the label selecting the candidate judged more similar to the reference.NIGHTS supplies the training and validation examples.Source: §A.9.1 · p. 17
• To prevent overlap with BLINK Visual Similarity, near-duplicate NIGHTS triads are removed using CLIP pre-filtering followed by perceptual hashing and SSIM confirmation.After deduplication, 1,000 training and 250 validation triads are retained; testing uses the 135-example BLINK Visual Similarity validation split.Source: §A.9.1 · p. 17

A.9.2. Prompt Template

• Visual Similarity is phrased as a two-way multiple-choice question over candidate images.The passage introduces a prompt template but does not provide its wording.Source: §A.9.2 · p. 18

Single-task experiments

• Each of nine single-task models trains on 1,000 examples: Direct SFT runs 10 epochs, while LIVR runs 4 masked-bottleneck and 6 unmasked-fine-tuning epochs with K = 16.The reported checkpoint is the one with the highest validation accuracy.Source: B. Training Setup · p. 18

Multi-task experiments

• For six-task Qwen3-VL-4B-Instruct training, each task contributes 1,000 examples; Direct SFT runs 5 epochs and LIVR runs 2 masked plus 3 unmasked epochs with K = 16.Performance is reported from the final checkpoint.Source: B. Training Setup · p. 18

Optimization details

• Training uses LoRA on language-backbone attention and MLP modules while freezing the vision encoder and projector; LIVR also unfreezes embeddings for its latent tokens.LoRA uses r = 16, α = 32, and dropout 0.05; AdamW uses learning rate 1 × 10−4, weight decay 0.01, batch size 8, 5% warmup, and cosine decay.Source: B. Training Setup · p. 18

Mirage comparison

• The Mirage comparison uses K = 4 and 10 epochs per stage for both methods, preserving each method’s own two-stage objective.It uses full language-backbone fine-tuning, a frozen vision encoder and projector, AdamW at 1 × 10−5, effective batch size 8, and per-stage 5% warmup with cosine decay.Source: B. Training Setup · p. 18

Compute resources

• Experiments use seed 42 and single-GPU training runs on one node with eight NVIDIA RTX 6000 Ada GPUs.Source: B. Training Setup · p. 18

C. Additional Visualizations

• Latent tokens largely overlap image tokens in t-SNE, while a compact latent cluster occupies a distinct region, suggesting both visual-feature alignment and specialized representations.The visualization uses final-layer states from the first 50 evaluation examples of each of six tasks after multi-task training Qwen3-VL-4B-Instruct.Source: C. Additional Visualizations · p. 18, Figure 4 · p. 19
• The t-SNE analysis extracts final-layer hidden states for latent, image, and text tokens from 50 evaluation examples per task across the six-task mixture.The states are embedded with t-SNE after multi-task training Qwen3-VL-4B-Instruct.Source: C. Additional Visualizations · p. 18, Figure 4 · p. 19

C.1. Additional Latent-to-Image Attention Visualizations

• In Visual Correspondence and Semantic Correspondence, latent-token attention focuses on the referenced parking line and toothbrush handle, respectively.The attention maps show the parking line indicated by the REF point and the toothbrush handle as the relevant features.Source: §C.1 · p. 19, Figure 5 · p. 20
• For Jigsaw, latent tokens attend to a diagonal table ledge in the masked reference and locate it in the correct choice, B.The figure marks the correct answer with a check mark for image-choice tasks.Source: §C.1 · p. 19, Figure 5 · p. 20
• In Relative Reflectance, latent-token attention focuses on the two points the model must compare.Source: §C.1 · p. 19, Figure 5 · p. 20
• For Visual Similarity and Art Style, correct-answer attention maps resemble the reference-image maps, illustrating useful latent representations that may be hard to define explicitly.The paper displays raw attention maps for these two tasks.Source: §C.1 · p. 19, Figure 5 · p. 20