Source-linked AI summary
Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
Mahtab Bigverdi, Zelun Luo, Cheng-Yu Hsieh, Ethan Shen, Dongping Chen, Linda G. Shapiro, Ranjay Krishna
TL;DR
Multimodal language models struggle with visual perception tasks that call for intermediate depth or object representations, while fine-tuning and external tools have limitations. AURORA trains models to use perception tokens as intermediate reasoning steps, and reports gains on relative-depth and counting benchmarks. These results position perception tokens as a way for MLMs to reason beyond language alone.
Problem
MLMs struggle with visual perception tasks such as depth reasoning and counting, while fine-tuning has shown marginal improvements and external tools add computation and memory costs.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
The conclusion describes perception tokens as a lightweight, scalable way to integrate depth maps and bounding-box coordinates into MLM reasoning without external tools or task-specific fine-tuning.
Source: §5 · p. 8Fine-tuning with the original training mixture raises computational costs and may be impractical when that data, particularly proprietary data, is unavailable, limiting scalability.
Source: §3.4 · p. 5
Abstract
from arXiv · showhide
Multimodal language models (MLMs) still face challenges in fundamental visual perception tasks where specialized models excel. Tasks requiring reasoning about 3D structures benefit from depth estimation, and reasoning about 2D object instances benefits from object detection. Yet, MLMs can not produce intermediate depth or boxes to reason over. Finetuning MLMs on relevant data doesn't generalize well and outsourcing computation to specialized vision tools is too compute-intensive and memory-inefficient. To address this, we introduce Perception Tokens, intrinsic image representations designed to assist reasoning tasks where language is insufficient. Perception tokens act as auxiliary reasoning tokens, akin to chain-of-thought prompts in language models. For example, in a depth-related task, an MLM augmented with perception tokens can reason by generating a depth map as tokens, enabling it to solve the problem effectively. We propose AURORA, a training method that augments MLMs with perception tokens for improved reasoning over visual inputs. AURORA leverages a VQVAE to transform intermediate image representations, such as depth maps into a tokenized format and bounding box tokens, which is then used in a multi-task training framework. AURORA achieves notable improvements across counting benchmarks: +10.8% on BLINK, +11.3% on CVBench, and +8.3% on SEED-Bench, outperforming finetuning approaches in generalization across datasets. It also improves on relative depth: over +6% on BLINK. With perception tokens, AURORA expands the scope of MLMs beyond language-based reasoning, paving the way for more effective visual reasoning capabilities.
1. Introduction
Language alone is poorly suited to some visual perception problems, while fine-tuning and external vision tools have limitations. The paper introduces perception tokens and AURORA, reporting gains on relative depth and counting tasks.
1. Introduction
- Depth estimation and bounding-box grounding provide abstractions for visual reasoning, but fine-tuning yields marginal improvements and external tools add computation and memory costs.The paper motivates depth maps for 3D reasoning and boxes for instance reasoning.Source: §1 · pp. 1–2
- Perception tokens provide intrinsic image representations for reasoning where language is insufficient, such as depth maps for relative-depth judgments and box locations for counting.The model can generate these representations as intermediate steps before producing its answer.Source: §1 · p. 2
- AURORA trains MLMs to use perception tokens as intermediate reasoning steps, tokenizing depth maps with a VQVAE and encoding bounding boxes as structured tokens.It uses multi-task training and curriculum learning to avoid catastrophic forgetting.Source: §1 · p. 2
- LLaVA-AURORA improves BLINK relative depth by 6.4% points over fine-tuning, and reports counting gains of 10.8% points on BLINK, 11.3% points on CVBench, and 8.3% on SEED-Bench.The paper also describes its results as state of the art on relative depth estimation and object counting.Source: §1 · p. 2
2. Related work
Related work includes unified end-to-end MLMs, tool-using systems, and models that generate visual outputs. The paper distinguishes perception-token reasoning as enabling MLMs to reason over their own generated visual representations.
Related work
- Modern MLMs use cross-attention or visual instruction tuning, yet still struggle with mid- and low-level tasks such as counting, depth reasoning, and segmentation.The passage distinguishes end-to-end MLMs from tool-using MLMs.Source: §2 · p. 2
- End-to-end MLMs use a unified architecture across tasks, and diverse training datasets can support generalization and nuanced visual representations.Training can be costly because these models require large amounts of multi-task data [27].Source: §2 · p. 2
- Tool-using MLMs route inputs to specialized vision modules, which can improve performance on many tasks but create multiple potential points of failure.The router selects a module for each input.Source: §2 · pp. 2–3
- Any-to-any systems can generate visual outputs, but the paper says they cannot reason over those generations; perception tokens instead support direct visual reasoning for tasks such as depth and counting.The passage names Unified-IO, LISA, and GSVA among related approaches.Source: §2 · p. 3
3. Perception Tokens & Aurora
AURORA augments multimodal language models with auxiliary perception tokens so they can use intermediate visual representations in reasoning. It combines token prediction and reconstruction objectives with representation-specific tokenization and a curriculum that progresses toward more complex reasoning.
3.1. Problem formulation
- Existing MLM vocabularies are incompatible with intermediate visual features such as depth maps and bounding boxes, motivating auxiliary perception tokens for richer visual reasoning.AURORA introduces these tokens into MLM reasoning and training.Source: §3.1 · p. 3, §3 · p. 3
3.2. Perception token prediction and reasoning
- Specialist-to-generalist distillation trains auxiliary-token predictions, while a lightweight decoder maps tokens to target features for reconstruction supervision.The reconstruction objective is optional; distillation alone can reduce computational overhead.Source: §3.2 · pp. 3–4
3.3. Tokenization
- AURORA uses discrete visual tokens for pixel-level features such as depth maps and masks, and structured tokens for features such as poses, boxes, and coordinates.The framework describes these as tailored schemes within a unified tokenization space.Source: §3.3 · p. 4
3.4. Curriculum learning with progressive CoTs
- Table 1 reports that LLaVA-AURORA significantly outperforms the fine-tuning baseline and base model on relative depth accuracy, especially for 3-, 4-, and 5-point configurations.The caption identifies the compared models and task but provides no exact scores.Source: Table 1 · p. 5
- Using the original training mixture raises computational costs and may be impractical when original data, particularly proprietary datasets, is unavailable.This constraint makes that approach less scalable for adding new tokens.Source: §3.4 · p. 5
- The curriculum begins with atomic token-prediction tasks and advances through intermediate tasks to full chain-of-thought reasoning.Task difficulty is defined by reasoning depth, from newly introduced token prediction to complete reasoning chains.Source: §3.4 · p. 5
4. Experiments
LLaVA-AURORA applies perception tokens to relative-depth reasoning and object counting, using depth-map and bounding-box representations as intermediate steps. It outperforms the reported baselines on both tasks, and its decoded depth maps capture spatial relationships.
4. Experiments
- The experiments use LLaVA 1.5 13B as the base model and evaluate relative depth with depth-map tokens and counting with bounding-box tokens.The two tasks represent 3D and 2D visual reasoning, respectively.Source: §4 · p. 5
4.1. 3D reasoning task
- Relative-depth reasoning asks which marked point is closest to the camera; the model uses discrete depth-map tokens to represent spatial depth.The task is evaluated on BLINK and the more challenging HardBLINK configurations with additional markers and modified prompts.Source: §4.1 · p. 6, §4.3 · pp. 6–7
4.2. 2D reasoning task
- For counting, the model predicts bounding boxes for each target-object instance before answering the count query.Boxes are encoded as four pixel-position tokens, from the top-left to bottom-right coordinates, after resizing images to 336x336 pixels.Source: §4.2 · p. 6
4.3. Benchmarks
- The evaluation covers BLINK relative depth and counting, HardBLINK depth configurations, and counting benchmarks CVBench and SEED-Bench.Counting evaluation removes multiple-choice options and requires exact counts.Source: §4.3 · pp. 6–7
4.4. Baselines
- Baselines include the unadapted LLaVA 1.5 13B model, direct-labeling fine-tuning, other open- and closed-source models, and tool-augmented GPT-4 Turbo.For depth, GPT-4 Turbo receives ground-truth depth maps; for counting, it uses Grounding DINO boxes before counting.Source: §4.4 · p. 7
4.5. How new tokens improve 3D reasoning?
- As marker complexity increases, LLaVA-AURORA’s depth-task advantage grows, and it surpasses GPT-4 Turbo and GPT-4 Turbo + Tool using ground-truth depth maps.It also outperforms the direct-labeling fine-tuning baseline and the original base model on relative-depth estimation.Source: §4.5 · pp. 7–8
4.6. How new tokens improve 2D reasoning?
- LLaVA-AURORA outperforms the direct-labeling and base-model baselines, LLaVA-OneVision, and GPT-4 Turbo + Tool on object counting.It does not yet surpass advanced closed-source models.Source: Table 2 · p. 7, §4.6 · p. 8
4.7. Perception token decoding
- Decoded depth maps are evaluated for relative-depth accuracy across images with 2, 3, 4, and 5 marked points, and outperform Unified-IO models.The maps are reconstructed from LLaVA-AURORA’s depth tokens using the pretrained VQVAE decoder; qualitative examples show spatial details, though the maps are imperfect.Source: Figure 4 · p. 7, §4.7 · p. 8, Table 3 · p. 8
5. Conclusion
AURORA integrates depth maps and bounding-box coordinates as perception tokens for intermediate reasoning in multimodal language models. The paper reports state-of-the-art results on object counting and relative-depth estimation, alongside improved generalization and interpretability without external tools or task-specific fine-tuning.
5. Conclusion
- AURORA integrates depth maps and bounding-box coordinates as lightweight, scalable perception tokens for intermediate reasoning in multimodal language models.The framework is described as adaptable to new perception tokens.Source: §5 · p. 8
6. Ablation study
The ablations examine CoT prompt steps, token formats for counting, and reconstruction loss. Both CoT steps and perception tokens improve their respective tasks, while reconstruction loss adds little in the evaluated setup.
6.1. Chain-of-thought steps
- Including both point-location identification and depth-map reasoning steps in prompts gives the best performance on the harder BLINK relative-depth datasets.The compared variants include direct labeling, step (2) alone, and AURORA with both steps.Source: §6.1 · p. 11, Table 4 · p. 12
6.2. Text tokens vs. Perception tokens
- Perception tokens outperform standard text tokens across BLINK, SEED-Bench, and CVBench for object counting.The PIXEL X tokens encode bounding-box pixel locations; the comparison replaces them with ordinary numeric text tokens.Source: §6.2 · p. 11, Table 5 · p. 12
6.3. Perception token reconstruction loss
- Adding reconstruction loss does not significantly improve depth-token generation performance in the evaluated setup.The comparison uses 20,000 depth-generation samples and evaluates MSE on BLINK and Visual Genome subsets.Source: §6.3 · p. 11, Table 6 · p. 12
- Because reconstruction loss adds decoder-related computation, omitting it may be more efficient when its performance contribution is marginal.The study identifies larger datasets and more complex tasks as directions for further evaluation.Source: §6.3 · pp. 11–12
7. Cross-task generalization
AURORA is tested on CVBench’s depth subtask as a different depth-related task from BLINK. It outperforms the base LLaVA 1.5 13B model and the fine-tuning baseline there.
7. Cross-task generalization
- The evaluation uses CVBench questions asking which of two red- and blue-boxed objects is closer to the camera, with answer options removed.AURORA is trained on depth generation and chain-of-thought data for relative depth.Source: §7 · p. 12
- On CVBench’s Depth subtask, AURORA outperforms both base LLaVA 1.5 13B and the fine-tuning baseline.The authors present this result as evidence of generalization across depth-related tasks.Source: §7 · p. 12
- Figure 5 qualitatively compares predicted depth maps generated with and without reconstruction loss.The figure concerns reconstruction loss rather than the CVBench cross-task evaluation.Source: Figure 5 · p. 12
8. Implementation details
The implementation extends LLaVA 1.5 13B with depth and bounding-box tokens, then fine-tunes it using LoRA and a staged curriculum. Training data and decoding are structured around the depth and counting tasks.
Computation resources
- Training runs use single-node machines with 8 A40 GPUs and complete in less than 10 hours.Source: §8 · pp. 12–13
Model architecture and token expansion
- The tokenizer adds 130 depth-map tokens and 336 bounding-box coordinate tokens, requiring expanded token embeddings and language-model output layers.These additions expand the original 32,000-token vocabulary.Source: §8 · p. 13
Fine-tuning approach
- Fine-tuning applies LoRA to the language model while freezing the vision backbone and fully training the token-embedding and language-model-head layers.The model is trained for 10 epochs using cross-entropy next-token prediction.Source: §8 · p. 13
Inference and decoding
- Inference uses deterministic generation and constrained decoding; depth maps are emitted as exactly 100 tokens between DEPTH START and DEPTH END.The decoding setup uses temperature 0.Source: §8 · p. 13
Curriculum learning for reasoning
- The depth curriculum begins with 20,000 depth-generation samples, then progressively shifts the training mix toward multi-task reasoning data.By epoch 10, the mix is 2,000 depth-generation and 18,000 multi-tasking samples.Source: §8 · p. 13
Fine-tuning data
- Depth and counting training each use three sub-datasets: generation or box prediction, chain-of-thought reasoning, and direct labeling.Figures 6 and 7 show representative samples for the depth and counting sub-datasets.Source: §8 · p. 13, Figure 6 · p. 14, Figure 7 · p. 15