PaperlayerView exact revision on arXiv

Source-linked AI summary

ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both

Ziyu Guo, Rain Liu, Xinyan Chen, Pheng-Ann Heng

arXiv:2605.15198v1cs.CVcs.AIcs.CL

TL;DR

Visual reasoning needs intermediate visual analysis, but existing image-generating, agentic, and latent approaches impose efficiency or training limitations. ATLAS uses discrete functional tokens as compact visual operations within ordinary autoregressive generation, with LA-GRPO stabilizing sparse-token reinforcement learning. The framework achieves strong performance on challenging benchmarks with reduced inference latency and memory usage.

  • Problem

    Existing visual reasoning approaches face limitations from costly intermediate image generation, external execution, or latent representations that complicate generalization and autoregressive training.

  • Method

    ATLAS represents visual operations as discrete vocabulary tokens generated by next-token prediction, and uses LA-GRPO to strengthen optimization of sparse functional tokens during reinforcement learning.

  • Results

    ATLAS achieves strong performance on challenging benchmarks with reduced inference latency and memory usage.

  • Takeaways & Limitations

    Compact functional tokens provide an interpretable visual reasoning paradigm compatible with standard autoregressive training and reinforcement learning.

Abstract

from arXiv · show

Visual reasoning, often interleaved with intermediate visual states, has emerged as a promising direction in the field. A straightforward approach is to directly generate images via unified models during reasoning, but this is computationally expensive and architecturally non-trivial. Recent alternatives include agentic reasoning through code or tool calls, and latent reasoning with learnable hidden embeddings. However, agentic methods incur context-switching latency from external execution, while latent methods lack task generalization and are difficult to train with autoregressive parallelization. To combine their strengths while mitigating their limitations, we propose ATLAS, a framework in which a single discrete 'word', termed as a functional token, serves both as an agentic operation and a latent visual reasoning unit. Each functional token is associated with an internalized visual operation, yet requires no visual supervision and remains a standard token in the tokenizer vocabulary, which can be generated via next-token prediction. This design avoids verbose intermediate visual content generation, while preserving compatibility with the vanilla scalable SFT and RL training, without architectural or methodological modifications. To further address the sparsity of functional tokens during RL, we introduce Latent-Anchored GRPO (LA-GRPO), which stabilizes the training by anchoring functional tokens with a statically weighted auxiliary objective, providing stronger gradient updates. Extensive experiments and analyses demonstrate that ATLAS achieves superior performance on challenging benchmarks while maintaining clear interpretability. We hope ATLAS offers a new paradigm inspiring future visual reasoning research.

1 Introduction

Current visual reasoning methods either generate intermediate images, rely on external tools, or use latent embeddings, while dynamic visual reasoning remains limited. ATLAS represents visual operations as discrete functional tokens, and LA-GRPO strengthens their optimization during reinforcement learning.

  • Motivation: Dynamic visual reasoning remains limited despite progress in direct visual understanding, because some problems require intermediate visual analysis such as constructing auxiliary lines.These intermediate visual states can reveal hidden relations that purely textual reasoning may miss.
  • Existing paradigms: Unified models generate pixel-level intermediate images but incur substantial inference cost, training difficulty, and architectural complexity.Image decoding and re-encoding consume model capacity.
  • Existing paradigms: Agentic methods use external code or tools, whereas latent methods reason through latent embeddings, creating distinct efficiency or training constraints.Agentic approaches can require verbose operation formulations and external execution, while latent approaches are difficult to train with standard autoregressive procedures.
  • ATLAS: ATLAS represents visual operations such as zooming, drawing, and labeling as single discrete vocabulary tokens generated through ordinary next-token prediction.The tokens function as both agentic operations and latent reasoning units within the same autoregressive sequence.
  • ATLAS: ATLAS internalizes code generation, tool calling, and external execution into a compact interface while requiring no image-level supervision and retaining standard SFT and RL compatibility.Functional tokens are optimized with the standard cross-entropy objective over token sequences.
  • Training: LA-GRPO addresses gradient dilution by adding a statically weighted auxiliary loss anchored on the sparse functional-token vocabulary.The auxiliary objective supplies a persistent learning signal for functional tokens during RL.
  • Results: ATLAS enables compact single-token visual reasoning with strong benchmark performance and substantially reduced overhead.The authors report reduced inference latency and memory usage alongside strong performance on complex benchmarks.

2 ATLAS

ATLAS integrates visual reasoning into a standard autoregressive sequence through discrete functional tokens that represent visual operations without generating intermediate images or invoking external tools. It combines supervised training on functional-token trajectories with GRPO-based reinforcement learning and LA-GRPO to strengthen optimization of sparse functional tokens.

  • Functional-token architecture: ATLAS avoids intermediate image generation and external tool execution while preserving compatibility with existing VLM pipelines and efficient parallel training.The approach also avoids pixel-level intermediate supervision and recurrent latent dependencies.
  • Functional-token architecture: ATLAS represents visual operations as discrete functional tokens in the tokenizer vocabulary and generates them through ordinary next-token prediction.Functional tokens remain within the same autoregressive sequence as natural-language tokens.
  • Functional-token architecture: The five-token vocabulary Vfunc contains <|Manip|>, <|Shape|>, <|Line|>, <|Arrow|>, and <|Text|>, which abstract common visual operations into general categories.For example, <|Shape|> can represent bounding boxes, masks, cropping, and zooming.
  • Two-stage training: ATLAS-178K provides supervised reasoning trajectories with functional-token annotations, followed by standard GRPO for reward-guided optimization.The dataset covers visual reasoning trajectories derived from visual operations and supports the SFT stage.
  • Latent-Anchored GRPO: 2.3% of generated tokens are functional tokens on average, creating gradient dilution because sequence-level GRPO signals are spread across many ordinary text tokens.The reported average response contains 203.7 generated tokens, including only 4.8 functional tokens.
  • Latent-Anchored GRPO: LA-GRPO retains sequence-level GRPO while adding a token-level auxiliary objective focused on Vfunc to provide stronger, persistent updates for functional-token optimization.The auxiliary objective concentrates rollout-advantage updates on positions containing functional tokens.

3 Experiments

ATLAS improves visual-reasoning performance across challenging benchmarks while using functional tokens to support interpretable, task-relevant reasoning. Its LA-GRPO training and reward constraints improve stability, efficiency, and answer quality.

  • Quantitative Results: 51.3% BLINK average accuracy is achieved by ATLASLA-GRPO, compared with 22.8% for Qwen2.5-VL.
  • Quantitative Results: 45.0% WeMath and 51.3% BLINK average accuracy make LA-GRPO the strongest ATLAS variant on these benchmarks.LA-GRPO also raises multi-view reasoning from 43.6% to 53.4% relative to standard GRPO.
  • Qualitative Analysis: Functional tokens are invoked at meaningful reasoning steps, with Shape localizing regions, Arrow guiding evidence, and Text supporting counting and labeling.
  • Efficiency: ATLAS reduces per-query generation overhead and latency while improving accuracy relative to explicit agentic reasoning.
  • Attention Analysis: Functional tokens tend to attend to relevant visual regions, including geometry height segments and cat regions used for spatial comparison.
  • Ablation Study: 51.3 BLINK average accuracy is obtained by the full LA-GRPO objective, while removing token-spam penalties lowers it to 47.0.Removing the length penalty increases average sequence length by 43.8%, and removing token-spam penalties can produce up to 18.7 functional tokens per sequence.

4 Related Work

Related work follows two main routes for visual reasoning: agentic systems that use external tools and latent methods that encode intermediate computation in hidden representations. Both approaches improve efficiency in some settings but retain important limitations in verbosity, supervision, generalization, or compatibility with standard autoregressive training.

  • Agentic Visual Reasoning: Agentic visual-reasoning systems generate executable programs or code to invoke specialized vision modules and external tools.
  • Agentic Visual Reasoning: Agentic methods can require verbose code or tool calls, increasing output length and inference latency even for simple visual operations.
  • Latent Visual Reasoning: Latent visual reasoning moves intermediate computation from explicit text into compact hidden representations or latent visual trajectories.
  • Latent Visual Reasoning: Existing latent methods may rely on auxiliary supervision, reconstruction, or distillation targets, restricting flexibility and potentially limiting generalization beyond training setups.
  • Latent Visual Reasoning: Recurrent or non-standard computation patterns in latent methods deviate from standard next-token prediction and reduce compatibility with highly optimized autoregressive pipelines.

5 Conclusion

ATLAS represents visual operations as compact discrete functional tokens, avoiding intermediate image generation and external tool execution while preserving standard autoregressive training. Latent-Anchored GRPO stabilizes sparse-token optimization, and experiments report strong benchmark performance with lower inference latency and memory usage.

  • ATLAS represents visual operations as discrete functional tokens within the standard autoregressive vocabulary.The framework internalizes visual reasoning into compact tokens rather than generating intermediate images or using external tools.
  • ATLAS avoids intermediate image generation, external tool execution, and verbose operation formulations.These design choices reduce the need for expanded visual or tool-mediated reasoning sequences.
  • Latent-Anchored GRPO stabilizes optimization for sparse functional tokens during GRPO training.The method addresses gradient dilution associated with sparse functional-token updates.
  • ATLAS achieves strong performance on complex benchmarks with reduced inference latency and memory usage.
Loading 2605.15198v1…