PaperlayerView exact revision on arXiv

Source-linked AI summary

Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought

Yunze Man, De-An Huang, Guilin Liu, Shiwei Sheng, Shilong Liu, Liang-Yan Gui, Jan Kautz, Yu-Xiong Wang, Zhiding Yu

arXiv:2505.23766v1cs.CV

TL;DR

Vision-centric reasoning can falter when accurate visual perception is needed, while explicit language-guided visual engagement remains underexplored. Argus uses grounded visual chain-of-thought to direct attention to relevant regions, and reports strong performance across multimodal reasoning and object-centric grounding benchmarks.

  • Problem

    Existing MLLMs can underperform in scenarios where accurate visual perception determines success in subsequent multimodal reasoning, and explicit language-guided visual engagement is underexplored.

    Source: §1 · pp. 1–2

  • Method

    Argus uses text-to-box object grounding as an intermediate reasoning stage, with predicted boxes acting as visual chain-of-thought signals to guide attention to prompt-relevant regions.

    Source: §1 · p. 2

  • Results

    Argus excels across diverse multimodal visual reasoning and object-centric visual grounding benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings suggest a promising direction for MLLM architectures that emphasize vision-centric mechanisms and visual chain-of-thought.

    Source: §5 · p. 8

    The study uses an 8-billion-parameter Llama3 decoder backbone, which may limit how generally its findings apply to larger model configurations.

    Source: D. Limitations and Future Work · p. 17

Abstract

from arXiv · show

Recent advances in multimodal large language models (MLLMs) have demonstrated remarkable capabilities in vision-language tasks, yet they often struggle with vision-centric scenarios where precise visual focus is needed for accurate reasoning. In this paper, we introduce Argus to address these limitations with a new visual attention grounding mechanism. Our approach employs object-centric grounding as visual chain-of-thought signals, enabling more effective goal-conditioned visual attention during multimodal reasoning tasks. Evaluations on diverse benchmarks demonstrate that Argus excels in both multimodal reasoning tasks and referring object grounding tasks. Extensive analysis further validates various design choices of Argus, and reveals the effectiveness of explicit language-guided visual region-of-interest engagement in MLLMs, highlighting the importance of advancing multimodal intelligence from a visual-centric perspective. Project page: https://yunzeman.github.io/argus/

1. Introduction

Argus addresses the underexplored problem of directing visual attention to the image regions most relevant to a language prompt. It uses object-centric grounding as visual chain-of-thought signals and reports strong performance across multimodal reasoning and object-centric grounding benchmarks.

1. Introduction

  • Existing MLLMs can underperform when accurate visual perception is essential to subsequent reasoning, such as for object relations or region properties.The paper notes that explicit language-guided visual engagement remains underexplored.Source: §1 · pp. 1–2
  • The paper asks how to introduce language-directed visual attention into MLLMs and whether explicit visual engagement benefits reasoning beyond perception tasks.These are the two questions posed in the introduction.Source: §1 · p. 2
  • Argus uses text-to-box object grounding as an intermediate reasoning stage, with predicted boxes serving as visual chain-of-thought signals for later reasoning.This explicit top-down search directs attention to image regions relevant to the prompt.Source: §1 · p. 2
  • Argus excels across diverse multimodal visual reasoning and object-centric visual grounding benchmarks.The introduction reports results across both task families.Source: §1 · p. 2

2. Related Work

Related work traces progress in multimodal reasoning, perception, and chain-of-thought, while identifying explicit goal-driven visual attention as a remaining gap. Argus positions grounded visual chain-of-thought as a way to connect visual grounding with vision-centric question answering.

Visual Reasoning with MLLMs

  • Recent MLLM work advances visual reasoning through alignment, high-resolution inputs, and dataset curation, alongside general-purpose proprietary models.The cited examples include GPT-4V, Claude 3, and Gemini.Source: §2 · p. 2
  • Vision-centric systems such as Cambrian-1 and Eagle study visual encoders and specialized processing, but the cited approaches lack explicit goal-driven visual token extraction.This gap motivates Argus's explicit visual attention mechanism.Source: §2 · p. 2

Visual Perception with MLLMs

  • Visual perception work includes expert-controller agents and unified MLLMs, but the related-work section says their synergy between perception and reasoning has received less focus.Argus instead explores grounded chain-of-thought for vision-centric reasoning.Source: §2 · p. 3

Chain-of-thought Reasoning

  • Prior multimodal chain-of-thought methods use detectors, multi-turn visual instructions, or interleaved segmentation and question answering; Argus studies explicit visual attention engagement as visual chain-of-thought signals.The paper frames this as bridging grounding and vision-centric VQA.Source: §2 · p. 3

3. Model

Argus augments the usual visual-token pipeline with prompt-conditioned region search and visual context re-engagement. The model compares implicit and explicit ways to use grounded regions as context for reasoning.

3. Model

  • Argus adds explicit language-directed region-of-interest search and context re-engagement to the usual visual-token and language-token autoregressive MLLM pipeline.Its visual encoder suite combines CLIP, ConvNeXt, and EVA-02 features using a mixture-of-vision-experts strategy.Source: §3 · p. 3, §3.1 · p. 3

3.1. Architectural Designs

  • For reasoning prompts, Argus predicts normalized text-format bounding boxes for relevant regions, including arbitrary regions beyond well-defined objects, and uses them to crop image context.The text-format boxes avoid additional box or mask decoding heads.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2. Directed Visual Context Re-engagement

  • The model compares implicit global self-attention and text-only box guidance with explicit visual re-engagement of grounded regions.Implicit box guidance uses predicted boxes as text or special tokens, while explicit methods supply actual visual tokens.Source: §3.2 · p. 4
  • Re-encoding processes expanded RoI crops as new images for tokenization, whereas re-sampling retrieves overlapping patch embeddings from the cached visual tokens.Re-sampling reduces encoding overhead and preserves positional context from the original image.Source: §3.2 · pp. 4–5, Figure 4 · p. 5

4. Experiments

The experiments evaluate Argus on multimodal reasoning and referring-expression grounding, then use controlled ablations to examine its design. Argus performs strongly across both task families, while the analyses identify benefits and trade-offs among visual re-engagement choices.

4.1. Training Pipeline

  • The training pipeline combines image-text pre-training with supervised fine-tuning on general multimodal, visual-CoT, and grounding data.Visual-CoT samples structure interactions around RoI prediction, visual-context signals, and a final response; fine-tuning updates the vision encoder, projectors, and decoder.Source: §4.1 · p. 5

4.2. Implementation Details

  • Argus-X3 uses Llama3-8B with three vision experts, and its RoI boxes are output as normalized text coordinates.The reported setup uses 1024×1024 inputs for ConvNeXt and EVA-02, 448×448 for CLIP, and trains one epoch per stage.Source: §4.2 · p. 5

4.3. Baseline Models and Benchmarks

  • The evaluation compares Argus with similarly sized MLLMs and grounding specialists across multimodal benchmarks, using benchmark metrics or prior MLLM settings when needed.Reported model performances come from official public reports; grounding uses Acc@0.5.Source: §4.3 · pp. 5–6

4.4. Main Performance

  • Argus achieves state-of-the-art performance among comparable-scale public MLLMs and leading performance among comparable-scale generalist models on referring grounding.The authors also report substantial improvements in vision-centric and text-understanding tasks and competitive performance against Grounding-DINO-L, a larger grounding specialist.Source: Table 1 · p. 6, §4.4 · p. 6, Table 2 · p. 7

4.5.1 Visual Attention Re-engagement Analysis.

  • The two explicit visual RoI re-engagement mechanisms lead the four-mechanism comparison, and explicit CoT guidance improves on implicit attention.CoT reasoning consistently enhances visual and text-based reasoning; re-sampling generally outperforms re-encoding, except on V-Star, where re-encoding preserves more small-object detail.Source: Table 3 · p. 7, §4.5.1 · p. 7
  • Higher-capacity vision encoders improve performance, while re-encoding is less dependent than re-sampling on the quality of initial visual features.Larger square contexts benefit re-encoding but can hurt re-sampling on text-centric OCR tasks; non-shared MLPs marginally improve re-sampling and produce the best overall performance.Source: §4.5.1 · pp. 7–8, Table 4 · p. 7

4.5.2 Grounding and Reasoning.

  • Adding the CoT dataset and re-engagement improves reasoning over Eagle-X3, and adding grounding datasets yields further gains in Argus.The authors attribute the latter gains to stronger object-centric perception and improved bounding-box predictions.Source: §4.5.2 · p. 8, Table 5 · p. 8

4.5.3 Computational Efficiency.

  • Re-sampling has a computational-efficiency advantage over re-encoding through reuse of patch embeddings, with fewer visual encoding operations and additional tokens.The authors report faster inference because LLM prediction is not blocked by visual encoding.Source: §4.5.3 · p. 8, Table 6 · p. 8

5. Conclusion

Argus combines grounded chain-of-thought with grounding-driven visual attention re-engagement. The paper reports strong performance in multimodal reasoning and referring-object grounding and presents vision-centric reasoning as a promising direction for MLLMs.

5. Conclusion

  • Argus introduces grounded chain-of-thought and grounding-driven visual attention re-engagement, with reported strong performance in multimodal reasoning and referring-object grounding.The authors suggest vision-centric mechanisms and visual chain-of-thought as a promising direction for future MLLM architectures.Source: §5 · p. 8

Supplementary Material

RoI context expansion has different effects across the two visual re-engagement strategies.

Supplementary Material

  • Larger context regions improve re-encoding, while re-sampling favors the original bounding-box size on ChartQA and V-Star.The passage describes the comparison as using RoI context expansion ratios on those benchmarks.Source: Supplementary Material · p. 14

A.1. RoI Context Expansion

Argus’s RoI context results depend on the re-engagement strategy: re-encoding benefits from moderate expansion, while re-sampling performs best with the original box. The appendix also reports multi-RoI gains, backbone comparisons, and the vision experts used in Argus.

A.1. RoI Context Expansion

  • Re-encoding performs best with 20–40% context expansion on ChartQA and V-Star, while re-sampling performs best at 0% expansion.Moderate expansion can compensate for tight or slightly inaccurate boxes and provide local and global context; resampling already uses overlapping patches intersecting the box.Source: §A.1 · p. 14
  • For both re-engagement strategies, overly large context regions hurt performance because irrelevant information distracts from relevant RoIs.The reported effect counteracts the advantages of RoI grounding and diminishes overall effectiveness.Source: §A.1 · p. 14
  • Argus generalizes across Vicuna 8B, Vicuna 13B, and Llama 8B backbones, with stronger backbones generally improving visual reasoning performance.The comparison includes vision-centric, text-understanding, and general task categories.Source: §A.1 · pp. 14–15
  • The multi-RoI extension improves performance by large margins on two visual reasoning benchmarks.It prompts for question-relevant RoIs, performs CoT reasoning for each object, then merges grounding boxes and textual thoughts into one joint signal.Source: §A.1 · p. 15, Table C · p. 15
  • The authors attribute CV-Bench3D’s performance discrepancy to multi-RoI data that is not extensively covered in training.They attribute MMMU and GQA discrepancies to language biases and suggest language-oriented training-data curation could improve performance.Source: §A.1 · p. 15

B.1. Visual Foundation Models and LLMs

  • Argus uses CLIP, ConvNeXt, and EVA-02 as vision experts, with inputs processed at 448 × 448 for CLIP and 1024 × 1024 for ConvNeXt and EVA-02.The implementation uses a ViT-L/14 CLIP checkpoint, ConvNeXt-XXLarge, EVA-02-L/16, and Meta-Llama-3-8B-Instruct.Source: §B.1 · p. 15

B.2. Implementation Details

The implementation describes two training stages, their compute requirements, and how visual encoder features are unified and projected into visual tokens.

B.2. Implementation Details

  • Argus uses stage 1 pre-training followed by stage 2 supervised fine-tuning, with stage 1 initializing vision experts from pre-aligned checkpoints and randomly initializing the MLP projector.Stage 2 initializes both vision experts and MLP projectors from stage 1 checkpoints; the global hyperparameters are listed in Table D.Source: §B.2 · pp. 15–16, Table D · p. 16
  • Stage 1 uses 32 NVIDIA A100 GPUs for 4 hours, and stage 2 uses 64 NVIDIA A100 GPUs for 28 hours.The second stage is supervised fine-tuning.Source: §B.2 · p. 16
  • Encoder embeddings are resized to 32×32, concatenated into a 32×32×5120 tensor, and projected to produce 1024 visual tokens.The multimodal MLP projector maps feature channels to the LLM hidden dimension of 4096.Source: §B.2 · p. 16

B.3. Training Dataset

Argus training combines multimodal instruction, visual CoT, and grounding data from several sources. The appendix also points to additional qualitative benchmark visualizations.

B.3. Training Dataset

  • Pre-training uses LLaVA-595K, while supervised fine-tuning uses a diverse mixture of datasets from multiple sources.The passage introduces the training-data section but does not enumerate the full mixture there.Source: §B.3 · p. 16
  • Eagle-1.8M contributes conversational data spanning visual question answering, documents, charts, and other reasoning scenarios.Its named sources include LLaVA-Instruct, DocVQA, ChartQA, GQA, and Geo170K, among others.Source: §B.3 · p. 16
  • VCoT provides paired bounding-box annotations and image-question pairs across datasets including TextVQA, DocVQA, GQA, and Visual7W.The listed collection also includes referring, document, and infographic tasks.Source: §B.3 · p. 16
  • GRIT links caption noun phrases and referring expressions to visual regions, while Shikra contributes 326K visual-grounding instruction samples.GRIT retains 756K samples after filtering entries with CLIP scores below 0.35; Shikra draws from RefCOCO-family datasets, Visual Genome, Visual-7w, and Flickr30K.Source: §B.3 · p. 16

C. Additional Qualitative Visualization

D. Limitations and Future Work

The authors identify limits in model scale, visual CoT data diversity, and evaluation coverage, and describe extensions for future study.

D. Limitations and Future Work

  • The study focuses on visual CoT mechanisms with grounding signals using an 8-billion-parameter Llama 3 decoder, leaving generalizability to larger scales unvalidated.The authors identify evaluation across a spectrum of model scales as a future extension.Source: D. Limitations and Future Work · p. 17
  • Available visual CoT signals remain limited in diversity, and the authors state that larger, higher-quality visual CoT data could yield substantial gains.They propose existing visual perception datasets or targeted human annotation as possible sources.Source: D. Limitations and Future Work · p. 17
  • Evaluation covers visual question answering and referring object grounding, but not open-world detection, which the authors leave for future work.They cite the substantial computational resources and time required for that investigation.Source: D. Limitations and Future Work · p. 17
Loading 2505.23766v1…