PaperlayerView exact revision on arXiv

Source-linked AI summary

Artemis: Structured Visual Reasoning for Perception Policy Learning

Wei Tang, Yanpeng Sun, Shan Zhang, Weihao Bo, Xiaofan Li, Piotr Koniusz, Wei Li, Na Zhao, Zechao Li

arXiv:2512.01988v2cs.CV

TL;DR

Perception-policy learning has relied on intermediate language reasoning and task-specific designs, leaving open how to support generalization across perceptual tasks. Artemis represents intermediate reasoning with verifiable label–bounding-box pairs and trains with rewards for reasoning and outcomes. The authors report generalization across perception tasks and visual domains, including zero-shot counting performance of 69.8 accuracy, compared with UniVG-R1 (59.7) and No-Thinking-RL (61.8).

  • Problem

    Existing perception-policy methods are task-specific, and the paper asks how to support consistent operation across diverse tasks and visual domains.

    Source: §1 · p. 2

  • Method

    Artemis uses label–bounding-box pairs as intermediate visual reasoning states and trains a unified perception policy with grounding, detection, and rewards for intermediate reasoning and final outcomes.

    Source: §1 · p. 2

  • Results

    Artemis generalizes across perception tasks and visual domains; on Pixmo-Count, it achieves 69.8 accuracy versus UniVG-R1 (59.7) and No-Thinking-RL (61.8) in zero-shot counting.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The authors conclude that aligning intermediate reasoning with spatial, object-centric evidence supports stronger perception across the evaluated settings.

    Source: §5 · p. 12

Abstract

from arXiv · show

Recent reinforcement-learning frameworks for visual perception policy usually incorporate intermediate reasoning chains expressed in natural language. Empirical observations indicate that such purely linguistic intermediate reasoning often reduces performance on perception tasks. We argue that the core issue lies not in reasoning per se but in the form of reasoning: while these chains perform semantic reasoning in an unstructured linguistic space, \textbf{visual perception requires reasoning in a spatial and object-centric space}. In response, we introduce \textbf{Artemis}, a perception-policy learning method that performs structured visual reasoning, where each intermediate step is represented as a (label, bounding-box) pair capturing a verifiable visual state. This design enables explicit tracking of intermediate states, direct supervision for proposal quality, and avoids ambiguity introduced by language-based reasoning. Building upon verifiable and spatially grounded reasoning chains, Artemis provides a unified architecture for diverse perceptual tasks, without requiring the task-specific designs relied upon by prior perceptual policy models. Trained using grounding and detection sampeles in natural image domains, Artemis generalizes to counting and geometric perception tasks. At its core, a spatially grounded, object-centric chain rule provides a principled foundation for scalable and general perceptual policies.

1 Introduction

Artemis argues that perception policies need spatial, object-centric reasoning rather than ungrounded linguistic reasoning. It represents intermediate reasoning as label–bounding-box pairs and uses a unified reinforcement-learning framework to support diverse perceptual tasks.

1 Introduction

  • Prior perception-policy methods use task-specific designs, limiting generalization across tasks such as grounding and counting and across natural images and diagrams.The paper identifies consistent operation across diverse perceptual tasks and visual domains as a goal for general perception systems.Source: §1 · p. 2
  • The paper frames the central question as which factors constrain perceptual generalization and what form of thinking benefits perception.Its motivating contrast is between language-centric reasoning and structured visual reasoning, rather than reasoning versus no reasoning.Source: §1 · pp. 2–3
  • Artemis represents intermediate reasoning as verifiable (label, bounding-box) pairs, grounding visual thought in spatial and object-centric evidence.This representation enables direct supervision of intermediate reasoning and the final answer.Source: §1 · p. 2
  • Artemis trains on visual grounding and object detection under a unified rule-based RL framework, with rewards for intermediate visual reasoning and final outcomes.Experiments span 7 perception and 10 general multimodal benchmarks; the authors report generalization across perception tasks and visual domains.Source: §1 · p. 2

2 Related Works

Prior RL-based reasoning work largely targets language or task-specific visual reasoning, while perception-policy methods have explored linguistic reasoning, no-think strategies, and grounded evidence. The paper positions unified, object-centric reasoning across perception tasks as an open direction.

2 Related Works

  • RLVR methods such as GRPO use verifiable rewards for reasoning, but MLLM approaches mainly target task-specific visual reasoning such as visual math or document understanding.The cited examples are visual math reasoning [33] and document understanding [35].Source: §2 · p. 3
  • Early perception-policy methods use explicit natural-language reasoning and improve task-specific abilities, but show limited gains on general perception tasks.The related-work discussion attributes the limitation to their linguistic search space.Source: §2 · p. 3
  • No-think and adaptive-think methods reduce linguistic reasoning, while grounded-reasoning approaches add visual evidence but lack direct supervision of intermediate object-centric states.The paper identifies how to represent and supervise perceptual reasoning as unresolved.Source: §2 · p. 3

3 Artemis Framework

Artemis uses GRPO to optimize perception policies with rewards for both final outcomes and structured intermediate visual evidence. Its reward design treats visual grounding and detection as spatially verifiable tasks and assigns greater weight to evidence matching the target object.

3 Artemis Framework

  • The framework section introduces GRPO, critiques outcome-only rewards, and then presents Artemis's reward design.Its reward components include format validity and task-specific answer accuracy, including spatial localization and label consistency.Source: §3 · pp. 3–4, §3.3 · p. 6

3.1 Preliminaries

  • GRPO compares each sampled output's reward with the group's mean and optimizes the policy using relative advantages, clipping, and KL regularization.Its rule-based rewards are directly verifiable and do not require a separate reward model; the passage reports efficiency and generalization advantages over SFT [18].Source: §3.1 · p. 4

3.2 Rethinking Rewards for Perception Policies

  • Outcome-only rewards provide sparse, delayed feedback on whether the model gathers or verifies visual evidence, and can favor plausible linguistic rationales over grounded reasoning.The paper argues that aligning rewards with intermediate perceptual steps matters because the reward objective shapes learning trajectories.Source: §3.2 · p. 4

3.3 Artemis Reward Design

  • Artemis combines outcome rewards with a structured visual reasoning reward that supervises intermediate object-centric evidence as well as final predictions.The reasoning output uses label–bounding-box pairs; outcome rewards supervise output format and final answers.Source: §3.3 · pp. 5–6
  • The key object receives the highest evidence weight, while other annotated objects receive smaller positive weights to retain contextual information.The weighting parameter η controls the key object's importance; the paper sets η = 0.8 empirically.Source: §3.3 · p. 5
  • For structured evidence, Artemis matches predicted boxes and labels to ground truth with the Hungarian algorithm, then rewards matched pairs using IoU and ROUGE thresholds.The reasoning reward is bounded in [0, 1]; high thresholds select high-quality evidence as informative contexts to stabilize RL training.Source: §3.3 · p. 5

4 Experiments

Experiments train Artemis on balanced grounding and detection data, then evaluate it on in-domain and zero-shot out-of-domain perception tasks. Ablations examine its reward components, reasoning format, training strategy, and transfer from natural images to counting and mathematical diagrams.

4.1 Implementation Details

  • Training uses roughly 77k Artemis-RFT instances with roughly balanced grounding and detection data, plus 80k non-overlapping COCO detection instances for cold start.The evaluation includes in-domain grounding and detection, zero-shot counting, mathematical diagrams, and general multimodal benchmarks.Source: §4.1 · p. 6

4.2 Experimental Results

  • Artemis reaches 31.0 mAP and 48.0 AP50 on COCO2017 val, while improving visual grounding over the base model across RefCOCO/+/g splits and general visual comprehension across multimodal benchmarks.Its AP50 and AR100 are 48.0 and 46.6, respectively, versus 46.7 and 41.2 for Perception-R1; the grounding comparison reports larger gains than task-specific RL-based MLLMs, especially at higher IoU thresholds.Source: §4.2 · pp. 7–9, Table 1 · p. 7, Table 2 · p. 8
  • On zero-shot Pixmo-Count and LISA evaluation, Artemis outperforms methods trained on Pixmo and reaches 69.8 accuracy on LISA, versus 59.7 for UniVG-R1 and 61.8 for No-Thinking-RL.Artemis enumerates instances during the think phase, while the compared counting approaches use detect-then-count post-hoc aggregation.Source: §4.2 · p. 8, Table 3 · p. 8
  • Artemis scores 49.3 on MATHGLANCE’s zero-shot average and transfers structured visual reasoning learned from natural images to mathematical visual scenes.MATHGLANCE covers plane geometry, solid geometry, and graph-based problems.Source: §4.2 · pp. 8–9, Figure 3 · p. 9

4.3 Ablation Study of Artemis

  • Grounding rewards alone particularly improve out-of-domain MATHGLANCE, detection rewards improve COCOmAP but reduce grounding accuracy, and adding structured visual reasoning improves all reported metrics.The full combination of grounding, detection, and structured visual reasoning rewards achieves the best balanced performance; removing reasoning or using linguistic reasoning yields poorer out-of-domain generalization than structured visual reasoning.Source: §4.3 · pp. 9–10, Table 5 · p. 9, Table 6 · p. 10
  • Unified grounding-and-detection training achieves the strongest results across out-of-domain perception benchmarks, although sequential training is competitive on in-domain grounding.On RefCOCOgval, Grd. →Det. scores 87.2 versus 86.9 for Det. →Grd.; the authors attribute unified training’s transfer to balanced supervision across tasks.Source: §4.3 · p. 10, Table 7 · p. 10
  • GRPO-based RL improves both detection and grounding under the same training samples where SFT variants yield limited or imbalanced gains, and cold-starting on detection achieves the best overall results among the reported cold-start strategies.Grounding-only SFT decreases grounding performance, while detection-only SFT improves detection but harms grounding; balanced detection data recovers performance when added to grounding cold-start data.Source: §4.3 · pp. 10–11, Figure 4 · p. 11, Figure 5 · p. 11

4.4 More In-depth Analysis of Artemis

  • Structured object-centric reasoning transfers to counting and mathematical diagrams, including zero-shot counting without counting-specific training or post-hoc processing and diagram tasks involving geometric shapes and relations.The paper reports a +11.3 improvement over VisionReasoner on counting; Artemis reasons by listing target objects and locations, while Qwen2.5-VL may duplicate or miss boxes and relies on post-processing for counts.Source: §4.4 · pp. 11–12

5 Conclusion

The conclusion presents Artemis as a unified rule-based RL framework that guides perception learning through structured visual reasoning. It reports stronger perception and broad in-domain and out-of-domain generalization.

5 Conclusion

  • Artemis guides visual perception learning through structured, verifiable reasoning supervision in a unified rule-based RL framework.The conclusion frames this approach as inspired by human perception of complex scenes.Source: §5 · p. 12
  • The model demonstrates substantial generalization across a broad range of in-domain and out-of-domain visual perception tasks.The conclusion attributes this to aligning intermediate reasoning with spatial and object-centric representations.Source: §5 · p. 12
  • The conclusion identifies spatial and object-centric intermediate reasoning as important to perception-policy learning.Source: §5 · p. 12

A Relation to Grounded Reasoning Methods

Artemis represents intermediate evidence with structured label–box states and keeps the visual input fixed, enabling deterministic process supervision during RL. It achieves similar inference time to VLM-R1 while delivering stronger perception and reasoning capabilities.

A Relation to Grounded Reasoning Methods

  • Artemis keeps visual input fixed and uses structured label–box states for intermediate evidence and deterministic process supervision during RL.Pixel Reasoner, ViGoRL, and DeepEyes rely mainly on visual revisiting or re-encoding, while Molmo/PixMo uses perception-aligned data and supervised fine-tuning.Source: A Relation to Grounded Reasoning Methods · p. 17, Table 8 · p. 17

B.1 Time Cost Comparison

  • Artemis achieves similar inference time to VLM-R1 while delivering much stronger perception and reasoning capabilities.The comparison measures runtime on RefCOCOg using batch size 1 on one NVIDIA A100 80GB GPU; Perception-R1 without intermediate reasoning is fastest but has comparatively lower perception performance.Source: §B.1 · p. 17

B.2 Results on Revisit-style Benchmarks

On revisit-style benchmarks, Artemis improves over Qwen2.5-VL-3B without explicitly revisiting or re-encoding image regions during inference. The accompanying reward details specify task formats and answer-scoring mechanisms for grounding and detection.

B.2 Results on Revisit-style Benchmarks

  • Artemis improves over Qwen2.5-VL-3B on V* and HR-Bench4K without explicit revisit-style inference.It achieves comparable relative gains over its own 3B backbone to Look-back-7B, suggesting complementary benefits to revisit-based methods.Source: §B.2 · pp. 17–18, Table 10 · p. 17

C.1 Format Rewards

  • Grounding outputs require structured think and answer blocks, whereas detection omits the think block and uses its own format validity scoring.The unified formulation accommodates both tasks while respecting their individual format requirements.Source: §C.1 · p. 18

C Details of Outcome Rewards

  • Outcome rewards assess perceptual correctness through format and answer components, with answer rewards measuring final-prediction accuracy.Grounding uses localization quality, while detection supplies dense scene-level supervision and evaluates localization and label consistency.Source: §C.2 · pp. 18–19

C.2 Answer Rewards

  • Detection answer scoring matches predicted and ground-truth boxes with the Hungarian algorithm, then combines localization, label match, and coverage.Source: §C.2 · p. 19
  • Detection reward applies quantity and missing-box penalties to the base score to encourage matching box counts and penalize unmatched ground-truth boxes.The quantity penalty targets agreement between predicted and ground-truth box counts; the missing penalty penalizes each unmatched ground-truth box.Source: §C.2 · p. 19

D Dataset Construction

Artemis-RFT is constructed from MS-COCO with grounding and detection subsets that provide spatial localization and scene-level supervision. The paper also specifies prompts and inference procedures for its tasks and comparisons.

D Dataset Construction

  • Artemis-RFT contains 39,651 grounding samples from 7,037 images and 37,446 detection samples from 37,446 images.The subsets average 5.6 and 1 sample per image, respectively.Source: D Dataset Construction · p. 19, Table 11 · p. 19
  • The dataset combines referring expressions, localized region captions, and standardized box annotations, ensuring each answer corresponds to a reasoning object.It uses MS-COCO images and annotations from LLaVA-665k and Vision-R1; detection annotations add scene-level supervision.Source: D Dataset Construction · pp. 19–20, Figure 6 · p. 20

E Prompt Settings

  • The paper provides task prompts and uses official evaluation prompts for MATHGLANCE and general visual-comprehension benchmarks.For other compared methods marked with †, it uses suitable official checkpoints and recommended prompt settings when available.Source: E Prompt Settings · p. 20, Table 12 · p. 21

F Qualitative Results

The qualitative results complement quantitative findings with examples spanning visual grounding, counting, and additional perception tasks.

F Qualitative Results

  • The appendix presents qualitative examples of visual grounding, visual counting, MATHGLANCE, COCO Val 2017, and LISA grounding.These visualizations complement the paper's quantitative results.Source: F Qualitative Results · p. 21
  • Table 13 provides Artemis's default prompt for object detection.Source: Table 13 · p. 22
  • Figure 8 compares Artemis with Perception-R1 and VLM-R1 on a RefCOCO/+/g grounding case.Purple marks Artemis reasoning boxes, red its answer, green ground truth, blue Perception-R1, and pink VLM-R1.Source: Figure 8 · p. 22

F.1 Qualitative Comparisons on RefCOCO/+/g

On RefCOCO/+/g, the qualitative comparisons contrast Artemis's grounding with Perception-R1 and VLM-R1. They show errors from skipping reasoning and mismatches between linguistic reasoning and final boxes.

F.1 Qualitative Comparisons on RefCOCO/+/g

  • Perception-R1's grounding is noticeably inaccurate in these cases, and skipping reasoning may limit its ability to learn perception beyond answer-box matching.The passage attributes this training tendency to supervision only through answer rewards and notes poor generalization to out-of-domain data or tasks.Source: §F.1 · pp. 22–23
  • VLM-R1 sometimes describes a scene correctly but selects the wrong or imprecise object, while other cases begin with an incorrect description.The examples include identifying two zebras when the image contains three and shifting a person target to a horse.Source: §F.1 · pp. 24–25
  • Artemis produces more precise and correct grounding results than Perception-R1 and VLM-R1 in the visualized cases.The comparisons include RefCOCO/+/g examples in Figures 8–10, with Artemis reasoning boxes and answer boxes shown separately.Source: Figure 9 · p. 23, Figure 10 · p. 24, §F.1 · p. 25

F.2 Qualitative Comparisons on Visual Counting

The visual counting cases show Artemis counting without task-specific training, while the compared methods use detections and exhibit box errors.

F.2 Qualitative Comparisons on Visual Counting

  • Although not trained on counting tasks or related data, Artemis internally enumerates instances and produces accurate counts in the shown Pixmo cases.Perception-R1 and Qwen2.5-VL-3B derive counts from detected objects instead.Source: §F.2 · p. 25
  • Perception-R1 over-detects one box and Qwen2.5-VL-3B produces duplicate boxes in Case 1, while both miss instances in Case 2.Source: §F.2 · p. 25
  • The authors describe Artemis's counting behavior in these cases as remarkably similar to human intuition.Source: §F.2 · pp. 25–26

F.3 Additional Qualitative Results

Additional examples show transfer from natural-image perception to mathematical diagrams, alongside object detection and reasoning-based grounding results.

F.3 Additional Qualitative Results

  • Artemis transfers perception capabilities learned from natural images to mathematical diagrams, including counting and relation-identification examples on MATHGLANCE.It identifies the quadrilateral and relational configuration correctly where Qwen2.5-VL makes incorrect choices.Source: §F.3 · p. 26, Figure 14 · p. 27
  • In the shown COCO Val 2017 detection cases, Artemis does not miss any ground-truth objects.Source: §F.3 · p. 26, Figure 15 · p. 28
  • On LISA grounding, Artemis locates relevant visual evidence for knowledge-intensive reasoning, such as the person and ladder used to infer an answer.The examples illustrate reasoning-based grounding across diverse tasks.Source: §F.3 · p. 27, Figure 16 · p. 28
Loading 2512.01988v2…