PaperlayerView exact revision on arXiv

Source-linked AI summary

ARGenSeg: Image Segmentation with Autoregressive Image Generation Model

Xiaolong Wang, Lixiang Ru, Ziyuan Huang, Kaixiang Ji, Dandan Zheng, Jingdong Chen, Jun Zhou

arXiv:2510.20803v2cs.CV

TL;DR

Existing MLLM segmentation methods rely on boundary-point representations or dedicated decoders, limiting pixel-level detail. ARGenSeg uses autoregressive image generation with direct image-token prediction and multi-scale generation, achieving state-of-the-art segmentation without extra segmentation heads and faster inference.

  • Problem

    Prior MLLM segmentation methods use boundary-point representations or dedicated decoders, limiting the MLLM’s understanding of pixel-level information.

    Source: §1 · p. 2

  • Method

    ARGenSeg integrates segmentation into a unified MLLM framework that directly predicts image tokens and uses multi-scale, coarse-to-fine generation.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    ARGenSeg achieves state-of-the-art segmentation without extra task-specific heads and significantly faster inference than existing MLLM-based methods.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The general-purpose visual tokenizer also supports extending the framework to image generation through fine-tuning on a small amount of image-generation data.

    Source: §1 · p. 2

Abstract

from arXiv · show

We propose a novel AutoRegressive Generation-based paradigm for image Segmentation (ARGenSeg), achieving multimodal understanding and pixel-level perception within a unified framework. Prior works integrating image segmentation into multimodal large language models (MLLMs) typically employ either boundary points representation or dedicated segmentation heads. These methods rely on discrete representations or semantic prompts fed into task-specific decoders, which limits the ability of the MLLM to capture fine-grained visual details. To address these challenges, we introduce a segmentation framework for MLLM based on image generation, which naturally produces dense masks for target objects. We leverage MLLM to output visual tokens and detokenize them into images using an universal VQ-VAE, making the segmentation fully dependent on the pixel-level understanding of the MLLM. To reduce inference latency, we employ a next-scale-prediction strategy to generate required visual tokens in parallel. Extensive experiments demonstrate that our method surpasses prior state-of-the-art approaches on multiple segmentation datasets with a remarkable boost in inference speed, while maintaining strong understanding capabilities.

1 Introduction

ARGenSeg frames segmentation as image generation within a unified MLLM, addressing the challenge of dense visual perception while retaining multimodal understanding. It predicts visual tokens for mask generation and uses next-scale prediction to accelerate inference.

1 Introduction

  • Prior MLLM segmentation methods either discretize masks into boundary points, producing incomplete masks and unnatural boundaries, or rely on dedicated decoders with insufficient pixel-level understanding.These decoder-based approaches condition on textual prompts [12] or MLLM hidden states [29] [45] [75], adding architectural complexity.Source: §1 · p. 2
  • ARGenSeg uses continuous image features as input and directly predicts quantized image tokens with a frozen pre-trained VQ-VAE, avoiding extra segmentation heads and downstream decoders.The universal tokenizer lets segmentation rely on the MLLM’s pixel-level visual understanding; direct image-token prediction is important for high segmentation accuracy.Source: §1 · pp. 2–3
  • ARGenSeg supports complex reasoning segmentation [29], anomaly detection, and other segmentation tasks [75], while fine-tuning on a small amount of image-generation data unlocks image generation.The paper contrasts its general-purpose visual tokenizer with UniGS [43] and HiMTok [57], whose approaches have limitations in understanding ability or generality and extensibility, as well as inference speed.Source: §1 · p. 2
  • Next-scale prediction generates visual tokens in parallel for over 4× speedup over sequential generation [19] [59], while coarse-to-fine mask generation boosts segmentation robustness.The multi-scale process follows coarse localization with fine-grained boundary refinement.Source: §1 · pp. 2–3
  • ARGenSeg outperforms existing MLLM-based segmentation methods with faster inference and substantially less segmentation data than prior state-of-the-art [57].The paper also reports state-of-the-art segmentation results without extra segmentation heads.Source: §1 · pp. 2–3

2 Related Work

Prior MLLM segmentation representations trade shape expressiveness or pixel-level detail for task-specific decoding, while unified generation and image-tokenization work offer components for an autoregressive alternative.

2 Related Work

  • Prior MLLM segmentation uses polygon sequences or semantic embeddings with dedicated decoders, often favoring high-level semantics over pixel-level understanding; HiMTok removes these components but has a limited binary-mask tokenizer.The section frames segmentation representation in MLLMs as an open challenge that autoregressive image generation may address.Source: §2 · p. 3
  • Unified understanding-generation models use diffusion-conditioned embeddings, joint token and diffusion prediction, shared discrete visual spaces, decoupled encoders, or next-scale generation with an extra visual decoder.The cited approaches include VARGPT, which uses next-token prediction for understanding and next-scale prediction for generation.Source: §2 · p. 3
  • Image tokenization converts autoregressive outputs into images: VQ-VAE quantizes downsampled latents, VQGAN improves reconstruction, TiTok reduces token counts, and VAR uses efficient next-scale prediction.TiTok also reports that more latent tokens consistently improve reconstruction quality.Source: §2 · p. 3

3 Method

ARGenSeg integrates segmentation into an MLLM by having it generate visual tokens that a VQ autoencoder detokenizes into images. It uses coarse-to-fine multi-scale generation to support pixel-level accuracy and efficient inference.

3.1 Preliminary

  • The VQ autoencoder quantizes images into discrete visual tokens and reconstructs them by looking up codebook embeddings and decoding the resulting latent features.The framework trains the MLLM to output these image tokens directly, grounding segmentation in generated pixels.Source: §3 · p. 3, §3.1 · p. 4
  • VAR’s next-scale prediction paradigm [50] generates each scale’s tokens in parallel, enabling coarse-to-fine refinement in K autoregressive steps.A 256×256 image can be represented with 680 visual tokens; the paper adopts this strategy to improve generation efficiency and quality.Source: §3.1 · p. 4

3.2 Architecture

  • ARGenSeg uses one autoregressive framework for text understanding and image generation, with a unified prediction head producing text or visual tokens based on the instruction.Special <gen_start> and <gen_end> tokens mark image generation; the visual tokens are detokenized by the visual tokenizer to reconstruct the image.Source: §3.2 · p. 5

3.3 Training Procedure

  • Training freezes the vision encoder and visual tokenizer, initializes components from InternVL 2.5 [13] and VAR [50], and uses single-stage supervised fine-tuning on segmentation and multimodal understanding data.Cross-entropy supervision teaches when to start generation and how to predict visual tokens, with ground-truth tokens obtained from the VQ-VAE encoder and quantizer.Source: §3.3 · p. 5

3.4 Inference

  • At inference, <gen_start> initiates the first scale, and upsampled, projected tokens from each prior scale query parallel prediction at the next scale.After all K scales are generated, the VAR tokenizer decodes them into the final image, and <gen_end> is added to continue inference.Source: §3.4 · pp. 5–6

4 Experiments

ARGenSeg unifies segmentation and multimodal understanding, outperforming prior methods on referring-expression benchmarks while retaining or improving understanding ability. Its parallel generation is faster than sequential approaches, and the framework extends to interactive segmentation and image generation.

4.1 Experimental Setup

  • Training uses 402K publicly available segmentation samples and 1.25M multimodal-understanding samples; the segmentation set is a strict subset of HiMTok’s 2.91M samples.The model is jointly trained on segmentation and multimodal-understanding data in a single-stage supervised finetuning setup.Source: §4.1 · p. 6

4.2 Referring Segmentation

  • ARGenSeg consistently outperforms HiMTok on RefCOCO/+/g across mixed-data and in-domain-finetuned versions, despite using fewer segmentation samples and no dedicated segmentation head.The comparison supports the effectiveness of the unified multimodal understanding and generation framework.Source: §4.2 · p. 6
  • On gRefCOCO, ARGenSeg outperforms all prior approaches that rely on dedicated segmentation heads.This benchmark includes instructions referring to multiple objects or none at all.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Multumodal Understanding

  • Adding segmentation supervision retains and slightly enhances visual grounding, while improving POPE object-hallucination performance over an understanding-only baseline.The evaluation uses RefCOCO/+/g for referring expression comprehension and POPE for object hallucination.Source: §4.3 · p. 7, Table 3 · p. 8

4.4 Function Extension

  • Finetuning enables interactive segmentation from point, scribble, and text-encoded box prompts, and class-conditional and instruction-based image generation without a pretrained generation model.Image generation uses 20k class-based and 30k instruction-based training iterations; interactive segmentation adapts quickly from pretrained segmentation capabilities.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5 Efficiency Analysis

  • Parallel inference is more than 10× faster than sequential Emu3 generation, and ARGenSeg is approximately 2× more efficient than VARGPT.ARGenSeg directly predicts VAR codebook token IDs with the classification head, avoiding an additional transformer-based visual decoder; PixelLM is slightly faster but has significantly lower segmentation performance, while ARGenSeg is faster than HiMTok at superior performance.Source: §4.5 · pp. 8–9, Table 4 · p. 9

4.6 Ablation Study

  • Ablations favor understanding-data supervision, a single-stage training strategy, and the multi-scale visual tokenizer.Understanding data particularly improves reasoning-based segmentation on RefCOCO+/g; a two-layer projector is unnecessary, and projector-only pretraining brings only marginal gains on RefCOCO+/g and little impact on RefCOCO while increasing complexity. The multi-scale tokenizer improves speed and robustness over a single-scale tokenizer through coarse-to-fine refinement.Source: §4.6 · p. 9

5 Conclusion

ARGenSeg integrates segmentation into an MLLM through image generation, with the MLLM directly producing image tokens and using coarse-to-fine multi-scale generation. The paper reports state-of-the-art segmentation without task-specific segmentation heads.

5 Conclusion

  • ARGenSeg demonstrates state-of-the-art segmentation with a unified MLLM and no extra task-specific segmentation heads.The authors describe this as an effective technical pathway toward unified AGI.Source: §5 · p. 9
  • The framework has the MLLM directly output image tokens to support pixel-level accuracy.This integrates image segmentation into multimodal large language models through an image-generation paradigm.Source: §5 · p. 9
  • Multi-scale image generation uses coarse-to-fine refinement for high responsiveness and robustness.Source: §5 · p. 9

NeurIPS Paper Checklist

The checklist answers yes that the abstract and introduction reflect the paper’s contributions. It states that claims should match the paper’s results and scope.

NeurIPS Paper Checklist

  • The authors answer yes, stating that the abstract and introduction accurately reflect the paper’s contribution.The justification says their contents clearly reflect the contribution.Source: NeurIPS Paper Checklist · p. 15
  • The checklist says claims should state contributions and important assumptions or limitations, and match the results and expected generalizability.Aspirational goals are acceptable when clearly presented as goals not attained by the paper.Source: NeurIPS Paper Checklist · p. 15

2. Limitations

The authors say they analyzed computational efficiency and placed a Limitations subsection in the appendix; they report no theoretical results and no error bars because of resource limitations.

2. Limitations

  • The authors report analyzing computational efficiency and placing a Limitations subsection in the appendix.Source: §2 · p. 15

3. Theory assumptions and proofs

4. Experimental result reproducibility

  • The authors say the method structure, dataset, and training details are explained.Source: §2 · p. 16

5. Open access to data and code

  • The data and underlying model are publicly accessible, according to the authors.Source: §2 · p. 17

6. Experimental setting/details

  • The authors say training and test details are explained in the Experiment section.Source: §2 · p. 17

7. Experiment statistical significance

  • No error bars are reported because of resource limitations.Source: §2 · p. 17

8. Experiments compute resources

  • The paper provides the platform and experiment running time in the main text.Source: §2 · p. 18

10. Broader impacts · Appendix of ARGenSeg

The appendix reports that the paper discusses societal impacts and summarizes its release plans, asset documentation, human-subjects status, and use of LLMs in core methods.

10. Broader impacts

  • The authors affirm that the paper discusses both potential positive and negative societal impacts, without specifying them in these passages.Source: §10 · p. 18
  • The authors plan to release no data; they currently have no model-release safeguards but say they will provide guidelines and instructions when releasing the model.Source: §10 · p. 19
  • The paper introduces no new assets, so accompanying documentation for new assets is marked not applicable.Source: §10 · p. 20
  • The paper involved neither crowdsourcing nor human-subjects research, so the authors mark participant instructions and IRB-related disclosures not applicable.Source: §10 · p. 20
  • The authors mark LLM usage not applicable because core method development does not use LLMs as important, original, or non-standard components.Source: §10 · p. 21

A Implementation Details

ARGenSeg uses substantially smaller segmentation and multimodal-understanding training subsets than HiMTok [57], with image-generation data reserved for an optional extension stage. At inference, it reconstructs outputs from visual-token logits, using task-specific sampling strategies.

Datasets

  • 402K segmentation and 1.25M multimodal-understanding samples are used, versus HiMTok’s 2.91M and 4.2M, respectively; image-generation data are reserved for an optional function-extension stage.The subsets are drawn from data used by HiMTok [57] to ensure a fair comparison.Source: A Implementation Details · p. 22

Inference Details

  • Visual outputs come exclusively from MLLM visual-token logits; segmentation uses deterministic argmax, while image generation uses classifier-free guidance followed by top-k sampling.The visual-token constraint maintains compatibility with the tokenizer and enables image reconstruction; top-k sampling is used to enhance generation diversity and quality.Source: A Implementation Details · p. 22

B Additional Qualitative Results · C Additional Quantitative Results

ARGenSeg’s qualitative results show that its multi-scale generation follows distinct segmentation instructions and, compared with single-scale HiMTok, can localize targets among similar objects and refine masks to recover occluded regions.

B Additional Qualitative Results

  • ARGenSeg correctly follows distinct instructions to segment similar objects in the same image, demonstrating understanding of target locations and semantic relationships.The multi-scale mask-generation visualizations illustrate this instruction-following ability (Fig. 5).Source: B Additional Qualitative Results · p. 22
  • Compared with HiMTok [57], ARGenSeg’s coarse localization identifies the target object in scenarios containing multiple similar objects.The comparison highlights cases where ARGenSeg succeeds and HiMTok fails (Fig. 6).Source: B Additional Qualitative Results · p. 22
  • ARGenSeg’s multi-scale refinement progressively improves mask precision after the target is identified.This coarse-to-fine process is presented as an advantage over single-scale generation.Source: B Additional Qualitative Results · p. 22
  • For a partially occluded teddy bear, ARGenSeg’s fine-grained refinement reconstructs the whole object while excluding the occluder.Both HiMTok and ARGenSeg’s initial coarse stage segment only the visible part.Source: B Additional Qualitative Results · p. 22

C.1 Performance on Multimodal Understanding · C.2 Results on Interactive Segmentation · D Additional Ablation Studies

ARGenSeg retains multimodal reasoning despite slight benchmark drops attributed to the smaller, lower-quality fine-tuning corpus. On COCO-Interactive, it outperforms SAM and performs comparably to PSALM with less fine-tuning; additional ablations examine MLLM backbones and image-generation strategies.

C.1 Performance on Multimodal Understanding

  • Segmentation data does not cause ARGenSeg to lose reasoning capability, although some multimodal benchmarks show slight performance drops.Table 8 reports multimodal understanding results across benchmarks.Source: §C.1 · pp. 22, 24, Table 8 · p. 24
  • The authors attribute the slight drops to fine-tuning on 1.25M lower-quality understanding samples rather than segmentation, compared with InternVL-2.5’s 16.3M samples [13].Fine-tuning InternVL-2.5 on the same understanding data for increasing steps produced a monotonic decline, mirroring the joint-training trend.Source: §C.1 · p. 24
  • Table 8 covers ARGenSeg’s multimodal understanding results across benchmarks.The supplied passage does not provide benchmark values.Source: Table 8 · p. 24

C.2 Results on Interactive Segmentation

  • Interactive-segmentation evaluation uses COCO-Interactive [75] and cIoU; HiMTok is omitted from the main experiments because it was not trained on this task.ARGenSeg received one fine-tuning epoch due to computational constraints, while PSALM used ten under its official implementation.Source: §C.2 · p. 24
  • On COCO-Interactive, ARGenSeg significantly outperforms SAM and achieves performance comparable to PSALM [75] after one fine-tuning epoch versus ten for PSALM.The comparison reports cIoU; PSALM’s results are sourced from its paper.Source: Table 9 · p. 24, §C.2 · p. 24

D Additional Ablation Studies

  • Additional ablations examine MLLM backbones and image-generation strategies, measuring segmentation performance with cIoU.The supplied table caption gives the ablation scope but no outcomes.Source: Table 10 · p. 24

D.1 Ablation on MLLM Backbone … F Broader Impacts

The ablations support backbone portability and direct visual-token generation for pixel-level segmentation; the paper also identifies future extensions, potential broader applications, and deployment risks.

D.1 Ablation on MLLM Backbone

  • Replacing InternVL-2.5 with LLaVA-1.5 [33] successfully transfers segmentation capability, supporting the VQVAE-codebook approach’s model-agnostic design.The paper describes the VQVAE codebook as integrated into the MLLM token space.Source: §D.1 · p. 24
  • With the same InternVL-2.5 backbone, ARGenSeg outperforms HiMTok, while LLaVA-1.5’s weaker understanding ability corresponds to lower segmentation performance.The comparison indicates that the gains are not merely due to a stronger backbone.Source: §D.1 · p. 24

D.2 Ablation on Image Generation Strategy

  • ARGenSeg-DiT produced roughly localized masks but severe spatial-shift and inflation artifacts, underscoring the need for direct discrete-token generation for pixel-level precision.The alternative uses MLLM-generated learnable queries mapped through a connector to pre-trained SANA-1.5 1.6B features, inspired by MetaQuery [41].Source: §D.2 · p. 25, Figure 7 · p. 25

E Limitations

  • The unified autoregressive-generation framework shows strong performance across segmentation tasks, while interactive segmentation, text-to-image generation, image editing, and depth estimation remain future directions.Resource constraints left exploration of these extensions beyond the scope of this work.Source: E Limitations · p. 25

F Broader Impacts

  • By integrating dense segmentation into multimodal understanding and generation, ARGenSeg may inspire more generalizable, modular, efficient visual-language models with fewer task-specific components.Potential applications include human-robot interaction, assistive vision systems, and real-world visual understanding under low supervision.Source: F Broader Impacts · p. 25
  • ARGenSeg may inherit biases from pre-trained components or datasets, so real-world deployment warrants fairness and robustness evaluation, especially in healthcare or surveillance.Source: F Broader Impacts · p. 25

Related papers

Loading 2510.20803v2…