Source-linked AI summary
Towards Pixel-Level VLM Perception via Simple Points Prediction
Tianhui Song, Haoyu Lu, Hao Yang, Lin Sui, Haoning Wu, Zaida Zhou, Zhiqi Huang, Yiping Bao, Y. Charles, Xinyu Zhou, Limin Wang
TL;DR
MLLMs have strong semantic abilities but remain limited in precise pixel-level localization, motivating a language-native segmentation approach. SimpleSeg predicts point sequences and trains them with SFT→RL, achieving performance comparable to or often surpassing complex decoder-based systems while exposing latent fine-grained perception in standard architectures.
Problem
MLLMs struggle to precisely localize and delineate fine structures, while existing segmentation approaches often rely on specialized decoders that leave fine-grained perception outside the language-native architecture.
Method
SimpleSeg reframes segmentation as textual point-sequence prediction and uses a two-stage SFT→RL pipeline with IoU-based rewards to refine generated geometry.
Results
SimpleSeg achieves performance comparable to or often surpassing complex decoder-based systems, demonstrating precise pixel-level perception beyond real-world objects.
Takeaways & Limitations
Precise spatial understanding can emerge from simple point prediction within a standard MLLM architecture, supporting more unified multimodal perception and reasoning.
Takeaways & Limitations
Long sequences bottleneck high-resolution, highly curved objects, with errors clustering at sharp corners and thin structures under aggressive sparsification.
Abstract
from arXiv · showhide
We present SimpleSeg, a strikingly simple yet highly effective approach to endow Multimodal Large Language Models (MLLMs) with native pixel-level perception. Our method reframes segmentation as a simple sequence generation problem: the model directly predicts sequences of points (textual coordinates) delineating object boundaries, entirely within its language space. To achieve high fidelity, we introduce a two-stage SF$\to$RL training pipeline, where Reinforcement Learning with an IoU-based reward refines the point sequences to accurately match ground-truth contours. We find that the standard MLLM architecture possesses a strong, inherent capacity for low-level perception that can be unlocked without any specialized architecture. On segmentation benchmarks, SimpleSeg achieves performance that is comparable to, and often surpasses, methods relying on complex, task-specific designs. This work lays out that precise spatial understanding can emerge from simple point prediction, challenging the prevailing need for auxiliary components and paving the way for more unified and capable VLMs. Homepage: https://simpleseg.github.io/
1 Introduction
SimpleSeg reframes segmentation as language-space point-sequence prediction, showing that standard MLLM architectures can support precise pixel-level perception without specialized decoders. Its SFT→RL training and benchmarks indicate strong generalization and performance comparable to or exceeding complex decoder-based systems.
- Motivation: MLLMs remain largely image-level, struggling to localize and delineate fine structures such as object boundaries and thin parts.The paper links this limitation to segmentation’s historical reliance on specialized decoders and complex architectural designs.
- Approach: SimpleSeg predicts sequential textual coordinates for segmentation masks entirely within the language space, without specialized decoders.This preserves the standard MLLM architecture and unifies points, boxes, and masks through a human-readable interface.
- Training: The two-stage SFT→RL pipeline uses supervised fine-tuning to learn the task format and IoU-based reinforcement learning to refine sequence-level geometry.RL targets fidelity and closure of the generated shape without changing the MLLM architecture.
- Results: SimpleSeg demonstrates precise pixel-level perception on both natural photographs and digitally generated or in-screen content, including anime, charts, and infographics.The examples are presented as evidence that its perception is not confined to real-world objects.
- Results: SimpleSeg achieves performance comparable to or exceeding complex decoder-based systems on standard referring segmentation benchmarks.The model is also reported to generalize across diverse domains and resolutions.
- Implications: The approach requires no specialized modules, produces interpretable coordinate sequences, and can support precise spatial localization across vision-language tasks.The output is described as useful for debugging, interactive editing, and tool use.
2 Related Work
Prior pixel-level MLLM methods largely divide between hybrid systems with specialized decoders and unified systems that represent masks as language sequences. The former compromise architectural simplicity, while the latter have struggled with segmentation fidelity.
- Hybrid approaches: Hybrid approaches augment general MLLM backbones with specialized, task-specific decoders for dense pixel-level perception.These designs create a tension between segmentation performance and architectural integrity.
- Comparison: The related work frames pixel-level MLLM perception as a central dilemma between specialized architectural additions and language-native mask representations.The two paths differ in how they balance task performance with preservation of the generalist MLLM design.
- Unified approaches: Unified approaches keep outputs in language space by representing masks as text sequences such as RLE or polygons.This aligns with an end-to-end LLM philosophy but has been associated with fidelity challenges.
3 Methodology
SimpleSeg keeps segmentation in the language space by predicting textual point trajectories, then trains them with supervised formatting followed by sequence-level reinforcement learning. Its representation unifies points, boxes, and masks while preserving interpretable, controllable outputs.
- Representation: SimpleSeg represents each mask as an explicit sequence of 2D contour coordinates predicted directly in the language space.The decoder-free design uses a textual point trajectory rather than a specialized segmentation head.
- Representation: Contour trajectories provide human-readable coordinates, compositional outputs, and token budgets that scale with vertices rather than image resolution.The representation sparsely samples boundaries instead of using dense per-pixel encodings.
- Task formulation: The unified query interface models targets as [text, point, bbox, mask] and recombines these elements into queries such as text→bbox and point→mask.This multiplies supervision sources and standardizes outputs for instruction tuning and reinforcement learning.
- Data construction: An automatic annotation pipeline uses object detection, SAM mask extraction, contour-coordinate conversion, and optional VLM description tagging to produce instance-level labels.The pipeline is designed to scale the framework with large-scale web data.
- Training pipeline: Training first uses SFT to teach valid coordinate formats and grounding priors, then GSPO-based RL optimizes sequence-level perception with location-aware and format rewards.The reward design includes mask IoU, centroid-distance IoU, and valid polygon formatting.
- Training pipeline: RL evaluates rendered masks rather than exact token matches, allowing alternative valid trajectories while improving closure and thin-structure adherence.This aligns optimization with the end segmentation metric because multiple contour sequences can represent the same mask.
4 Experiment
SimpleSeg is evaluated across referring segmentation, comprehension, training-stage, and exploration studies. It matches or exceeds complex decoder-based systems, while experiments show benefits from RL and a moderate point density.
- Main results: SimpleSeg achieves strong referring expression segmentation performance, comparable to or exceeding methods that use complex task-specific decoders.The evaluation covers refCOCO, refCOCO+, refCOCOg, and refCLEF.
- Main results: 87.2 average Acc@0.5 establishes state-of-the-art referring expression comprehension performance, exceeding the closest competitor Text4Seg.Predicted masks are converted to bounding boxes with min-max operations for this task.
- Training-stage ablation: Adding RL after SFT raises gIoU by 9.7, 9.8, and 10.5 points across three validation datasets.SFT alone reaches 65.5, 60.8, and 60.4 gIoU, while SFT+RL reaches 75.2, 70.6, and 70.9 respectively.
- Exploration studies: Moderate point density performs best: 221 tokens outperforms both 78-token underfitting and 859-token long-horizon decoding.The reported cIoU values are 35.6 at 78 tokens and 72.5 at 859 tokens; the moderate-density setting yields the best score.
- Exploration studies: Distance-based rewards provide an average gain of around 0.2, whereas hard length constraints degrade performance.The reward-design study compares distance and length-penalty components.
- Exploration studies: During RL, the model adaptively adjusts output length to balance token efficiency and mask fidelity without length-related rewards.Higher point density reduces redundant vertices, while a low token budget can prompt slightly longer responses for refinement.
- Exploration studies: Clockwise sampling order provides a valid, lower-entropy polygon target, whereas alternative orders can prevent mask derivation or confuse the model.The ordering study uses Suzuki–Abe boundary tracing to obtain clockwise contours.
- Extended tasks: The unified interface extends beyond referring queries to SAM-like point→mask and bbox→mask tasks, demonstrating broader perception-task generality.These extended-task results are visualized in the appendix.
5 Conclusion
SimpleSeg shows that predicting point sequences can unlock latent pixel-level perception in standard MLLM architectures. Its SFT→RL-trained model performs comparably to, and often surpasses, complex decoder-based systems.
- SimpleSeg reframes segmentation as point-sequence prediction to unlock native pixel-level perception in standard MLLM architectures.
- The model is cultivated through a novel SFT→RL pipeline.
- SimpleSeg achieves performance comparable to, and often surpassing, complex decoder-based systems.
A Limitations and Diagnostics
SimpleSeg’s main limitation is sequence length: high-resolution, highly curved objects remain difficult when many boundary points are required.
- Long point sequences remain a bottleneck for high-resolution, highly-curved objects.
- Aggressive sparsification concentrates errors at sharp corners and thin structures.
- Future diagnostics should supplement cIoU/Acc@0.5 with boundary F-score, vertex-wise Chamfer distance, and token-per-mask analyses across object scales.
B.1 Data Details
The paper uses large-scale open-source and web data for pre-training, while benchmark metrics are based exclusively on RefCOCO-derived SFT and RL data for fair comparison.
- Pre-training uses large-scale open-source and web data, including LAION and Coyo.
- SFT training uses 800k samples from train splits of refCOCO, refCOCO+, refCOCOg, and refCLEF.
- RL uses a 400k-sample prompt set derived from the RefCOCO series.
- Metrics in Tables 1 and 2 come from models trained only with SFT and RL on RefCOCO datasets, while web-data pre-training is evaluated in Table 3.
B.2 Training Details
Training hyperparameters are documented separately for the SFT and RL stages, with Table 4 specifically covering SFT settings.
- Tables 4 and 5 report the training hyperparameters for the SFT and RL stages.
- The training details distinguish SFT settings from RL settings across separate tables.
- Table 4 presents the training settings used for the SFT stage.
C Additional qualitative results
SimpleSeg is illustrated across diverse spatial tasks, including point-, box-, text-, panoptic-, multi-object, and multi-part segmentation. The examples also include failure cases involving holes and texture confusion.
- SimpleSeg provides qualitative results across point-to-mask and bounding-box-to-mask tasks.These examples visualize position information in the image.
- The examples extend to bounding-box-to-mask and text-to-mask tasks with visualized position information.
- Additional results cover text-to-point and text-to-bounding-box tasks.The position information is visualized in the image.
- The examples include polygon prompts and point-coordinate outputs for objects such as stairs, donuts, crosses, and staffs.
- Further qualitative examples address panoptic segmentation, multiple-object segmentation, and multi-part object segmentation.
- Failure cases include objects with holes and texture confusion, while other examples show text-guided recognition and point localization.