WeVisOne-VL: Unified Visual Perception from Objects to Pixels

1 Introduction

• The work addresses persistent VLM errors including hallucinated objects, misinterpretation of unfamiliar descriptions, and overlooked non-salient instances.The paper links these errors to incomplete recall and unsatisfactory real-world deployment performance.Source: §1 · p. 2
• WeVisOne-VL combines object- and pixel-level perception through a natural-language interface, structured next-token prediction, and a lightweight pixel decoder.It is built on Qwen3-VL at 2B and 4B scales and trained on approximately 100M images with fine-grained annotations.Source: §1 · p. 2
• The model uses dedicated coordinate tokens and an <OBJ> token whose hidden state lets a lightweight decoder produce dense predictions from shared ViT features.The design supports discrete and dense predictions without relying on a separate large segmentation model.Source: §1 · p. 2
• Training proceeds through three SFT stages, then mixed-task RL on uncertain samples to improve localization accuracy and suppress repeated or hallucinated predictions.The stages establish task alignment, add high-resolution and dense perception data, and then incorporate pixel-level tasks.Source: §1 · p. 2
• WeVisOne-VL demonstrates highly competitive performance against state-of-the-art general-purpose models and specialized vision systems across multiple visual benchmarks.Reported strengths include object-level and pixel-level tasks, crowded-scene recall, and rejection when a target is absent.Source: §1 · pp. 2–3

2 WeVisOne-VL Data Engine

• WDE combines over 100 million images and approximately 3.6 billion box annotations, then links acquisition, task-specific localization, mask generation, matting, and reviewed-error refinement.Its sources include GRIT, CC3M, LAION, SA-1B, and task-specific datasets.Source: §2 · p. 3

2.1 Concept Discovery and Targeted Acquisition

• A shared knowledge graph organizes category concepts and compositional referring expressions, guiding targeted acquisition toward semantic gaps and difficult visual conditions.Category concepts support category-based localization, while referring expressions select instances through attributes, states, actions, and relations.Source: §2.1 · p. 4

2.2 Detection Annotation

• Detection annotation merges proposals from SAM3 and an open-vocabulary DINO-style detector, then uses Qwen3-VL to verify matches, remove duplicates, and resolve conflicts.Accepted boxes also define the target instances for subsequent segmentation annotation.Source: §2.2 · p. 4

2.3 Referring Annotation

• For referring annotation, SAM3 and Rex-Omni propose regions, and Qwen3-VL checks each against the expression’s attributes, states, actions, and relations before binding targets.The semantic check distinguishes intended targets from objects mentioned only as context and handles whole objects or parts.Source: §2.3 · p. 4

2.4 Mask Annotation

• Each localized instance receives a SAM2 binary mask linked to its box query, providing paired box and mask supervision for detection, localization, and segmentation tasks.A subset is manually spot-checked for correct instance links and mask accuracy, with identified defects corrected.Source: §2.4 · p. 5

2.5 Hard-Example Construction

• Hard negatives are reviewed to ensure no object satisfies every query condition; hard positives capture missed targets, referent errors, and difficult localization.Negative examples retain empty target sets, while reliable positive annotations are kept; sampling emphasis is reassessed as model errors change.Source: §2.5 · p. 5

2.6 Matting Data Engine

• The matting pipeline combines expert-assisted annotation, a dedicated annotator, and corpus re-annotation to produce alpha labels linked to original instance descriptions and boxes.The annotator’s predictions replace initial labels, while the instance links keep language grounding and matting supervision on the same target.Source: §2.6 · pp. 5–6

2.7 Training Feedback and Data Refinement

• Reviewed errors drive targeted acquisition, hard-example sampling, or annotation correction, while unresolved cases are excluded and formal benchmarks remain isolated from data construction.Model disagreement triggers investigation; annotations are changed only after review.Source: §2.7 · p. 6

3 Model Design

• WeVisOne-VL extends Qwen3-VL with dedicated task and coordinate tokens and a lightweight decoder that converts discrete representations into segmentation and matting predictions.The architecture is designed to handle a broad range of visual tasks as structured next-token prediction.Source: §3 · p. 7

3.1 Task Formulation

• All tasks use structured next-token prediction: coordinate tokens encode normalized points or boxes, None marks absent targets, and pixel tasks append <OBJ> after each box.Box coordinates use [x0, y0, x1, y1], while point outputs use [x0, y0].Source: §3.1 · p. 7, Figure 3 · p. 7

3.2 Model Architecture

• For pixel tasks, the decoder combines each <OBJ> hidden state and predicted box with multi-level ViT features to produce full-image masks or alpha mattes.Segmentation and matting share the decoder components; matting adds a lightweight refinement module for fine boundaries and fractional opacity.Source: §3.2 · pp. 7–8, Figure 4 · p. 8

3.2.1 Visual Feature Adapter

• The visual feature adapter projects intermediate ViT features into a shared decoder space, adds positional and level information, and concatenates levels into visual memory.The features are taken before ViT merging to preserve fine-grained visual information for dense prediction.Source: §3.2.1 · p. 8

3.2.2 Box-guided Query Decoder

• Instance queries come from VLM outputs rather than static learned embeddings: each content query projects an <OBJ> state, and its positional query encodes the predicted box.Six deformable blocks use self-attention among instance queries and multi-scale deformable cross-attention over visual memory.Source: §3.2.2 · pp. 8–9

3.2.3 Instance-Conditioned Dense Prediction

• The dense prediction stage fuses multi-level visual features, predicts instance masks, and refines them across scales; the input image adds shallow detail features for color, texture, and edges.An instance-aware dynamic kernel produces pixel-wise logits, followed by coarse-to-fine refinement.Source: §3.2.3 · p. 9

4 Training Pipeline

• Three-stage SFT establishes a shared output interface, scales resolution and density, and connects autoregressive localization with pixel-level prediction before mixed-task RL.RL optimizes complete-prediction rewards rather than individual token likelihoods.Source: §4 · p. 10, §4.2 · p. 11

4.1 Curriculum Supervised Fine-Tuning

• Stage 1 trains task alignment and structured outputs; Stage 2 unfreezes the vision encoder and raises capacity to 1,960 visual tokens and 500 boxes or points per response.Stage 1 limits examples to at most 50 boxes or points per category to mitigate repetitive output collapse.Source: §4.1 · p. 10
• Stage 3 bridges the VLM and pixel decoder through <OBJ>, then alternates next-token and pixel-level training before specializing the decoder for alpha prediction.Decoder warm-up uses 250M segmentation samples; matting fine-tuning uses 130M samples while keeping the VLM frozen.Source: §4.1 · pp. 10–11

4.2 Mixed-Task Reinforcement Learning

• Uncertainty-aware selection favors prompts with high reward variance and mixed rollout success, selecting about 140K samples before targeted examples bring the RL set to 178K.The selection score targets moderately difficult samples that provide stronger relative-advantage signals.Source: §4.2 · p. 11
• GRPO evaluates complete structured responses using box grounding rewards based on one-to-one Hungarian matching within semantic classes.The reward penalizes missed and duplicate objects and cubes IoU to emphasize improvements at high overlap; segmentation and matting RL use box rewards only.Source: §4.2 · p. 11

5.1.1 Object Detection

• WeVisOne-VL consistently outperforms VLMs in mean F1 across general, long-tailed, and dense detection benchmarks, with especially pronounced gains on Dense200 and VisDrone.Its results at IoU=0.95 also indicate improvements in precise localization.Source: Table 2 · p. 12, §5.1.1 · p. 12

5.1.2 Referring Object Detection

• WeVisOne-VL outperforms existing VLMs across single-target, multi-target, and absent-target referring detection settings, localizing targets more accurately and rejecting absent queries more reliably.Table 3 reports mean F1 and rejection rates for absent-target queries.Source: §5.1.2 · pp. 12–13, Table 3 · p. 13

5.1.3 Object Counting

• 93.7%/89.6% exact-match accuracy on PixMo-Count validation/test and 42.0%/40.4% on MixCount validation/test; WeVisOne-VL has the highest accuracy on every reported benchmark and split.PixMo-Count covers natural images, while MixCount targets fine-grained open-vocabulary counting in mixed-object scenes.Source: Table 4 · p. 13, §5.1.3 · pp. 13–14

5.1.4 Object Pointing

• WeVisOne-VL achieves the best results on PointBench and PixMo-Points and performs strongly on detection-derived benchmarks.These results indicate that its pointing ability extends to diverse scenes.Source: Table 5 · p. 14, §5.1.4 · p. 14

5.1.5 OCR

• WeVisOne-VL consistently outperforms specialist OCR systems and general-purpose VLMs across benchmarks covering dense documents, scene text, curved text, and receipts.The reported results span diverse layouts and visual conditions.Source: §5.1.5 · p. 14, Table 6 · p. 15

5.1.6 Referring Segmentation

• WeVisOne-VL improves over all baselines across RefCOCO-family splits and produces accurate masks for present targets while rejecting absent-target expressions on Ref-ZOM.It also shows potential on reasoning segmentation without explicit chain-of-thought reasoning.Source: Table 7 · p. 15, §5.1.6 · p. 15

5.1.7 Promptable Matting

• WeVisOne-VL is competitive with task-specific matting models under box prompts, improves over the SAM2-Matting baseline, and performs strongly with category and referring prompts on WeRefMat.The evaluation uses box, category, and referring-expression prompts to specify target objects.Source: Table 8 · p. 16, §5.1.7 · pp. 16–17

Object detection

• WeVisOne-VL more often preserves instance-level boxes in dense scenes and recovers distant, partly obscured elephants missed by both baselines.Across carrot and dense-fruit examples, baselines merge adjacent instances into coarse boxes.Source: §6 · p. 17, Figure 5 · p. 17

Referring object detection

• WeVisOne-VL selects the intended target across referring queries and rejects an absent-target query for which both baselines predict a box.The examples test spatial position, world knowledge, fine-grained and negative attributes, and absent targets.Source: §6 · p. 18, Figure 6 · p. 18

Object counting

• WeVisOne-VL gives the ground-truth count in all four examples, while baselines sometimes count outside the requested subset or use the wrong unit.For fake green apples, Rex-Omni reports 537 rather than 42.Source: §6 · p. 19, Figure 7 · p. 19

Object pointing

• WeVisOne-VL points to queried instances or relations in examples where baselines miss instances, select distractors, or place redundant points.Queries include a pair of people gazing at each other and a curved hook rather than its handle.Source: §6 · p. 20, Figure 8 · p. 20

OCR

• On dense-text examples, WeVisOne-VL localizes and transcribes curved sign lettering and covers more small text than either baseline.The comparison also includes a receipt and an instruction sheet.Source: §6 · p. 21, Figure 9 · p. 21

Referring segmentation

• WeVisOne-VL segments referred objects and multiple requested instances, while returning no mask for an absent-target query that prompts masks from both baselines.Examples include attribute, relational, multi-instance, and absent-target queries.Source: §6 · p. 22, Figure 10 · p. 22

Image matting

• In box- and text-prompted matting examples, WeVisOne-VL retains fine boundary details and intended subjects that competing methods miss or render transparent.Examples include rabbit whiskers, camel coat, hair strands, an apron logo, chickens, icebergs, two people in black, and a named balloon.Source: §6 · pp. 23–24, Figure 11 · p. 23, Figure 12 · p. 24

Grounded caption

• Grounded captions link detailed descriptions of people, actions, and objects to their corresponding image regions.The examples include a baseball scene and an outdoor table; the figure maps colored phrases and coordinate tokens to matching boxes.Source: §6 · p. 25, Figure 13 · p. 25

Visual Grounding

• Visual grounding progresses from open-vocabulary detectors to language-interface models, with approaches using text–box links, sequence generation, point supervision, quantized coordinates, or parallel box and point decoding.The cited methods include Kosmos-2, Florence-2, Molmo/PixMo, Rex-Omni, and LocateAnything [104].Source: §7 · p. 25

Referring Segmentation

• Referring segmentation work connects language reasoning to pixels through segmentation tokens, interleaved text and masks, absent-target rejection, serialized masks, and unified autoregressive supervision.The passage surveys SAM, SAM 2, SAM 3, LISA, GLaMM, GSVA, Text4Seg, UniPixel, Sa2VA, and Youtu-VL [107].Source: §7 · p. 25

Instance Matting

• Prior matting evaluations often assume a salient foreground or provide a ground-truth mask prompt; this work evaluates boxes, category names, and referring expressions in multi-object scenes.The passage also surveys automatic, promptable, reference-image, and video matting methods.Source: §7 · p. 26

8 Limitations

• Limited general-purpose training data may compromise general capabilities relative to the base model.Direct structured outputs without chain-of-thought potentially limit complex reasoning, and the current approach is single-turn.Source: §8 · p. 26
• Benchmark prompts are typically clear and unambiguous, while real-world instructions may be vague or hesitant and make target localization more challenging.The authors identify strong intent reasoning as essential for a reliable vision system.Source: §8 · p. 26

9 Conclusion

• Across evaluated benchmarks, WeVisOne-VL improves mean detection F1 and counting accuracy over comparable-scale VLMs and surpasses evaluated baselines for referring segmentation and image matting.The conclusion also reports advances in text grounding and referring localization.Source: §9 · p. 26
• Dedicated coordinate tokens and an <OBJ> token connect instance predictions to a lightweight pixel decoder for masks and alpha mattes.The three-stage supervised curriculum is followed by mixed-task reinforcement learning to optimize grounding outputs.Source: §9 · p. 26

Appendix

• The appendix describes the WeVisOne-VL training corpus and provides implementation details.It says the following sections describe data for SFT, segmentation and matting training, and RL.Source: Appendix · p. 34, A WeVisOne-VL Training Corpus · p. 34

• Table 9 breaks down SFT data by task and annotation source.The sources include public annotations and annotations generated by the WeVisOne-VL Data Engine.Source: §A.1 · p. 34
• Table 9 summarizes training data volumes across the three SFT stages.Source: Table 9 · p. 34
• The table covers both the amount of data at each stage and its task and source composition.Source: Table 9 · p. 34, §A.1 · p. 34

A.2 Segmentation and Matting Data

• Public matting datasets provide high-resolution images with high-quality alpha mattes, while matting fine-tuning samples public and WDE data at a 1:4 ratio per batch.Source: §A.2 · p. 34
• Table 10 summarizes data sources and sample counts for segmentation and matting.Source: Table 10 · p. 35

A.3 Reinforcement Learning Data

• The RL set is built by applying uncertainty-aware sample selection to SFT data and contains 178K examples.Source: §A.3 · p. 35
• Detection accounts for 44.0% of the RL set, dense detection 20.8%, referring localization 17.5%, OCR 7.9%, referring segmentation 5.8%, and pointing 4.0%.Source: §A.3 · p. 35

B.1 Training Hyperparameters

• All four training configurations use AdamW, weight decay 0.01, maximum gradient norm 1.0, and BF16 precision; SFT uses cosine learning-rate decay.SFT-S3 uses LoRA for the LLM, while other modules are fully trainable; the table also lists RL schedules and sequence limits.Source: Table 11 · p. 35, §B.1 · p. 35

B.2 Pixel Decoder Training

• After SFT-S2, the pixel decoder is warmed up for 100,000 segmentation-supervised steps, then its trained weights and the <OBJ> embedding are attached to the original SFT-S2 checkpoint.Other VLM weights updated during warm-up are discarded.Source: §B.2 · p. 36
• SFT-S3 alternates next-token prediction and segmentation steps at a 1:1 ratio, sampling 24 <OBJ> tokens per packed sequence per GPU for decoder prediction.Source: §B.2 · p. 36
• After training, the VLM is frozen and the decoder is fine-tuned for matting for 100,000 steps at 1024 × 1024 output resolution.The warm-up and matting phases use AdamW with cosine learning-rate decay, weight decay 0.05, and maximum gradient norm 5.0.Source: §B.2 · p. 36

B.3 RL Rewards

• For segmentation and matting, RL uses box-based grounding rewards rather than rewards from predicted masks or alpha mattes.For absent targets, reward is 1 only when the model predicts no object.Source: §B.3 · p. 36