PaperlayerView exact revision on arXiv

Source-linked AI summary

Xiaomi-Lens-0 Technical Report

Pei Fu, Yike Liu, Yue Sun, Hanmo Wang, Xiuwen Xi, Ruoceng Zhang, Shaojie Zhang, Chao Jiang, Hang Li, Bingrui Liu, Zhenbo Luo, Heng Qu, Changjian Xu, Ming Xu, Jian Luan

arXiv:2610.23279v1cs.CV

TL;DR

Fine-grained region understanding remains difficult for MLLMs, and text-only supervision does not specify which regions support an answer. Xiaomi-Lens-0 predicts masks from its shared visual representations and uses visual latents to ground intermediate reasoning regions. It gains 5.6 points on average over Qwen3.5 in REC across eight RefCOCO-family splits and surpasses its backbone on every fine-grained perception benchmark.

  • Problem

    Text-only supervision does not specify which image regions support an answer or how far those regions extend, a gap associated with fine-grained region-understanding errors.

    Source: §1 · p. 3

  • Method

    Xiaomi-Lens-0 decodes masks from its vision encoder's patch grid and language-model visual latents, jointly supervising shared representations and intermediate reasoning regions.

    Source: §1 · pp. 3–4

  • Results

    Across all eight RefCOCO, RefCOCO+, and RefCOCOg splits, Xiaomi-Lens-0 gains 5.6 points on average over Qwen3.5 in REC and surpasses its backbone on every fine-grained perception benchmark.

    Source: §1 · p. 3, §6 · p. 21

  • Takeaways & Limitations

    The results show that mask prediction from shared visual representations can accompany strong referring comprehension and fine-grained perception performance without an additional spatial encoder.

    Source: §1 · p. 4, §6 · p. 21

    On panoramic LensBench scenes with multiple targets, Box Prec@0.5 falls from 81.3 for explicit to 48.7 for implicit instructions, and cIoU falls from 71.8 to 48.9.

    Source: §4.3 · p. 18

Abstract

from arXiv · show

Multimodal large language models (MLLMs) answer questions about whole images well, yet they still struggle with fine-grained region understanding. A key reason is that text-only supervision never specifies which regions support an answer. To close this gap, we present Xiaomi-Lens-0, a 4B MLLM built on Qwen3.5 that predicts masks from its own visual representations. Specifically, a lightweight decoder combines vision-encoder patch features with visual latents, the language model's mask-token states, so mask supervision reaches the shared backbone without an additional segmentation encoder. Because each latent decodes to a mask, the model can interleave reasoning with region-specific latents and thus think with visual latents, while grounded reasoning traces supervise the regions it visits during chain-of-thought. As a result, across all eight RefCOCO, RefCOCO+, and RefCOCOg splits, Xiaomi-Lens-0 gains 5.6 points on average over Qwen3.5 in referring comprehension and achieves the highest segmentation cIoU among compared models. Moreover, it surpasses Qwen3.5 across all perception benchmarks. Homepage: \href{https://seerray-lab.github.io/Xiaomi-Lens-0}{this https URL}

1 Introduction

Xiaomi-Lens-0 addresses fine-grained region understanding by training a MLLM to predict masks from its own visual representations, with grounded reasoning traces extending mask supervision to intermediate regions. It reports gains over its Qwen3.5 backbone across referring comprehension and perception benchmarks, and introduces LensBench for broader referring evaluations.

1 Introduction

  • Text-only supervision does not specify which image regions support an answer or their extent, while masks provide pixel-level targets for region extent.The paper links this gap to fine-grained errors such as miscounting nearby objects and misidentifying small-region attributes.Source: §1 · p. 3
  • Xiaomi-Lens-0 is a 4B MLLM that predicts masks from its own visual representations, allowing mask loss to supervise visual features shared with language generation.It uses the vision encoder's patch grid before spatial merging and requires neither an additional hierarchical visual encoder nor multi-scale feature fusion.Source: §1 · pp. 3–4
  • The model interleaves textual reasoning with visual latents decoded as masks, and grounded reasoning traces supervise the intermediate regions referenced in chain-of-thought.This makes the spatial extent of intermediate reasoning evidence explicit and inspectable.Source: §1 · pp. 3–4
  • Across all eight RefCOCO, RefCOCO+, and RefCOCOg splits, Xiaomi-Lens-0 gains 5.6 points on average over Qwen3.5 in REC, achieves the highest cIoU on every RES split, and leads in threshold-averaged F1 for COCO 2017 multi-instance grounding.It also outperforms its backbone on every fine-grained perception benchmark.Source: §1 · pp. 3–4
  • LensBench pairs single and multiple targets with explicit and implicit instructions across four visual domains, with human review of each paired unit.Xiaomi-Lens-0 achieves the highest cIoU among compared models in aerial, driving, and panoramic domains, but implicit multiple-target selection in panoramic scenes remains challenging.Source: §1 · p. 4
  • Ablations show that adding mask supervision to box supervision further improves all three evaluated general perception benchmarks and REC validation splits of RefCOCO, RefCOCO+, and RefCOCOg.Grounded reasoning traces further increase REC and RES scores.Source: §1 · p. 4

2 Method

The method decodes masks from the shared vision encoder's patch grid using language-model states as region queries, so mask supervision reaches representations used for language generation. It extends this decoding to intermediate reasoning regions and trains the decoder first with a frozen backbone, then jointly fine-tunes the full model.

2.1 Mask Prediction from Shared Visual Features

  • The method uses the vision encoder's patch grid before spatial merging, preserving patch resolution and giving mask loss a direct path to visual features used for language generation.This avoids an additional encoder or multi-scale feature fusion.Source: §2 · p. 4, §2.1 · p. 5

2.2.1 Shared Visual Features and Visual Latents

  • The vision encoder produces a spatial patch grid, while the language model consumes spatially merged visual tokens and yields contextual states for generated mask tokens.Each mask-token state represents the region requested by the instruction or referenced in the preceding response.Source: §2.2.1 · p. 5

2.2.2 Mask Decoding

  • A lightweight decoder refines patch features, projects each mask-token state into the same feature space as a dynamic patch classifier, then upsamples patch scores into a dense mask.The shared patch features are computed once per image and reused across visual latents, changing only the query.Source: §2.2.2 · p. 6

2.2.3 Joint Language and Mask Supervision

  • Joint training combines next-token prediction with pixel-wise binary cross-entropy and Dice mask losses, propagating mask loss to both vision features and language-model states.The two mask-loss terms supervise individual pixels and region overlap, respectively.Source: §2.2.3 · p. 6

2.3 Thinking with Visual Latents

  • During reasoning, each region-specific visual latent is decoded into a mask, grounding the region discussed in preceding text; traces supervise intermediate regions as well as final segmentation targets.Inference does not feed decoded masks, crops, or region features back into the language model.Source: §2.3 · pp. 6–7

2.4 Training Recipe

  • Training has two stages: decoder pretraining with frozen backbone representations, followed by joint fine-tuning of the backbone and decoder.Stage 1 uses in-house data and public sources covering objects, parts, and referring expressions; Stage 2 includes referring-expression and grounded-reasoning examples, plus public datasets.Source: §2.4 · p. 7, §2.4.1 · p. 7, §2.4.2 · p. 7

2.4.1 Stage 1: Mask Decoder Pretraining

  • In Stage 1, the backbone is frozen while only the mask decoder is optimized to decode regions from fixed image features and visual latents.The public data include Objects365, LVIS, PACO, PhraseCut, RefCOCO, RefCOCO+, and RefCOCOg.Source: §2.4.1 · p. 7

2.4.2 Stage 2: Supervised Fine-Tuning

  • In Stage 2, the backbone is unfrozen and the full model is optimized, with language loss supervising responses and mask loss updating shared visual features and visual latents.The data include in-house referring-expression and grounded-reasoning examples, plus COCO 2017, RefCOCO, RefCOCO+, RefCOCOg, RoboRefIt, and ReasonSeg.Source: §2.4.2 · p. 7

3 Data Pipeline

The data pipeline supplies referring-expression supervision for final target masks and grounded-reasoning supervision for intermediate regions. LensBench evaluates target selection and region extent as instruction semantics, target cardinality, and visual domain vary.

3 Data Pipeline

  • Referring-expression data pair instructions with final target masks, while grounded-reasoning data pair intermediate chain-of-thought region references with masks.Both pipelines fix or verify region annotations before accepting accompanying language.Source: §3 · p. 7
  • Figure 4 first localizes candidate objects and generates masks, then synthesizes referring instructions and checks each instruction against its intended target.The masked view and original image guide instruction synthesis; re-grounding and contextual verification check target identification.Source: Figure 4 · p. 8

3.1 Referring-Expression Training Data

The referring-expression pipeline establishes each object’s spatial annotation before generating descriptions, then filters instructions for localization and semantic agreement. Accepted examples pair one referent with both box- and pixel-level supervision.

3.1 Referring-Expression Training Data

  • The pipeline fixes each candidate’s box and mask before generating instructions, allowing multiple descriptions to share one spatial annotation.Each instruction is checked against the annotated region before retention.Source: §3.1 · p. 8, §3.1.3 · p. 9

3.1.1 Visual Grounding

  • Candidate regions are discovered as box–description pairs, crop-checked against the image, and given masks before instruction generation.The crop–text check verifies each description against its image crop.Source: §3.1.1 · p. 8

3.1.2 Instruction Synthesis

  • Instructions are generated from a masked target view and the original image, using target appearance and surrounding scene context.Descriptions vary across location, appearance, action, function, or contrasts while retaining the same annotated target.Source: §3.1.2 · p. 8

3.1.3 Consistency Filtering

  • Accepted samples pair each instruction with its target’s tight mask box and pixel-level mask after localization and meaning checks.The semantic check screens for references to nearby instances or properties absent from the target, complementing box overlap.Source: §3.1.3 · p. 9

3.2 Grounded Reasoning Data

Grounded reasoning data attach masks to regions referenced at intermediate steps, extending spatial supervision beyond final answers. The pipeline synthesizes and independently verifies questions, then retains answer-matched traces with mask-annotated region references.

3.2 Grounded Reasoning Data

  • Grounded reasoning examples pair an image and question with a trace, final answer, and masks for regions referenced along the trace.This extends spatial training signal from final segmentation targets to regions discussed during reasoning.Source: §3.2 · p. 9

3.2.1 Question–Answer Synthesis

  • Questions about counting, attributes, and spatial relations use screened image regions and temporary box IDs to distinguish instances or combine information.An MLLM generates multiple-choice or numeric answers for exact-match verification.Source: §3.2.1 · p. 9

3.2.2 Question–Answer Verification

  • An independent checker must exactly reproduce the reference option or numeric answer after filters remove flawed questions and options are shuffled.The checker answers from the image without seeing the reference answer.Source: §3.2.2 · p. 10

3.2.3 Trace Generation and Mask Annotation

  • Complete traces with valid box references and the verified answer are retained, and each accepted box is paired with a mask whose bounding box passes an IoU threshold.When no instance mask is available, a mask is generated by SAM 3 [4]; the mask supervises the box reference’s contextual state.Source: §3.2.3 · p. 10

3.3 LensBench

LensBench evaluates referring across four visual domains, varying target cardinality and explicit versus implicit wording in paired samples. Complete units undergo automated checks and human review before inclusion.

3.3 LensBench

  • LensBench contains 1,200 paired units and 4,800 instruction-level samples across general, aerial, driving, and panoramic imagery.Each unit has two target masks and four instructions crossing single or multiple targets with explicit or implicit wording.Source: §3.3 · p. 10

3.3.1 Benchmark Design

  • Explicit instructions name the category and specify the target set, while implicit instructions rely on function, purpose, or commonsense relations without naming the category.The four instructions within an image enable paired comparisons because explicit and implicit wording share the same ground-truth mask within each target cardinality.Source: §3.3.1 · p. 10

3.3.2 Benchmark Construction

  • Targets and masks are fixed before instruction synthesis, with instance curation checking identity and visible extent and resolving duplicates.Units group countable instances by category and select a focus instance before generating instructions from the image and surrounding crop.Source: §3.3.2 · pp. 10, 12

3.3.3 Consistency Filtering

  • Automated checks assess mask quality, instruction semantics, and re-grounding, followed by human review of each complete four-sample unit.Only complete units enter LensBench, preserving paired comparisons.Source: §3.3.3 · pp. 12–13

4 Experiments

The experiments evaluate spatial prediction, fine-grained perception, and instruction-conditioned grounding across six tasks. Xiaomi-Lens-0 improves over its Qwen3.5 backbone across referring and perception benchmarks, while LensBench and ablations expose remaining challenges and the contributions of mask supervision and grounded reasoning traces.

4.2.1 Referring Expression Comprehension

  • Xiaomi-Lens-0 gains 5.6 points over Qwen3.5 on average across all eight REC splits, ranking first on seven and second on RefCOCOg test.Its F1@mIoU scores rank first across RefCOCO, RefCOCO+, and RefCOCOg, with gains of 8.8, 11.6, and 9.5 points over the backbone.Source: §4.2.1 · pp. 14–15, Table 1 · p. 14, Table 2 · p. 15

4.2.2 Referring Expression Segmentation

  • Xiaomi-Lens-0 achieves the highest cIoU on all eight RES splits, averaging 81.4 versus 78.6 for Qwen3-VL-Seg4B.It also exceeds SAM 3 by 7.0–12.9 points and MLLMSegInternVL2.5-4B by 2.8–5.5 points across the splits.Source: Table 3 · p. 15, §4.2.2 · p. 16

4.2.3 Multi-Instance Grounding

  • On COCO 2017, Xiaomi-Lens-0 achieves the highest multi-instance grounding F1 at mIoU, reaching 63.9 and exceeding GroundingPI by 0.9 points.At IoU 0.95 its F1 is 27.4, above GroundingPI by 4.6 points and GroundingDINO by 1.1 points.Source: §4.2.3 · p. 16, Table 4 · p. 16

4.2.4 Reasoning Segmentation and Embodied Referring

  • On ReasonSeg, Xiaomi-Lens-0 leads in gIoU and cIoU on both splits; on RoboRefIt, it exceeds InstructSAM in gIoU by 4.5 and 7.1 points.ReasonSeg test scores are 68.0 gIoU and 62.5 cIoU; RoboRefIt scores 87.0/88.4 gIoU/cIoU on TestA and 81.5/81.2 on TestB.Source: §4.2.4 · p. 17, Table 5 · p. 17

4.2.5 Fine-Grained Perception

  • Xiaomi-Lens-0 exceeds its Qwen3.5 backbone and all compared open-source models on all eight fine-grained perception benchmarks.Its CountQA accuracy rises from 43.8 to 50.6, and its HR-8K score improves by 5.0 points over the backbone.Source: §4.2.5 · pp. 17–18, Table 6 · p. 17

4.3 LensBench Results

  • On LensBench, Xiaomi-Lens-0 leads in cIoU on aerial, driving, and panoramic imagery, but implicit instructions remain harder, especially for panoramic multi-target scenes.It scores 57.1, 68.6, and 62.3 cIoU in those domains; on panoramic multi-targets, cIoU falls from 71.8 for explicit to 48.9 for implicit instructions.Source: §4.3 · p. 18, Table 7 · p. 18

4.4 Ablation Study

  • Against BBox-only, mask supervision adds 1.1–2.4 REC and 0.2–1.4 perception points; grounded traces further add 0.1–0.6 REC and 0.5–0.8 RES points.The ablations isolate mask loss and grounded reasoning data as separate contributions.Source: §4.4 · p. 20, Table 8 · p. 20, Table 9 · p. 20

5 Related Work

Prior work grounds language through boxes, masks, supplementary segmentation branches, or MLLM representations, while visual reasoning work explores image interactions and latent representations. Xiaomi-Lens-0 connects latent reasoning with pixel-level supervision of intermediate regions.

Visual Grounding in MLLMs

  • Box-coordinate interfaces leave region extent implicit, whereas RES makes it explicit with pixel-level masks.Existing MLLM grounding includes coordinate-token approaches and sequence prediction; Youtu-VL also extends autoregressive supervision to visual targets.Source: Visual Grounding in MLLMs · pp. 20–21

Segmentation with MLLMs

  • Segmentation systems variously use dedicated segmenters or MLLM-derived features; Xiaomi-Lens-0 studies mask supervision from a single shared patch grid and its effects on text-answer perception.Prior MLLM-feature approaches include PixelLM and MLLMSeg, which keep the vision encoder frozen.Source: Segmentation with MLLMs · p. 21

Grounded and Latent Visual Reasoning

  • Thinking with visual latents links intermediate reasoning references to masks, so pixel-level supervision reaches those regions and keeps them inspectable.This formulation connects explicit intermediate grounding with latent visual reasoning.Source: Grounded and Latent Visual Reasoning · p. 21

6 Conclusion

Xiaomi-Lens-0 decodes masks from shared vision-encoder patches and language-model visual latents, extending pixel-level supervision to regions named in grounded reasoning. It performs strongly across referring, segmentation, multi-instance grounding, and perception evaluations, with implicit multi-target panoramic selection remaining challenging.

6 Conclusion

  • The 4B model achieves the highest cIoU on all eight RES splits, improves REC on every split, and leads threshold-averaged F1 for COCO 2017 multi-instance grounding.It also surpasses its backbone on every fine-grained perception benchmark.Source: §6 · p. 21
  • Mask supervision adds gains across evaluated REC and perception metrics, while grounded reasoning further raises REC and RES scores.On LensBench, implicit instructions remain harder, especially for multiple targets in panoramic scenes.Source: §6 · p. 21

A Conversation Format and Examples

The examples show how responses serialize boxes and masks, and how a mask token’s contextual state supplies a visual latent to the decoder. They contrast explicit identification with reasoning from a function-based instruction.

A Conversation Format and Examples

  • A box is serialized in a <bbox> block, while a mask block contains <|mask|>; the token’s contextual state is passed to the decoder as a visual latent.The response format examples illustrate these outputs.Source: A Conversation Format and Examples · p. 28
  • For an explicit clothing description, the response returns a single box after an empty reasoning block.The example instruction asks to locate the lady with the blue shirt.Source: A Conversation Format and Examples · p. 28
  • A function-based question prompts a reasoning trace that identifies a helmet, then returns its box and a mask token.The contextual state at the mask token represents the selected target.Source: A Conversation Format and Examples · p. 28

B Qualitative Analysis

The qualitative examples show Xiaomi-Lens-0 selecting and delineating requested regions, while linking intermediate visual-latent references to reasoning and final answers. They cover target selection, multi-instance grounding, functional requests, clutter, spatial comparison, color recognition, and counting.

B Qualitative Analysis

  • The VQA visualizations link each region discussed in a reasoning trace to its decoded mask and final answer, making intermediate references inspectable.The stored completion is replayed through the same model to obtain the contextual states used for mask decoding.Source: B Qualitative Analysis · p. 29

Referring Expression Grounding and Multi-Object Localization

Grounding Functional Descriptions

  • Figure 12 shows functional descriptions guiding selection among visible candidates, including traces that consider alternatives or return multiple boxes.Examples include choosing a spacecraft after considering two candidates and returning both gift-packaging boxes.Source: B Qualitative Analysis · p. 30, Figure 12 · p. 30

Object Selection in Cluttered Scenes

  • Figure 13 shows category, attribute, and position cues selecting objects in clutter, with masks delineating the requested instances.Positional phrases distinguish the requested cup and cleanser from other objects of the same categories.Source: B Qualitative Analysis · p. 31, Figure 13 · p. 31

Spatial Comparison and Color Recognition

  • Figure 14 links visual-latent references to boxes and masks while comparing chair and bicycle positions and recognizing a small vehicle’s color.An enlarged view shows the small target.Source: B Qualitative Analysis · p. 32, Figure 14 · p. 32

Counting with Instance References

  • Figure 15 traces five climbing stones and six leaves individually, linking each counting step to a predicted box and mask.The examples include overlapping leaves and spatially separated stones.Source: B Qualitative Analysis · p. 33, Figure 15 · p. 33

Related papers

Loading 2610.23279v1…