MAOAM: Unified Object and Material Selection with Vision-Language Models

Jaden Park; Valentin Deschaintre; Jason Kuen; Kangning Liu; Iliyan Georgiev; Krishna Kumar Singh; Yong Jae Lee; Michael Fischer

Source-linked AI summary

MAOAM: Unified Object and Material Selection with Vision-Language Models

Jaden Park, Valentin Deschaintre, Jason Kuen, Kangning Liu, Iliyan Georgiev, Krishna Kumar Singh, Yong Jae Lee, Michael Fischer

arXiv:2606.04880v1Jun 2, 2026cs.CV

TL;DR

Existing selection systems are largely object-centric and limited in interaction modality, despite the need to select objects or materials through precise, semantically rich prompts. MAOAM unifies click- and text-based selection with VLM reasoning, generated material descriptions, and multi-task selection-plus-VQA training. It demonstrates strong material selection while preserving object-selection performance, with improvements from complementary task supervision and limitations from VLM reasoning and SAM decoding.

Problem
Existing VLM-based selection methods are object-centric and typically support one interaction modality, while material datasets often lack rich text annotations.
Method
MAOAM combines click- and text-based object/material selection with VLM-generated material descriptions and joint segmentation and VQA training.
Results
MAOAM substantially outperforms GLaMM across material and object selection under text and click interactions, with the full multi-task configuration achieving the highest mIoU on RealMat among compared models.
Takeaways & Limitations
Unified training preserves material understanding while adding object selection and joint object-material reasoning, supporting varied material descriptions and interaction scenarios.
Takeaways & Limitations
Performance can be limited by VLM reasoning and SAM mask decoding; the Sa2VA-based variant has higher quantitative metrics, while the GLaMM-based variant generalizes better and is more robust during inference.

Abstract

from arXiv · show

Selection is a core operation in interactive image editing. To be practical, a user should be able to specify and disambiguate the desired selection region through either text or click-based interactions, and the system should support selecting not only objects but also other criteria, such as materials. Material-based selection is valuable for tasks like re-texturing surfaces or editing instances of a specific material. However, existing vision-language-model (VLM) based selection methods are object-centric and typically support a single interaction modality, limiting their applicability. In this work, we thus present Mask Any Object And Material (MAOAM), a unified selection framework that enables precise object and material-level selection across both text- and click-based interactions. MAOAM leverages a VLM with a segmentation head to produce pixel-accurate masks from user prompts: the VLM interprets the user's selection intent (object or material-level) and encodes visual entities, attributes, and spatial relations, while the segmentation head decodes the output token into a mask. A key challenge is the lack of material selection datasets with text annotations. We propose a scalable data generation pipeline: we collect real and synthetic images with material masks, and leverage VLMs to generate material descriptions with rich visual-semantics. We train MAOAM with a multi-task objective over click and text-based selection, along with an auxiliary VQA task derived from the material descriptions to facilitate deeper material understanding. Despite being trained with uni-modal prompts, our model exhibits an emergent improvement in selection when combining text and clicks at inference, enabling flexible image editing workflows. Experiments demonstrate accurate and coherent selections across diverse objects, materials, and interaction scenarios, highlighting robustness in practice.

1 INTRODUCTION

MAOAM addresses the gap between object-centric, single-modality selection systems and interactive editing scenarios requiring object or material selection through text, clicks, or both. It combines unified VLM-based selection with scalable material-description generation and multi-task training.

Motivation: Existing selection models mostly support one modality and object-level criteria, while prior material methods are limited to click-based local cues.These limitations hinder global or relative semantic queries and object-versus-material disambiguation.
Contribution: MAOAM supports object- and material-level segmentation through both text- and click-based interactions.Its VLM interprets the selection intent and produces a segmentation token decoded into a mask.
Data and Training: Multi-task training combines selection with VQA to strengthen material understanding while preserving object-selection performance.The paper reports improved material selection and understanding with competitive object selection against strong baselines.
Data and Training: The method uses a scalable pipeline that collects real and synthetic material-mask data and generates rich material descriptions with VLMs.The descriptions include visual semantics and spatial information for training selection and VQA tasks.
Significance: MAOAM unifies object and material selection with text and click inputs, unlike prior approaches that separate these capabilities.Its goal is precise interactive selection suitable for image-editing workflows.

3 METHOD

MAOAM uses a shared VLM-and-mask-decoder pipeline for click- and text-based selection, with a special token carrying the user’s selection specification. Joint training adds VQA supervision alongside selection losses.

Input Processing: The model accepts an image, a selection prompt, and either a star-marked click or referring text as a unified input interface.The star overlay grounds the VLM in the clicked region, while text selection uses the original image without an auxiliary marker.
Inference: The same image can yield different masks because the VLM encodes selection intent from the interaction prompt before decoding the [SEG] token.A click-based material prompt can select all matching regions, whereas text can identify a specific spatially described object.
Input Processing: A CLIP vision encoder and MLP map image features into the LLM token space for fusion with the user prompt.This avoids a separate cross-modal module and remains compatible with pretrained LLM backbones.
Selection Representation: The LLM emits a [SEG] token whose hidden representation summarizes intent, attributes, and spatial relations for the target selection.The single-token bottleneck provides a task-relevant representation compatible with both click and text interactions.
Mask Decoding: The SAM prompt encoder and mask decoder convert the [SEG] embedding into a dense, high-resolution selection mask.The decoder receives no other task-specific conditioning, isolating intent understanding in the VLM while using SAM’s mask prior for boundaries.
Training Objective: Training jointly combines click selection, referring-text selection, and VQA objectives.Selection uses language-modeling and per-pixel BCE and DICE losses, while VQA uses token-level cross-entropy.

4 DATASET

The dataset combines dense material masks from real and synthetic images with VLM-generated descriptions and verification, supporting material selection and auxiliary VQA training.

Existing material datasets lack aligned text descriptions or contain semantically inconsistent material assignments, limiting their suitability for material reasoning.
The collection combines real images, rendered synthetic scenes, and SAMa data to provide diverse material masks and material-light responses.RealMat contains approximately 8K images and 49K masks; SynMat and SAMa add rendered or multi-view data.
The combined datasets provide approximately 104K material annotations from 15K images, and training on real plus synthetic data improves both evaluation sets over either subset alone.
VLM-generated descriptions are verified with model-based checks and targeted human review to scale annotation while maintaining supervision quality.
Descriptions cover entity labels, spatial relations, and self-contained material attributes in six 10∼50-word variants sampled during training.
The training data supports four-way VQA with hard negatives and mixes object-segmentation datasets, yielding approximately 190K training samples at an approximate 1:1 material-to-object ratio.

5 EVALUATION

MAOAM is evaluated across material and object selection with text and click inputs, alongside VQA and qualitative tests. It substantially improves material selection while retaining competitive object selection and enabling multimodal refinement.

Quantitative evaluation: MAOAM achieves substantial material-selection gains, including 67.5% average mIoU over Sa2VA for text selection and 35.5% average mIoU over Materialistic for click selection.
Quantitative evaluation: MAOAM improves object-selection average mIoU over GLaMM, LISA, and Sa2VA by 33.9%, 14.6%, and 3.2%, respectively.The reported results cover RefCOCO, RefCOCO+, RefCOCOg, and EntitySeg with text- and click-based inputs.
VQA: MAOAM achieves high VQA accuracy on both question variants, while Sa2VA and Qwen2.5-VL-7B perform poorly on visually differing material stimuli.
VQA: MAOAM’s stronger Q2 than Q1 performance and improved downstream selection with VQA training indicate fine-grained material understanding.
Model properties: Combining text and clicks at inference improves mask quality despite training exclusively on unimodal prompts.
Model properties: The model handles spatial relations, multi-object material queries, prompt-dependent object-versus-material selection, and fine-grained boundaries for image editing.

6 ABLATION & DISCUSSION

Ablations show that MAOAM’s data composition, description verification, prompt flexibility, and click representation materially affect selection quality, while qualitative tests expose both capabilities and limitations.

Description generation: Verification consistently improves 8B-generated material descriptions across all three datasets, while gains are less consistent for the 235B model.The authors nevertheless use verification by default because it produces more grounded descriptions and better instruction following.
Training data composition: Joint Materials + Objects training preserves competitive material-selection performance while adding object selection and joint object-material reasoning.Some click-based results improve despite adding object data.
Training data composition: Training on combined real and synthetic material data improves both evaluation sets by up to 9.15%, while full training improves over single-dataset training by up to 21.49%.The authors attribute the benefit to complementary supervision from synthetic data transferring to real images.
Prompt robustness: Performance remains stable across short, medium, and long prompts, indicating that MAOAM grounds material cues without relying on a fixed prompt length or template.The evaluation uses six descriptions ranging from 10∼50 words across three material benchmarks.
Click representation: Star-overlay clicks achieve the best performance across all three material benchmarks and provide a direct grounding cue in the VLM’s [SEG] token.Coordinates enter only at decoding, whereas bounding boxes may include multiple materials and confuse the model.
Qualitative behavior and limitations: MAOAM selects material-specific subsets, full object sets, and joint criteria such as “brown eggs,” but can confuse mortar with bricks and produce inaccurate masks.The reported failure modes are attributed to VLM reasoning limits and coarse image-encoder resolution; mask quality is also limited by the SAM decoder.

7 CONCLUSION

MAOAM unifies material and object selection across click- and text-based prompts using a scalable automatic annotation pipeline for rich material descriptions. The method demonstrates strong material selection while matching or outperforming object-centric segmentation methods.

Conclusion: MAOAM unifies material- and object-selection with click- and text-based prompts.Its automatic annotation pipeline generates a large corpus of rich material text descriptions for visual grounding.
Conclusion: The framework demonstrates strong material selection while matching or outperforming object-centric segmentation methods.

Supplemental Material for MAOAM: Unified Object & Material Selection with

The supplement provides dataset, implementation, model, qualitative evaluation, and ablation details deferred from the main paper.

Supplementary scope: The supplementary material contains additional dataset, implementation, model, qualitative evaluation, and ablation details.

S1 DATASET DETAILS

The dataset section documents source-image and annotation counts for material and object selection data, totaling approximately 197K training masks across the combined corpus.

Material datasets: The material selection datasets provide click- and text-based prompts paired with precise material masks.
Material datasets: RealMat contains 7,848 training images and 46,646 training material annotations, plus 395 validation images and 2,214 validation annotations.
Material datasets: SynMat contains 5,532 training images and 54,315 training material annotations, plus 352 validation images and 3,071 validation annotations.
Material datasets: SAMa contains 1,292 training images and 3,294 training material annotations, plus 141 validation images and 346 validation annotations.
Object datasets: RefCOCO, RefCOCO+, and RefCOCOg supply diverse text-based referring expressions with conversational, appearance-based, and linguistically richer descriptions.
Object datasets: EntitySeg provides approximately 37K click-based object masks from approximately 8K real-world images after filtering invalid and very small masks.
Combined corpus: The combined material and object training data contains approximately 197K masks with varied criteria and orientations.

S2 TRAINING DETAILS

MAOAM is trained through unified click- and text-based object/material selection tasks, auxiliary VQA, and material-focused prompting and data strategies. It uses two VLM–segmentation configurations and full fine-tuning is especially beneficial for material understanding.

Architecture: The model is implemented with Sa2VA and GLaMM backbone configurations paired with segmentation heads.Sa2VA combines Qwen2.5-VL-7B with SAM 2, while GLaMM combines LLaVA-v1.5 with SAM.
Multi-task Training: MAOAM trains on click-based selection, text-based selection, and VQA using a weighted multi-task objective.The task weights are λ_click = 0.4, λ_ref = 0.4, and λ_vqa = 0.2.
Click Interaction: The star overlay supplies spatial information for click-based selection, with training augmentations designed to improve signals near thin or boundary regions.Stars are randomly placed, and target masks are eroded before sampling to keep most stars inside the selected area.
Prompt Design: Material prompts distinguish material identity from appearance variation and support both click-based and text-based queries.The formulation treats color changes as different materials while treating lighting, shading, and shadows as the same material.
Fine-tuning: Full VLM fine-tuning outperforms LoRA on text-based selection and VQA, whereas LoRA remains comparable or better for click-based selection.The reported pattern attributes the text-side advantage to adapting representations for intricate material descriptions and visual alignment.

S3 FULL EVALUATION RESULTS

The full evaluation covers material and object selection with both text and click prompts, plus VQA where applicable. MAOAM substantially improves material-centric understanding while preserving or improving object-centric grounding.

Material-Centric Understanding: MAOAM substantially outperforms GLaMM and Sa2VA on material segmentation and VQA across the RealMat, SynMat, and SAMa benchmarks.The evaluation includes click- and text-based material selection and two VQA question types targeting material understanding.
Object-Centric Grounding: MAOAM preserves and often improves the base Sa2VA model’s performance on RefCOCO, RefCOCO+, RefCOCOg, and EntitySeg.These benchmarks cover text-based referring expression segmentation and click-based object selection.

S4 DISCUSSION AND ABLATION STUDIES

Ablations show that training choices strongly affect the balance between text, click, VQA, material, and object selection. Multi-task supervision and the Sa2VA backbone provide broad gains, while practical performance remains competitive with reduced data or training.

LoRA vs. Standard Fine-tuning: Standard fine-tuning is clearly advantageous for intricate text descriptions and VQA, while LoRA remains competitive for local click-based mask prediction.The authors therefore use standard fine-tuning for GLaMM and retain LoRA for Sa2VA because of VRAM requirements.
Effect of Multi-task Training: Full multi-task training produces the strongest overall material-selection results by combining click selection, text selection, and VQA.Adding VQA improves text-based selection and can also improve click-based selection, including the highest RealMat mIoU among the ablations.
GLaMM vs Sa2VA: Sa2VA substantially outperforms GLaMM on material and object selection across text and click interactions despite fewer training epochs.VQA performance is comparable between the two backbones.
Model Variants: The Sa2VA variant yields higher quantitative metrics, whereas the GLaMM variant generalizes better and is more robust during inference.Quantitative results consequently use Sa2VA, while qualitative examples use GLaMM.
Data Scaling: Training with 50% of the randomly sampled material data remains competitive with full-scale training across all three material benchmarks.The data-generation pipeline therefore retains useful performance under reduced training-data scale.
Training Epochs: Five to ten training epochs already achieve competitive results, although longer training continues to improve performance.This provides flexibility to adjust training duration to the available compute budget.

S5 FURTHER APPLICATIONS IN MEDICAL IMAGING DATA

MAOAM is evaluated beyond image-editing imagery on surgical medical images. Despite no medical-data training, click interactions produce pixel-accurate masks for surgical tools.

Out-of-Domain Generalization: MAOAM generalizes to surgical imagery and produces pixel-accurate masks for surgical tools from simple click interactions.The model was never trained on medical data, so the result tests transfer to an out-of-domain setting.