Referring Image Matting

1. Introduction

• Referring Image Matting (RIM) extracts a high-quality alpha matte for the specific foreground object that best matches a natural-language description.This is presented as controllable matting with a more natural and simpler instruction than conventional approaches.Source: §1 · p. 1
• RefMatte contains 47,500 images, 118,749 expression-region entities, and 474,996 expressions across 230 object categories.The dataset is built using public matting datasets, labeled entities, and generated visual attributes.Source: §1 · p. 1
• RefMatte-RW100 provides a real-world test set of 100 images with human-annotated expressions to assess out-of-domain generalization.The expressions are introduced as diverse and the images as high-resolution in the paper context.Source: §1 · p. 2, Figure 2 · p. 2
• CLIPMat combines a context-embedded text prompt, text-guided visual semantic features, and multi-level detail extraction for RIM.It uses CLIP text and visual backbones with matting decoders; the detail module exploits shallow-layer features and the input image.Source: §1 · p. 2
• Prior visual-grounding methods target segmentation and do not produce promising fine-detail alpha mattes when directly applied to RIM.The introduction motivates a RIM-specific baseline rather than direct reuse of those methods.Source: §1 · p. 2
• CLIPMat outperforms representative methods in both keyword-based and expression-based RefMatte settings, with experiments also covering real-world images.The introduction reports subjective and objective validation.Source: §1 · p. 2

2. Related Work

• Scribble- and click-guided matting can indicate a specific foreground, but predictions are limited by low-level color propagation and sensitivity to input density.Automatic alternatives extract foregrounds without manual input, often for specific categories or salient objects.Source: §2 · pp. 2–3
• Language-guided matting of a specific foreground object remains unexplored in the cited prior work, motivating the RIM task.Earlier controllable examples select humans or human-object interactions rather than arbitrary objects through natural-language descriptions.Source: §2 · p. 3
• Public matting datasets provide diverse foreground categories and backgrounds that can be reused to construct customized datasets such as RefMatte.The cited work describes synthetic composites as a way to create trainable data when original backgrounds are unavailable.Source: §2 · p. 3
• RefMatte uses synthetic images with multiple semantically similar foregrounds and fine details, alongside a real-world test set with human-annotated expressions.The real-world set is intended to assess out-of-domain generalization.Source: §2 · p. 3
• Referring image segmentation methods produce coarse, non-fine-grained masks because their task and annotations do not require pixel-level text-semantic alignment or alpha mattes.The paper positions CLIPMat as a baseline designed specifically for high-quality alpha matte prediction.Source: §2 · p. 3

3.1. Preparation of Matting Entities

• 13,187 filtered foreground entities were labeled across 230 categories and annotated with at least three attributes, or six for human entities.The selected source datasets were required to provide high-resolution foregrounds with clear alpha-matte detail.Source: §3.1 · pp. 3–4

3.2. Image Composition and Expression Generation

• Each composite places two or three entities in six spatial relations, with overlaps simulated for front/behind relations and avoided for the other four.The foregrounds are alpha-blended onto BG-20k backgrounds.Source: §3.2 · p. 4
• Descriptions cover basic attributes, absolute image position, or relative position to another entity.Relative expressions can combine attributes of both the target and the related entity.Source: §3.2 · p. 5

3.3. Dataset Split and Task Settings

• 45,000 training and 2,500 test composites support keyword and expression settings; keyword-setting images exclude ambiguous semantic entities.The entity split reserves 11,799 entities for training and 1,388 for testing, with long-tailed categories moved to training and duplicated entities used to balance category proportions.Source: §3.3 · p. 5
• RefMatte-RW100 contains 100 high-resolution natural images with two or three entities per image and specialist-annotated expressions and alpha mattes.It is intended to evaluate out-of-domain generalization.Source: §3.3 · p. 5

4.1. Overview

• CLIPMat takes an image and keyword or expression, then uses a dual-decoder framework to predict a trimap and the target's alpha matte in the transition area.The text and image encoders are from CLIP; the two decoders handle global semantics and local detail, respectively.Source: §4.1 · p. 6

4.2. CP: Context-embedded Prompt

• The context-embedded prompt adds matting-related context before tokenization for keywords and learnable context after tokenization for both keywords and expressions.The post-embedding context has length eight in both settings.Source: §4.2 · p. 6

4.3. TSP: Text-driven Semantic Pop-up

• TSP uses text-guided cross-attention to extract visual semantic information before the matting semantic decoder.It processes text and visual features, then uses self-attention to further refine the visual feature.Source: §4.3 · p. 6

4.4. MDE: Multi-level Details Extractor

• MDE combines the original image with features from all four CLIP image-encoder transformer blocks to supply local details to corresponding decoder layers.The features are fused through convolution, batch normalization, and ReLU, then passed via residual connections; the detail and semantic decoder outputs are merged by a collaboration module.Source: §4.4 · p. 7

5.1. Experiment Settings

• The benchmark compares CLIPMat with MDETR and CLIPSeg, trained on RefMatte, in keyword and expression settings; evaluation averages SAD, MSE, and MAD over test entities.Inputs are resized to 512 × 512; CLIPMat uses Adam at a fixed 1e-4 learning rate, and P3M can optionally refine predicted mattes.Source: §5.1 · p. 7

5.2.1 Keyword-based Setting

• In keyword-based RefMatte evaluation, the best CLIPMat reduces MDETR error by about 75% and CLIPSeg error by about 50%.The dandelion comparison shows CLIPMat extracting fine detail where both competing methods produce incomplete, blurry mattes.Source: §5.2.1 · pp. 7–8, Figure 5 · p. 8

5.2.2 Expression-based Setting

• In expression settings, CLIPMat reduces MDETR error on RefMatte by over 50% and CLIPSeg error on RefMatte-RW100 by about 60%.On RefMatte-RW100, CLIPMat finds the described woman while CLIPSeg selects the wrong foreground; CLIPMat also preserves more hair detail than MDETR.Source: §5.2.2 · p. 8, Figure 5 · p. 8

5.3. Ablation Studies

• Each proposed module improves keyword-setting metrics; adding CP after MDE and TSP reduces SAD from 22.88 to 14.55 and then to 9.91.The ablations use ViT-B/16; the authors attribute CP's contribution to matting context for the text encoder.Source: §5.3 · p. 8, Table 3 · p. 8

6. Conclusion

• RefMatte and RefMatte-RW100 together support in-domain and out-of-domain evaluation of RIM methods.The conclusion credits RefMatte's scale, high-quality images, abundant annotations, and two experiment settings with making it a suitable test bed.Source: §6 · p. 8
• CLIPMat demonstrates the value of task-specific designs for RIM and serves as a reference for model design.Source: §6 · p. 8
• The authors hope the study will provide insights to the image-matting community and inspire follow-up research.Source: §6 · p. 8

• The listed authors are Jizhizi Li, Jing Zhang, and Dacheng Tao.Source: Referring Image Matting (Supplementary… · p. 11
• The listed affiliation is the University of Sydney, Sydney, Australia.Source: Referring Image Matting (Supplementary… · p. 11
• The material is identified as arXiv:2206.05149v3, dated 22 March 2023.Source: Referring Image Matting (Supplementary… · p. 11

1. The Significance of RIM

• Unlike automatic matting limited to predefined categories or salient objects, RIM targets a specific foreground through straightforward language descriptions.The paper presents this flexibility as useful for applications including interactive editing, human-machine interaction, virtual and augmented reality, film production, and e-commerce.Source: §1 · p. 11
• The paper identifies visual-text alignment, the composite-to-real domain gap, and distinguishing same-type objects as open RIM research directions.RefMatte and CLIPMat are presented as an initial starting point for these studies.Source: §1 · pp. 11–12
• In the interactive-editing case study, text input supports highlighting or pasting selected objects and separating adherent foregrounds.The examples include selecting a cat by fur description or a man taking a photo.Source: Figure 1 · p. 11, §1 · p. 12
• Unlike RIS, RIM predicts fine foreground detail from high-resolution images as pixel-level regression rather than coarse contours as binary classification.The paper says RIM datasets should preserve detail and methods need both global semantic and local detail features.Source: §1 · p. 12

2.1. The Distribution of Matting Entities

• The entity distribution is balanced across humans, animals, and objects at 5:1:1 by duplicating entities, while retaining long-tail categories in training.Source-dataset splits are otherwise largely preserved.Source: §2.1 · pp. 12–13, Table 1 · p. 12

2.2. Linguistic Details in RefMatte

• RefMatte defines transparent and non-salient entities from category synonyms, while humans and animals are salient and non-transparent.Human references also draw on age- and gender-specific synonyms, with additional candidate words used randomly.Source: §2.2 · p. 13, Table 2 · p. 13
• Basic expressions use different syntax templates for humans and other entities; position expressions describe absolute or relative relations using candidate relationship phrases.“Middle” is used only for absolute-position expressions.Source: §2.2 · pp. 13–14, Table 3 · p. 13, Table 4 · p. 14

2.3. The Statistics of RefMatte

• The keyword setting has 30,391 training images and 1,602 test images; average text length is about 1 for keywords versus about 17 and 12 for expressions in RefMatte and RefMatte-RW100.The keyword setting removes images with multiple entities of the same category to avoid ambiguity; humans and animals form a large share of RefMatte, and relationship words cover varied relations.Source: §2.3 · p. 14, Table 5 · p. 14, Figure 2 · p. 15

2.4. More Examples of RefMatte

• The examples show composition relations and keyword, basic, absolute-position, and relative-position descriptions for RefMatte, plus free expressions for RefMatte-RW100.Green dots mark the target objects.Source: §2.4 · p. 15

Context-embedded prompt

• Context-embedded prompting uses matting-related prefix templates, including keyword-only, photo, foreground, mask, and alpha-matte forms.The templates are intended to reduce the gap between CLIP’s pretraining sentences and single-word keyword inputs.Source: §3 · p. 15

Text-driven semantic pop-up

• The text-driven semantic pop-up aligns visual and text feature dimensions, guides visual features with text through cross-attention, then refines and reshapes them.Self-attention and a residual connection refine the fused features before reshaping them into a spatial map.Source: §3 · p. 15

Multi-level details extractor

• The multi-level details extractor combines features from CLIP image-encoder blocks with the original image, then supplies features to decoder layers through residual connections to preserve details.At each level, image information is resized to match the visual feature scale before the features are combined.Source: §3 · pp. 15–16

4.1. More Details of Experiment Settings

• CLIPSeg and MDETR use VGPhraseCut-fine-tuned weights, whereas CLIPMat uses CLIP pretraining directly; all methods train for 50 epochs at 512 × 512 input resolution.An optional P3M post-refiner is trained on RefMatte; CLIPMat performs well without it, though refinement can improve results.Source: §4.1 · p. 16

4.2. Further Evaluation of the Main Results

• CLIPMat achieves the best results with both tested backbones across the RefMatte settings and RefMatte-RW100; larger backbones and the post-matting refiner improve results further.The refiner improves CLIPMat less than MDETR or CLIPSeg, which the authors attribute to CLIPMat’s ability to preserve details.Source: Table 6 · p. 17, §4.2 · p. 17

4.3.1 Error Bars from Multiple Runs with Different Seeds

• CLIPMat’s SAD results are reported as stable across random seeds on both RefMatte and RefMatte-RW100.The authors report mean SAD and standard deviation for keyword and expression settings.Source: §4.3.1 · p. 17, Table 7 · p. 18

4.3.2 Impact of Input Texts

• CLIPMat is robust across prompt templates, performs best with “an image of a {keyword},” and performs worst on human-written free expressions in RefMatte-RW100.Relative-position expressions perform best on both test sets; basic expressions fare worse than position-based ones, while free prompts’ diverse logic, grammar, and wording are identified as a challenge.Source: §4.3.2 · p. 18, Table 8 · p. 18, Table 9 · p. 19

4.3.3 The Impact of Pre-training and Freezing

• Fine-tuning CLIP’s image and text encoders yields SAD 9.91 versus 20.05 when both encoders are frozen; VGPhraseCut pretraining yields SAD 11.33 versus 9.91 without it.The authors hypothesize that fine-tuning reduces the visual and text gap between RefMatte and CLIP’s dataset.Source: §4.3.3 · p. 19, Table 10 · p. 19

4.4. Failure Cases

• CLIPMat can select the wrong foreground under complex or ambiguous expressions and may omit foreground details, especially around occlusions.The authors identify improved expression understanding, detailed boundaries, and reduced synthetic-to-real domain gaps as future work.Source: §4.4 · pp. 19–20, Figure 3 · p. 19

A.1. Motivation

• RefMatte was created to support RIM, a task for extracting a specific foreground using a linguistic keyword or expression, for which visual-grounding methods were limited to segmentation-level outputs.The authors constructed synthetic images from public high-quality matting foregrounds using an image-composition and expression-generation engine.Source: §A.1 · p. 26

A.2. Composition

• RefMatte covers 230 object categories, 47,500 images, 118,749 expression-region entities with high-quality alpha mattes, and 474,996 expressions.Each instance includes a synthetic image, an entity matte, a keyword, and expressions describing that entity.Source: §2 · p. 26, §A.2 · p. 26

2. How many instances are there in total (of each type, if appropriate)?

• The dataset stores relationships among instances and uses manual train/test splits that keep long-tail categories in training; the authors note that some expression labels may be inaccurate.The authors describe such label noise as potentially similar to subjectivity and error in human-generated language.Source: §2 · pp. 26–27

A.3. Collection Process

• Matting entities come from public datasets, and the final images are generated with the authors’ composition engine using data assembled semi-automatically from model-predicted attributes and manual annotations.The listed source datasets include AM-2k, P3M-10k, AIM-500, SIM, DIM, and HATT.Source: §2 · p. 27, §A.3 · p. 27

A.4. Preprocessing/cleaning/labeling

• The preprocessing pipeline cleans existing matting data and uses composition and expression engines to generate synthetic images with linguistic labels.The authors state that their processing code will be released.Source: §A.4 · p. 28

A.5. Uses

• The authors identify referring image matting, domain adaptation, referring image localization, and one-shot or zero-shot referring image matting as potential uses.They report no tasks for which the dataset should not be used.Source: §3 · p. 28

A.6. Distribution

• The dataset is planned for public distribution on the project website under a CC BY-NC license.The authors report no export controls or other regulatory restrictions.Source: §A.6 · p. 28, §3 · p. 28

2. How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

• The authors report no erratum or planned dataset updates, and provide no mechanism for contributions or extensions.The stated answers also indicate that older versions will not be maintained.Source: §2 · p. 29