Source-linked AI summary
Indices Matter: Learning to Index for Deep Image Matting
Hao Lu, Yutong Dai, Chunhua Shen, Songcen Xu
TL;DR
Dense prediction needs upsampling that can recover both regions and boundaries, yet existing operators behave differently across tasks. The paper learns index functions from feature maps and uses them to guide pooling and upsampling through IndexNet. On Composition-1k, its best MobileNetv2 model improves over the VGG-16 deep matting baseline by at least 16.1%.
Problem
Existing upsampling operators have different task-dependent behaviors, and bilinear interpolation performs worse than unpooling for boundary-sensitive image matting.
Method
IndexNet dynamically learns indices from local feature maps without supervision and uses them to guide indexed pooling and indexed upsampling.
Results
At least 16.1% improvement was achieved on Composition-1k by the best MobileNetv2 model over the VGG-16-based deep matting baseline.
Takeaways & Limitations
Learned indices capture boundaries and textural patterns and support effective image matting and other visual tasks.
Takeaways & Limitations
IndexNet has substantial room for improvement, and its applicability to other dense prediction tasks remains planned future work.
Abstract
from arXiv · showhide
We show that existing upsampling operators can be unified with the notion of the index function. This notion is inspired by an observation in the decoding process of deep image matting where indices-guided unpooling can recover boundary details much better than other upsampling operators such as bilinear interpolation. By looking at the indices as a function of the feature map, we introduce the concept of learning to index, and present a novel index-guided encoder-decoder framework where indices are self-learned adaptively from data and are used to guide the pooling and upsampling operators, without the need of supervision. At the core of this framework is a flexible network module, termed IndexNet, which dynamically predicts indices given an input. Due to its flexibility, IndexNet can be used as a plug-in applying to any off-the-shelf convolutional networks that have coupled downsampling and upsampling stages. We demonstrate the effectiveness of IndexNet on the task of natural image matting where the quality of learned indices can be visually observed from predicted alpha mattes. Results on the Composition-1k matting dataset show that our model built on MobileNetv2 exhibits at least $16.1\%$ improvement over the seminal VGG-16 based deep matting baseline, with less training data and lower model capacity. Code and models has been made available at: https://tinyurl.com/IndexNetV1
1. Introduction
Upsampling operators behave differently across visual tasks, and bilinear interpolation can lose boundary details in image matting. The paper proposes learning index functions from local feature maps to guide both downsampling and upsampling, improving matting quality.
- Motivation: Upsampling is essential for dense prediction, but common operators exhibit task-dependent behavior.The paper discusses transposed convolution, unpooling, periodic shuffling, and interpolation followed by convolution.
- Motivation: Bilinear interpolation performs worse than unpooling on boundary-sensitive image matting.Unpooling uses max-pooling indices that can record boundary locations from shallow-layer responses.
- Learning to index: The paper frames upsampling operators as forms of index functions that can be modeled and learned.The learned index function is conditioned on local feature maps and supports boundary and region prediction.
- Learning to index: IndexNet dynamically predicts indices without supervision for indexed pooling and indexed upsampling.The framework generalizes SegNet by using learned indices to guide corresponding encoding and decoding stages.
- Results: 16.1% improvement was achieved by the best MobileNetv2 model over the VGG-16-based deep matting model on Composition-1k.The result used less training data and a more compact model, with faster inference speed.
2. Related Work
Prior work includes dense-prediction upsampling operators and deep matting systems, with unpooling retaining boundary information but introducing sparsity. IndexNet extends this idea while targeting the computational limitations of heavier matting backbones.
- Upsampling in Deep Networks: Upsampling research includes transposed convolution, resize-plus-convolution, perforate, unpooling, and periodic shuffling.Resize-plus-convolution became standard in state-of-the-art semantic segmentation models, while periodic shuffling targets efficient super-resolution.
- Upsampling in Deep Networks: Unpooling and perforate preserve boundary information through sparse indices but require subsequent densification.Large-filter convolutional layers are needed to densify the sparse upsampled feature maps.
- Relation to IndexNet: IndexNet generalizes unpooling by learning index functions beyond fixed max-pooling indices.The framework uses learned indices to retain spatial information during downsampling and upsampling.
- Deep Image Matting: Classical matting methods rely heavily on color cues, which can fail when colors are unreliable in natural scenes.Deep matting methods emerged partly to use learned features, though early systems retained conventional matting backends.
- Deep Image Matting: SegNet is effective for matting but computationally expensive and memory-inefficient for high-resolution inference.The cited limitation makes lightweight alternatives practically relevant.
3. An Indexing Perspective of Upsampling
The paper unifies common upsampling operators as index functions, showing that their indices can be modeled and learned to adapt to local spatial patterns.
- Unpooling, average pooling, bilinear interpolation, deconvolution, and periodic shuffling can each be expressed through an index-function view.Bilinear interpolation and deconvolution use soft indices, while periodic shuffling performs indexing through feature-map rearrangement.
- Average-pooling indices used during upsampling are equivalent to nearest-neighbor interpolation.
- Bilinear interpolation and deconvolution share the same index-function form, differing because only deconvolution learns its filter weights.
- Periodic shuffling indexes a 1×1×r^2 tensor into an r×r matrix through one-hot channel-position encoding.
- Because upsampling operators are index functions, an index function can in principle be learned to capture local spatial patterns adaptively.
4. Index-Guided Encoder-Decoder Framework
The framework learns soft indices from feature maps and uses them to guide both pooling and upsampling. IndexNet supports holistic and depthwise designs, including nonlinear and weak-context variants with different capacity and flexibility trade-offs.
- 4. Index-Guided Encoder-Decoder Framework: IndexNet dynamically predicts indices from feature maps, while indexed pooling and indexed upsampling use them for downsampling and reconstruction.The framework generalizes SegNet and can combine multiple IndexNet modules analogously to max-pooling layers.
- 4.2. Holistic Index Networks: Holistic index networks output one spatial index map shared across channels, while depthwise networks output channel-specific index maps.
- 4.1. Learning to Index, to Pool, and to Upsample: Encoder indices use local softmax normalization for magnitude consistency, whereas decoder indices use a sigmoid to produce values in (0, 1).
- 4.1. Learning to Index, to Pool, and to Upsample: Indexed pooling computes weighted sums of local activations, making max pooling and average pooling special cases of the same operator.
- 4.1. Learning to Index, to Pool, and to Upsample: Indexed upsampling combines nearest-neighbor-expanded features with local indices, unlike deconvolution, which applies a fixed kernel across local regions.
- 4.2. Holistic Index Networks: Holistic designs include linear and nonlinear variants, with nonlinear networks adding batch normalization, ReLU, and point-wise convolution.
- 4.3. Depthwise Index Networks: Depthwise networks use one-to-one or many-to-one channel assumptions, implemented with grouped or standard convolution, respectively.Under the one-to-one assumption, each index slice corresponds to its feature-map slice; under many-to-one, each index slice uses all channels.
- 4.4. Learning with Weak Context: Weak-context indexing enlarges the receptive field so surrounding features can distinguish boundary points from isolated noise.The approach is applicable to both depthwise and holistic index networks.
5. Results and Discussions
IndexNet is evaluated on natural image matting using a MobileNetv2-based encoder-decoder and several ablations, benchmarks, and visualizations. Learned indices improve matting quality, preserve boundary and texture details, and also transfer to other visual tasks.
- Experimental Setup: The study evaluates IndexNet on Adobe Image Matting, using 43,100 training images, 1,000 Composition-1k testing images, and SAD, MSE, Grad, and Conn errors.The dataset contains 431 foreground objects composited with backgrounds from MS COCO and Pascal VOC.
- Experimental Setup: The customized MobileNetv2 encoder-decoder replaces stride-2 convolutions with max pooling to extract indices, while IndexNet can replace pooling and unpooling with indexed operators.The architecture follows the encoder-decoder paradigm and is designed for high-resolution inference with a lightweight backbone.
- Ablation Studies: Indices significantly benefit matting, low-level feature fusion helps, concatenation outperforms skip connections at higher computation, and context encoding generally improves results.The ablation compares 11 baselines and reports that MobileNetv2 can match VGG-16 with appropriate design choices.
- Index Network Results: 16.5% relative improvement is achieved by M2O DIN with “Nonlinear+Context” over the B11 DIN baseline on Composition-1k.The best HIN also improves over HMI by at least 12.3%, and the best model outperforms DeepMatting while using less memory.
- Index Visualization: Learned index maps capture complex structural and textural patterns, including dog fur and water bubbles, whereas random indices delineate edges poorly.The visualized maps provide an interpretable view of what IndexNet learns from feature maps.
- Online Benchmark: On the alphamatting.com benchmark, IndexNet Matting ranks first in gradient error among published methods and produces better hair mattes qualitatively.The model is tested directly after training on the Adobe Image Dataset, without fine-tuning.
- Extensions to Other Visual Tasks: IndexNet consistently improves performance in image classification, monocular depth estimation, and scene understanding in addition to image matting.The extensions evaluate classification on CIFAR-10 and CIFAR-100, depth estimation on NYUDv2, and scene understanding on SUN-RGBD.
6. Conclusion
The paper presents IndexNet as a learned index-function framework that unifies upsampling operators and guides downsampling and upsampling across visual tasks. It reports state-of-the-art image-matting performance while identifying substantial room for improvement.
- 6. Conclusion: IndexNet unifies existing upsampling operators through learned index functions and uses them to guide downsampling and upsampling.The framework includes IP and IU operators, while IndexNet can be designed according to the task.
- 6. Conclusion: The framework was demonstrated on image matting, image classification, depth prediction, and scene understanding.The paper reports state-of-the-art performance on image matting using a modified MobileNetv2-based model on Composition-1k.
- 6. Conclusion: The model is simple and has substantial room for improvement, making it a possible baseline for future research.The authors plan to explore IndexNet on other dense prediction tasks.