Task-Adaptive Feature Transformer with Semantic Enrichment for Few-Shot Segmentation

• Few-shot segmentation addresses semantic segmentation of previously unseen classes using only a few examples, with segmentation labels containing more information than classification labels.Source: §1 · p. 1
• TAFT extends encoder-decoder segmentators by transforming task-specific high-level features into task-agnostic features that the decoder can use to generate masks.Source: §1 · p. 1
• TAFT's task-varying transformation maps class prototypes toward meta-learned, task-independent references that provide stable targets for the decoder.Source: §1 · pp. 1–2
• Semantic enrichment combines pixel-wise self-attention with an auxiliary multi-class segmentation loss to improve high-level features for adaptation to novel classes.Source: §1 · p. 2
• The authors do not expect the add-on modules to match tailored state-of-the-art methods in extremely low-shot settings such as one-shot, and target stronger competition as shots increase, such as to five.Source: §1 · p. 2
• On 5-shot PASCAL-5i and COCO-20i testing, TAFT-SE with Deeplab V3+ beats prior state-of-the-art few-shot segmentation algorithms by substantial margins, although its 1-shot performance is not state of the art.Source: §1 · p. 2

2.1. Problem Definition

• Each episode samples non-overlapping training or test classes, adapts from labeled support pairs, and predicts query masks; meta-training updates model parameters from mask loss.For 1-way N-shot segmentation, the support set contains N image-label pairs.Source: §2.1 · p. 2

2.2. Task-Adaptive Feature Transformer

• TAFT forms support foreground and background prototypes using average-pooled soft labels, then computes task-specific P to map them near task-independent reference vectors.The references are stable across episodes, while the prototypes depend on the current task's inputs.Source: §2.2 · pp. 2–3

2.3. Semantic Enrichment

• SE refines high-level features with pixel-wise self-attention and trains the encoder with auxiliary multiclass segmentation supervision to enrich semantic features.The auxiliary labels derive from training-image segmentation labels, with test classes treated as background.Source: §2.3 · pp. 3–4

2.4. Combination with Segmentation Algorithm

• During episodic training, the reference vectors are trained with regression loss, while the decoder and auxiliary decoder receive segmentation and auxiliary losses.Inference follows the episode process but omits loss computation, the auxiliary decoder, and parameter updates.Source: §2.4 · p. 4
• In the Deeplab V3+ integration, TAFT transforms high-level features before ASPP, while the decoder also uses low-level features; the auxiliary decoder predicts multiclass masks.TAFT-SE does not process the low-level feature; the auxiliary decoder is used only during training.Source: §2.4 · p. 4, §2.3 · p. 4
• Qualitative PASCAL-5i examples show 1-shot query masks for aeroplane, bus, dog, and sheep classes across the dataset splits.The figure presents support images and labels alongside query images and predictions.Source: §2.4 · p. 4

3.1. Semantic Segmentation

• Semantic segmentation architectures use convolutionalized classification networks, multi-scale context, and dilated convolutions, but do not readily adapt to unseen classes from few samples.Source: §3.1 · pp. 4–5
• In an episode, TAFT downsamples support labels, forms foreground and background prototypes, then constructs P to transform encoder features into task-agnostic features.Source: §3.1 · p. 5
• Training updates the encoder, attention module, decoder, references, and auxiliary decoder using the specified segmentation, regression, and auxiliary losses.Source: §3.1 · p. 5

3.2. Few-Shot Learning

• Few-shot classification literature divides into metric-space mapping and optimization-based meta-learning, where a meta-learner supports rapid adaptation.Source: §3.2 · p. 5
• Unlike TapNet’s class-pair alignment and error-nulling objective, TAFT uses least-squares fitting for segmentation and performs inference in a task-agnostic space.Source: §3.2 · pp. 5–6

3.3. Few-Shot Segmentation

• Prior few-shot segmenters span masked-average prototypes, multiple prototypes, support-query graph attention, cosine-similarity prior masks, and learned memory embeddings.Source: §3.3 · p. 6
• TAFT transforms task-specific features into stable task-agnostic features and reuses one matrix P across query images, enabling parallel mask prediction.Source: §3.3 · p. 6
• Unlike CWT, SE applies self-attention among pixels in one feature and uses all-class segmentation only as an auxiliary loss to train the encoder.Source: §3.3 · p. 7

4.1. Dataset and Evaluation Metric

• PASCAL-5i and COCO-20i use four disjoint test splits of five and 20 classes, respectively; the reported metrics are mIoU and FBIoU for PASCAL-5i and mIoU for COCO-20i.mIoU averages per-class foreground IoU; FBIoU averages foreground and background IoU over test images.Source: §4.1 · p. 7, Table 2 · p. 8

4.2. TAFT-SE Architecture

• TAFT is implemented as a 1 × 1 convolution, while SE uses single-head scale-dot-product attention and an auxiliary decoder.The attention dimensions match the input feature dimension; the auxiliary decoder differs from the main decoder at its final pixel-classification layer.Source: §4.2 · p. 7

4.3. Experimental Settings

• Training used 30,000 episodes with 12 queries and ImageNet-pretrained ResNet encoders at a 10× smaller encoder learning rate; PASCAL-5i evaluation averaged three scales.Inputs were resized to 512 × 512, and evaluation used 30,000 episodes.Source: §4.3 · p. 7

4.4. Experimental Results

• On PASCAL-5i, TAFT-SE trails prior methods on 1-shot mIoU but exceeds their state-of-the-art performance by a fair margin at 5 shots.Its increase with shot count is largest for ResNet-101 and, among ResNet-50 experiments, except for PPNet; it also has the best FBIoU at 5 shots with ResNet-50 and at both shot counts with ResNet-101.Source: §4.4 · pp. 7–8, Table 1 · p. 8

5. Conclusion

• TAFT transforms semantic high-level features into task-agnostic features intended to help the decoder generate segmentation masks.The modules are designed to plug into existing semantic segmentation algorithms such as Deeplab V3+.Source: §5 · p. 8
• Semantic enrichment improves high-level features through pixel-wise self-attention and an auxiliary network trained for multi-class semantic segmentation.Source: §5 · p. 8
• TAFT-SE is not always state-of-the-art at one shot, but improves as shots increase and beats tailored few-shot segmentation models at five shots with Deeplab V3+.Source: §5 · p. 8

A. Additional Experimental Results

• Tables 4–6 report PASCAL-5i mIoU and FBIoU, and COCO-20i mIoU.Source: A. Additional Experimental Results · p. 11
• The comparisons include prior methods using a VGG-16 backbone and the concurrent Cycle-Consistent Transformer work.Source: A. Additional Experimental Results · p. 11
• The section also presents TAFT-SE combined with Fully Convolutional Network (FCN).Source: A. Additional Experimental Results · p. 11

A.1. Combination with FCN

• FCN treats the classification network's feature extractor as its encoder and its convolutionalized layers and upsampling layers as its decoder.The TAFT transformation acts on attended high-level features, while the convolutional decoder uses task-specific high-level features to produce masks.Source: §A.1 · p. 11, Figure 4 · p. 11
• FCN training used ResNet-50 and ResNet-101 encoders, with seven queries per 1-shot episode and three per 5-shot episode.The learning rate was 3 × 10^-3, with other hyperparameters matching the Deeplab V3+ experiments.Source: §A.1 · p. 11
• For FCN, 5-shot PASCAL-5i mIoU and FBIoU are comparable to most prior methods; on COCO-20i, ResNet-101 mIoU exceeds prior methods.FCN's PASCAL-5i FBIoU exceeds prior works except PPNet with ResNet-50 and exceeds all other methods with ResNet-101; on COCO-20i, FCN has the largest Δ except for TAFT-SE on Deeplab V3+.Source: §A.1 · p. 11, Table 4 · p. 12

A.2. Comparison with Concurrent Work

• TAFT-SE adds few-shot segmentation capability to semantic segmentation models such as Deeplab V3+, whereas CyCTR enhances the existing few-shot architecture PFENet.TAFT transforms features linearly; CyCTR transforms them nonlinearly through an attention layer (Zhang et al., 2021b).Source: §A.2 · p. 12
• CyCTR has higher 1-shot mIoU on both datasets and both backbones, while TAFT-SE has higher PASCAL-5i FBIoU for both shot counts.For 5-shot mIoU, TAFT-SE leads on COCO-20i and on PASCAL-5i with ResNet-101; PASCAL-5i ResNet-50 is similar.Source: §A.2 · p. 12
• Both methods use multi-head scaled-dot-product attention, but TAFT-SE applies self-attention within one feature while CyCTR attends across query and support features.TAFT-SE’s attention captures within-feature context; CyCTR’s cross-attention correlates different features (Zhang et al., 2021b).Source: §A.2 · p. 12

A.3. Additional Experiments with Various Numbers of Shots

• On PASCAL-5i, TAFT-SE with Deeplab V3+ improves quickly as shot count increases, starts outperforming PFENet at 3 shots, and its performance gain grows with additional shots.The comparison uses mean mIoU scores from 1-shot to 10-shot.Source: §A.3 · p. 12
• PFENet gains only slightly as shots increase, and its 10-shot performance is worse than its 7-shot performance.The comparison covers 1-shot through 10-shot mean mIoU on PASCAL-5i.Source: §A.3 · p. 12

A.4. Visualization of Prototype and Reference Vectors

• Across meta-training, prototype changes are consistently much larger than reference-vector changes, whether or not the sampled class changes between episodes.The comparison tracks foreground and background vectors from the first through the 10,000th episode.Source: §A.4 · p. 13
• The experiment uses TAFT on Deeplab V3+ without SE and a ResNet-50 encoder to isolate reference-vector behavior.It measures Euclidean distances between vectors across episodes and visualizes their percentage changes in Figure 5.Source: §A.4 · p. 13

A.5. Ablation Study on Transforming Low Level Feature

• The main TAFT-SE experiments omit low-level transformation because those features contain shape information considered general.The low-level transformation experiment omits SE to isolate the effect of that transformation.Source: §A.5 · pp. 13–14
• Adding low-level feature transformation degrades 1-shot and 5-shot mIoU by 1.39 and 0.74, and 1-shot and 5-shot FBIoU by 0.70 and 0.61.The comparisons are on PASCAL-5i; the tables report TAFT on Deeplab V3+ with and without low-level feature transformation.Source: §A.5 · p. 14, Table 8 · p. 14, Table 9 · p. 14
• The tested low-level transformation adds a reference-vector set and trains it with an additional regression loss.The transformation matrix uses reference vectors and prototypes computed from low-level support features.Source: §A.5 · p. 14

B. Additional Qualitative Results

• Segmentation quality improves from 1-shot to 5-shot in the displayed TAFT-SE results.Source: B. Additional Qualitative Results · p. 14
• Figure 6 presents 1-shot segmentation results for TAFT-SE on Deeplab V3+ on PASCAL-5i.Source: Figure 6 · p. 15
• Figure 7 presents 5-shot segmentation results for TAFT-SE on Deeplab V3+ on PASCAL-5i.Source: Figure 7 · p. 16