AVSegFormer: Audio-Visual Segmentation with Transformer

• AVS asks models to locate audible frames and delineate sounding objects, requiring fine-grained relationships between image pixels and audio.Its sub-tasks include single-source S4, multi-source MS3, and multi-category semantic segmentation AVSS.Source: Introduction · p. 1, Figure 1 · p. 1
• Existing methods for other audio-visual tasks face difficulty on AVS, while simple fusion struggles with dense mixed audio and changing sources.Dense segmentation also struggles to separate multiple coexisting sounding objects for one-to-one localization.Source: Introduction · pp. 1–2
• AVSegFormer combines a transformer encoder, audio-visual mixer, query generator, and cross-modal transformer decoder for audio-visual segmentation.The mixer uses an auxiliary loss; the decoder forms sparse queries to separate sources and match visual features.Source: Introduction · p. 2, Figure 2 · p. 2
• The framework combines bidirectional cross-modal feature fusion with dense pixel-level mixing and sparse instance-level decoding.The complementary representations are intended to adapt to multiple sound sources and objects.Source: Introduction · p. 2
• AVSegFormer significantly outperforms existing state-of-the-art methods on the AVS benchmark.Source: Introduction · p. 2

Multi-Modal Tasks

• Audio-visual correspondence, event localization, and sound-source localization are established multimodal tasks; transformer architectures are also used in unified multimodal models.The paper describes AVS as sharing commonalities with these tasks and says pioneering work inspired AVSegFormer.Source: Multi-Modal Tasks · p. 2

Vision Transformer

• Vision transformers developed from ViT and advanced through models such as DETR, Deformable DETR, and DINO, inspiring their application to AVS.Source: Vision Transformer · pp. 2–3

Image Segmentation

• Transformer segmentation models including Mask2Former, Mask DINO, and OneFormer unify image-segmentation tasks and inform the segmentation design of AVSegFormer.Image segmentation partitions an image into distinct regions.Source: Image Segmentation · p. 3

Overall Architecture

• The query-based architecture uses audio queries and a transformer encoder to produce features for a decoder that separates potential sparse objects.The audio-visual mixer amplifies relevant features and suppresses irrelevant ones, with an auxiliary mixing loss supervising the reinforced features.Source: Overall Architecture · p. 3

Multi-Modal Representation

• The visual backbone extracts hierarchical visual features from video frames, while the audio pipeline uses VGGish features derived from a mel spectrum.The audio is resampled to 16 kHz mono and mapped to a 64th-order mel filter bank before VGGish extraction.Source: Multi-Modal Representation · pp. 3–4
• The adopted channel-attention mixer uses audio features to adjust visual channels and performed better than the initial cross-attention mixer.Source: Figure 4 · p. 4

• The query generator uses initial queries with audio features as keys and values to produce sparse audio-conditioned queries, then combines them with learnable queries for the decoder.Source: Query Generator · p. 4
• Learnable queries provide dataset-level context and let the model adjust attention allocated to different sounding targets.Source: Query Generator · p. 4

Transformer Encoder

• Backbone features at 1/8, 1/16, and 1/32 resolution are flattened and concatenated as transformer-encoder input queries.Source: Transformer Encoder · p. 4
• The encoder outputs are reshaped to their original spatial shapes, and the 1/8-scale features are upsampled by 2×.Source: Transformer Encoder · p. 4
• The upsampled 1/8-scale features are added to the visual backbone’s 1/4-scale features to form the mask feature.The mask feature has dimensions T × D × h × w, with h = H/4 and w = W/4.Source: Transformer Encoder · p. 4

Audio-Visual Mixer

• Because audio semantics vary widely, a static network may fail to capture relevant information and identify inconspicuous sounding objects.Source: Audio-Visual Mixer · p. 4
• The mixer uses channel attention and audio-visual cross-attention to learn weights that amplify or suppress visual channels based on audio features.Source: Audio-Visual Mixer · p. 4
• The mixer takes audio features and the initial mask feature as inputs and produces a mixed mask feature.The number of attention heads is set to 8 by default.Source: Audio-Visual Mixer · p. 4

Transformer Decoder

• The decoder uses mixed queries as input queries and multi-scale visual features as keys and values to match queries with visual features.Source: Transformer Decoder · p. 4
• As decoding proceeds, output queries aggregate visual features and combine auditory and visual modalities while containing target information.Source: Transformer Decoder · p. 4
• The model multiplies the mixed mask feature by output queries, integrates channels with an MLP, and predicts the segmentation mask through a fully connected layer.The output mask has dimensions T × Nclass × h × w, where Nclass is the number of semantic classes.Source: Transformer Decoder · p. 4

Loss Function

• The auxiliary mixing loss supervises the mixer to improve target-object localization in complex scenes.Source: Loss Function · p. 4
• The mixing loss compares a binary mask predicted from the channel-integrated mixed mask feature with foreground labels using Dice loss.Source: Loss Function · p. 5
• The final segmentation mask is supervised against ground truth with Dice loss as the IoU loss, and the total loss adds the weighted mixing loss.The auxiliary-loss coefficient λ is set to 0.1 because it performs best.Source: Loss Function · p. 5

Discussion

• AVSegFormer adapts a Mask2Former-like segmentation framework to handle multimodal information for audio-visual segmentation.The paper notes that this capability is not inherent in Mask2Former.Source: Discussion · p. 5
• Dual-tower backbone networks extract visual and auditory features.Source: Discussion · p. 5
• A dense audio-visual mixer and sparse audio-visual decoder are designed to leverage auditory cues effectively and enhance segmentation performance.Source: Discussion · p. 5

Dataset

• AVSBench-Object supplies pixel-level annotations for S4 and MS3, while AVSBench-Semantic adds semantic labels and longer videos for AVSS.The Object videos are 5 seconds with one frame per second; Semantic videos are 10 seconds with 10 frames extracted.Source: Dataset · p. 5

Implementation Details

• Training uses AdamW, batch size 2, and a 2 × 10^-5 initial learning rate; MS3 trains for 60 epochs, while S4 and AVSS train for 30.The encoder and decoder each have six layers and 256-dimensional embeddings; the mixing-loss coefficient is 0.1.Source: Implementation Details · p. 5

Comparison with Prior Arts

• Against AVSBench, AVSegFormer improves S4 by 3.66 mIoU and 1.1 F-score (R50), and AVSS by 6.89 mIoU and 6.8 F-score (PVTv2), among gains across all tasks.On MS3, AVSegFormer-PVTv2 improves by 1.65 mIoU and 5.0 F-score; AVSS R50 improves by 4.75 mIoU and 4.1 F-score.Source: Table 1 · p. 6, Comparison with Prior Arts · pp. 6–7
• AVSegFormer-PVTv2 reaches 82.06 mIoU and 89.9 F-score on S4, and improves over prior-task methods by 15.47 mIoU and 14.9 F-score on MS3.On AVSS, AVSegFormer-PVTv2 surpasses AOT by 11.26 mIoU and 11.0 F-score; on S4, it exceeds LGVT by 7.12 mIoU and 2.6 F-score.Source: Comparison with Prior Arts · p. 6, Table 1 · p. 6

Ablation Study

• In experiments varying decoder-input queries among 1, 100, 200, and 300, performance increased with query count, and the best performance was obtained with 300 queries.Source: Ablation Study · p. 7
• Learnable decoder queries bring relatively small gains on S4 but significant gains on MS3, which the authors attribute to sounding-object complexity.The ablation compares S4 and MS3.Source: Ablation Study · p. 7
• The channel-attention mixer outperforms cross-attention; visualizations show it enhances sounding-object features and suppresses non-sounding features, while mixing loss helps final predictions.The visualization examples include a sounding girl and dog and non-sounding objects or background.Source: Ablation Study · p. 7
• Qualitative comparisons across three AVS tasks show better sounding-object localization, semantic understanding, and segmentation by AVSegFormer than AVSBench.The visualizations include multiple-sound scenes.Source: Ablation Study · p. 7, Figure 6 · p. 7

Conclusion

• AVSegFormer combines a transformer encoder, audio-conditioned query generator, dense audio-visual mixer, and sparse audio-visual decoder.The components build mask features, adjust visual features, and separate audio sources while matching visual features.Source: Conclusion · p. 7
• The framework is designed to improve audio-visual cross-modal representations and AVS performance across different scenarios.Source: Conclusion · p. 7
• Experiments show that AVSegFormer outperforms existing state-of-the-art methods.Source: Conclusion · p. 7