Less is More: Encoder-only Audio-Visual Segmentation

1. INTRODUCTION

• Image-segmentation components can increase model size and computation without substantially improving segmentation quality, motivating simpler, more modular architectures.The cited recent studies examine redundant components in image segmentation models.Source: §1 · p. 1
• EASE removes task-specific building blocks in favor of a plain ViT, enabling integration of Vision Foundation Models while reducing complexity.The paper tests PVTv2 and DINOv3 visual backbones.Source: §1 · p. 1
• EASE achieves SotA AVSS performance with 3× higher throughput, while Fig. 1 contrasts its simplified approach with common Transformer-based AVS methods.The scalable vanilla ViT design also supports optimized attention implementations such as FlashAttention for faster training and inference.Source: §1 · p. 1, Figure 1 · p. 1
• The paper contributes a simple AVS baseline, model-agnostic cross-modal fusion with audio feature enhancement, and experiments across all AVS sub-tasks.The experiments assess the balance between segmentation performance and efficiency.Source: §1 · p. 1

2. BACKGROUND

• AVS predicts pixel-wise masks for sound-producing objects and comprises S4, MS3, and AVSS; AVSS also assigns class labels to pixels.S4 and MS3 focus on binary sound-producing-versus-silent predictions, and their data are subsets of AVSS.Source: §2 · p. 1
• Prior AVS methods use cross-attention with audio as a reference query [4] [23] or bidirectional attention with audio as keys and values [5, 27].The initial TPAVI network used CNN-based temporal pixel-wise audiovisual interaction.Source: §2 · pp. 1–2
• EASE clusters encoded audio into semantic centers, generates sparse audio queries, joins them with visual features, and fuses the sequence through remaining encoder blocks.Its mask module predicts masks from visual tokens and classes from sparse audio-query tokens.Source: Figure 2 · p. 2
• Other AVS approaches adapt Mask Transformer through semantic alignment, while others explore selective state-space models or shared modality spaces.The shared-space approaches use a conditional variational autoencoder or diffusion model.Source: §2 · p. 2
• Inherited hierarchical backbones, feature pyramids, and localized attention add complexity and make AVS models incompatible with VFM pretraining without significant re-engineering.This motivates a simpler, efficient alternative for audio-visual segmentation.Source: §2 · p. 2

3. METHOD

• EASE builds AVS on a plain ViT, adding audio feature enhancement, mid-backbone audio query injection, and bidirectional audio-visual fusion.The simplified architecture is compatible with modern vision foundation models.Source: §3 · p. 2
• Learned semantic centers cluster frozen VGGish frame features end-to-end with Gumbel-Softmax to capture variation across sound categories.The resulting enhanced audio features are intended to reduce semantic bias from intra-class variation, without dataset-level precomputation.Source: §3 · p. 2
• The plain-ViT mask design provides the structural foundation, while mid-backbone audio-query injection adapts it to multimodal AVS.Final masks come from visual mask features and class labels from audio queries; intermediate masks provide localization supervision, with masked attention annealed during training because it is expensive.Source: §3 · p. 2
• After L1 blocks encode visual tokens independently, enhanced audio generates sparse queries that join the visual sequence for later ViT processing.The queries are formed by cross-attending enhanced audio features with a learnable query vector.Source: §3 · p. 2
• The fusion module maps visual mask features and enhanced audio features to updated visual tokens using only dimension-D token sequences, without constraining the backbone.Source: §3 · p. 3
• Visual clusters score how well audio is grounded in the scene to suppress off-screen sound, then audio conditions visual-token updates across ViT blocks.Fusion modules share weights across blocks, keeping added parameter count negligible.Source: §3 · pp. 2–3

4.1. Setup

• AVSBench experiments use PVTv2 and DINOv3 at 224 × 224 or 384 × 384 inputs; AVSS training takes up to 15 and 11 GPU hours, respectively.Videos provide one frame per second; efficiency is measured in FPS and GFLOPs on an RTX 4090.Source: §4.1 · p. 3

4.2. Main Results

• EASE achieves SotA AVSS performance with ViT and superior throughput; with PVTv2, it outperforms methods with more complex architectures.The AVSS comparison includes efficiency evaluations.Source: §4.2 · p. 3, Table 1 · p. 3
• With ViT-L, EASE achieves the best MS3 Jaccard index and S4 F-score among compared specialized methods, while those methods retain advantages on other metrics in low-data regimes.The comparison spans S4, MS3, and AVSS across backbone models and input sizes.Source: §4.2 · p. 4, Table 2 · p. 4

4.3. Ablations

• At 224 × 224, S4 pretraining raises MS3 Jaccard about 9% and F-score 2%; at 384 × 384, Jaccard rises about 4% while F-score falls about 0.4%.The PVTv2 model is pretrained on S4 and fine-tuned on MS3.Source: §4.3 · p. 4, Table 3 · p. 4
• Learned clustering-based audio enhancement significantly outperforms both the DDESeg feature bank and a single linear projection baseline.The static feature bank does not improve over the linear projection and sometimes reduces performance.Source: §4.3 · p. 4, Table 4 · p. 4
• Clustering-based bidirectional fusion significantly outperforms cross-attention and CAVP bidirectional attention, capturing complex relationships between modalities.The comparison uses EASE with DINOv3-B at 224 × 224.Source: §4.3 · p. 4, Table 5 · p. 4

• EASE replaces task-specific complexity with a plain ViT backbone and pairs learned audio enhancement with lightweight fusion to achieve SotA accuracy at 3× the speed of prior methods.Source: §5 · p. 4