PaperlayerView exact revision on arXiv

Source-linked AI summary

Taming Modality Entanglement in Continual Audio-Visual Segmentation

Yuyang Hong, Qi Yang, Tao Zhang, Zili Wang, Zhaojin Fu, Kun Ding, Bin Fan, Shiming Xiang

arXiv:2510.17234v3cs.MMcs.AIcs.CV

TL;DR

Existing multimodal continual-learning methods largely focus on coarse-grained tasks, leaving fine-grained audio-visual segmentation unaddressed. The paper introduces CAVS and CMR, combining consistency-based sample selection with collision-based rehearsal. It reports state-of-the-art performance across three class-incremental datasets.

  • Problem

    Existing continual audio-visual methods focus on coarse-grained tasks and do not address fine-grained pixel-level segmentation under continual learning.

    Source: §1 · p. 2

  • Method

    CMR addresses semantic drift and co-occurrence confusion with modal-consistency sample selection and collision-based adjustment of rehearsal frequency.

    Source: §1 · p. 2

  • Results

    CMR achieves state-of-the-art performance on three class-incremental datasets.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The experiments report CMR's effectiveness in addressing semantic drift and co-occurrence confusion in CAVS.

    Source: §1 · p. 3

    The current CAVS setup follows class-incremental labeling, treating old-class pixels in new training data as background.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Recently, significant progress has been made in multi-modal continual learning, aiming to learn new tasks sequentially in multi-modal settings while preserving performance on previously learned ones. However, existing methods mainly focus on coarse-grained tasks, with limitations in addressing modality entanglement in fine-grained continual learning settings. To bridge this gap, we introduce a novel Continual Audio-Visual Segmentation (CAVS) task, aiming to continuously segment new classes guided by audio. Through comprehensive analysis, two critical challenges are identified: 1) multi-modal semantic drift, where a sounding objects is labeled as background in sequential tasks; 2) co-occurrence confusion, where frequent co-occurring classes tend to be confused. In this work, a Collision-based Multi-modal Rehearsal (CMR) framework is designed to address these challenges. Specifically, for multi-modal semantic drift, a Multi-modal Sample Selection (MSS) strategy is proposed to select samples with high modal consistency for rehearsal. Meanwhile, for co-occurence confusion, a Collision-based Sample Rehearsal (CSR) mechanism is designed, allowing for the increase of rehearsal sample frequency of those confusable classes during training process. Moreover, we construct three audio-visual incremental scenarios to verify effectiveness of our method. Comprehensive experiments demonstrate that our method significantly outperforms single-modal continual learning methods. Code can be seen at https://github.com/cqu-student/CAVS-CMR.

1 Introduction

The introduction frames continual audio-visual segmentation as a fine-grained setting that existing coarse-grained continual methods do not address. It presents CAVS and the CMR framework for tackling semantic drift and co-occurrence confusion, with experiments reported across three class-incremental datasets.

1 Introduction

  • Existing continual audio-visual methods focus mainly on coarse-grained tasks, leaving pixel-level segmentation under continual learning unaddressed.CAVS requires sequential audio-visual segmentation while retaining knowledge of previously seen classes; its challenges include learned sounds labeled as background and confusion between co-occurring classes.Source: §1 · p. 2, Figure 1 · p. 2
  • CMR uses Multi-modal Sample Selection (MSS) for high-modal-consistency rehearsal and Collision-based Sample Rehearsal (CSR) to increase rehearsal for classes with frequent prediction collisions.CSR adjusts rehearsal frequency using discrepancies between old-model predictions and ground-truth labels.Source: §1 · pp. 2–3
  • The paper introduces Continual Audio-Visual Segmentation (CAVS), extending continual learning to audio-visual segmentation.It identifies semantic drift and co-occurrence confusion as two forms of modality entanglement in this setting.Source: §1 · pp. 2–3
  • CMR achieves state-of-the-art performance on three class-incremental datasets, and experiments report effectiveness across three CAVS task setups.The introduction identifies AVSBench-CI, AVSBench-CIS, and AVSBench-CIM as the three setups.Source: §1 · p. 3

2 Related Works

Related work covers continual learning, multimodal continual learning, and audio-visual segmentation. The paper positions CAVS at their intersection: continual pixel-level audio-visual segmentation.

2.1 Continual Learning.

  • Continual learning research addresses forgetting through regularization and rehearsal, while continual semantic segmentation extends the problem to pixel-level classification.The review contrasts single-modality semantic-shift work with the paper’s focus on multimodal semantic drift.Source: §2.1 · p. 3

2.2 Multi-modal Continual Learning

  • Existing multimodal continual methods address classification, separation, or grouping, rather than fine-grained pixel-level segmentation.The paper describes CAVS as the first continual audio-visual segmentation setting, with pixel-level modality entanglement challenges.Source: §2.2 · p. 4

2.3 Audio Visual Segmentation

  • Audio-visual segmentation localizes sound sources through pixel-level prediction, and prior work develops benchmarks and models for audio-guided segmentation.The review describes approaches using audio semantics, transformer queries, cross-modal interaction, and shared or modality-specific representations.Source: §2.3 · p. 4

3 Methods

Section 3 defines CAVS as class-incremental audio-visual segmentation and develops CMR’s two rehearsal components, MSS and CSR, for that setting.

3 Methods

  • CMR uses MSS to select modality-consistent rehearsal samples and CSR to adjust rehearsal frequency according to collisions between old predictions and current ground truth.The framework is based on the AVSBench ResNet50 architecture.Source: Figure 2 · p. 4, §3 · p. 5

3.1 Revisiting Continual Semantic Segmentation (CSS)

  • CSS presents tasks sequentially with disjoint category sets, and later stages may lack access to earlier training data while retaining the previous model.The paper extends this setting to audio-visual segmentation.Source: §3.1 · p. 5

3.2 Problem Setup and Notation of CAVS

  • CAVS pairs audio with 10 consecutive video frames; new-task labels cover new classes and background, while stored old-task samples retain their original annotations.Previously learned sounding objects are labeled background in new data, creating a need to retain their segmentation ability without their original training data.Source: §3.2 · pp. 5–6

3.3 Multi-modal Sample Selection

  • MSS replays selected samples in later tasks, reinforcing cross-modal associations and reducing drift for previously learned classes mislabeled as background.The paper reports that this approach outperforms random selection by 2.0 mIoU.Source: §3.3 · pp. 6–7
  • MSS selects, for each newly added class, the k samples with the smallest absolute audio-contribution deviation; default k = 5, and consistency is required over all T frames.Small |∆(Sa)| reflects consistent audio-visual alignment, not minimal audio contribution.Source: §3.3 · p. 7
  • MSS’s selection criterion beats random selection by 2.0 mIoU, while rehearsal with resampling outperforms direct rehearsal.The 2.0 mIoU comparison is reported for the selection criterion in an ablation study.Source: §3.3 · p. 7, §3.4 · p. 8

3.4 Collision-based Sample Rehearsal

  • CSR counts spatial conflicts where the old model predicts a learned class but the current ground truth is a new class, tracking which old classes collide most.It records significant collisions when a class’s collision ratio exceeds the mean ratio across learned classes, which is computed adaptively at each stage.Source: §3.4 · pp. 7–8
  • CSR smooths and normalizes collision frequencies into replay probabilities, giving collision-heavy classes more rehearsal while limiting any single class’s dominance.Classes without collisions are assigned frequency 1; 20% of the existing memory is sampled using the resulting distribution before combining it with that memory.Source: §3.4 · p. 8

4 Experiments

The experiments evaluate CMR on three class-incremental AVSBench datasets and compare it with adapted continual-learning baselines. Results cover overall performance, rehearsal ablations, architectural transfer, qualitative segmentation, and analyses of open-vocabulary learning and co-occurrence confusion.

4.1 AVSBench Datasets; 4.2.1 Baselines

  • AVSBench-CI contains 70 categories split into 60-10, 60-5, and 65-1 streams, with overlapped and disjoint settings; adapted classification and segmentation baselines share a ResNet-50 backbone.CIS and CIM separate single-target and multi-target videos; evaluation uses mIoU.Source: §4.1 · p. 9, §4.2.1 · p. 9

4.3 Main Results

  • CMR achieves the best performance across AVSBench-CI settings; on 60-10 overlapped, gains are 11.3 mIoU on CIS versus 1.5 mIoU on CIM.The reported improvement is larger on the single-target CIS dataset than on multi-target CIM.Source: §4.3 · pp. 9–10, Table 2 · p. 9, Figure 5 · p. 10

4.5 Additional Ablation: Number of Rehearsal Samples

  • Increasing rehearsal samples per class reduces forgetting of old classes, but excessive rehearsal can inhibit learning new samples; the authors select five samples per class.The ablation compares 3, 5, and 7 samples per class.Source: §4.5 · p. 11, Table 4 · p. 11

4.6.1 Qualitative Analysis of AVSBench-CI

  • Qualitative comparisons show more complete, finer segmentation and improved audio-based distinction between similar old and new classes after replaying more easily confused learned classes.Examples include airplane, train, and handpan segmentation.Source: §4.6.1 · p. 11, Figure 6 · p. 12

4.8 Analysis of Multi-modal Semantic Drift and Co-occurrence Confusion

  • Collision-heavy categories correspond to frequently co-occurring objects, and CSR increases their rehearsal frequency to reduce confusion between co-occurring class pairs.The analysis gives “guitar” and “man” as an example of a frequently co-occurring pair.Source: §4.8 · p. 12

4.7 Comparison with Open-vocabulary AVS

  • Open-vocabulary AVS scores 8.6 mIoU on novel classes, while CMR scores 35.3 mIoU on novel classes in the 60-10 disjoint comparison.Open-vocabulary AVS scores 15.9 mIoU on base classes; base and novel denote classes 1–60 and 61–71, respectively.Source: §4.8 · p. 12, Table 7 · p. 12, Figure 7 · p. 13

5 Conclusion

The conclusion presents CAVS as a fine-grained continual-learning task with semantic drift and co-occurrence confusion, addressed by CMR’s MSS and CSR strategies. Experiments support the framework’s effectiveness, while the current setup remains limited to fixed-length 10-second clips.

5 Conclusion

  • CAVS introduces fine-grained continual audio-visual segmentation and identifies multi-modal semantic drift and co-occurrence confusion as its two challenges.CMR combines multi-modal sample selection and collision-based sample rehearsal to mitigate incorrect modality-semantic associations.Source: §5 · p. 13
  • The current CAVS setup assumes fixed-length 10-second clips, leaving long-video settings with temporally varying audio-visual alignment as future work.The conclusion notes that semantic drift may be more severe in long videos.Source: §5 · p. 13

A.1 Multimodal Learning

The related-work discussion contrasts multimodal-learning approaches with this paper’s focus on rehearsal for continual audio-visual segmentation.

A.1 Multimodal Learning

  • Unlike Finger, which focuses on foreground-background distinction and transferring unimodal knowledge, this work selects consistent samples and replays them according to collision frequency.At the task level, Finger integrates new classes with limited incremental samples, while this work focuses on avoiding interference with old-task knowledge during new-task training.Source: §A.1 · p. 16

A.2 Evaluation Metrics

The evaluation metric is mean Intersection-over-Union (mIoU), calculated using class-level true-positive, false-positive, and false-negative counts.

A.2 Evaluation Metrics

  • mIoU is the evaluation metric, and its per-class calculation uses true positives, false positives, and false negatives.The passage defines these counts as correct predictions, incorrect predictions, and missed class predictions, respectively.Source: §A.2 · p. 16

A.3 Implementation Details

The implementation uses a PLOP-based model with rehearsal and specifies the training, input, and evaluation setup.

A.3 Implementation Details

  • Training uses ResNet-50 pretrained on ImageNet, ASPP fusion, 224 × 224 frames, and 30 epochs per task.The batch size is 2 per GPU across 4 Nvidia L40 48GB GPUs.Source: §A.3 · p. 16
  • Rehearsal selects 5 samples per class and resamples memory samples at 20% of the total sample size.Memory samples are excluded from the otherwise shared data augmentation.Source: §A.3 · p. 16
  • All tasks share a common test set containing all learned classes, while single-modal training uses only visual-modal data.The model builds on PLOP combined with memory.Source: §A.3 · p. 16

A.4 Baseline Methods

The section describes continual-learning baselines and provides context on collision pairs and the multi-object evaluation setting.

A.4 Baseline Methods

  • The classification baselines are LwF, LwF-MC, and ILT, using output distillation, multiple binary classifiers, and dual-space distillation, respectively.LwF-MC combines two weighted binary cross-entropy losses.Source: §A.4 · p. 16
  • The incremental segmentation comparisons include MiB, PLOP, and EIR, with the study adopting PLOP-based EIR for a fair comparison.The authors report that PLOP-enhanced EIR outperforms vanilla EIR; fine-tuning each task and offline training on all classes are also included as baselines, with offline training serving as an upper bound.Source: §A.4 · p. 16
  • On AVSBench-CIM, visualizations show CMR performing robustly and comparatively well when segmenting multiple co-occurring objects.The dataset presents higher demands because multiple objects often emit sounds simultaneously.Source: Figure 8 · p. 17
  • Table 9 reports collision-pair counts and identifies guitar and man as the categories with the highest collision rate.Table 8 presents the 60-10 category configuration of AVSBench under the disjoint setting.Source: Table 9 · p. 17, Table 8 · p. 17

A.5.1 Qualitative Analysis of AVSBench-CIM

The qualitative analyses illustrate CMR’s performance and the two challenges it targets, alongside sample-selection behavior and failure cases. The discussion describes how its principles may extend to long videos while identifying additional challenges in that setting.

A.5.1 Qualitative Analysis of AVSBench-CIM

  • In AVSBench-CIM examples, CMR segments both ukulele and man more completely with less class confusion, and segments most of the piano where prior methods miss it.The figure presents three multi-target cases.Source: §A.5.1 · p. 17

A.5.2 The Details of Category

  • For AVSBench-CI, the disjoint 70-category setup uses Louvain graph partitions to allocate minimally overlapping classes across steps, while the overlapped setup follows sequential category order.Table 8 presents the 60-10 category configuration under the disjoint setting.Source: §A.5.2 · pp. 17–18

A.5.3 Qualitative Analysis of Collision Classes

  • Collision classes often co-occur in prior tasks, and the authors attribute their perceived semantic similarity to the model lacking prior knowledge of new classes.The authors describe subsequent forgetting as correction of this bias, which often leads to catastrophic forgetting.Source: §A.5.3 · p. 18

A.5.4 Examples of Multi-modal Semantic Drift

  • When violin and woman are introduced in step 1, previously learned guitar and drum are labeled background, associating their audio with background semantics.The authors identify this as multi-modal semantic drift.Source: §A.5.4 · p. 18

A.5.5 Effect Analysis of Multi-modal Sample Selection (MSS)

  • MSS favors single-target samples with consistently present targets and better audio-visual alignment, which typically exhibit less semantic drift and aid retention of old classes.The figure compares MSS selections with a maximum-modality-discrepancy strategy.Source: Figure 10 · p. 18, §A.5.5 · p. 19

A.6 Failure Case Analysis

  • CMR still struggles with pixel-level confusion between visually similar classes with distinct audio and with ambiguous audio in multi-target scenes.These failure cases motivate future research directions.Source: §A.6 · p. 19, Figure 11 · p. 19

A.7 Discussion on Long-Video Applicability

  • The current CAVS setup inherits AVSBench’s 10-second, 10-frame assumption; long videos may intensify semantic drift, require segment-level modality metrics, and call for memory-buffer adaptation.Storing full long videos is memory-intensive, making segment-level selection a more practical possibility.Source: §A.7 · p. 19
  • CMR’s modality-consistent selection and collision-based rehearsal are not inherently limited to fixed-length inputs; the authors suggest segment-level MSS and temporally aware collision detection for long videos.MSS averages modality discrepancy over frames, while CSR uses collision statistics aggregated across samples.Source: §A.7 · p. 19

Related papers

Loading 2510.17234v3…