Source-linked AI summary
SEAL: Mixture-Closed Additive Reconstruction and Refinement-Aware Expert Routing for Efficient Speech Separation
Shao-Chun Hu, Zi-Xiang Lin, Jeih-Weih Hung, Hung-Shin Lee
TL;DR
Compact separators that mask and iteratively refine a mixture face limits at cancellation bins and from applying shared weights across tokens. SEAL combines mixture-closed additive reconstruction with bounded, refinement-aware expert routing. On EchoSet, it improves on TIGER (small) with lower parameter and MAC costs, and approaches TIGER (large) SI-SDRi with fewer MACs.
Problem
A bounded multiplicative mask can limit estimates where components cancel, and a shared cell applies the same weights to tokens with differing acoustic and refinement evidence.
Source: §1 · p. 1Method
SEAL combines a zero-sum additive residual bounded by local mixture amplitude with token routing to six residual experts using acoustic and inter-step evidence and a capped step cue.
Source: §1 · p. 1, §2.4 · p. 3Results
0.31 dB SI-SDRi gain over TIGER (small) with 28% fewer parameters and 2.9 times fewer MACs; SEAL (large) trails TIGER (large) by 0.07 dB SI-SDRi at 3.1 times fewer MACs.
Source: §4.1 · p. 3Takeaways & Limitations
Retrained ablations indicate that both proposed mechanisms contribute to separation quality.
Source: §5 · p. 4The evaluation is limited to EchoSet and two-speaker mixtures.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Compact time-frequency separators that mask the mixture and refine through a shared cell face two limits. First, a bounded multiplicative mask only scales a mixture bin, so where overlapping components cancel, the estimate stays small. Second, a shared cell applies the same weights to every time-frequency token at every step, so enlarging it adds compute everywhere. We present SEAL (Sparse Expert routing with Additive Latent reconstruction) to address both. For reconstruction, a zero-sum additive residual bounded by the local mixture amplitude lets estimates be nonzero where components cancel yet still sum to the mixture. For routing, a query built from acoustic and inter-step evidence sends each token to one of six residual experts, and a norm cap keeps the step cue from overriding clear acoustic evidence. On EchoSet, SEAL (small) surpasses TIGER (small) by 0.31 dB SI-SDRi with 28% fewer parameters and 2.9 times fewer MACs, and SEAL (large) is within 0.07 dB SI-SDRi of TIGER (large) at 3.1 times fewer MACs.
1. INTRODUCTION
The introduction identifies limits in mixture-masked reconstruction and shared-cell refinement, then presents SEAL as a shared separator combining additive reconstruction with token-specific expert routing. Its EchoSet operating points compare favorably with TIGER in quality and compute cost.
1. INTRODUCTION
- Single-channel separation progressed from discriminative embeddings and permutation-invariant training to time-domain, dual-path, multiscale, and group-communication models; TIGER is the EchoSet efficiency reference.EchoSet is described as the noisy reverberant corpus introduced by TIGER.Source: §1 · p. 1
- A bounded multiplicative mask limits estimates at cancelled bins, while shared-cell weights serve every token and step despite differing acoustic states and refinement evidence.Enlarging the shared cell to address those differences adds compute everywhere.Source: §1 · p. 1
- SEAL combines a mixture-closed allocation with an additive residual bounded by local mixture amplitude, enabling nonzero estimates at cancelled bins while preserving the mixture sum.Prior work includes complex masking and spectral mapping, mixture consistency, and noise estimation for noisy mixtures.Source: §1 · p. 1
- SEAL combines mixture-closed additive reconstruction and refinement-aware sparse routing in one shared cell, with two EchoSet cost–quality operating points against TIGER.SEAL (small) exceeds TIGER (small) with fewer MACs, while SEAL (large) is within 0.07 dB SI-SDRi of TIGER (large) at 3.1 times fewer MACs.Source: §1 · p. 1
- Its routing query uses inter-step evidence and sends each token to one of six experts; bounding the step cue addresses its ability to override acoustic evidence.The cited control without the norm cap reached 1.7–1.9 times the fused evidence's mean norm at the first two steps [Section 4.4].Source: §1 · p. 1
2. PROPOSED METHODOLOGY
The methodology combines a four-step shared refinement backbone with additive mixture-closed reconstruction and token-wise expert routing guided by acoustic and inter-step evidence.
2.1. Shared Refinement Backbone
- Four refinement steps share one cell’s parameters, with the backbone adapting GTCRN and reinitializing GRU hidden states at each step.After the final step, the decoder restores full frequency resolution and re-injects a gated observation adapter.Source: §2.1 · pp. 1–2
2.2. Mixture-Closed Additive Reconstruction
- Zero-sum complex corrections bounded by local mixture amplitude let speech estimates remain nonzero at cancelled bins while the residual sink closes the mixture sum.The sink absorbs what speech groups do not claim and is not supervised as a noise estimate.Source: §2.2 · p. 2
2.3. Sparse Residual Routing
- Six stateless residual experts correct the shared readout, while the uncorrected readout passes to the next step to form the inter-step delta.A separate predictor sets correction strength from the routing query and summaries of recurrent features, readout, and delta.Source: §2.3 · p. 3
2.4. Refinement-Aware Query and Bounded Cue
- Five acoustic and inter-step evidence branches fuse with a step cue capped at ρ = 0.15 to form a cosine-similarity query routing tokens among six experts.The cue changes prototype gaps by at most 2ρ, preserving signs when |g_jk| > 2ρ; branch biases and fusion embeddings remain unconstrained.Source: §2.4 · p. 3
3. EXPERIMENTAL SETUP
Experiments use EchoSet’s official split and report separation, computation, latency, and fixed-checkpoint diagnostic measures. SEAL is trained with permutation-invariant training and a composite objective, with each configuration reported from one run.
3.1. Dataset and Evaluation Metrics
- EchoSet’s official 20,268/4,604/2,650 train/validation/test split is used; SI-SDRi, SDRi, and MACs are reported.EchoSet is a 16 kHz two-speaker noisy corpus with reverberation simulated using SoundSpaces 2.0.Source: §3.1 · p. 3
- Batch-1 fp32 latency is measured on one host, while fixed-checkpoint diagnostics use utterance-bootstrap 95% confidence intervals.Timing averages warm-up-excluded waveform-to-waveform passes; longer inputs use fewer passes.Source: §3.1 · p. 3
- Fig. 2 labels “Traj.” as within-step temporal statistics and marks its score profile as schematic.Source: Figure 2 · p. 3
3.2. Training Configuration
- SEAL (small, R = 4) and SEAL (large, R = 8) are trained with Adam, permutation-invariant training, and a composite objective; each configuration is a single run without seed repeats.Training runs up to 500 epochs with early stopping on validation loss.Source: §3.2 · p. 3
4. RESULTS AND DISCUSSION
SEAL’s EchoSet results pair lower compute with competitive separation scores, while ablations test its reconstruction residual, expert assignment, and refinement-aware routing. The results also show the residual’s benefit on cancelled bins and the step cue’s selective influence under a norm cap.
4.1. Comparisons with State-of-the-Art Systems
- Both SEAL models stay below TIGER counterparts through 32 s on CPU and GPU; SEAL GPU latency is nearly flat to 8 s but rises beyond 16 s.For 1 s inputs, SEAL (small) takes 32.7 vs. 43.5 ms on GPU and 247 vs. 1,211 ms on one CPU thread against TIGER (small).Source: §4.1 · p. 3, Figure 3 · p. 4
4.2. Reconstruction Accuracy and Cancellation
- Removing the additive residual lowers test SI-SDRi by 1.04 dB while saving 990 parameters; the single-atom variant is 0.55 dB below Full.The single-atom gap reflects factorized parameterization, not reachability; fixed-checkpoint interventions measure dependence and need not match retrained gaps.Source: §4.2 · p. 3, §4.1 · p. 3
4.1. Comparisons with State-of-the-Art Systems
- 0.31 dB SI-SDRi over TIGER (small) with 28% fewer parameters and 2.9 times fewer MACs; SEAL (large) trails TIGER (large) by 0.07 dB SI-SDRi at 3.1 times fewer MACs.SEAL (small) reaches 12.89 dB SI-SDRi and 13.62 dB SDRi; SEAL (large) leads TIGER (large) by 0.20 dB SDRi.Source: §4.1 · p. 3, Table 1 · p. 4
4.2. Reconstruction Accuracy and Cancellation
- On 4,604 validation utterances, adding the residual reduces cancelled-bin mean NMSE from 0.891 to 0.552, a 38.0% relative reduction.Heavily cancelled bins fall below 0.325 under the stated speech-energy and source-ratio filters; disabling the residual costs 1.28 dB validation SI-SDRi.Source: §4.2 · pp. 3–4
4.3. Conditional Capacity and Expert Assignments
- Fixed-hash dispatch reaches 12.57 dB test SI-SDRi versus 11.99 dB for dense readout and 12.89 dB for Full; learned assignment adds 0.32 dB at identical cost.The comparison separates the expert bank’s capacity from how tokens are assigned.Source: §4.3 · p. 4, Table 2 · p. 4
- Permuting Full’s token–expert matches lowers mean validation SI-SDRi by 0.64 dB over 20 permutations, while the same protocol costs 0.01 dB on fixed-hash.Expert counts, weights, and correction strengths are held fixed during the permutations.Source: §4.3 · p. 4
4.4. Refinement Evidence and Step-Cue Influence
- Disabling inter-step delta and temporal-statistics routing branches gives 12.10 dB SI-SDRi (∆= −0.79 dB); removing the additive step cue costs 0.30 dB.Both controls retain learned assignment; the delta summary still reaches βs in the branch-ablation control.Source: §4.4 · p. 4
- With ρ = 0.15, zeroing the step cue changes 33.6–44.8% of lowest-quartile-margin assignments but at most 0.01% of highest-margin assignments.Without the norm cap, the step cue reaches 1.7–1.9 times the mean fused-evidence norm at the first two steps, and that control is 0.32 dB below Full.Source: §4.4 · p. 4
5. CONCLUSION AND FUTURE WORK
On EchoSet, SEAL (small) outperforms reported TIGER (small) scores on both metrics with fewer parameters and MACs, and both SEAL configurations have lower measured latency. The conclusion also highlights the mechanisms’ contributions and limits the evaluation to EchoSet and two-speaker mixtures.
5. CONCLUSION AND FUTURE WORK
- SEAL (small) outperforms reported TIGER (small) scores on both metrics with fewer parameters and MACs, while both SEAL configurations achieve lower measured latency.Retrained ablations confirm that both proposed mechanisms contribute to separation quality.Source: §5 · p. 4
- The additive residual relaxes per-bin scaling while preserving the mixture sum; the router selects token-specific corrections from acoustic and inter-step evidence, with a norm cap limiting the step cue to low-margin decisions.Source: §5 · p. 4
- The evaluation is limited to EchoSet and two-speaker mixtures; future work will examine additional corpora and speaker counts.The authors also plan to replace quadratic temporal attention to reduce GPU latency for inputs longer than 16 s.Source: §5 · p. 4