Source-linked AI summary
Lead Vocal Separation from Vocal Ensemble Mixtures Using Phoneme Alignment
Yuma Narahata, Tomohiko Nakamura, Yuki Saito, Hiroshi Saruwatari
TL;DR
Vo separation is difficult because the lead and interfering parts are similar singing voices, yet separating them supports lead-voice processing and minus-one accompaniment. This paper conditions BS-RoFormer on frame-level Vo phoneme alignment and finds improved separation over audio-only and activity-only conditioning. The phoneme-label advantage is larger when fewer other parts share Vo’s phoneme.
Problem
Vo separation must distinguish a melody-carrying lead from other overlapping singing voices with similar acoustic characteristics, despite its potential uses in lyric recognition and minus-one accompaniment.
Source: §I · p. 1Method
The method uses FiLM to condition BS-RoFormer intermediate representations on frame-level phoneme labels aligned to the Vo part.
Source: §I · p. 1Results
The phoneme-conditioned method improved SI-SDRi over the audio-only baseline for Vo and Other and achieved higher average SI-SDRi than activity-only conditioning.
Source: D. Results · pp. 4–5, §V · p. 5Takeaways & Limitations
The phoneme-label advantage is larger when fewer remaining vocal parts share the same phoneme as Vo.
Source: E. Phoneme-Overlap Analysis · p. 5Activity-only conditioning exceeded the proposed method on two pieces performed by the same singer group, and analysis of singer- or ensemble-dependent factors is left for future work.
Source: D. Results · p. 5
Abstract
from arXiv · showhide
Contemporary a cappella singing often has a lead-and-accompaniment texture, where the lead vocal (Vo) part carries the main melody and the remaining vocal parts provide accompaniment. Owing to their distinct roles, separating the Vo part from the remaining vocal parts, referred to as Vo separation, enables downstream applications such as lyric recognition and minus-one accompaniment generation for vocal ensemble music. Despite these potential applications, acoustic cues for this task are limited because the target and interfering sources are all singing voices with similar acoustic characteristics and often overlap in time, making Vo separation challenging. In this paper, we propose a Vo separation model that uses phoneme alignment of the Vo part as auxiliary information. The proposed model is based on band-split RoPE Transformer (BS-RoFormer), a state-of-the-art music source separation model, and introduces frame-level phoneme labels into its intermediate representations using feature-wise linear modulation (FiLM). Experimental results show that phoneme-alignment conditioning improves Vo separation performance over an audio-only baseline and yields larger average gains than conditioning only on Vo singing/silence activity. Further analysis suggests that the advantage of phoneme-label information is larger when fewer remaining vocal parts share the same phoneme as Vo.
I. INTRODUCTION
In contemporary a cappella, the lead vocal carries the main melody and lyrics while the remaining parts provide accompaniment, making separation useful for lead-voice processing and minus-one accompaniment. Because both target and interfering parts are singing voices, the paper uses Vo phoneme alignment as an auxiliary cue and examines its effects alongside activity-only conditioning and phoneme overlap.
I. INTRODUCTION
- Vo separation isolates the lead vocal from the remaining vocal parts, supporting lead-voice processing and Vo-minus-one accompaniment applications.The lead carries the main melody and lyrics; the other parts provide accompaniment and percussive sounds.Source: §I · p. 1
- Unlike source-type separation, Vo separation targets a musical role, so an audio-only model must infer the Vo/non-Vo distinction from acoustic cues among similar voices.This motivates using role-specific auxiliary information to specify the Vo part more directly.Source: §I · p. 1
- Because other parts may share Vo’s lyrics or phonemes, the effectiveness of lyric-derived cues is uncertain and motivates analysis by phonetic overlap.The paper draws an analogy to lyric-informed singing voice separation, where the target carries the melody and lyrics.Source: §I · p. 1
- The proposed method conditions BS-RoFormer intermediate representations on frame-level Vo phoneme labels using FiLM.The paper also compares phoneme labels with singing/silence activity and analyzes conditions with different phoneme overlap.Source: §I · p. 1
II. RELATED WORK
Vo separation shares its mixture type with vocal ensemble separation but resembles singing voice separation in targeting a melody-carrying vocal part. Prior vocal ensemble separation work has largely estimated individual parts from the mixture, while part-associated Fo conditioning presents a different comparison setting.
II. RELATED WORK
- Vo separation operates on vocal ensemble mixtures like vocal ensemble separation, but resembles singing voice separation in separating a melody-carrying part from other sources.These are the two relationships highlighted for Vo separation.Source: §II · p. 2
A. Vocal Ensemble Separation
- Prior vocal ensemble separation studies adapted speech- and music-separation networks to estimate individual vocal parts, mainly using the mixture without auxiliary information.Examples include DPTNet, MRDLA, X-UMX, Hybrid Transformer Demucs, and SepReFormer.Source: A. Vocal Ensemble Separation · p. 2
- Conditioning on part-specific Fo trajectories has been proposed, but obtaining trajectories assigned to vocal parts requires multi-Fo estimation and part tracking or oracle trajectories.That makes this approach less directly comparable to the setting considered here.Source: A. Vocal Ensemble Separation · p. 2
B. Auxiliary Information for Singing Voice Separation
Lyric-derived information can cue the lead role because Vo typically carries the main lyrics; aligned phonemes provide a controlled form of that information. The proposed method incorporates phoneme alignment into BS-RoFormer, whose band-split Transformer architecture estimates a Vo mask and obtains the remaining parts by subtraction.
B. Auxiliary Information for Singing Voice Separation
- Because Vo typically carries the main lyrics, lyric-derived information can provide a direct cue for Vo separation.Prior work used time-aligned lyrics or phonemes, while other studies estimated or jointly modeled alignment and separation; this paper uses given phoneme alignment to isolate conditioning effects from alignment errors.Source: B. Auxiliary Information for Singing… · p. 2
III. PROPOSED METHOD
- The proposed method incorporates Vo phoneme alignment into BS-RoFormer using FiLM-based conditioning.The figure marks the phoneme-alignment conditioning mechanism in green; other modules match the original BS-RoFormer.Source: §III · p. 2, Figure 2 · p. 3
A. Network Architecture of BS-RoFormer
- BS-RoFormer processes a monaural input’s complex spectrogram by splitting it into frequency bands and mapping each to an intermediate representation, then alternates temporal and frequency-band Transformer blocks.The Transformer modules use RoPE to encode relative positional information.Source: A. Network Architecture of BS-RoFormer · p. 3
- After the Transformer modules, a mask-estimation module estimates a complex mask for Vo, and the remaining-vocal spectrogram is computed by subtracting the estimated Vo spectrogram from the input.The estimated Vo spectrogram is obtained by element-wise multiplication of the mask and input spectrogram.Source: A. Network Architecture of BS-RoFormer · p. 3
B. Incorporating Phoneme Alignment Using FiLM
The proposed network conditions BS-RoFormer’s intermediate representations on frame-level Vo phoneme labels using FiLM. It preserves the baseline’s input-output format while inserting conditioning throughout the network.
B. Incorporating Phoneme Alignment Using FiLM
- FiLM layers are inserted after the band-split module and every Transformer module, conditioning each intermediate representation on the Vo phoneme label at that frame.The conditioned network keeps BS-RoFormer’s input-output format.Source: B. Incorporating Phoneme Alignment… · p. 3
- Vo phoneme alignment is represented as a frame-level sequence of label indices, with labels including phonemes, silence, and padding.Silence marks valid frames where Vo is not singing; padding is reserved for dummy batching frames.Source: B. Incorporating Phoneme Alignment… · p. 3
- A phoneme-label index is converted to an embedding, and fully connected layers generate frame-specific FiLM scale and bias vectors.The FiLM layers are indexed after the band-split module and after each Transformer module.Source: B. Incorporating Phoneme Alignment… · p. 3
- FiLM modulates each intermediate representation using the frame’s scale and bias, which do not vary across frequency bands.The modulated representations are passed to the subsequent module.Source: B. Incorporating Phoneme Alignment… · p. 3
- Because FiLM modifies only intermediate representations, the conditioned model retains the BS-RoFormer output format and uses the same loss formulation as the baseline.Source: B. Incorporating Phoneme Alignment… · p. 3
IV. EXPERIMENTS
The experiments examine the effect of phoneme alignment on Vo separation and compare it with conditioning on Vo singing/silence activity alone.
IV. EXPERIMENTS
- The experiments investigate the impact of phoneme alignment on Vo separation.Source: §IV · p. 4
- An activity-only condition derived from the phoneme alignments is evaluated alongside phoneme-alignment conditioning.Source: §IV · p. 4
- The activity-only evaluation tests the effect of Vo singing/silence information.Source: §IV · p. 4
A. Data
The study uses jaCappella recordings to create Vo-versus-Other mixtures and manually corrected Vo phoneme alignments. Its activity-only condition converts phoneme labels into singing, silence, and padding labels.
A. Data
- jaCappella contains 50 vocal ensemble pieces with six parts, mixtures, isolated part recordings, and musical scores.The pieces are arrangements of Japanese children’s and school songs, divided into ten five-piece genre subsets.Source: A. Data · p. 4
- Each piece is reorganized into a Vo source and an equal-gain mix of the other five parts, with 40 pieces for training and 10 for evaluation.The evaluation set includes one piece from each subset, and no phoneme label appears only in evaluation.Source: A. Data · p. 4
- Because jaCappella lacks phoneme alignments, Vo alignments are initialized with lyric-to-recording dynamic time warping and manually corrected by trained annotators.The experiments use these corrections under the idealized assumption of accurate alignment; there are 36 labels.Source: A. Data · p. 4
- The activity-only condition maps phoneme labels to singing while retaining silence and padding labels.Source: A. Data · p. 4
B. Compared Models
The comparison uses an audio-only BS-RoFormer baseline, an activity-only condition, and the proposed phoneme-conditioned model. All share the same BS-RoFormer backbone configuration.
B. Compared Models
- The three compared models are an audio-only baseline, an activity-only model, and the proposed phoneme-conditioned model.The activity-only condition tests whether Vo activity alone accounts for phoneme-alignment effects.Source: B. Compared Models · p. 4
- The activity-only model uses three conditioning labels, while the proposed model uses 36 phoneme labels; both use embedding dimension 36.Source: B. Compared Models · p. 4
- All models use the same BS-RoFormer backbone configuration, including 62 frequency bands, six Transformer modules, dimension 192, and eight attention heads.The STFT uses a 2048-sample window and 512-sample frame shift.Source: B. Compared Models · p. 4
C. Training Conditions
All models used shared training conditions, with random segment cropping and independent gain augmentation. Training used a combined signal and multi-resolution spectrogram loss, and the final-epoch models were evaluated.
C. Training Conditions
- Training samples were random 2.74-second crops, with Vo and Other gains independently varied from 0.5 to 1.5.Batch size was 8.Source: C. Training Conditions · p. 4
- The loss combined mean absolute signal error with multi-resolution STFT loss across five Hann-window lengths.The window lengths were 256, 512, 1024, 2048, and 4096 samples.Source: C. Training Conditions · p. 4
- Models were trained for 1,000 epochs with 1,000 minibatch updates per epoch, and the final-epoch models were evaluated.Optimization used Adam with a learning rate of 5 × 10^-5.Source: C. Training Conditions · p. 4
D. Results
Phoneme-conditioned Proposed achieved the strongest average SI-SDRi for both separated outputs, while Activity-only also improved on the audio-only baseline. The advantage was not uniform across all evaluation pieces.
D. Results
- Proposed achieved the highest average SI-SDRi for both Vo and Other and beat Baseline on every evaluation piece.SI-SDRi measures improvement over the input mixture.Source: D. Results · p. 4, Table I · p. 5
- Activity-only ranked second in average SI-SDRi for both outputs and outperformed Baseline on every evaluation piece.This supports a benefit beyond the audio-only baseline, while Proposed had higher average performance.Source: D. Results · p. 5
- Activity-only exceeded Proposed on Dongurikorokoro and Kisha, both performed by the same singer group.The authors suggest singer or ensemble configuration may affect phoneme-conditioning gains; detailed analysis is left for future work.Source: D. Results · p. 5
E. Phoneme-Overlap Analysis
Phoneme alignment improved SI-SDRi over the audio-only baseline across the analyzed overlap conditions. Its advantage over activity-only tended to shrink as more other parts shared Vo’s phoneme.
E. Phoneme-Overlap Analysis
- For each Vo-active frame, the analysis counted other parts matching Vo’s phoneme and grouped frames by that count.Condition-wise SI-SDRi averaged song-level values weighted by the duration of selected samples.Source: E. Phoneme-Overlap Analysis · p. 5
- Proposed achieved higher SI-SDRi than Baseline across all phoneme-overlap conditions, with relatively consistent improvement.The table reports duration-weighted average SI-SDRi for each condition.Source: Table II · p. 5, E. Phoneme-Overlap Analysis · p. 5
- The gap between Proposed and Activity-only tended to decrease as more other vocal parts shared Vo’s phoneme.The authors interpret this trend as phoneme-category information being most useful when it distinguishes Vo from the other parts.Source: E. Phoneme-Overlap Analysis · p. 5
V. CONCLUSION
The conclusion reports that phoneme-alignment conditioning improves Vo and Other separation over an audio-only baseline and outperforms activity-only on average. Its advantage over activity-only narrows with greater phoneme overlap; practical usability and subjective listening remain for future work.
V. CONCLUSION
- On the jaCappella corpus, Proposed improved SI-SDRi over the audio-only baseline for both Vo extraction and Vo removal.Activity-only also improved performance, but Proposed achieved higher average SI-SDRi.Source: §V · p. 5
- The activity-only comparison suggests phoneme labels contribute information beyond Vo singing/silence activity.The overlap analysis found that Proposed’s advantage over Activity-only became smaller when more parts shared Vo’s phoneme category.Source: §V · p. 5
- Future work will investigate using lyrics without accurately aligned phoneme labels to improve practical usability.Subjective listening evaluation is also left for future work.Source: §V · p. 5