Open-Set Ego-Noise Separation for Legged-Robot Audition via Annotation-Free Adaptation and Pretrained-Model Transfer

I. INTRODUCTION

• Acoustic sensing can reveal events beyond camera view, but microphones also capture footstep impacts, joint-backlash rattling, and motor noise.Legged-robot mobility, including stair traversal, makes these platforms promising for patrol applications.Source: §I · p. 1
• Open-set ego-noise separation removes robot-specific noise while preserving arbitrary environmental sounds whose classes are unspecified in advance.Source: §I · p. 1
• Robot-specific dictionary separators require separately recorded ego-noise-only data, creating substantial adaptation work across robots, gaits, and operating conditions.Source: §I · p. 1
• Adaptation recordings represent a target domain that may span sites and routes; the separator is adapted offline before its parameters remain fixed during online patrol.Source: §I · p. 1
• The proposed two-stage framework selects ego-noise-dominant clips from unlabeled recordings, mixes them with diverse environmental sounds, and adapts a pretrained separator.RecurGraph supplies the selected clips, and Transfer-DiT adapts the general-purpose separator.Source: §I · p. 2

A. Platform-Dependent Characteristics of Ego-Noise

• Legged-robot ego-noise varies across robots, gaits, floors, and recording conditions, making a fixed generic model inadequate for high-fidelity suppression.Source: A. Platform-Dependent Characteristics… · p. 2

B. Ego-Noise Reduction and Separation

• Reducing footstep noise through walking policies trades off against walking speed and energy use, and complete elimination of footstep and actuator sounds remains difficult.Source: B. Ego-Noise Reduction and Separation · p. 2
• Multichannel separation can use spatial cues, but microphone layout, channel count, calibration, reflections, and robot-body occlusion constrain it, motivating this study's single-channel focus.Source: B. Ego-Noise Reduction and Separation · p. 2
• Dictionary-based methods accommodate unknown target sounds but struggle with complex sounds overlapping broadband transient ego-noise; prior neural robot separators focused on speech, not open-set sounds.The cited dictionary approaches are,, and prior neural approaches are,.Source: B. Ego-Noise Reduction and Separation · p. 2

C. Generic Audio Separation

• Generic zero-shot separators provide broad audio priors, but semantic prompts cannot specify target-robot acoustics well enough for high-fidelity ego-noise removal.Source: C. Generic Audio Separation · pp. 2–3

A. Concept

• The framework derives robot-specific noise supervision from unlabeled pre-deployment recordings and samples preservation targets from a large-scale sound-event dataset.RecurGraph selects recurring ego-noise clips; Transfer-DiT adapts a pretrained general-purpose separator.Source: A. Concept · p. 3

B. RecurGraph: Recurrence-Guided Seed Initialization and Graph-Based Score Propagation

• RecurGraph estimates ego-noise-dominant scores because using every clip as ego-noise training data could teach the separator to remove environmental sounds that should be preserved.Source: B. RecurGraph: Recurrence-Guided Seed… · p. 3
• A pretrained audio encoder maps several-second clips to embeddings, and global-average pooling forms a centroid representing the recording-wide shared acoustic structure.The encoder uses a VAE encoder followed by a Transformer encoder; recurrent ego-noise is expected to contribute more consistently than intermittent environmental sounds.Source: B. RecurGraph: Recurrence-Guided Seed… · p. 3
• Cosine similarity to the centroid identifies clips expected to contain more of the recurring acoustic component, reflecting target robot, gait, and recording-environment characteristics.Source: B. RecurGraph: Recurrence-Guided Seed… · p. 3
• The upper and lower ρ fractions of clips in the centroid-similarity ranking become positive and negative seeds, while middle-ranked clips remain unassigned.Using only the ranking ends avoids hard assignments for ambiguous clips and avoids reliance on a fixed similarity threshold that may not transfer across settings.Source: B. RecurGraph: Recurrence-Guided Seed… · p. 3
• PCA-reduced embeddings define a neighborhood graph because ego-noise-dominant clips are hypothesized to form locally coherent groups amid more dispersed environmental sounds.The graph uses compact representations of dominant embedding structure.Source: B. RecurGraph: Recurrence-Guided Seed… · pp. 3–4, Figure 2 · p. 4
• Graph propagation assigns scores to unassigned clips using positive and negative seeds, and high-scoring clips are used for separator training without prompts or manual annotations.Source: B. RecurGraph: Recurrence-Guided Seed… · p. 4, Figure 2 · p. 4

C. Open-Set Ego-Noise Separation

• Open-set ego-noise separation defines the removal target as robot-specific ego-noise, while leaving the environmental sounds to preserve unrestricted by predefined classes.The separator is defined by what should be removed rather than what should be preserved.Source: C. Open-Set Ego-Noise Separation · pp. 4–5
• Selected ego-noise-dominant clips are mixed with diverse environmental sounds, which serve as targets for paired mixture–target supervision.A large-scale sound-event dataset approximates the broad environmental-sound distribution and supplies diverse temporal structures, spectra, and source types.Source: C. Open-Set Ego-Noise Separation · p. 5
• Minibatch augmentation varies the SNR, applies room impulse responses, and perturbs ego-noise playback speed by a few percent.These augmentations expose the separator to differing relative signal levels, reverberation, and slight gait-related pitch and rhythm changes.Source: C. Open-Set Ego-Noise Separation · p. 5

D. Transfer-DiT: Pretrained-Model Transfer for High-Fidelity Separation

• Transfer-DiT adapts a general-purpose zero-shot separator’s pretrained DiT backbone to robot-specific ego-noise separation using conditional Flow Matching.The model learns a velocity field that transforms a Gaussian latent sequence into an environmental-sound latent sequence.Source: D. Transfer-DiT: Pretrained-Model… · p. 5
• A frozen VAE encodes the mixture into a latent sequence, and the decoder reconstructs environmental sound after Transfer-DiT-guided numerical integration.The VAE fixes the latent sequences’ temporal sampling rate at 25 Hz.Source: D. Transfer-DiT: Pretrained-Model… · p. 5
• Transfer-DiT removes text and cross-attention target-conditioning modules, while trainable BAL and LoRA adapt the fixed pretrained backbone.BAL learns robot-specific residual transformations, and LoRA updates selected linear projections.Source: D. Transfer-DiT: Pretrained-Model… · pp. 5–6, Figure 4 · p. 6
• Training interpolates Gaussian noise and environmental-sound latents at a sampled flow time and minimizes conditional Flow-Matching loss against the target velocity.The target velocity is the difference between the environmental-sound latent and the sampled noise; the loss expectation is over paired training latents and sampled noise and time.Source: D. Transfer-DiT: Pretrained-Model… · p. 6

A. Experimental Design Overview

• The protocol evaluated RecurGraph clip selection, adapted a pretrained separator, and then tested separation on reference-audio mixtures without further parameter updates.Adaptation used each robot’s unlabeled pre-deployment clips as its only robot-specific data; ego-noise-only labels were reserved for evaluating clip selection.Source: A. Experimental Design Overview · p. 6
• Reference-audio evaluation used ego-noise recorded at sites disjoint from adaptation sites, with specified SNRs and isolated environmental-sound references.The controlled protocol supported consistent comparisons of separation quality and downstream Environmental Sound Classification performance.Source: A. Experimental Design Overview · p. 6
• A complementary physical playback evaluation tested separator behavior through a loudspeaker–room–microphone chain while the robot walked.Source: A. Experimental Design Overview · p. 6

B. Robots and Recording Conditions

• Experiments used Unitree G1 bipedal and Go1 quadrupedal robots, each carrying a microphone, recorder, and shock mount.The shock mount reduced direct transmission of structure-borne vibrations to the microphone.Source: B. Robots and Recording Conditions · p. 6, Figure 6 · p. 7
• Four sites within one building and its surroundings spanned five floor types; plaza and atrium recordings supported adaptation, while lobby and corridor recordings were held out.The site-disjoint evaluation combined shifts in room acoustics, floor conditions, and gait within the target operating domain.Source: B. Robots and Recording Conditions · pp. 6–7

C. Adaptation Data and Ego-Noise Clip-Selection Evaluation

• Each robot’s 256 five-second adaptation clips were half ego-noise-only and half mixed with FSD50K sounds at SNRs from −10.0 to 10.0 dB.RecurGraph received neither ego-noise-only labels nor isolated source signals; controlled tests varied the ego-noise-only ratio and recurring environmental-class ratio over 20 seeds.Source: C. Adaptation Data and Ego-Noise… · p. 7

D. Separator Training

• Selected ego-noise clips were mixed with FSD50K training sounds at randomly sampled −12 to 12 dB SNRs, with 20% of pairs held out for validation.The checkpoint with the lowest validation Flow-Matching loss was evaluated.Source: D. Separator Training · p. 7
• Supervised neural and dictionary-based baselines used the same selected clips; NMF dictionary rank was chosen by held-out output SNR.SB-INMF adapted an environmental-sound dictionary per mixture, while Harmonic NMF used an oracle ego-noise fundamental-frequency trajectory.Source: D. Separator Training · p. 7
• Transfer learning initialized DiT from a pretrained backbone and optimized BAL and LoRA, whereas the scratch model was randomly initialized and fully optimized.Both were trained for at most 200 epochs and evaluated on reference-audio mixtures at 0 dB.Source: D. Separator Training · pp. 7–8

E. Reference-Audio Evaluation

• Reference-audio evaluation used 256 base sounds per robot, with 26 anomalous and 230 normal sounds, mixed at five SNRs from −10 to 10 dB.The same sounds, ego-noise segments, and offsets were reused across SNR conditions so only the mixing ratio changed.Source: E. Reference-Audio Evaluation · p. 8
• Transfer-DiT was compared with zero-shot, supervised neural, and dictionary-based separators, as well as unprocessed mixtures.The listed comparison methods were CLAPSep and SAM-Audio; Conv-TasNet, Dual-Path RNN, and Conformer; and SB-INMF and Harmonic NMF.Source: E. Reference-Audio Evaluation · p. 8
• Evaluation measured semantic similarity, perceptual separation quality, anomalous-sound discrimination, and 52-class accuracy.The anomaly score averaged similarities within normal and anomalous class sets before computing their normalized ratio; the same scoring rule was used for every separator and unprocessed mixtures.Source: E. Reference-Audio Evaluation · p. 8
• Fig. 7 depicts environmental sounds played through a loudspeaker and recorded with ego-noise from a walking quadrupedal robot.This is a physical playback setup rather than the isolated-reference condition used in the reference-audio evaluation.Source: Figure 7 · p. 8

F. Physical Playback Evaluation

• The physical playback mixture was recorded through a loudspeaker–room–microphone chain at a site excluded from training-mixture construction, with no further robot-specific adaptation.This complements controlled reference-audio evaluation by physically realizing source paths and recording conditions.Source: F. Physical Playback Evaluation · p. 8
• Fourteen playback clips contained six anomalous and eight normal events, with robot speed, position, and heading varying across clips.An operator manually controlled the walking quadruped throughout recording.Source: F. Physical Playback Evaluation · p. 8
• Because playback recordings contained no isolated environmental reference, evaluation used CLAPT and reference-free SAJ rather than reference-based CLAPA.Harmonic NMF was inapplicable because its oracle fundamental-frequency trajectory required isolated ego-noise.Source: F. Physical Playback Evaluation · pp. 8–9
• Fig. 8 plots mean AUPRCego and standard deviations over 20 seeds as the dominant environmental-class ratio varies at fixed ego-noise-only ratios.It compares RecurGraph against direct PE AV similarity using embedding-centroid and prompt-reference conditions.Source: Figure 8 · p. 9

G. Implementation Details

• RecurGraph used a 0.1 seed ratio, 128-dimensional PCA embeddings, a 64-nearest-neighbor graph, and a selection threshold above 0.9.It used no text prompt; prompt-conditioned baselines used “mechanical footsteps” for the biped and “rapid mechanical footsteps” for the quadruped.Source: G. Implementation Details · p. 9
• Transfer-DiT trained for 20 epochs with BAL and LoRA ranks of 16, while inference solved the ODE with 16 midpoint steps.Training used cosine learning-rate annealing, random playback-speed perturbation of −2.5% to +2.5%, and RIR augmentation with probability 0.25.Source: G. Implementation Details · p. 9

A. Ego-Noise Clip Selection

• At ego-noise-only ratio 0.25, increasing dominant environmental-class prevalence reduced centroid AUPRCego, but RecurGraph remained more robust than direct PE AV similarity.Embedding-centroid variants approached ceiling performance as the ego-noise-only ratio increased; the centroid supplies recording-specific initial scores, and graph propagation improves classification beyond reference similarity alone.Source: A. Ego-Noise Clip Selection · pp. 9–10
• RecurGraph requires a sufficiently large subset of ego-noise-dominant clips, and the needed proportion varies with the prevalence of recurring environmental classes.The paper expects recordings across diverse times, locations, and walking trajectories to reduce recurring-class prevalence and increase available ego-noise-dominant clips.Source: A. Ego-Noise Clip Selection · p. 10

B. Ego-Noise Separation and ESC

• Without processing, AUPRCanom declined particularly at low SNR and CLAPA was especially low for the quadrupedal robot; Transfer-DiT improved quadrupedal AUPRCanom in all conditions.Its CLAPA improvement over no processing was larger for the quadrupedal robot at every SNR level.Source: B. Ego-Noise Separation and ESC · p. 10
• Zero-shot separators had limited, unstable effects, while RecurGraph-supervised neural separators varied by architecture; Transfer-DiT remained best or competitive across robot and SNR conditions.Conv-TasNet and Conformer generally improved separation quality, whereas Dual-Path RNN often reduced CLAPA and AUPRCanom relative to no processing.Source: B. Ego-Noise Separation and ESC · pp. 10–11
• For both robots, transfer learning stopped substantially earlier than training from scratch and achieved considerably higher CLAPA, SAJ, Acc52, and AUPRCanom values.Table I compares initialization and adaptation at 0 dB.Source: B. Ego-Noise Separation and ESC · p. 10, Table I · p. 11
• Transfer-DiT achieved the best or competitive ESC performance across robot and SNR conditions, with improvement over no processing clearest at low SNR.The results tables summarize classification performance across evaluation conditions.Source: B. Ego-Noise Separation and ESC · p. 11

C. Physical Playback Evaluation

• The physical-playback ranking differed from Table II; recording process, source-specific acoustic paths, and possible site acoustics or floor conditions may have contributed.The physical recording was collected at a different site from the controlled reference-audio evaluation.Source: C. Physical Playback Evaluation · p. 11, Table II · p. 12
• Transfer-DiT tied for the highest CLAPT and AUPRCanom and achieved the highest SAJ and Acc52 in the physical playback evaluation.Table IV reports separation and classification performance for a physical playback recording with the walking quadrupedal robot.Source: C. Physical Playback Evaluation · pp. 11–12, Table IV · p. 12
• Future work proposes periodic offline adaptation using deployment recordings, while excluding clips associated with detected anomalous events and mitigating catastrophic forgetting.The paper recommends covering as many representative sites and routes as practical.Source: C. Physical Playback Evaluation · p. 12

• Experiments with bipedal and quadrupedal robots showed reliable ego-noise-dominant clip selection and improved separation quality and downstream classification performance over baseline separators.Source: §VI · p. 12
• RecurGraph automatically selects ego-noise-dominant clips from unlabeled pre-deployment recordings, which are mixed with diverse environmental sounds to provide paired mixture–target supervision.Source: §VI · p. 12
• Transfer-DiT adapts a pretrained general-purpose separator to remove robot-specific ego-noise with high fidelity.Source: §VI · p. 12