Source-linked AI summary
Cross-Modal Knowledge Distillation for Acoustic Pedestrian Detection
Yonghyun Kim, Chaeyeon Han, Sancho Gatungay, Subhrajit Guhathakurta, Alexander Lerch
TL;DR
Audio-only pedestrian detection is limited by weak acoustic cues, and the contribution of video-to-audio distillation under severe class imbalance and a wide modality gap is unclear. The paper introduces Trust-Filtered Distillation and interprets its logit formulation as conditional label smoothing under a shared temperature. Across ASPED configurations, modest macro-accuracy gains generally accompany shifts toward the majority class, with only small PR-AUC changes.
Problem
The contribution of video-to-audio distillation to audio-only pedestrian detection is unclear when acoustic cues are weak and the task has severe class imbalance and a wide modality gap.
Source: §1 · p. 1Method
The paper introduces Trust-Filtered Distillation, a ground-truth-conditioned gate that suppresses minority-class KD, and interprets its logit formulation as conditional label smoothing under a shared temperature.
Source: §1 · p. 1, §3.2 · p. 2Results
Across ten ASPED KD configurations, modest macro-accuracy gains often accompany lower pedestrian recall and little improvement in distinguishing pedestrian presence from absence.
Source: §1 · p. 1, §5.2 · p. 4Takeaways & Limitations
In this setting, improved probability calibration need not imply better detection, and operating-point shifts should be distinguished from discrimination gains.
Source: §1 · p. 1, §6 · p. 4The gate targets the observed 91:9 imbalance; its optimality at other class ratios and its causal role in the observed shift are not established.
Source: §5.4 · p. 4
Abstract
from arXiv · showhide
Audio-only pedestrian detection is attractive for urban sensing but limited by weak acoustic cues. An appealing strategy is cross-modal knowledge distillation, in which a video teacher supervises the audio student during training so that the deployed model runs on audio alone. Under this task's severe class imbalance and wide video-audio modality gap, however, what such distillation contributes is unclear. We introduce Trust-Filtered Distillation (TFD), which selectively suppresses teacher supervision on pedestrian samples, and interpret its logit formulation as conditional label smoothing under a shared temperature. Across ten distillation configurations under five-fold cross-session validation on ASPED, most methods yield modest gains in macro accuracy accompanied by small changes in PR-AUC. The main effect is higher no-pedestrian accuracy at the cost of lower pedestrian recall. Adding TFD to logit distillation strengthens this trade-off without improving mean PR-AUC. These findings clarify the benefits and limitations of selective cross-modal supervision by distinguishing operating-point shifts from discrimination gains in imbalanced acoustic detection.
1. INTRODUCTION
Audio-only pedestrian detection offers sensing where vision fails, but weak acoustic cues and urban noise limit its reliability. The paper studies video-to-audio distillation under class imbalance and a modality gap, finding modest accuracy gains that often coincide with lower pedestrian recall and little discrimination improvement.
1. INTRODUCTION
- Audio sensors can complement cameras in darkness, fog, or occlusion, but weak pedestrian sounds are easily masked by urban noise.Reliable pedestrian detection supports urban safety, autonomous driving, and smart-city planning.Source: §1 · p. 1
- The study examines video-to-audio distillation, using visual supervision during training while retaining audio-only inference.The video teacher observes the scene directly and provides a discriminative signal.Source: §1 · p. 1
- The setting combines severe class imbalance, including 8.5% pedestrian presence in ASPED v.a, with a wide video-audio modality gap.The paper asks how distillation affects discrimination and operating points, and whether selective supervision changes those effects.Source: §1 · p. 1
- Across ten ASPED distillation configurations, modest macro-accuracy gains often accompany lower pedestrian recall, with little improvement in distinguishing pedestrian presence from absence.The paper frames this as distinguishing accuracy gains from discrimination gains.Source: §1 · p. 1
- Trust-Filtered Distillation uses a ground-truth-conditioned gate to suppress minority-class KD and admits a conditional label-smoothing interpretation under a shared temperature.The paper also reports that improved probability calibration need not imply better detection in this setting.Source: §1 · p. 1
2. RELATED WORK
Prior work addresses acoustic pedestrian detection, cross-modal transfer, distillation objectives, and selective supervision. This paper focuses on video-to-audio pedestrian presence detection under severe class imbalance.
2. RELATED WORK
- Earlier acoustic pedestrian datasets pair audio with video and labels; reported approaches use weighted sampling and loss, while audio-only accuracy remains limited.Prior work reports faint pedestrian sounds and labels heavily skewed toward no-pedestrian.Source: §2 · p. 1
- Cross-modal methods include visual-to-audio supervision, sample selection by teacher-student agreement, and feature-domain alignment, while modality differences can limit transfer.Related detection work distills visual teachers for sound-only object detection and LiDAR teachers for audio-visual pedestrian awareness.Source: §2 · p. 1
- Distillation variants match logits, intermediate features, contrastive objectives, or sample relations; related work also links logit distillation to label smoothing and calibration.A cited study reports teacher calibration error correlating more strongly with student accuracy than teacher accuracy for image classification on CIFAR-100.Source: §2 · p. 1
- Selective-supervision methods use class-dimension masks, calibration criteria, ground-truth agreement, or trust scores, leaving systematic cross-modal detection under severe imbalance open.The stated gap combines sample-level selection, ground-truth guidance, and class-specific exemption.Source: §2 · pp. 1–2
3. METHOD
The method combines logit distillation with a per-sample mask that selectively removes KD supervision, then interprets logit-based TFD as conditional label smoothing under specified assumptions. It situates that gate among prior selective-distillation approaches.
3.1. KD Loss under Class Imbalance
- The logit KD loss matches softened student predictions to the teacher alongside ground-truth labels, and the total loss adds KD to cross-entropy with weight α.The sampler balances the two classes in expectation; the observed teacher mean probability on the correct class is lower for pedestrian than no-pedestrian samples, about 0.75 versus 0.85.Source: §3.1 · p. 2
3.2. Trust-Filtered Distillation
- TFD gates each sample's KD term, leaving only its cross-entropy term when the mask is zero, and suppresses selected minority-class samples.The default threshold is τ=0.4, is not tuned on the folds, and masking renormalizes the surviving KD weight; the gate never masks majority-class samples.Source: §3.2 · p. 2
3.3. TFD as Conditional Label Smoothing
- Under a shared temperature, logit-based TFD can be read as conditional, teacher-dependent label smoothing: masked samples retain hard labels and unmasked samples use a label-teacher mixture.The identity assumes ordinary unweighted cross-entropy and shared temperatures; experiments instead use task loss at T=1 and KD at T=3, and the Hybrid cosine term is not covered.Source: §3.3 · p. 2
3.4. Relation to Prior Selective KD
- Unlike prior selective methods, TFD combines a sample-level binary gate with an explicit ground-truth class condition and suppresses KD for minority samples above a teacher-probability threshold.The comparison concerns how methods select supervision and does not claim mathematical equivalence to the label-smoothing identity.Source: §3.4 · p. 2
4. EXPERIMENTAL SETUP
The experiments use ASPED’s five outdoor sessions for five-fold leave-one-session-out evaluation, comparing ten distillation configurations with an audio-only baseline. Metrics cover class-wise and threshold-free detection, calibration, and Brier-score decomposition.
Dataset
- ASPED contains five outdoor urban pedestrian sessions, with labels comprising 91.5% no-pedestrian and 8.5% pedestrian samples.A sample is labeled Ped when a detected person lies within 6 m of the recorder.Source: §4 · pp. 2–3
Models
- The audio student uses a frozen VGGish encoder, trainable Transformer, and binary head, while a frozen video teacher supplies soft logits during training only.The teacher is fitted per fold on the four training sessions; inference uses no teacher.Source: §4 · p. 3, Figure 1 · p. 3
Compared configurations
- Ten KD configurations span logit, imbalance-aware, feature-based, and relation-based methods, with LogitKDTFD and HybridTFD as the proposed gated variants.HybridTFD adds the KDCos cosine term to the logit loss; configurations share the optimizer, schedule, and student architecture.Source: §4 · p. 3
Metrics
- Evaluation reports 0.5-threshold macro accuracy and F1Ped alongside PR-AUC and calibration metrics, separating operating-point outcomes from ranking and calibration.The Brier score is decomposed into reliability and resolution using ten equal-width bins; resolution is treated as corroborating PR-AUC.Source: §4 · p. 3
Experimental protocol
- Five-fold leave-one-session-out evaluation on ASPED uses four training sessions per fold and reports five-fold means, with per-fold consistency and effect sizes emphasized.The held-out session is scored separately; a paired t-test is supplementary because the exact sign-flip test cannot reach p<0.05 with five folds.Source: §4 · p. 3
5. RESULTS AND ANALYSIS
Across ten distillation configurations, macro-accuracy gains generally accompany a shift toward the majority class, while PR-AUC changes little. TFD strengthens that operating-point trade-off, but neither its advantage over ungated KD nor large gains in student ranking are established.
5. RESULTS AND ANALYSIS
- Across the main cross-session comparison, no configuration gained more than one macro-accuracy point, and fold-to-fold standard deviation exceeded those gains.The Table 1 paired t-test results did not survive Bonferroni correction across ten tests.Source: §5 · p. 3, Table 1 · p. 4
5.1. Cross-Session Method Comparison (Experiment 1)
- PR-AUC ranged from 0.324 to 0.333 across configurations, against a baseline of 0.328; LogitKD had the largest mean gain, +0.004.Its descriptive 95% bootstrap CI was [0.002, 0.008], while the intervals for LogitKDTFD and HybridTFD included zero.Source: §5.1 · p. 4
- TFD configurations had the largest standardized paired macro-accuracy effects against baseline, but LogitKD with and without TFD showed no significant difference in macro accuracy or F1Ped.The comparison does not isolate sample selection from effective KD weight because the gate renormalizes the surviving KD term.Source: §5.1 · p. 4
5.2. Per-Class Operating-Point Shift (Experiment 1)
- KD raised AccNo-Ped and lowered AccPed (LogitKD: 63.5→70.8 and 82.3→76.4), trading pedestrian recall for majority-class accuracy.The direction held across configurations except LogitKDAdj, which over-corrected toward Ped; most of the F1Ped rise was reproduced by baseline thresholding at the same false-positive rate.Source: §5.2 · p. 4
5.3. Calibration vs. Probability Ranking (Experiment 1)
- Calibration metrics improved mainly through reduced over-prediction, while PR-AUC changed little and resolution remained 0.011.Positive-class ECE changed from 0.366 to 0.316 and Brier reliability from 0.163 to 0.110; mean predicted pedestrian probability fell from 0.45 to 0.40.Source: §5.3 · p. 4
5.4. Teacher Comparison (Experiment 2)
- The sampler balanced classes in expectation, while TFD's gate targeted the observed 91:9 imbalance; its optimality at other class ratios and causal role in the shift remain unestablished.No random-mask control was run.Source: §5.4 · p. 4
- The video teacher reached 89.9% LOSO macro accuracy and 53% F1Ped, versus 72.9% and 28.6% for the student, yet student PR-AUC changed only slightly.The single pipeline does not isolate the modality gap from teacher calibration, student capacity, or optimization effects.Source: §5.4 · p. 4
6. CONCLUSION
The conclusion finds modest macro-accuracy gains alongside majority-class operating-point shifts and small PR-AUC changes across ten ASPED configurations. It presents TFD as selective supervision, while bounding the evidence to one dataset and noting unresolved effects of selectivity and imbalance severity.
6. CONCLUSION
- Across ten KD configurations on ASPED, modest macro-accuracy gains generally accompanied a majority-class operating-point shift and only small PR-AUC changes.Source: §6 · p. 4
- TFD uses a ground-truth-conditioned gate to suppress minority-class supervision, and its logit formulation admits a conditional label-smoothing interpretation under a shared temperature.Source: §6 · p. 4
- Evidence is limited to ASPED, one campus, five folds, binary detection, and single runs without multiple-comparison correction; the gate's selectivity and performance at other imbalance severities remain unresolved.The conclusion identifies these as matters for future work alongside closing the modality gap.Source: §6 · p. 4