Source-linked AI summary
Sounding Surgical Instrument Localisation through Audio-Visual Correspondence
Dominic Behling, Jonas Hein, Matthias Seibold
TL;DR
Surgical sound localization has relied on microphone arrays, while audio-visual correspondence had not been investigated for surgery. This paper adapts weakly supervised localization to single-camera, mono-audio surgical recordings, training without spatial annotations. It reports localization on an independent corpus recorded with different cameras and microphones, as well as separation of tool activity from background noise and correct attribution when two tools are present.
Problem
Surgical sound-localization methods use microphone-array beamforming, while it was unknown whether learned audio-visual correspondence could localize instruments in surgery.
Source: §1 · pp. 2–3Method
The model learns from single-camera images and mono audio, extending weakly supervised AV-SSL with silence and composite-frame training terms and no spatial supervision.
Source: §1 · p. 3, §3.2 · pp. 6–7Results
The model localizes the sounding instrument on an independent corpus recorded with different cameras and microphones, separates tool activity from background noise, and attributes sound correctly when two tools are present.
Source: §4 · p. 8, §5 · pp. 9–10Takeaways & Limitations
Audio-visual correspondence learned from in-domain recordings supports surgical tool detection and localization using commodity hardware without localization supervision.
Source: §6 · p. 10The implementation assumes the sounding instrument is in view and does not handle visual occlusions; audible but invisible instruments are not addressed.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Surgical scene understanding is a prerequisite for context-aware and intelligent systems in the operating room. Current methods rely mostly on visual data, while acoustic sensing has shown potential to provide complementary information on surgical activity. First approaches for the localisation of surgical activity relied on microphone arrays and acoustic beamforming. This work investigates whether the active instrument can be localized from a single microphone and a camera by learning the correspondence between the appearance and the sound of surgical instruments, without annotating its position. We adapt weakly supervised audio-visual localisation, which learns from the co-occurrence of image and sound, to the surgical setting. We record simulated surgical actions, sawing, drilling, and chiseling, and train the model on these recordings without any spatial annotation to produce a heatmap over the sounding tool. We extend the training objective by two terms: a silence term, which suppresses the output for silence and background noise, and a composite term, which trains the model to select the sounding one of two visible instruments. The model localizes the active instrument on an independent corpus recorded with a different camera and microphone, in which instrument positions were obtained by optical tracking. It separates tool activity from silence and background noise and can attribute sound correctly when two tools are present. Audio-visual correspondence learned from in-domain recordings connects the sound of a surgical action to the instrument that produces it, using a single microphone and a camera. This proof of concept introduces a novel paradigm that enables the detection and localisation of surgical activity without manual annotations.
1 Introduction
Surgical scene understanding matters for intelligent operating-room systems, but current approaches rely largely on vision and do not locate where acoustically detected activity occurs. This work investigates weakly supervised audio-visual correspondence as a way to detect and localize active surgical instruments without microphone arrays or spatial annotations.
1 Introduction
- Surgical scene understanding supports the next generation of operating-room systems, building on work in phase recognition and semantic scene graphs.Computer-assisted surgery aims to improve procedural accuracy, safety, and reproducibility.Source: §1 · p. 2
- Beamforming-based surgical localization uses bulky, costly microphone arrays, can produce reflection artefacts, and separates detection from localization.It can detect only part of an instrument, such as a drill engine or hammer impact point.Source: §1 · p. 2
- AV-SSL has not been investigated in surgery, where visible-but-silent instruments and similar-looking tools complicate localization.The approach could use a single camera and microphone and jointly detect and localize the full sounding object, rather than only its physical emission point.Source: §1 · p. 3
- The paper investigates learned audio-visual correspondence for surgical activity and introduces a simulated-surgery dataset with sequence-level labels.It also presents a proof of concept for localization without microphone arrays and adds loss terms for silence and multi-tool scenes.Source: §1 · p. 3
2 Related Work
Prior surgical sound-localization work uses microphone-array beamforming, while audio-visual sound source localization learns which image regions produce sound. Whether that learned correspondence transfers to surgery remains unclear.
2 Related Work
- Surgical beamforming has localized sound with acoustic cameras and dynamic RGB-D representations, but relies on phased arrays and can be costly, slow, artefact-prone, and limited to the physical emission point.Prior work also treats detection and localization as separate stages.Source: §2 · p. 3
- General-domain AV-SSL predicts which image regions emit sound from unlabelled or weakly labelled video, but its transfer to surgery is uncertain.The surgical setting includes visible inactive instruments, similar-looking tools, and substantial background noise.Source: §2 · pp. 3–4
3 Methodology
The method learns to localize the sounding instrument from a single RGB frame and mono audio, using sequence-level instrument labels rather than spatial supervision. Its training objective addresses negative audio and scenes with multiple visible tools.
3.1 Data
- The action-camera corpus supplies training and development data, while an independently recorded acoustic-camera corpus is used for localization testing.Both contain simulated drilling, sawing, and chiseling in the same operating room; the acoustic-camera corpus provides optically tracked instrument ground truth for testing only.Source: §3.1 · pp. 4–5
- The action-camera corpus contains 54 roughly one-minute sequences from three operators, with static and handheld views and three two-tool test sequences.Bounding boxes are annotated only on validation and test sequences, and the data-split caption specifies their use for model selection and evaluation.Source: §3.1 · p. 5, Table 1 · p. 5
3 Methodology
- The model takes one RGB frame and a 3-second mono-audio clip, then outputs a heatmap for the sounding tool or a flat map for silence or background noise.Training requires sequence-level class labels, not spatial supervision.Source: §3 · p. 4, §3.2 · p. 5
3.2 Model
- The model adapts SLAVC with a higher-resolution visual branch: 448 × 448 input, DINOv2 ViT-S/14 features, and a pooled 16 × 16 grid.The 7 × 7 SLAVC grid was considered too coarse for small instruments; the audio encoder and projection layers are retained.Source: §3.2 · p. 6
- The silence term penalizes nonzero maps for silence and background noise, while the area term drives otherwise unconstrained similarities toward zero and sets map magnitude.The silence term alone did not separate silence from tool activity; together the terms keep negative-audio maps near zero while retaining high magnitude for tool audio.Source: §3.2 · p. 6
- Composite frames place crops showing different instruments side by side or vertically, while retaining one crop’s audio to train selection of the sounding tool.This addresses the single-instrument setting, where the model could otherwise highlight any visible instrument regardless of the audio.Source: §3.2 · p. 7
4 Experiments
The experiments evaluate localisation on an independently recorded acoustic-camera corpus, alongside activity detection and two-tool selection. Results show complete separation of tool activity from background noise and strong selection performance; ablations identify which loss terms support detection and selection.
4 Experiments
- Localisation is evaluated on 853 annotated acoustic-camera frames using IoU and median heatmap-maximum distance to the ground-truth center.The distance is measured in pixels at 224 × 224 resolution; Table 2 reports results per instrument and overall.Source: §4 · p. 7, Table 2 · p. 8
- Qualitative heatmaps show responses to the hammer-swinging hand during chiseling and to an assistant’s empty hand during sawing.Figure 2 also shows the ground-truth box and drill, chisel, and saw examples.Source: Figure 2 · p. 8
- 1.00 AUC separates tool activity from background noise for all three seeds and instruments.AUC ranks activity scores and does not depend on a threshold.Source: §4 · pp. 7–8
- With each frame’s own audio, the active instrument is correctly recovered in 93±5% of frames; with audio from the idle instrument’s class, inactivity is detected in 87±10% of samples (mean and standard deviation over three seeds).Source: §4 · p. 8
- Ablations indicate that silence and area terms are required for detection, while the composite term is required for tool selection.The ResNet-18 visual encoder has similar IoU but is weaker at selecting between two instruments.Source: §4 · p. 8, Table 3 · p. 9
- The two-tool visualization contrasts background noise, matched saw audio, and swapped chisel audio, marking the active saw green and idle chisel grey.Heatmaps are shown as deviations from their means on one common colour scale.Source: Figure 3 · p. 9
5 Discussion
The discussion reports cross-corpus localization from inexpensive weakly labeled recordings while identifying limits in action coverage, recording conditions, and model design.
5 Discussion
- Trained on about an hour of weakly labeled action-camera video, the model localizes on an independent corpus recorded with different cameras and microphones.The authors report that general-domain pretrained baselines do not transfer to the operating-room setting, while the training data is cheap to record.Source: §5 · p. 9
- Saw and drill localization reaches a median distance of about 10 px from the ground-truth box center, while chisel results are partly limited by the tracked target.The heatmap responds to the hammer-swinging hand and pose, but the ground-truth box covers only the untracked hammer's chisel.Source: §5 · p. 9
- On this limited dataset, the composite loss is required to condition the model on audio rather than let single-instrument frames drive its heatmap.The two-instrument selection test demonstrates that the model makes its selection based on audio.Source: §5 · p. 10
- Training covers only three exemplary orthopedic actions, and robustness to speech, monitoring beeps, quieter surgical sounds, and other procedures remains untested.The recordings use one operating room and contain no other operating-room sounds; extension beyond orthopedics requires investigation.Source: §5 · p. 10
- The 16 × 16 heatmap localizes a region over 3 s, and composite training relies on active instruments being in the central half of the image.The authors identify the output's coarse spatial and temporal resolution as a limitation.Source: §5 · p. 10
- The current implementation assumes the sounding instrument is visible and does not handle visual occlusions.Empty hands can also trigger a heatmap response; audible instruments outside the image are not addressed.Source: §5 · p. 10
6 Conclusion
The conclusion presents a proof of concept for detecting sounding surgical objects through audio-visual correspondence.
6 Conclusion
- The proof of concept localizes and disambiguates surgical tools using commodity-hardware recordings without localization supervision.The authors position it as enabling detection and localization of surgical activity for advanced multimodal scene understanding.Source: §6 · p. 10