PaperlayerView exact revision on arXiv

Source-linked AI summary

EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments

Jacob Donley, Vladimir Tourbabin, Jung-Suk Lee, Mark Broyles, Hao Jiang, Jie Shen, Maja Pantic, Vamsi Krishna Ithapu, Ravish Mehra

arXiv:2107.04174v2cs.SDcs.CVcs.LGeess.ASeess.SP

TL;DR

EasyCom addresses a gap in realistic data for research on improving conversations in noisy environments. The paper releases synchronized multimodal egocentric recordings and evaluates a targeted-speech enhancement baseline, which improves most reported metrics over the reference microphone.

  • Problem

    The cocktail-party problem motivates improving conversations in noisy environments, and existing datasets lacked the multimodal egocentric noisy recordings described as needed to close the dataset gap.

    Source: §II · p. 2, §VII · p. 8

  • Method

    The authors release more than five hours of synchronized multimodal egocentric noisy recordings with annotations, define a benchmark task, and evaluate a real-time baseline using acoustic and positional information.

    Source: §VII · p. 8

  • Results

    The Baseline Method improves all metrics over Reference Mic except PESQ on Noise + Interferer (1.172 vs. 1.168), with average gains of 1.2–3.7 dB in SNR-based metrics, 0.05–0.11 in most intelligibility metrics, 32–36 bits in SIIB, and 0.03–0.10 in quality metrics.

    Source: §VI · p. 7

  • Takeaways & Limitations

    The authors open-source the dataset and its annotated labels to facilitate research on multimodal AR solutions to the cocktail-party problem.

    Source: §VII · p. 8

    Evaluation uses close-microphone signals as minimally noisy clean references, but these are not identical to the true clean speech components captured by the AR glasses microphones.

    Source: §VI · p. 7

Abstract

from arXiv · show

Augmented Reality (AR) as a platform has the potential to facilitate the reduction of the cocktail party effect. Future AR headsets could potentially leverage information from an array of sensors spanning many different modalities. Training and testing signal processing and machine learning algorithms on tasks such as beam-forming and speech enhancement require high quality representative data. To the best of the author's knowledge, as of publication there are no available datasets that contain synchronized egocentric multi-channel audio and video with dynamic movement and conversations in a noisy environment. In this work, we describe, evaluate and release a dataset that contains over 5 hours of multi-modal data useful for training and testing algorithms for the application of improving conversations for an AR glasses wearer. We provide speech intelligibility, quality and signal-to-noise ratio improvement results for a baseline method and show improvements across all tested metrics. The dataset we are releasing contains AR glasses egocentric multi-channel microphone array audio, wide field-of-view RGB video, speech source pose, headset microphone audio, annotated voice activity, speech transcriptions, head bounding boxes, target of speech and source identification labels. We have created and are releasing this dataset to facilitate research in multi-modal AR solutions to the cocktail party problem.

I. INTRODUCTION

EasyCom addresses the shortage of realistic egocentric, multimodal recordings for cocktail-party research by releasing a dataset and defining a target-speech enhancement task. Its baseline beamformer improves the three benchmark measures with minimal target-speech distortion.

I. INTRODUCTION

  • Existing datasets lack a combination of egocentric perspective, multimodal data, and noisy conversations representative of the cocktail-party problem.Prior resources are missing video, noise, annotations, head-mounted recordings, or rigid wearable microphone arrays in different combinations.Source: §I · p. 1
  • EasyCom releases egocentric multichannel audio, video, positional data, voice activity, speech transcriptions, and face bounding boxes recorded in a noisy restaurant-like setting.The modalities support tasks including audio-visual speech enhancement, while audio and annotations provide ground truth for intrusive metrics.Source: §I · pp. 1–2

II. CONVERSATIONAL FOCUS TASK

  • The conversational-focus task targets enhancement of a chosen speaker while improving SNR, speech quality, and intelligibility without distorting the target speech.Practical solutions should handle sensor movement, operate causally and in real time with minimal latency, and suit conversations in noisy social settings.Source: §I · p. 2, §II · p. 2

I. INTRODUCTION

  • The diffuse-noise-optimized beamformer improves signal-to-noise ratio, speech quality, and intelligibility with minimal target-speech distortion.It dynamically steers toward the desired speech source, works across dataset wearers, and runs in real time.Source: §I · p. 2

III. THE EASYCOM DATASET

EasyCom records natural conversations in a noisy restaurant-like room, with wearable audio, video, and participant tracking. Its sessions include multiple participants and more than five hours of recordings.

III. THE EASYCOM DATASET

  • Sessions involved three to five participants around a table, with one wearing AR glasses that recorded multichannel audio and video.The microphone array included two binaural microphones and four additional microphones distributed around the glasses.Source: §III · p. 2
  • The dataset contains 12 sessions totaling approximately 5 h and 18 min of conversational recordings, plus three extra sessions with minor errors.Sessions ran in a roughly 6 m × 7 m × 3 m room with RT60 of 645 ms; ten loudspeakers played restaurant-like noise at approximately 71 dB SPL.Source: §III · p. 2
  • Wide-angle video was recorded at 1920 × 1080 and 20 frames per second, while participant head poses and headset microphone audio were also recorded.The camera had an approximate horizontal field of view of 120°; headset microphones recorded at 48 kHz and 32-bit depth.Source: §III · p. 3

A. Methods and Procedure

The recording procedure staged improvised conversations and interaction tasks in a restaurant-like noisy room. The dataset also includes human annotations and calibration data for using its modalities together.

A. Methods and Procedure

  • The team prepared a restaurant-like room with acoustic panels, optical tracking, a table, and a separate recording area before each session.Participants were assigned seats, fitted with recording or tracking equipment, and exposed to gradually increased restaurant-like noise before recording began.Source: A. Methods and Procedure · p. 3
  • Participants improvised introductions, menu ordering, puzzle solving, and an “I spy” game, followed by reading command-like sentences for speech-recognition tasks.The activities were designed to sustain conversation and, for “I spy,” simultaneous discussion and head movement.Source: A. Methods and Procedure · pp. 3–4
  • The dataset includes calibration data for inter- and intra-modality relationships.The data supports acoustic-array use, links acoustic and visual modalities, reduces optical distortions, and refines tracking estimates.Source: A. Methods and Procedure · p. 4

B. Annotated Labels

  • Human annotators labeled voice activity, transcribed speech, and labeled target of speech; face and head detection models generated bounding boxes.Annotators used audio mixed from the glasses microphones and participant headset microphones to reduce background noise during labeling.Source: B. Annotated Labels · p. 4

1) Voice Activity:

The annotation process combines human labeling of activity, transcripts, and speech targets with automated face and head detection followed by targeted manual review. These procedures provide time-aligned labels and participant identities across recordings.

1) Voice Activity:

  • Voice activity covers any participant utterance, including speech, laughter, coughing, and hesitations, labeled by time segment and participant ID.Annotators labeled participants sequentially, using millisecond-precision time segmentation.Source: 1) Voice Activity · p. 4

2) Speech Transcriptions and Target of Speech:

  • Transcription and target-of-speech labels extend the voice-activity segments, with labels for individual, group-targeted, non-targeted, and secondary speech targets.Transcription conventions include punctuation and markers for laughter, coughing, throat clearing, unintelligible speech, hesitations, and interruptions; annotators reviewed one another’s work for consistency.Source: 2) Speech Transcriptions and Target of… · pp. 4–5

3) Face Bounding Boxes:

  • Face IDs are assigned by matching detected faces to tracked glasses, then grouping detections into tracklets for manual review.Tracklet-level review reduces workload by a factor of 15 compared with frame-level annotation; a third rater resolved disagreements.Source: 3) Face Bounding Boxes · p. 5
  • Head tracking extends candidate trajectories to match detections, using head-box positions, optical-flow homographies, and appearance features.The method uses a homography to compensate for large egocentric camera motion.Source: 3) Face Bounding Boxes · p. 5

4) Head Bounding Boxes:

The method detects head candidates and associates them over time to form trajectories, handling missed detections and false alarms. Trajectories receive IDs through face detections, and Fig. 4 shows labels and tracked poses projected into image space.

4) Head Bounding Boxes:

  • Head tracking detects candidate heads and associates detections through time to form person trajectories, despite missed detections and false alarms.For each video, trajectory IDs are automatically assigned using corresponding face IDs.Source: 4) Head Bounding Boxes · p. 5
  • The assignment optimization uses binary variables to indicate trajectory-to-candidate matches and a threshold to avoid the trivial all-zero solution.Matching uses a primal-dual method.Source: 4) Head Bounding Boxes · p. 5
  • Matching cost combines motion-compensated bounding-box position with a deep feature that distinguishes head appearances.The method estimates a homography between consecutive frames using optical flow; the appearance feature comes from a modified ResNet-18 trained on VoxCeleb2.Source: 4) Head Bounding Boxes · p. 5
  • Fig. 4 shows dataset annotations and labels in an example frame, with tracked poses projected into image space.Source: Figure 4 · p. 6
  • When a trajectory has no matched detection, it is extended at a homography-predicted position while its remaining life decreases; matched tracks have their life restored.The example maximum life is 20; trajectories are removed when life falls below zero.Source: 4) Head Bounding Boxes · pp. 5–6
  • Only trajectories longer than a threshold, such as five frames, are output to filter unstable false-alarm detections.Source: 4) Head Bounding Boxes · p. 6
  • Each trajectory receives the face ID with the majority vote among face detections matched to its head boxes by maximum IoU.Each head detection in the trajectory is then labeled with that trajectory ID.Source: 4) Head Bounding Boxes · p. 6

IV. DATASET AVAILABILITY

The dataset is downloadable and comprises 323 one-minute segments totaling 5 hours and 18 minutes. Its files are organized by data type, with compressed video available at the provided link.

IV. DATASET AVAILABILITY

  • 323 one-minute segments provide 5 hours and 18 minutes of data in a dataset approximately 79 GB in size.Source: §IV · p. 6
  • The dataset contains 3,245 files, organized into separate directories for each data type.Source: §IV · p. 6
  • The video at the provided download link is compressed; uncompressed video details are available by contacting the authors.Sampling rates and the dataset's exact structure are described in the root directory.Source: §IV · p. 6

V. BASELINE METHOD

The baseline uses a maximum directivity index beamformer to produce distortionless enhanced speech from the glasses’ microphone array. It updates the target transfer-function estimate as participants move and filters audio frame by frame.

V. BASELINE METHOD

  • The maximum directivity index beamformer maximizes directivity while keeping sound arriving from the target direction undistorted.The method is analyzed as a baseline for producing enhanced speech from noisy acoustic-array signals.Source: §V · p. 6
  • The beamformer coefficient vector depends on the target acoustic transfer functions and the multichannel covariance matrix for a spherically isotropic unit-power noise field.The distortionless constraint is set by g; choosing a target-to-microphone transfer function for g retrieves that microphone’s target speech without distortion.Source: §V · p. 6
  • The covariance matrix can be approximated from the dataset’s array transfer functions, while the desired participant’s transfer function is updated as relative positions change.OptiTrack positions determine the relative angle used to select the closest measured array transfer function.Source: §V · pp. 6–7
  • At every frame, the method updates beamformer coefficients and filters multichannel audio using weighted overlap-add.Source: §V · p. 7

VI. BASELINE RESULTS AND DISCUSSION

The baseline improves nearly all evaluated metrics over the reference microphone signal, with one PESQ exception. Results use two test cases and a minimally noisy close-microphone reference that is not identical to the true clean speech component.

VI. BASELINE RESULTS AND DISCUSSION

  • Evaluation uses 11 intrusive objective metrics spanning SNR, speech intelligibility, and speech quality.The metrics include SNR variants, STOI, ESTOI, HASPI, SIIB, PESQ, HASQI, and ViSQOL.Source: §VI · p. 7
  • Noise tests exclude competing talkers, while Noise + Interferer includes active target speech with or without a competing talker; both always include noise.Close-microphone signals serve as clean references but are not identical to the true clean speech component, so metric values may not be meaningful alone.Source: §VI · p. 7
  • Average gains are 1.2–3.7 dB in SNR-based metrics, 0.05–0.11 in most intelligibility metrics, 32–36 bits in SIIB, and 0.03–0.10 in quality metrics.Source: §VI · p. 7
  • The dataset conditions make improvement over the reference microphone difficult, while other available signals could outperform the reported baseline.Source: §VI · pp. 7–8
  • Across test cases, the baseline improves every metric over Reference Mic except PESQ for Noise + Interferer (1.172 vs. 1.168).The authors suggest interfering speech in the metric reference likely explains the PESQ exception.Source: §VI · p. 7, Table III · p. 8

VII. CONCLUSIONS

The authors release more than five hours of synchronized, noisy, egocentric multimodal recordings with annotations. They propose a benchmark and baseline for targeted speech enhancement, and call for further work to outperform the baseline.

VII. CONCLUSIONS

  • The released dataset contains more than five hours of synchronized multimodal egocentric recordings of natural conversations in noisy settings, with annotations across modalities.Source: §VII · p. 8
  • The authors propose a dataset benchmarking task and analyze a baseline that leverages acoustic and positional information for real-time targeted speech enhancement.The baseline can instantaneously switch between target speech sources.Source: §VII · p. 8
  • The baseline yields consistent improvements in speech quality, intelligibility, and SNR-based metrics without distorting target speech.Source: §VII · p. 8
  • The authors challenge the research community to outperform the baseline using the dataset’s benchmarking task.Source: §VII · p. 8

APPENDIX A POST-PROCESSING

Appendix A describes post-processing intended to reduce dataset size, align clean speech references, and improve annotation accuracy.

APPENDIX A POST-PROCESSING

  • Post-processing reduced dataset size, aligned clean speech reference signals, and facilitated more accurate annotations.Source: Appendix A · p. 8

A. Video Compression

  • Video compression reduced raw uncompressed files, several terabytes in size, to approximately 42 gigabytes; two binaural array channels were embedded in the compressed video.The videos were encoded using MPEG4 with x264 settings including the ‘veryslow’ preset and CRF 17.Source: A. Video Compression · p. 8

B. Audio Alignment

  • Headset microphone recordings were first coarsely aligned to timestamps, then refined using GCC-PHAT between headset signals and an AR-glasses reference microphone.Source: B. Audio Alignment · p. 8

APPENDIX B CALIBRATION DATA

The appendix provides calibration data for using the dataset across modalities and refining tracking estimates. It describes acoustic transfer functions, acoustic-optical alignment, camera correction, and tracking-marker locations.

APPENDIX B CALIBRATION DATA

  • Calibration data supports acoustic-array use, links acoustic and visual modalities, reduces optical distortion, and refines tracking estimates.The described calibration data is not a depiction of any future product.Source: Appendix B · p. 8

A. Acoustic Array Transfer Functions

  • Acoustic array transfer functions, also called steering vectors or array manifolds, were measured on a head-and-torso simulator in an anechoic chamber at discrete spherical positions.Source: A. Acoustic Array Transfer Functions · p. 8

B. Spatial Acoustic-Optical Alignment

C. Optical Camera Intrinsics

D. Tracking Marker Locations

  • The dataset provides marker positions relative to the center of mass of tracked glasses markers, allowing tracked positions to be realigned to a specific marker.Source: D. Tracking Marker Locations · p. 9
Loading 2107.04174v2…