Source-linked AI summary
VAANI Noise Event Dataset: A curated spontaneous speech dataset annotated with timestamps for noise events
Pavan Kumar J, Agneedh Basu, Pranav Bhat, Sujith Pulikodan, Suryansh Shukla, Nihar Desai Prasanta K. Ghosh
TL;DR
Public corpora provide limited combinations of realistic speech, diverse Indian acoustic conditions, and timestamped overlapping noise events, motivating VAANI. The paper introduces a derived annotation layer over spontaneous Project VAANI recordings, with broad release coverage and staged quality control. VAANI therefore offers an auditable resource for noise-robust ASR, sound event detection, and speech enhancement research.
Problem
Existing resources rarely combine naturally co-occurring speech and noise, overlapping span-level noise timestamps, and spontaneous multilingual speech.
Method
VAANI adds seven-class, overlapping start/end noise-event annotations to spontaneous Project VAANI speech recordings and applies staged sanity checks, re-timestamping, and independent auditing.
Results
72,756 release-quality segments totaling 122.17 hours cover 38,541 speakers, 58 languages, 30 states, and 162 districts.
Takeaways & Limitations
VAANI provides an auditable resource for noise-robust ASR, sound event detection, and speech enhancement in authentic Indian acoustic conditions.
Abstract
from arXiv · showhide
Most public sound-event corpora are optimized either for general audio tagging or for clean speech separation, and comparatively few provide strong timestamped noise annotations layered directly on top of spontaneous, real-world speech. We present the VAANI Noise Event Timestamp Dataset, a derived annotation layer built on Project VAANI field recordings of spontaneous speech collected across 165 Indian districts in 105 languages. Unlike synthetically mixed corpora, VAANI captures speech and ambient noise in situ and simultaneously, and annotates each recording with fine-grained start/end timestamps for overlapping background noise events organized into a compact seven-class semantic taxonomy: animal, traffic, baby/child, music, signal/alarm, appliance, and non-speech human. This combination of spontaneous multilingual Indic speech, authentic regional soundscapes, and span-level noise tags that may overlap with speech targets tasks that existing datasets address only partially: noise-robust Automatic Speech Recognition (ASR), sound event detection (SED), and speech enhancement. We position VAANI against nine widely used corpora and benchmarks, including WHAM!, AVA-Speech, MUSAN, FSD50K, CHiME-6, AudioSet, DESED, the India-specific iNoise noise database, and the Kathbath-Noisy noisy-ASR benchmarks, and describe the annotation protocol and quality-control procedure used to produce the timestamped tags.
I. INTRODUCTION
Real-world Indian speech contains diverse, non-stationary background events whose timing and overlap affect recognition and enhancement. VAANI addresses this need with timestamped noise annotations over spontaneous speech for noise-robust ASR, sound event detection, and speech enhancement.
- Indian speech recordings commonly contain traffic, animals, children, appliances, music, alarms, and human non-speech sounds.
- Noise-event onset, duration, and overlap with speech influence recognition errors and perceived enhancement quality.
- Existing corpora cover different combinations of synthetic mixing, clip-level labels, isolated noise, domain mismatch, or India-specific noisy speech.
- VAANI adds exact start/end timestamps for background events to spontaneous Project VAANI recordings alongside transcripts.
II. RELATED DATASETS
The reviewed resources address separate aspects of noisy speech and sound-event research, but differ in recording realism, speech content, annotation granularity, and construction. VAANI is positioned against these complementary design choices.
- Table I compares nine resources by recording method, speech content, noise annotation type, and scale.
- In-situ audio records speech and background noise together in the field rather than mixing separately recorded signals afterward.
- Span-level annotation records each noise event’s exact start and end timestamps instead of only clip-level or frame-level labels.
- WHAM! uses clean read speech with separately recorded urban noise mixed at controlled signal-to-noise ratios.
B. Frame- and clip-level general-audio corpora
General-audio and spontaneous-speech corpora provide useful labels or realistic recordings, but they do not jointly offer VAANI’s event-level timing and target setting. Their limitations differ by corpus.
- AVA-Speech labels movie audio at frame level using four mutually exclusive speech-activity states.
- FSD50K and AudioSet provide weak clip-level multi-label tags without timestamp information.
- These corpora sacrifice the event-level timing precision preserved by VAANI’s category, tag, start, end format.
- CHiME-6 contains unscripted overlapping conversational speech but targets English Western domestic ASR and diarization without noise-event annotations.
D. Indian noise and noisy-ASR resources
India-specific resources address environmental or noisy-speech conditions, but their labels do not provide timestamped noise events co-occurring with spontaneous speech. VAANI targets that remaining annotation gap.
- iNoise contains ten Indian environmental noise categories recorded across five outdoor and five indoor settings.
- iNoise is noise-only and lacks co-occurring speech and event timing relative to an utterance.
- Kathbath-Noisy benchmarks evaluate noisy ASR with transcriptions and word-error-rate degradation by language.
- Kathbath-Noisy labels speech transcriptions rather than noise events.
E. Positioning
Across nine corpora and benchmarks, no single resource combines in-situ speech–noise co-occurrence, overlapping span-level noise timestamps, and spontaneous multilingual speech from Indian mobile-device recordings.
- No single compared resource combines real in-situ speech–noise co-occurrence, overlapping event timestamps, and spontaneous multilingual Indian field speech.
- The comparison includes nine resources spanning synthetic mixtures, clip-level labels, noise-only recordings, and noisy-ASR benchmarks.
- VAANI addresses this combination while Table I positions it against representative noise and sound-event corpora.
III. DATASET DESCRIPTION
The release-quality dataset contains 72,756 short speech segments with transcripts and timestamped background-noise events, totaling 122.17 hours from 38,541 speakers across India.
- 72,756 speech segments total 122.17 hours and come from 38,541 distinct speakers.
- Each short utterance carries a speech transcript and one or more timestamped background-noise events.Segment durations range from 0.79 to 23.49 seconds, with a 6.05-second mean.
- The release-quality subset excludes structural-issue, held-out evaluation, and synthetic-data segments, retaining verified_timestamps and unverified_timestamps tiers.
- Coverage spans 58 languages, 30 states, and 162 districts of India.
A. Linguistic and geographic coverage
The corpus provides broad linguistic and geographic coverage, combining Hindi-dominant data with a long tail of Indian languages and recordings distributed across diverse regional environments.
- Hindi dominates with 83.9 hours and 47,080 segments, while the corpus retains substantial Telugu, Bengali, and Marathi coverage.
- The dataset includes lower-resource languages such as Chakma, Garo, and Mizo among 57 languages.
- Bihar, Andhra Pradesh, Uttar Pradesh, West Bengal, and Maharashtra contribute the largest regional totals across 30 states.
- The remaining hours span rural, semi-urban, and urban Indian acoustic environments rather than a single locale.
B. Noise taxonomy and annotation format
VAANI represents noise at both segment and event levels, using seven top-level categories and exact timestamps for overlapping events within spontaneous speech recordings.
- Each segment stores a multi-label NoiseCategory list, while each event stores category, tag, start, and end fields.Timestamps use verbatim-precision strings, and underlying tags such as barking or lip smacking are preserved.
- Seven top-level noise categories organize the annotation taxonomy.
- Non-speech human sounds cover 37.8% of segments and account for 37,739 events, but only 4.5 event-hours at a 0.42-second mean duration.
- Appliance/machine events are rarest but longest on average at 6.1 seconds, whereas animal and traffic events contribute most annotated noise duration.
- 106,892 noise events occur across 72,756 segments, with 17,426 segments containing two or more co-occurring categories.
IV. ANNOTATION AND QUALITY CONTROL
VAANI uses staged annotation and quality control to produce timestamped noise-event labels, separating a broad unverified tier from a smaller, independently audited verified tier.
- Annotation pipeline: Roughly 150+ hours of spontaneous-speech segments are sampled for noise-event annotation by trained freelancers marking each audible event’s start, end, and category.
- Annotation pipeline: A structural sanity check rejects failed freelancer batches and releases about 100 hours as unverified_timestamps while forwarding at least 20 hours for verification.
- Quality control: An internal team re-timestamps the candidate subset using the unverified annotations as reference before independent auditing.
- Quality control: A second reviewer audits a random 10% sample, and any disagreement sends the batch back for rework before verified_timestamps release.
- Quality distinction: Both tiers contain fully timestamped events, but the staged sanity gate, internal re-timestamping, and audit give verified_timestamps higher trust.
V. CONCLUSION
VAANI concludes by presenting a large, geographically and linguistically broad release-quality dataset that pairs authentic multilingual speech with overlapping timestamped noise events and auditable quality tiers.
- Conclusion: VAANI contributes a span-level annotation layer over in-situ spontaneous multilingual Indic speech, unlike synthetic mixtures or clip-level noise-label corpora.
- Conclusion: The release-quality subset contains 72,756 segments totaling 122.17 hours from 38,541 speakers across 58 languages, 30 states, and 162 districts.
- Conclusion: The dataset organizes naturally co-occurring background noise into seven classes with exact, overlapping start/end timestamps.
- Conclusion: Non-speech human, animal, and vehicle/traffic events dominate segment coverage, while animal and vehicle/traffic events contribute most annotated noise duration.
- Conclusion: The explicit distinction between verified_timestamps and unverified_timestamps provides downstream users with an auditable quality choice for ASR, sound event detection, and speech enhancement research.