Source-linked AI summary
NELS -- Never-Ending Learner of Sounds
Benjamin Elizalde, Rohan Badlani, Ankit Shah, Anurag Kumar, Bhiksha Raj
TL;DR
Many online recordings have undescribed sound content, and the web poses challenges involving sound-language relations, diverse classes, and large-scale evaluation. NELS mines web audio continuously to index sounds and associated knowledge; by October 2017, it had indexed over 300 hours across 4 million short segments.
Problem
Online recordings contain largely undescribed sound content, while sound-language relations, diverse classes, and large-scale evaluation pose challenges for automatic analysis.
Source: §1 · pp. 1–2Method
Results
Over 300 hours of audio across 4 million 2.3-second video segments had been crawled and indexed as of October 20, 2017.
Source: §2 · p. 2Takeaways & Limitations
The indexed audio is available for search and retrieval through the NELS website.
Source: §2 · p. 2, §2.3 · p. 3YouTube is NELS's first source, with use of other sound-web archives left for the future.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Sounds are essential to how humans perceive and interact with the world and are captured in recordings and shared on the Internet on a minute-by-minute basis. These recordings, which are predominantly videos, constitute the largest archive of sounds we know. However, most of these recordings have undescribed content making necessary methods for automatic sound analysis, indexing and retrieval. These methods have to address multiple challenges, such as the relation between sounds and language, numerous and diverse sound classes, and large-scale evaluation. We propose a system that continuously learns from the web relations between sounds and language, improves sound recognition models over time and evaluates its learning competency in the large-scale without references. We introduce the Never-Ending Learner of Sounds (NELS), a project for continuously learning of sounds and their associated knowledge, available on line in nels.cs.cmu.edu
1 Introduction and Related Work
The growth of online recordings makes automatic sound indexing and retrieval important, but web audio presents challenges that curated datasets do not capture. NELS applies continuous learning to sounds and their associated knowledge by mining the web.
1 Introduction and Related Work
- Recognizing sounds can support applications from privacy-sensitive care and safer driving to surveillance, digital assistants, and video retrieval.The growing volume of online video makes sound recognition important for organizing and understanding multimedia data.Source: §1 · p. 1
- Web audio is largely unstructured, has an unbounded range of sound classes, grows continually, and is often insufficiently described.The paper contrasts these conditions with curated recordings that have defined tasks, limited classes and samples, and richer descriptions.Source: §1 · p. 1
- Existing sound-recognition systems learn from finite curated sources and therefore do not improve their learning over time.Never-ending learning architectures instead reuse previously learned knowledge and apply self-reflection to avoid stagnation.Source: §1 · pp. 1–2
- Never-ending learning had been explored for text and images, but not sound, where tasks include linking sounds with language, growing vocabularies, and evaluating recognition without source references.These tasks address sound-language associations, recognizer robustness, and evaluation subjectivity.Source: §1 · p. 2
- NELS is a web-mining project for large-scale continuous learning of sounds and associated knowledge, with a working framework for indexing and searching audio.Associated knowledge can include semantics about objects, events, actions, places, cities, and qualities.Source: §1 · p. 2
2 NELS Framework
NELS continuously crawls YouTube audio and metadata, applies sound recognition and human feedback, and indexes the resulting content for search. Its framework includes a crawl stage, a classifier stage, and an online site for evaluating recognition and searching indexed audio.
2 NELS Framework
- NELS continuously crawls YouTube audio and metadata to build a content-based index using a vocabulary of 600 sounds.The framework is described as operating 24/7.Source: §2 · p. 2
- Over 300 hours across 4 million 2.3-second segments were crawled and indexed using metadata, recognizer predictions, and human feedback.Recognizers were trained from varied sources, including web audio, and the indexed content is available for search and retrieval.Source: §2 · p. 2
2.1 Crawl
- Crawl queries use 605 sound-event labels from four datasets and extract 12 video metadata attributes; YouTube was selected for its sound diversity and available metadata.Queries combine each sound-event label with “sound”; videos longer than ten minutes or shorter than two seconds are discarded.Source: §2.1 · pp. 2–3
2.2 Hear & Learn
- Classifiers trained on 605 events from four annotated datasets run on unlabeled crawled segments, and their predictions also index the audio.The current classifier setup uses CNNs with 2.3-second recordings and log-scaled mel-spectrogram features.Source: §2.2 · p. 3
2.3 Website
- The website supports human-feedback evaluation and indexed-audio search, with text queries mapped to sound classes and YouTube links yielding dominant sounds.Users can mark displayed segments Correct or Incorrect to assess whether a predicted class is present.Source: §2.3 · p. 3
3 Discussion
The discussion examines sound–language relations, continuous semi-supervised learning, and large-scale evaluation for NELS. It reports linguistic consistency and label discovery, a self-training gain that plateaued, and search-query precision trends close to human feedback.
Relation between sounds and language
- Despite label subjectivity, sound events showed some consistency with adjective-noun and verb-noun pairs across 1,000 pair-labels.The labels were derived from different audio ontologies.Source: §3 · p. 4
- A “sounds of <Y>” query and rule-based filtering yielded over 100,000 sound labels, while dependency-path features linked labels to acoustic scenes.For example, forests were associated with bird singing, breaking twigs, cooing, and falling water.Source: §3 · p. 4
Continuous semi-supervised learning of sounds
- About 1.4% overall precision improvement was achieved with 200,000 unlabeled YouTube segments, but learning plateaued, possibly due to data mismatch, initial class bias, and ambiguous audio.The study used US8K, with about 8,000 labeled samples across 10 sounds.Source: §3 · p. 4
Evaluation of the learning quality
- Less than a 10% absolute precision difference separated search-query and human-feedback references, suggesting queries could provide a preliminary lower-bound evaluation.The comparison used the highest-40 recognized segments per sound class.Source: §3 · p. 4