PaperlayerView exact revision on arXiv

Source-linked AI summary

Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition

Rishabh Jain, Aristeidis Papadopoulos, Zhaofeng Lin, Naomi Harte

arXiv:2609.10394v1eess.AScs.CVcs.MM

TL;DR

Existing AVSR benchmarks largely use clean, scripted speech that omits important properties of natural conversation. This paper introduces Candor-LR, a conversational benchmark derived from CANDOR, and evaluates pretrained models across conversational, cross-domain, and noisy conditions. The results show that conversational data exposes substantial weaknesses in audio-only recognition while increasing the value of visual cues and improving robustness.

  • Problem

    Current AVSR benchmarks rely heavily on clean, scripted speech and omit overlapping speech, spontaneous turn-taking, unscripted vocabulary, and variable acoustic conditions found in natural conversation.

  • Method

    The paper constructs Candor-LR from 1,656 natural dyadic videoconferences, using a custom pipeline and pretrained AVSR models for benchmark and robustness evaluation.

  • Results

    Visual gains on Candor-LR reach +6.54 in the combined training setup, while combined LRS2+LRS3+Candor-LR training reaches 9.83% AV WER on Candor-LR.

  • Takeaways & Limitations

    Candor-LR provides a challenging conversational benchmark in which in-domain exposure and visual information help narrow the gap between scripted-data systems and real dialogue.

Abstract

from arXiv · show

Current audio-visual speech recognition (AVSR) benchmarks, like LRS3, rely heavily on clean, scripted and rehearsed speech. They fail to reflect the complexity of natural conversation, which involves overlapping speech, spontaneous turn-taking, unscripted vocabulary and variable acoustic conditions. To shift the field toward realistic dialogue, we introduce Candor-LR, a conversational benchmark derived from the CANDOR corpus of 1,656 natural dyadic videoconferences. Our custom data preparation pipeline yields 713.5, 10.1, and 60.1 hours of training, validation, and test data, respectively. Evaluating pretrained AVSR models on Candor-LR reveals that audio-only accuracy drops sharply compared to LRS3, but visual cues compensate effectively, driving much larger performance gains on Candor-LR than on LRS3. Furthermore, training on this corpus significantly improves cross-domain robustness under both clean and noisy conditions, as its realistic conversational data captures broader audio-video features. We open-source our pipeline to ensure reproducibility, establishing Candor-LR as a challenging benchmark for conversational AVSR.

I. INTRODUCTION

Existing AVSR benchmarks emphasize clean, scripted speech and omit key features of natural conversation. Candor-LR addresses this gap with a large conversational benchmark and open-source preparation pipeline.

  • Motivation: Scripted benchmarks such as LRS2 and LRS3 use professional, well-articulated speech recorded under relatively consistent conditions.They contain no overlapping speech, no spontaneous interruptions, and little acoustic variability.
  • Motivation: Natural conversation introduces interruptions, overlapping speech, mid-sentence turn-aways, and variable room acoustics absent from many current AVSR benchmarks.These differences limit how directly scripted-benchmark performance reflects everyday dialogue.
  • Motivation: Low WER can reflect dataset patterns and language modeling rather than genuinely stronger visual speech representations.Prior analyses also question whether current benchmarks adequately measure visual information use.
  • Related work: Recent datasets add harder conditions including in-the-wild speech, degraded audio, reverberation, overlapping speakers, and silent-face segments.These efforts shift evaluation toward more realistic multi-speaker and acoustic settings.
  • Contribution: Candor-LR derives from 1,656 natural dyadic videoconferences and includes spontaneous turn-taking, unscripted vocabulary, and realistic acoustics.The benchmark contains 713.5 training hours, 10.1 validation hours, and 60.1 test hours.
  • Contribution: The authors open-source the pipeline that converts the original CANDOR corpus into AVSR-ready data.This supports reproducibility of the benchmark construction process.

II. CANDOR CLEANING FOR AVSR

CANDOR supplies long, unscripted dyadic conversations, but its original transcripts and organization require additional processing for AVSR. The cleaning workflow uses word-level Speechmatics alignments to produce suitable training segments.

  • CANDOR cleaning: CANDOR contains 1,656 natural dyadic conversations with over 850 hours of video and approximately 7 million words.Sessions last about 26–30 minutes and include overlapping speech, broad vocabulary, and frequent head movements.
  • Transcript source: CANDOR’s existing AWS Transcribe-based turn splits lack the millisecond-level word timestamps required to align audio and video for AVSR.The pipeline therefore uses Speechmatics outputs for precise timing and segmentation.
  • Access constraint: The dataset is not distributed with the paper, so users must obtain their own license from the CANDOR authors.The authors disclaim responsibility for licensing.
  • Transcript source: Speechmatics-ASR outputs are organized separately from the original CANDOR data for downstream processing.The paper presents this organization as part of the AVSR preparation workflow.

B. Speaker-Video Mapping

Because CANDOR identifies speakers by video User IDs while transcripts use audio-channel indices, the pipeline relies on channel metadata to align each speaker’s transcript and video.

  • Speaker-Video Mapping: CANDOR speaker videos use unique User IDs, whereas Speechmatics transcripts are indexed by audio channel number.Without a direct link, a transcript cannot be automatically assigned to its speaker video.
  • Speaker-Video Mapping: The channel_map.json file maps channel 0 to the left video and channel 1 to the right video.This mapping resolves the speaker-to-channel correspondence needed for multimodal extraction.
  • Segmentation: Long CANDOR sessions are segmented into 2–5 second clips by grouping words into phrases.Breaks occur at punctuation or inter-word gaps longer than 0.5 seconds, producing clips from 0.8 to 5.0 seconds.

D. Face Detection and Mouth ROI Extraction

The preprocessing pipeline extracts stable mouth-region video and synchronized per-speaker audio, then normalizes transcripts and filters unsuitable phrases for standard AVSR frameworks.

  • Face Detection and Mouth ROI Extraction: RetinaFace detects faces at threshold 0.8, after which 68 landmarks identify and crop the mouth region.The mouth crop is resized to 96×96 pixels at 25 fps, with a 3-frame moving average reducing jitter from spontaneous head movement.
  • Audio alignment: Per-speaker audio is extracted using the same phrase timestamps as the video alignment.Dedicated tracks eliminate cross-talk from the other participant before conversion to mono, resampling to 16 kHz, and WAV storage.
  • Transcript cleaning: Transcripts are lowercased, stripped of punctuation and redundant whitespace, and cleared of remaining disfluency markers.Sequential phrase-level filters then remove artifacts unsuitable for AVSR training.

G. Data Splitting and Output Format

Candor-LR uses speaker-aware splitting and provides a substantially larger, demographically characterized dataset than LRS3. Its test set is speaker-disjoint and contains 60.1 hours across 58,718 utterances.

  • G. Data Splitting and Output Format: 701 single-occurrence speakers are reserved exclusively for testing, ensuring no test speaker appears in training or validation.The remaining multi-session speakers are partitioned into training and validation sets.
  • G. Data Splitting and Output Format: Candor-LR contains 783.7 hours and 787,670 utterances, split into 713.5 training, 10.1 validation, and 60.1 test hours.These splits represent 91.2%, 1.3%, and 7.5% of the total data, respectively.
  • G. Data Splitting and Output Format: 1,554 Candor-LR speakers have valid demographic metadata, with 1,521 unique speakers after accounting for 33 training-validation overlaps.Test speakers remain disjoint from training and validation speakers.
  • G. Data Splitting and Output Format: 60.1 hours and 58,718 utterances make Candor-LR’s test set substantially larger than LRS3’s approximately one-hour, 1,321-utterance test set.The larger test set supports evaluation on considerably more conversational material.
  • Demographics: 46% of Candor-LR training speakers are aged 25–35, compared with 26% in LRS3, while older age groups are less concentrated in Candor-LR.Candor-LR has 11% of speakers aged 45–55 and 5% aged 55+, versus 23% and 12% in LRS3.
  • Demographics: Candor-LR has a more balanced gender distribution than LRS3, with 47% female training speakers and 57% female test speakers.The training-set female proportion is 36% in LRS3, compared with 47% in Candor-LR.

IV. EXPERIMENT AND RESULTS

Pretrained AVSR models are evaluated across video-only, audio-only, and audio-visual modalities on LRS3 and Candor-LR. Candor-LR is substantially harder for audio-only and video-only recognition, while visual gains remain modest in these zero-shot results.

  • Benchmarking Candor-LR with pretrained AVSR Models: Three pretrained AVSR models are evaluated on Candor-LR without downstream finetuning, using zero-shot cross-domain generalization.The models are AV-HuBERT, Llama-AVSR, and Auto-AVSR configurations.
  • Benchmarking Candor-LR with pretrained AVSR Models: 16.83–24.32% AO WER on Candor-LR contrasts with 0.74–1.95% on LRS3, demonstrating a substantially harder conversational benchmark.Llama-AVSR has the best AO performance on both datasets: 0.74% on LRS3 and 16.92% on Candor-LR.
  • Benchmarking Candor-LR with pretrained AVSR Models: 70.48% VO WER for Auto-AVSR-L versus 78.35% for AV-HuBERT on Candor-LR shows that video-only recognition remains especially brittle for natural conversation.On LRS3, the corresponding values are 19.10% and 28.69%.
  • Benchmarking Candor-LR with pretrained AVSR Models: AV gains over AO remain marginal, with Candor-LR improvements bounded by ∆≤+1.56.The reported delta is defined as AO − AV, so positive values indicate reduced WER with visual information.
  • Benchmarking Candor-LR with pretrained AVSR Models: Llama-AVSR shows no visual benefit on Candor-LR, with ∆=0.00, and a −0.05 delta on LRS3.These zero-shot results make the contribution of visual information difficult to assess from its performance alone.
  • Benchmarking Candor-LR with pretrained AVSR Models: Auto-AVSR-L achieves the best overall Candor-LR AV performance at 16.69% WER with ∆=+0.57.Its Candor-LR AO WER is 17.26%, slightly higher than Auto-AVSR-S at 16.83%, despite nearly double the training data.

B. Cross-Domain Generalization and Multi-Dataset Training

Candor-LR training substantially improves conversational-domain recognition while preserving strong performance on scripted benchmarks. Including Candor-LR also makes visual information more valuable for Candor-LR recognition.

  • 14.81% AO WER and 10.50% AV WER are achieved on Candor-LR when training uses Candor-LR alone.
  • 9.83% AV WER is the best Candor-LR result under combined LRS2+LRS3+Candor-LR training.The same setup reaches 2.54% AV WER on LRS2 and 1.42% AV WER on LRS3.
  • Candor-LR-trained models retain strong cross-domain performance while improving the conversational test set.The combined setup places between-domain results at 2.54% AV WER on LRS2 and 1.42% AV WER on LRS3.
  • The visual benefit on Candor-LR rises from +1.77 with LRS3-only training to +6.54 with combined training.The corresponding gains are +1.40 for LRS2+LRS3, +4.31 for Candor-LR-only, and +6.54 for combined training.

C. Noise Robustness for Candor-LR

The noise-robustness evaluation compares LRS3 and Candor-LR across SNR levels under inference-only and joint train-and-test noise. It uses synthetic babble noise and reports WER trends for both datasets.

  • Synthetic babble noise is evaluated from −10 dB to +10 dB SNR on LRS3 and Candor-LR.Noise is applied either only during inference or during both training and inference.
  • Figures 4 and 5 plot WER across SNR levels for inference-only and joint train-and-test noise configurations.The figures compare both test sets and different training-data combinations.

1) Evaluation with Inference-Only Noise:

Inference-only noise exposes Candor-LR as more noise-sensitive than LRS3, while visual information becomes increasingly valuable as acoustic conditions worsen. Candor-LR training also generalizes better to LRS3 than LRS3 training generalizes to Candor-LR.

  • 126.66% WER is reached by Candor-LR at −10 dB SNR, compared with 100.73% for LRS3.At 10 dB, WER is 34.60% for Candor-LR and 8.39% for LRS3.
  • Visual cues become increasingly important as noise increases, reducing Candor-LR WER by 16.11% at 10 dB SNR.The comparison is between 34.60% AO WER and 18.49% AV WER for in-domain training and testing.
  • 48.7% WER separates Candor-LR audio-only and audio-visual models at 0 dB SNR.The audio-only model reaches 99.33% WER, while the audio-visual model reaches 50.63%.
  • 38.96% AV WER results from LRS3-to-Candor-LR transfer at 10 dB, versus 6.62% for Candor-LR-to-LRS3 transfer.Clean-condition Candor-LR-to-LRS3 transfer is 6.02% AV WER.

2) Evaluation with Joint Train-and-Test Noise:

Joint noise training changes how models use visual information and interacts strongly with training-data diversity. Broad training data helps avoid severe failure at extreme noise, although noise augmentation can harm narrow-domain transfer.

  • 69.53% WER results from noise-trained LRS3-only models on Candor-LR at 10 dB, versus 38.96% for clean-trained models.This indicates a penalty for applying noise augmentation to a narrow, single-domain dataset.
  • Table III benchmarks Candor-LR WER across SNR levels using LRS2+LRS3+Candor-LR training.The table treats ∞ as clean audio.
  • 1.84% AV WER is achieved on LRS3 at 10 dB when diverse training data is combined with noise augmentation.The LRS3-only AV WER falls from 6.48% to 1.84%.
  • 12.40% AV WER is achieved on Candor-LR at 10 dB with combined data and noise training, compared with 16.19% for Candor-LR-only training.The gain on Candor-LR is smaller than the corresponding gain on LRS3.
  • 82.00% AV WER at −10 dB is achieved by the fully optimized model, improving on 106.20% for Candor-LR-only training without noise augmentation.The result supports using diverse data with noise augmentation under severe interference.

V. CONCLUSION

Candor-LR provides an open-source, conversational AVSR benchmark built from natural dyadic videoconferences. Its evaluations show that unscripted speech challenges current models, while visual modality and in-domain exposure help narrow the gap.

  • Candor-LR contains 713.5 hours of training data, 10.1 hours of validation data, and 60.1 hours of test data.The benchmark is derived from natural dyadic videoconference data.
  • Candor-LR captures spontaneous turn-taking speech and realistic acoustic conditions largely absent from scripted datasets.This shifts AVSR evaluation toward conversational settings.
  • Current AVSR models struggle with unscripted conversational speech, while visual modality and in-domain exposure help narrow this gap.The evaluation examines cross-domain generalization under varying training configurations and noise conditions.
  • The open-source pipeline supports reproducible construction of AVSR-ready data from CANDOR.Candor-LR is presented as a challenging real-world benchmark for conversational speech understanding.
Loading 2609.10394v1…