PaperlayerView exact revision on arXiv

Source-linked AI summary

VākQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering

Bhavana Akkiraju, Ravi Sastry Kolluru, Sri Charan D, Srihari Bandarupalli, Santosh Kesiraju, Anil Vuppala

arXiv:2609.19879v1cs.CLeess.AS

TL;DR

Telugu spoken question answering lacked a benchmark, and evaluation reliability remained unquantified. VākQA introduces a benchmark and systematic evaluation study; proprietary systems consistently outperform open-weight models, while results vary with language and modality.

  • Problem

    Telugu SQA lacked a benchmark, and evaluation reliability was unresolved for correct answers that differ from references in surface form.

    Source: §I · p. 1

  • Method

    The authors construct a Telugu spoken QA benchmark from quiz-style YouTube audio across six domains, then evaluate judges and models across input conditions.

    Source: §II · p. 2, A. Data Collection and Audio Extraction · p. 2, §I · p. 2

  • Results

    Proprietary systems consistently outperform open-weight models, while Telugu text better preserves specificity than translation and speech introduces meaning-altering acoustic confusions.

    Source: §VI · p. 6

  • Takeaways & Limitations

    Reliable evaluation remains a bottleneck for Telugu spoken factoid question answering.

    Source: §VI · p. 6

    Gemini-as-a-judge’s non-uniform strictness limits fine-grained comparisons.

    Source: §VI · p. 6

Abstract

from arXiv · show

Question answering has advanced rapidly with large language models, but predominantly for high-resource languages, in both text and spoken settings. Spoken question answering (SQA) benchmark for Telugu remains unexplored, and the reliability of automatic evaluation in this setting remains unquantified. We introduce VākQA, a Telugu SQA benchmark of 2,001 factoid question-answer pairs across six domains, with 2.53 hours of speech audio, bilingual transcriptions, and human-verified reference answers. We first validate evaluation methods against human judgements: Gemini-as-a-judge best approximates human ratings but is non-uniformly strict, while open-weight judges systematically penalize correct Telugu answers that differ in surface form from the reference. Using this validated setup, we benchmark proprietary and open-weight models across input modality, language, and domain. We observe that Telugu phrasing retains cultural specificity that is lost in translation, speech input introduces phonetic confusions that alter question meaning, and cascaded ASR-MT errors compound progressively. VākQA is publicly released.

I. INTRODUCTION

VākQA introduces a Telugu spoken question-answering benchmark and examines evaluation reliability alongside model performance across input conditions. The introduction frames this work against a gap in Telugu speech QA and the limitations of existing evaluation practices.

I. INTRODUCTION

  • Telugu had text QA in TeQuAD, but speech variability remained unrepresented amid scarce annotated speech resources.The introduction describes Telugu spoken QA as largely unexplored.Source: §I · p. 1
  • Exact Match and F1 are brittle to paraphrase,, while model-based metrics are primarily English-trained; VākQA therefore quantifies LLM-judge reliability for Telugu spoken QA.The introduction notes that Indic QA datasets have not explored LLM-as-a-judge reliability and that prior work found judges inconsistent across languages and tasks,.Source: §I · p. 1
  • VākQA is presented as the first Telugu spoken QA benchmark, built from quiz-style spoken interaction across six domains with audio and bilingual transcriptions.It includes original transcriptions and English translations.Source: §I · pp. 1–2
  • Gemini-as-a-judge exhibits non-uniform strictness, and the paper analyzes how judge-model choice affects automatic evaluation reliability for Telugu QA.The authors frame this as a limitation of current evaluation practice.Source: §I · p. 2
  • The benchmark study isolates input language, modality, model size, and model tier, and examines cascaded ASR→MT errors and domain-wise QA performance.The study also evaluates performance across proprietary and open-weight models.Source: §I · p. 2

II. V ¯AKQA BENCHMARK · A. Data Collection and Audio Extraction

VākQA is constructed through a three-step pipeline, beginning with collection and audio extraction. The collection draws on Telugu quiz-style and multiple-choice videos spanning six subject categories.

II. V ¯AKQA BENCHMARK

  • The dataset pipeline comprises data collection and audio extraction, QA-pair extraction, and human validation and translation.Figure 1 provides an overview of the full workflow.Source: §II · p. 2

A. Data Collection and Audio Extraction

B. Semi-Automatic QA Pair Extraction · C. Human Verification and Translation

The pipeline automatically extracted Telugu question–answer pairs and aligned them to audio, then annotators verified the pairs and translated them into English. The resulting bilingual benchmark contains 2,001 spoken questions across six domains.

B. Semi-Automatic QA Pair Extraction

  • Pyannote VAD segmented speech into overlapping 7-second chunks, which a finetuned Seamless-large-v2 Telugu ASR model transcribed.The ASR model was trained on approximately 900 hours of data from IndicVoices, Kathbath, Google FLEURS, SyspIn TTS, and IndicTTS resources.Source: B. Semi-Automatic QA Pair Extraction · p. 2
  • Gemini extracted QA pairs verbatim from the merged chunk transcripts.The chunks retained their original timestamps.Source: B. Semi-Automatic QA Pair Extraction · p. 2
  • Whisper-timestamped, with IndicWhisper supplied word-level timestamps, and fuzzy matching aligned QA text to the ASR output.A Levenshtein ratio of at least 85% enabled adjustment of the QA audio segment boundaries.Source: B. Semi-Automatic QA Pair Extraction · p. 2

C. Human Verification and Translation

  • Five annotators checked whether extracted question and answer transcripts matched their corresponding audio segments.Source: C. Human Verification and Translation · p. 2
  • Annotators manually translated all Telugu QA pairs into English, creating bilingual question–answer pairs.Source: C. Human Verification and Translation · p. 2
  • The dataset comprises 2,001 spoken questions totaling 2.53 hours across six domains: Science (27%), General Knowledge (23%), Politics (16%), History (13%), Culture (12%), and Geography (10%).Six Telugu and eight English instances with long descriptive answers were retained despite potential effects on EM and F1 metrics.Source: C. Human Verification and Translation · p. 2

III. EXPERIMENTAL SETUP … B. Evaluation Metrics

The study evaluates proprietary and open-weight Telugu QA models across speech and text configurations, using human judgments as the reference for automatic answer-correctness metrics.

A. QA Models and Input Configurations

  • The evaluation covers a proprietary model that accepts speech or text and open-weight text-only models, since no open-weight Telugu speech LLM was available.Gemini-2.5-Flash is the proprietary model; open-weight models include Gemma-3 (4B, 12B, 27B), Llama-3.1, Hex-1, Sarvam-m, and Qwen-3-4B.Source: A. QA Models and Input Configurations · p. 2
  • The open-weight set comprises Gemma-3 (4B, 12B, 27B) [38], Llama-3.1 [39], Hex-1, Sarvam-m, and Qwen-3-4B [42].Gemini-2.5-Flash is the proprietary model.Source: A. QA Models and Input Configurations · p. 2

Input modality and language:

  • Models are evaluated by input modality and language; direct Telugu speech goes to Gemini, while open-weight models are excluded because they cannot accept Telugu speech.The configurations distinguish speech or text input and Telugu or English input.Source: Input modality and language · p. 2
  • For Telugu ASR-text evaluation, transcripts from Seamless FT or IndicWhisper are fed to the QA models.Source: Input modality and language · p. 2
  • The cascaded ASR→MT setup translates Telugu ASR transcripts into English with four configurations, while oracle Telugu and English text isolates ASR and MT error effects.The four cascades combine two ASR systems with two MT systems; testing both languages on identical questions distinguishes missing knowledge from language-access failures.Source: Input modality and language · p. 2

B. Evaluation Metrics

  • Human judgments provide the gold-standard answer-correctness reference, alongside Exact Match (EM), token-level F1, BLASER-2.0, and an LLM-as-a-judge correctness score.Source: B. Evaluation Metrics · p. 3

Human judgments: · LLM-as-a-judge: · Lexical and Embedding-based Metrics:

The section establishes human ratings as the reference for evaluating automatic answer metrics, using LLM judges alongside lexical and embedding-based measures. It describes the annotation sample, judge setup, and metric definitions.

Human judgments:

  • Gemini and three Gemma-3 variants were benchmarked on 100 Telugu textual questions, yielding 400 question-reference-candidate triplets.Five native Telugu speakers rated each candidate answer on a 1–5 scale using the rubric in Table II.Source: Human judgments · p. 3
  • After excluding 20 highly inconsistent items, human ratings reached Krippendorff’s α = 0.836, indicating strong inter-rater agreement.The excluded items had annotator scores spanning the full 1–5 range.Source: Human judgments · p. 3
  • Human annotations served as the gold-standard reference for assessing the reliability of automatic evaluation metrics.Source: Human judgments · p. 3

LLM-as-a-judge:

  • Gemini and three Gemma-3 variants judged answers by comparing each response with its question and reference answer under the same 1–5 rubric.The evaluation prompt was iteratively refined to maximize correlation with human ratings.Source: LLM-as-a-judge · p. 3

Lexical and Embedding-based Metrics:

IV. EVALUATION RELIABILITY

Gemini-as-a-judge aligns most closely with human judgments, but its ratings are not uniformly calibrated. Lexical and embedding-based metrics correlate poorly with human judgments and can penalize correct answers that differ in wording.

IV. EVALUATION RELIABILITY

  • Gemini-as-a-judge leads with ρ = 0.86 and τ = 0.77, outperforming open-weight judges; Gemma-3 12B and 27B follow at ρ = 0.81, τ = 0.71 and ρ = 0.80, τ = 0.70.The lexical metrics EM and F1 and embedding-based BLASER-2.0 have correlations below 0.5 with human judgments.Source: Table III · p. 3, §IV · p. 3
  • ME = -0.28 and LoA = -1.28 to 0.72: Gemini is slightly stricter on average, yet more lenient on low-quality answers and stricter on high-quality ones.These are the narrowest limits of agreement among the judges described.Source: Figure 2 · p. 3, §IV · p. 3
  • Gemma-3 4B is the weakest correlate with human judgments (ρ = 0.57, τ = 0.5).Gemma-12B shows positive bias (ME=0.34; LoA: -1.15 to 1.83), while Gemma-27B has near-zero bias (ME =-0.07; LoA: -1.57 to 1.43); 4B has the largest spread (LoA: -2.31 to 2.93).Source: §IV · pp. 3–4
  • EM and F1 assign 0 to a correct, more detailed answer naming cyclone-affected states, while BLASER-2.0 gives it 2.43.The reference names the broad affected region, so the candidate’s specific state names have low lexical overlap.Source: §IV · p. 3

Pairwise comparison of judge models: · V. BENCHMARKING QA MODELS · A. Proprietary vs Open-Weight Models

VākQA benchmarking shows Gemini outperforming open-weight models across input configurations, while judge choice and surface-form sensitivity affect score reliability. The evaluation also examines input language, modality, and cascaded-pipeline errors.

Pairwise comparison of judge models:

  • Switching the judge from Gemini to Gemma-12B worsened scores for 46.23% of Gemini QA answers, improved 21.14%, and left 32.63% unchanged.Source: Pairwise comparison of judge models · p. 4
  • For a correct candidate answer, Gemini-as-a-judge gave 5 while Gemma-3-12B gave 1, reflecting difficulty recognizing Telugu semantic equivalence across surface forms.The passage cites numeric versus spelled-out dates and correct short forms where the reference includes glosses.Source: Pairwise comparison of judge models · p. 4
  • Gemini is used as the primary evaluator in subsequent sections because reliable evaluation in this low-resource setting requires a sufficiently capable judge.Source: Pairwise comparison of judge models · p. 4

V. BENCHMARKING QA MODELS

  • The benchmark compares QA results across Telugu versus English input, speech versus text modality, and cascaded-pipeline errors.Source: §V · p. 4

A. Proprietary vs Open-Weight Models

  • 3.63 (1.71) is Gemini’s score on oracle Telugu text, above Gemma-27B’s 2.55 (1.79), Gemma-12B’s 2.01 (1.60), Sarvam-m’s 1.98 (1.60), and Gemma-4B’s 1.43 (1.11).The passage reports a consistent gap between Gemini and open-weight models across all input configurations.Source: A. Proprietary vs Open-Weight Models · p. 4
  • Telugu ASR transcripts lower scores versus oracle text, but Gemini remains ahead: 3.40/3.09 versus Gemma-27B’s 2.34/2.22 and Gemma-12B’s 1.84/1.77.Source: A. Proprietary vs Open-Weight Models · p. 4
  • Translated English inputs score lower still due to MT error propagation, although larger open-weight models remain stronger than smaller ones.Source: A. Proprietary vs Open-Weight Models · p. 4

B. Effect of Input Language · C. Effect of Input Modality · D. Effect of Cascaded ASR →MT Errors

Telugu phrasing preserves scope that translation can obscure, while speech and cascaded ASR→MT introduce errors that alter question meaning and reduce QA performance. The passages quantify these effects against oracle Telugu text and illustrate how they arise.

B. Effect of Input Language

  • 3.63 (1.71) for O1 versus 3.52 (1.74) for O2; switching Telugu to English degrades 18.8%, improves 16.5%, and leaves 64.7% unchanged.These scores and pairwise outcomes compare oracle Telugu and English text inputs to Gemini QA.Source: B. Effect of Input Language · p. 4
  • In Telugu, “mana” makes the question’s India-specific scope clear and elicits “Aryabhata”; in English, the reference is ambiguous and Gemini answers “Sputnik.”The example illustrates how translation can introduce ambiguity that changes the answer.Source: B. Effect of Input Language · p. 4

C. Effect of Input Modality

  • 3.63 (1.71) for O1 versus 3.28 (1.84) for S1; speech degrades 21.2%, improves 13.1%, and leaves 65.7% unchanged.The comparison holds the QA model to Gemini and contrasts Telugu oracle text with speech input.Source: C. Effect of Input Modality · p. 4
  • Speech confuses Telugu “pa ˙mdu” (fruit) with phonetically closer “pa ˙md.uga” (festival), leading Gemini to answer Bathukamma instead of mango.This acoustic confusion shifts the question’s meaning from Telangana’s state fruit to its state festival.Source: C. Effect of Input Modality · p. 4

D. Effect of Cascaded ASR →MT Errors

  • 2.74 (O3) vs. 3.63 (O1), and 2.44–2.80 for M1–M4, show QA scores fall after MT and further under full ASR+MT cascades.The reported results indicate that errors compound across pipeline stages.Source: D. Effect of Cascaded ASR →MT Errors · p. 4
  • 11.9% of questions worsen and 4.8% improve with Seamless FT ASR versus O1; at WER 30.25, a misrecognized ringworm question yields heat instead of fungus.The corrupted transcript makes the question ill-posed despite the oracle-text answer being correct.Source: D. Effect of Cascaded ASR →MT Errors · p. 4

E. Domain-wise Performance

The section examines domain-wise answer correctness across six domains under two input configurations: oracle Telugu text and oracle Telugu text passed through Seamless MT.

E. Domain-wise Performance

  • The domain-wise analysis compares average answer-correctness scores for QA models across six domains under O1 and O3 input configurations.O1 uses oracle Telugu text; O3 uses oracle Telugu text → Seamless MT.Source: E. Domain-wise Performance · pp. 4–5
  • Table V reports QA-model evaluations across input configurations using Gemini-as-a-judge on a 1–5 scale, with mean and standard deviation.The table identifies the evaluated Gemma-3, Llama-3.1, Qwen-3, and Sarvam-m models.Source: Table V · p. 5

Gemini performance across domains: · Cross-model comparison:

With oracle Telugu text, Gemini scores consistently across domains, with Culture strongest; translated English lowers Culture performance, though some Science questions improve. Larger open-weight models perform better than smaller ones but still lag behind Gemini.

Gemini performance across domains:

  • Gemini scores 3.54–3.76 across domains with oracle Telugu text, with Culture highest at 3.76 and Science and Politics lowest at 3.54.The passage attributes Culture's strength to preserved cultural cues and weaker Science and Politics results to specialized terminology.Source: Gemini performance across domains · p. 5
  • Specialized terminology weakens Gemini's Science and Politics answers; for example, it answers “geology” instead of “pedology.”Source: Gemini performance across domains · p. 5

Cross-model comparison:

  • Culture has Gemini's largest domain decrease, from 3.76 to 2.42, consistent with translation obscuring details such as in the Rigveda question.Source: Cross-model comparison · p. 5
  • English phrasing can improve Science answers: the soil-science question is answered as “pedology” after translation, rather than “geology.”Source: Cross-model comparison · pp. 5–6
  • Geography is strongest in O3 at 2.94, suggesting questions centered on place names and proper nouns transfer more reliably across languages.Source: Cross-model comparison · p. 6
  • Gemma-27B scores 2.31–2.80 in O1, while Gemma-12B scores 1.63–2.27; both lag behind Gemini.Source: Cross-model comparison · p. 6
  • Smaller Hex-1 and Llama-3 models often score below 1.5; among open-weight models, Science is generally easiest and Culture hardest.The passage characterizes larger open-weight QA models as performing better while still lagging behind Gemini.Source: Cross-model comparison · p. 6

VI. CONCLUSIONS

VākQA establishes a challenging benchmark for Telugu spoken factoid question answering, with results shaped by language, input modality, pipeline design, and domain. The conclusions also identify limits in transcript translation and automatic judging.

VI. CONCLUSIONS

  • VākQA is the first released Telugu spoken factoid question-answering benchmark, and proprietary systems consistently outperform open-weight models while both remain challenged.The benchmark contains 2,001 question-answer pairs across six domains, as described in the paper context.Source: §VI · p. 6
  • Telugu text better preserves scope and specificity than English translation; speech can introduce meaning-changing acoustic confusions, and cascaded ASR→MT progressively compounds errors.These findings concern input formulation and pipeline design.Source: §VI · p. 6
  • Culture is hardest for open-weight models and most sensitive to translation, whereas Science and Geography transfer more reliably because terminology and proper nouns are stable.Source: §VI · p. 6
  • Open-weight judges can miss Telugu semantic equivalence, Gemini judging is lenient at low scores but stricter at high scores, and transliterated Science and Geography answers remain unevaluated for scoring consistency.The conclusions also caution that Gemini's non-uniform strictness limits fine-grained comparisons.Source: §VI · p. 6
  • YouTube audio requires faithful transcript-based translation rather than clarified rewrites, which can introduce scope ambiguity, as in the English rendering of a Telugu possessive pronoun.Source: §VI · p. 6
Loading 2609.19879v1…