PaperlayerView exact revision on arXiv

Source-linked AI summary

AcoustiClaim: A Numeric Claim Benchmark with Instrument Ground Truth

Sheng-Tse Lin, Siyuan Zhai, Chien-Liang Kuo, Massa Baali, Bhiksha Raj

arXiv:2609.30483v1cs.SDcs.CLcs.LGeess.AS

TL;DR

Speech models often describe recordings with numeric claims such as SNR or jitter, and readers treat these numbers as measurements of the audio. AcoustiClaim extracts numeric claims from free text, scores them against quantity-defining instruments, and classifies whether each reference is present in the model input. Only 8 of 158 rankable cells exceeded the 0.3 bar, and calibrated withholding reduced mean mixture error on all ten quantities, while a linear baseline ordered errors at least as well.

  • Problem

    Speech models often describe recordings with numeric claims such as SNR or jitter, and readers treat these numbers as measurements of the audio.

    Source: §1 · p. 1

  • Method

    AcoustiClaim extracts numeric claims from free text, scores them against quantity-defining instruments, and classifies whether each reference is present in the model input.

    Source: §1 · p. 1

  • Results

    Only 8 of 158 rankable cells exceeded the 0.3 bar, and calibrated withholding reduced mean mixture error on all ten quantities, while a linear baseline ordered errors at least as well.

    Source: §1 · p. 1, §4 · p. 4

  • Takeaways & Limitations

    The findings show that instrument-grounded scoring can reveal poor numeric tracking and that abstention can reduce mixture error, without establishing better error ordering than a linear baseline.

    Source: §5 · p. 4, §4 · p. 4

Abstract

from arXiv · show

Audio language models state numbers for acoustic quantities, and neither human opinion nor a judge model says whether such a number is true of the signal. AcoustiClaim extracts each numeric claim from free text, scores it against the instrument that defines the quantity, and classes each quantity by where its reference can be read. Four open-weight systems and one closed model, asked for ten quantities five ways on two corpora, fill 207 cells. Of these, 49 emit fewer than five distinct values, and eight of the 158 cells that can be ranked exceed a rank correlation of 0.3, the bar we set, three with an interval clear of it, five of them one closed model reading pitch. Error sits at or above a constant-predictor floor in every ranked cell but three. The reference decoder we train declines the five voice quantities in prose on 95% of mixtures, with nothing withheld, and states them on the clean twins, reproducing its targets' rule from audio alone. With a calibrated threshold, withholding lowers error on all ten quantities on the mixtures in the mean and on eight at every split, against at most 0.6% from a random selector. A linear baseline orders errors at least as well as ours. F0 s.d. and shimmer stay above the constant floor.

1. INTRODUCTION

Audio language models often state acoustic numbers that readers take as measurements, but existing evaluations do not establish whether the numbers match the signal. AcoustiClaim scores free-text claims against defining instruments and classifies whether each reference is present in the model input.

  • Existing evaluations compare numbers with opinion scores or judge models, while readers take stated acoustic quantities as measurements of the audio.Source: §1 · p. 1
  • AcoustiClaim scores numeric claims against the instrument that defines each quantity and tiers quantities by whether their reference is present in the model input.Source: §1 · p. 1
  • The benchmark pairs a parser for numeric claims in free text with a reference system that carries the quantity tiering.Source: §1 · p. 1
  • Only 8 of 158 rankable cells exceeded the 0.3 bar, and error sat at or above the constant-predictor floor in all but three.Source: §1 · p. 1
  • With calibrated withholding, mixture error fell on all ten quantities in the mean and on eight at every split; random selection moved error by at most 0.6%.Source: §1 · p. 1
  • With nothing withheld, the reference decoder declined five voice quantities in prose on 95% of mixtures and stated them on clean twins.Source: §1 · p. 1

2. METHODOLOGY

The benchmark pairs acoustic claims with instrument-defined references across two corpora, then evaluates a reference system that predicts values and withholds uncertain claims. Its design distinguishes how each reference is obtained and how prediction error behaves for different quantity forms.

2.1. Benchmark

  • Libri2Mix mixtures with reverberation and independently mixed noise and AMI distant-microphone meetings form the two evaluation corpora.Each Libri2Mix mixture is paired with its clean twin; the reference system trains on Libri2Mix train-clean-100, making its AMI evaluation zero-shot.Source: §2.1 · pp. 1–2
  • References are tiered by whether they are fixed by the heard signal, mixing parameters, unseen clean stems, or mixture regions outside detected overlap.The mixture F0 references use masked windows, while AMI voice and timing references are read on the same distant-microphone channel the models hear.Source: §2.1 · p. 2
  • The parser extracts 13 quantities from prose by quantity name and unit, but only ten instrument readings are scored.Descriptions with no parsed value lower coverage.Source: §2.1 · p. 2
  • Evaluation reports coverage, selective risk, normalized MAE, and Spearman correlation, with cells screened for at least 25 claims, five distinct values, and ρ > 0.3.The constant predictor is the median reference and is recomputed on the retained subset when a threshold applies.Source: §2.1 · p. 2

2.2. Reference system

  • The reference system combines a frozen layer-7 WavLM encoder, convolutional compressor and adapter, and a LoRA-tuned Qwen3-8B decoder with value and confidence heads.The value and log-variance heads are trained with heteroscedastic Gaussian negative log-likelihood, scaled by each quantity’s training-set dispersion.Source: §2.2 · p. 2
  • Each stated value is an exact sum of per-token terms; sum-form quantities have bounded accumulated error, whereas quotient-form error is unbounded.For sums or rates, the bound depends on clip length and per-frame error; for rates it is flat in clip length, while count error grows linearly.Source: §2.2 · p. 2
  • A claim is stated only below a quantity-specific predicted log-variance threshold, calibrated to 0.75 coverage on held-out data and applied to evaluation splits.Readout A uses the decoder’s own coverage; readout B evaluates the confidence head at the calibrated cut, with Table 2 covering parsed mixture-only outputs on unseen test clips.Source: §2.2 · pp. 2–3, Table 2 · p. 3

3. EXPERIMENTAL SETUP

The external panel compares five audio language models across ten quantities, five elicitation styles, and two corpora. Its sampling and parsing procedure defines which cells enter the benchmark counts.

3. EXPERIMENTAL SETUP

  • The panel contains four open-weight systems and one closed model, each asked for ten quantities using five elicitation styles.The styles range from free descriptions and direct number requests to named quantities with units, a template, and a worked example.Source: §3 · p. 3
  • Eight sampling seeds each draw 120 clips under greedy decoding, yielding 4,800 generations per system and corpus pooled by clip.The pooled sample contains at most 839 Libri2Mix or 859 AMI panel clips.Source: §3 · p. 3
  • Truncation affects 0.7–15% of generations, while 55.6% parse to nothing; Granite-Speech-8B supplies no AMI cell meeting the 25-claim minimum.Its AMI row therefore enters no count.Source: §3 · p. 3

4. RESULTS AND DISCUSSION

The external panel yields few above-threshold rank correlations, while selective prediction lowers mixture error; a linear selector remains competitive. Internal diagnostics show that the reference decomposition and its predictability vary by quantity.

Tiers

  • On mixtures, SRMR and SNR hold, timing rows sit 0.11–0.18 below pooled values, and the five voice rows use clean twins.The estimator suite reproduces SRMR exactly and recomputes jitter and shimmer with intervals containing zero.Source: §4 · p. 3

External panel

  • 8 of 158 ranked cells exceed the 0.3 bar; only AMI F0 mean cells clear it cleanly, with +0.629 [0.57, 0.68] the largest.Five of the eight are Gemini 3.8 Flash reading pitch; three of five systems clear the bar when the five cells whose intervals contain it are counted, and one without them.Source: §4 · p. 3

Tiers

  • The matched ridge ranks better on seven of ten rows, five by under 0.05; nMAE favors ours on a different seven, by 0.075 or more on channel rows and pause count.The matched ridge is a linear probe on the same frozen encoder, fit on the clips retained for each row.Source: §4 · p. 3

Where the reference sits

  • The decomposition diagnostic clears chance for HNR (6.06 times), jitter (4.45), and speaking rate (4.05), while F0 s.d. reaches 1.63.SNR, F0 mean, and shimmer sit near chance; pause references are dropped before scoring because a frame-index ramp reaches 97% of the best agreement, above the 70% allowance.Source: §4 · p. 3
  • In Fig. 2, half the reference magnitude falls within 0.1–4 s, while the token decomposition holds 0.76–1.30 times an even split on each column.Boxes mark the shortest window containing half the reference magnitude; peaks show frame height relative to an even split.Source: §4 · p. 3, Figure 2 · p. 4

Withholding

  • On mixtures, withholding lowers risk for all ten quantities in the mean and for eight on every split; Fig. 3 shows reductions across 16–42% withheld.Across mixtures and twins, all ten quantities reduce risk on all five splits; a random selector at the same coverage moves risk by at most 0.6%.Source: Figure 3 · p. 4, §4 · p. 4
  • A residual-ridge selector orders errors at least as well as the confidence head, but the assembled system has the better mean risk-coverage predictor.The residual ridge wins 33 of 50 quantity-and-seed comparisons on mixtures against none for ours, and 15 of 50 against our one pooled win.Source: §4 · p. 4
  • With the confidence head’s calibrated cut, F0 s.d. (nMAE 1.093) and shimmer (1.055) remain above the constant-predictor floor.The other eight quantities sit below the floor on readout B.Source: §4 · p. 4

5. CONCLUSION

AcoustiClaim evaluates free-text acoustic numbers against the instruments that define them and classifies whether their references are available in the input. Most systems seldom exceed the benchmark’s correlation bar, while calibrated withholding lowers mixture error but does not outperform a linear baseline in error ordering.

  • AcoustiClaim scores each numeric claim against its defining instrument and records whether that quantity’s reference is present in the input.Source: §5 · p. 4
  • Eight of 158 rankable cells exceed the 0.3 bar, and error is at or above the constant-predictor floor in all but three.Source: §5 · p. 4
  • A calibrated threshold lowers mixture error on all ten quantities, but its advantage over a self-abstaining linear baseline lies in the mean predictor, not error ordering.Source: §5 · p. 4
  • All clips come from LibriSpeech or AMI; no participants were recruited, no new audio was recorded, and no audio is redistributed.Source: §5 · p. 5
Loading 2609.30483v1…