PaperlayerView exact revision on arXiv

Source-linked AI summary

CORA: A Protocol for Diagnosing Boundary Robustness in Text-to-Audio Retrieval under Query Reformulations

Jae Min Woo, Kyongmin Kong, Bogyung Jeong, Minjeong Kim, HaeJun Yoo, Du-Seong Chang

arXiv:2609.33433v1cs.SDcs.CLeess.AS

TL;DR

Text-to-audio retrieval is commonly evaluated with caption-style queries, although the same intent can be phrased in different ways. CORA compares ranks for the same target across five intent-preserving reformulations, and its diagnostics associate rank degradation more strongly with target alignment and boundary degradation than with query movement.

  • Problem

    Caption-style evaluation does not show whether the same target’s rank declines when a query is reformulated to preserve intent.

    Source: §1 · p. 1

  • Method

    CORA rewrites each source caption into five intent-preserving forms while fixing the target audio and candidate pool, then tracks the target’s rank across forms.

    Source: §1 · p. 1, §3.3 · p. 3

  • Results

    Across 66,100 CORA pairs, RankDrop correlates with raw query movement at r = 0.084, TAL at r = 0.508, and TBMD at r = 0.615.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The results suggest that robust T2A retrieval requires preserving the target audio’s score advantage over competitive non-target audio under reformulation.

    Source: §6 · p. 9

    CORA uses LLM-generated caption reformulations and CLAP models with selected external retrievers; its diagnostics are not causal and may behave differently in other retrieval settings.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Text-to-Audio (T2A) retrievers are typically evaluated with caption style queries, but the same user intent can be expressed in many forms. We introduce CORA (Caption-Offset Retrieval for Audio), a caption anchored diagnostic protocol that rewrites each source caption into five intent preserving forms (Command, Question, Indirect, Key phrase, and Statement) while fixing the target audio. By tracking the same target across query forms, CORA defines RankDrop, a metric revealing failures hidden by Recall@k. Using Pearson's correlation coefficient r, we find that RankDrop is weakly associated with raw text space movement (r=0.084), but strongly associated with Target Alignment Loss and Target Boundary Margin Degradation (r=0.508 and r=0.615). The same pattern appears in OEA retrievers, where RankDrop is better explained by boundary degradation (r=0.472/0.478) than by query movement (r=0.084/0.046). Overall, these results suggest that robust T2A retrieval requires preserving the target's boundary advantage over competing audio under reformulation.

1 Introduction

CORA tests whether text-to-audio retrievers preserve the same target when caption queries are rewritten in five intent-preserving forms. Its diagnostics associate rank degradation more strongly with target alignment and boundary degradation than with raw text-space movement.

1 Introduction

  • Standard Recall@k reports aggregate performance and cannot show whether a particular target’s rank declines when query form changes.The paper situates its diagnostic question alongside work on query-form variation and robustness.Source: §1 · p. 1
  • CORA fixes the target audio and candidate pool while comparing source captions with five intent-preserving query forms.The forms are Command, Question, Indirect, Key phrase, and Statement.Source: §1 · pp. 1–2
  • Query-form sensitivity depends on the model–dataset regime, so a form that is stable in one setting can be harmful in another.The paper frames CORA as a diagnostic for residual failures under intent-preserving reformulation.Source: §1 · pp. 1–2
  • Across 66,100 CORA pairs, RankDrop correlates weakly with raw query movement (r = 0.084) and more strongly with TAL (r = 0.508) and TBMD (r = 0.615).The same boundary-degradation pattern is reported for robustness-oriented retrievers.Source: §1 · pp. 1–2

2 Related Work

The related work covers representational limits in audio-language models, query-form robustness, and embedding geometry. CORA takes an instance-level diagnostic approach to residual failures rather than proposing a new training objective or architecture.

2.1 Audio-Language Models

2.2 Query Robustness in Audio and Text Retrieval

  • Where prior work studies aggregate robustification or macro-level evaluation, CORA tracks whether a specific target remains retrievable under systematic query-form shifts.CORA focuses on diagnosing residual failures through directional geometry, not on new training objectives or architectural changes.Source: §2.2 · p. 2

2.3 Contrastive Embedding Geometry and Hard Negatives

  • The paper uses hard negatives as inference-time probes of whether query shifts move toward the target and away from the decision boundary with its nearest competitor.This connects form-shift diagnostics to embedding geometry and hard-negative analysis.Source: §2.3 · p. 2

3 Method

CORA pairs source captions with semantically aligned reformulations and evaluates rank changes for the same target audio. It also defines embedding-space diagnostics for query movement and changes in the target’s advantage over hard negatives.

3.1 Caption-Offset Retrieval for Audio (CORA)

  • CORA evaluates five query types—Command, Question, Indirect, Key phrase, and Statement—to represent varied formulations of the same intent.Its taxonomy adapts UIQ and is used to evaluate retrieval robustness.Source: §3.1 · pp. 2–3

3.2 CORA Generation

  • CORA introduces Indirect as an independent query type and encourages broader formulation diversity than fixed opening patterns.The authors also verify that generated queries preserve source-caption semantics.Source: §3.2 · p. 3

3.3 Metric

  • RankDrop compares the same target audio’s 1-indexed rank under source and CORA queries, encoded directly without preprocessing or rewriting.Positive values indicate a lower target rank after reformulation; negative values indicate an improved rank.Source: §3.3 · p. 3

3.4 Embedding-Space Diagnostic Metrics

  • The text-space movement baseline measures how far the CORA query moves from the source query in text-embedding space.Target Boundary Margin Degradation measures the reduction in the target’s score advantage over its hard negative, decomposed into Target Alignment Loss and Hard Negative Pull.Source: §3.4 · pp. 3–4

3.5 Directional Projection Diagnostics

  • Directional projection diagnostics compare query-shift magnitude with projections onto the target-audio and source-anchored target-boundary directions.These projections are analyzed as associations with RankDrop.Source: §3.5 · p. 4

4 Experiment

CORA evaluations show that rank changes vary across query forms, datasets, and retrievers, while target alignment and boundary signals are more strongly associated with RankDrop than query movement. Projection analyses and tests on robustness-oriented retrievers support the same target-boundary pattern.

4.1 Experimental Setup

  • Across four datasets and four CLAP models, the paired evaluation uses five reformulations; 3,367 test items become 3,305 after excluding 62 with incomplete query forms.This yields 66,100 paired evaluations; the main diagnostic uses within-dataset candidate pools, while Figures 1 and Tables 2–3 use a pooled cross-dataset candidate set for sensitivity analysis.Source: §4.1 · p. 4, Table 1 · p. 5

4.2 Query-Form Sensitivity Depends on Model-Dataset Regime

  • Statements have the lowest average MeanRD and indirect queries the highest, but Clotho key phrases reach 114.92 versus MECAT −39.55, and model-specific rank changes also reverse within datasets.For example, on AudioCaps MS-CLAP has MeanRD −109.96 while LAION, M2D, and MGA show positive MeanRD.Source: Figure 1 · p. 4, §4.2 · p. 4, Table 2 · p. 5

4.3 TBMD Characterizes Rank Degradation Better than Query Movement

  • CORA compares query movement with target-alignment and target-boundary accounts, with dynamic-HN TBMD combining reduced target similarity and increased competition from a hard negative.The authors report that dynamic-HN TBMD associates with RD more strongly than two fixed hard-negative alternatives.Source: §4.3 · pp. 5–6

4.3.1 Main Diagnostic Analysis

  • Per-dataset, ∆move correlates weakly with RD (r = 0.084), versus TAL (r = 0.508) and TBMD (r = 0.615); TBMD’s score-derived association is diagnostic, not causal.On the matched UIQ subset, the ordering also holds: ∆move r = −0.075, TAL r = 0.508, and dynamic-HN TBMD r = 0.538.Source: §4.3.1 · p. 6

4.3.2 TBMD-Guided View Intervention

  • Choosing the minimum-TBMD view yields MeanRD −19.467, below minimum-TAL’s −18.568 and near the best-rank oracle’s −24.560; these target-aware oracles are not inference-time methods.By comparison, choosing the text-space-closest view yields MeanRD 1.875, versus 3.881 when averaging over all CORA forms.Source: §4.3.2 · p. 7, Table 4 · p. 7

4.4 Projection Analysis Validates the Directional Boundary Hypothesis

  • Projection features correlate more strongly with RD than shift magnitude: ρ = 0.594 for p_audio and ρ = 0.529 for p_boundary, supporting a directional account of target alignment and boundary preservation.The analysis describes p_audio as alignment with the target-audio direction and p_boundary as preserving separation from nearby hard negatives.Source: §4.4 · pp. 7–8

4.5 TBMD Characterizes Residual Rank Degradation in Query-Robust Retrievers

  • OEA variants reduce CORA degradation to MeanRD near zero, while RobustCLAP shows larger MeanRD; across all three, TBMD correlates more strongly with RD (0.472–0.508) than query movement (0.046–0.174).The authors interpret residual degradation in these external retrievers as following the same target-boundary pattern as in the CLAP models.Source: §4.5 · p. 8

5 Discussion

CORA pairs each source query with intent-preserving reformulations while tracking whether the same target audio is retained, a view that aggregate Recall@k does not provide. The discussion identifies target alignment and boundary degradation—not raw query movement—as more informative signals of RankDrop, while cautioning against treating them as inference-time objectives.

5 Discussion

  • CORA’s paired view checks whether the same target audio changes rank after reformulation, unlike aggregate Recall@k.Query-form sensitivity varies across model–dataset regimes rather than following a fixed ordering of surface forms.Source: §5 · p. 8
  • Target Alignment Loss and Target Boundary Margin Degradation are more informative RankDrop signals than raw text-space movement.They measure weakened target alignment and reduced score advantage over a Hard Negative; the findings do not establish them as inference-time objectives.Source: §5 · p. 8
  • Inference-time score fusion can improve aggregate R@5 without resolving query-form brittleness; preprocessing lowers MeanRD but preserves the TAL/TBMD-over-movement ordering.The discussion argues that robustness methods should be evaluated on preserving the target’s margin over competitive non-target audio under reformulation.Source: §5 · p. 8

6 Conclusion

CORA tests whether T2A retrievers preserve the same target audio under intent-preserving reformulations. Its findings associate residual RankDrop more strongly with target alignment and boundary degradation than with raw text-space movement, including in external robustness-oriented retrievers.

6 Conclusion

  • Across four datasets and multiple retrievers, query-form sensitivity depends on the model–dataset regime rather than an intrinsic property of a surface form.CORA is a caption-anchored diagnostic protocol for testing preservation of the same target audio under intent-preserving reformulations.Source: §6 · p. 9
  • Residual RankDrop is more closely associated with Target Alignment Loss and Target Boundary Margin Degradation than with raw text-space movement.The same pattern appears in external robustness-oriented retrievers, including OEA and RobustCLAP.Source: §6 · p. 9
  • Robust T2A retrieval requires preserving the target audio’s score advantage over competitive non-target audio under reformulation.CORA provides a testbed for analyzing target-boundary behavior and developing future boundary-aware robustness methods.Source: §6 · p. 9

Limitations

CORA’s controlled caption-reformulation design, generated query set, and evaluation scope define the boundaries of its evidence. The accompanying checks support query quality and five-form coverage, while not extending the diagnostic to unconstrained or specialized audio-search settings.

Limitations

  • CORA’s results are limited by LLM-generated queries, controlled caption reformulation, and evaluation on CLAP models and selected external retrievers.Generated queries may contain artifacts and underrepresent multilingual queries; the diagnostic does not cover unconstrained intent, multi-target relevance, or specialized domains, and TAL/TBMD are diagnostic rather than causal signals.Source: Limitations · p. 9

A.1 CORA taxonomy

  • The five query forms are Command, Question, Indirect, Key phrase, and Statement, covering imperative, interrogative, request-style, keyword-centric, and declarative formulations.The examples preserve the same sound-event content while varying query form.Source: §A.1 · p. 11

A.2 CORA Generation

  • CORA uses a training-free prompting strategy to diversify openings, and the evaluation set contains 3,305 test items with all five non-empty forms.The 5,567 source captions come from Clotho, AudioCaps, MECAT, and MACS; the final set excludes items with an empty query form.Source: §A.2 · p. 11

A.3.1 Query Quality Verification

  • Human verification found 98.3% semantic alignment and 97.7% query-type validity; overlapping annotation yielded mean pairwise Cohen’s κ of 0.970.The overlapping check recorded six single-annotator Task 2 errors, mostly at boundaries between query types.Source: §A.3.1 · p. 12

A.3.2 Five-Form Taxonomy Validation

  • Across all five leave-one-form-out tests, TBMD (r = 0.607–0.625) and TAL (r = 0.494–0.526) remained more strongly correlated with RankDrop than ∆move (r = 0.060–0.098).The pattern persisted regardless of which query form was excluded.Source: §A.3.2 · p. 12

A.3.3 Real-World Query Coverage

  • 97.6% of WildClaims information-seeking queries fit the five forms; CORA uses balanced reformulations to isolate query-form effects, not to reproduce real-world frequencies.The remaining 2.4% were classified as None, often because they lacked a clear or typical query form.Source: §A.3.3 · p. 12

A.4.1 Experimental Setup

The appendix compares CORA variants’ semantic diversity and same-pool retrieval behavior, alongside the experimental setup and diversity measures. The comparisons report gains in diversity from the taxonomy and prompting strategy, while the retrieval comparisons are explicitly descriptive.

A.4.1 Experimental Setup

  • The diversity comparison uses samples from AudioCaps, Clotho, and MECAT, with approximately 300 queries per type; MACS is excluded because it is absent from UIQ.UIQ is the original query benchmark, while CORA(w/o Prompt) removes the diversification strategy from CORA’s taxonomy.Source: §A.4.1 · p. 12

A.4.2 Metrics

  • Global diversity averages pairwise cosine distance between query embeddings; within-type diversity measures variation among queries of the same type.Within-type analysis covers command, indirect, and question queries because the prompting strategy is applied only to those forms.Source: §A.4.2 · p. 13

A.4.3 UIQ vs. CORA vs. CORA(w/o Prompt)

  • Global semantic diversity rose from 0.693 to 0.741 for CORA(w/o Prompt) versus UIQ, then from 0.741 to 0.750 with full CORA.The first comparison reflects the expanded query taxonomy; the second compares variants sharing that taxonomy and is attributed mainly to diversification prompting.Source: Figure 4 · p. 13, §A.4.3 · p. 13
  • Command-query diversity increased from 0.689 to 0.751 with diversification prompting, the largest within-type gain reported.Question queries showed smaller gains, with their yes/no structure described as constraining formulation variability.Source: §A.4.3 · p. 13, Figure 4 · p. 13

A.5 Same-Pool Retrieval Comparison

  • MeanRD was +1.10 for CORA versus CORA without diversification (95% cluster-bootstrap CI [0.75, 1.45]) and +11.06 versus UIQ ([9.35, 12.82]).These are descriptive comparisons on 2,805 test items with matched per-dataset candidate pools; the query sets differ in generation conditions and, for UIQ, query types.Source: Table 11 · p. 14, §A.5 · p. 14

B Experimental Setup Details

  • The retrieval study covers AudioCaps, Clotho, MACS, and MECAT and evaluates four CLAP-family retrievers without additional fine-tuning.The passage notes potential source overlap with WavCaps for AudioCaps and Clotho, while MECAT and MACS are described as relatively leakage-free benchmarks.Source: B Experimental Setup Details · p. 14

C Retriever-Level CORA Evaluation

Retriever-level CORA results show that aggregate recall changes can obscure paired target rank degradation, and that query-form sensitivity depends on the dataset–model regime. Controls support dynamic top-1 TBMD and show the TAL–RD association remains positive when the anchor or described content changes.

C Retriever-Level CORA Evaluation

D Full Dataset-by-Model-by-Form Retrieval Results

  • A query form stable for one dataset–model pair can be harmful for another, so aggregate ∆R@5 should be interpreted together with paired MeanRD.The full breakdown reports aggregate Recall@5 and paired MeanRD across dataset, model, and query form.Source: D Full Dataset-by-Model-by-Form… · p. 15, Table 14 · p. 16

E Interaction Analysis for Query-Form Sensitivity

  • Interaction terms improve RD prediction over additive model, dataset, and query-form terms, with the full model–dataset–query cell giving the highest correlation.This supports sensitivity being conditional on the model–dataset regime, not universal to a query form.Source: E Interaction Analysis for Query-Form… · p. 15

F Hard Negative Definition Sensitivity

  • Dynamic top-1 TBMD has the strongest association with RD because each query is compared against its own most competitive non-target audio.The fixed-source alternative can miss a competitor that becomes salient only after reformulation.Source: F Hard Negative Definition Sensitivity · p. 15

G Anchor and Content Controls for Target Alignment Loss

  • TAL–RD association weakens but remains positive when the source-query anchor is removed or acoustic content is fixed.These controls are robustness checks, not replacements for the primary source-query-anchored CORA setting.Source: G Anchor and Content Controls for… · pp. 16–17

H TBMD Control Analysis

The control analysis tests whether TBMD contributes diagnostic information beyond TAL and measured confounds. TBMD improves RD prediction over TAL alone, but the association is treated as diagnostic rather than causal.

H TBMD Control Analysis

  • The analysis uses grouped cross-validated Ridge regression to compare TAL alone against TAL plus dynamic-HN TBMD, with fixed effects for dataset, model, and query form.The pooled and per-dataset settings each use 66,100 pairs.Source: H TBMD Control Analysis · p. 17
  • Adding dynamic-HN TBMD to TAL improves RD prediction in both settings, and TBMD retains a positive partial correlation with RD after measured confounds are controlled.The pooled partial association is smaller than in the per-dataset setting.Source: H TBMD Control Analysis · p. 17
  • TBMD is treated as an additional diagnostic signal, not evidence that TBMD alone drives rank degradation.Its partial association remains positive after residualizing out TAL, dataset, model, query form, ∆move, source rank, and pool size.Source: H TBMD Control Analysis · p. 17

I Boundary-Margin Token Saliency Audit

The token-saliency audit finds that rank-worsened pairs have weaker target boundary margins, while coarse token categories do not consistently explain rank degradation. The authors treat saliency as descriptive rather than causal evidence.

I Boundary-Margin Token Saliency Audit

  • Across all four models, rank-worsened pairs have lower mean target boundary margins than rank-improved or stable pairs.The mean margin changes from −0.0928 to −0.2189 for LAION, −0.0697 to −0.1250 for MS-CLAP, −0.0169 to −0.0430 for M2D, and −0.0459 to −0.1138 for MGA.Source: I Boundary-Margin Token Saliency Audit · p. 19
  • No token category separates rank-worsened pairs consistently across all models, and token-group shares do not provide a stable explanation of RD.Form- is slightly higher for worsened pairs in LAION and MGA, but lower in MS-CLAP and M2D.Source: I Boundary-Margin Token Saliency Audit · p. 19
  • Token-group saliency alone predicts continuous RD weakly, and adding it to boundary features does not improve prediction.In the pooled setting, saliency alone reaches Pearson r = 0.090 and Spearman ρ = 0.037; boundary features reach r = 0.634 and ρ = 0.591.Source: I Boundary-Margin Token Saliency Audit · p. 19

J Dataset-Level Diagnostic Correlations

Dataset-level results support target-alignment and boundary-margin diagnostics over raw query movement. Projection analyses find that directional features predict RD more strongly than movement magnitude, while score fusion improves aggregate retrieval metrics but is not evidence of query-form robustness.

J Dataset-Level Diagnostic Correlations

K.2 Hard Negative Selection

  • The projection analysis selects a source-anchored hard negative from competitors retrieved in either the source or CORA query neighborhood.The union pool includes competitors salient before or after reformulation, while selection under the source query preserves the pre-reformulation anchor.Source: §K.2 · p. 21

K.3 Regression Details

  • The projection evaluation covers four datasets, four CLAP-family retrievers, and five CORA forms, using continuous RD and grouped cross-validation by audio.The split prevents forms derived from the same audio sample from appearing in both training and test folds.Source: §K.3 · p. 21

K.1 Mathematical Formulation

  • Projection features test whether the source-to-CORA embedding displacement points in a target-preserving or boundary-preserving direction, complementing realized TAL and TBMD changes.The boundary projection estimates whether the shift increases or decreases the local target-minus-hard-negative score difference.Source: §K.1 · pp. 20–21, §K.4 · p. 21

K.4 Interpretation

  • Adding movement magnitude gives only a small RD-prediction improvement, while target-audio and boundary projections provide substantially stronger prediction.This supports the directional boundary hypothesis rather than an account based simply on query displacement distance.Source: §K.4 · p. 21

L Additional Inference-Time Fusion Check

M Query Preprocessing

The preprocessing experiments test whether external rewriting can recover retrieval performance under CORA reformulations and whether the diagnostic pattern persists. CORA-aware rewriting reduces average rank degradation, but its gains are uneven across items and query forms.

M Query Preprocessing

  • The evaluation uses 500 balanced source items from four datasets, all five CORA query forms, four CLAP retrievers, and each dataset’s full candidate pool.The 125 items per dataset come from Clotho, AudioCaps, MECAT, and MACS.Source: M Query Preprocessing · p. 22
  • The three conditions are direct retrieval without canonicalization, general rewriting without CORA-form information, and CORA-aware rewriting guided by the query form and its definition.Because real-world systems may not know a query’s form, CORA-aware rewriting is treated as a type-aware diagnostic condition rather than a directly deployable method.Source: M Query Preprocessing · p. 22
  • CORA-aware rewriting reduces MeanRD from +6.33 to −0.03 and increases R@5 from 37.33% to 40.02%.General rewriting reduces MeanRD by 2.18 ranks (CI [0.27, 4.15]); CORA-aware rewriting reduces it by 6.36 ranks (CI [4.86, 7.91]).Source: M Query Preprocessing · p. 22, Table 26 · p. 22
  • CORA-aware rewriting improves MeanRD for 377 of 500 items, worsens it for 118, and leaves 5 unchanged; Key Phrase MeanRD rises from +3.03 to +3.78.Key Phrase R@5 still improves.Source: M Query Preprocessing · p. 22
  • After preprocessing, query movement remains weakly associated with RankDrop while TAL and TBMD remain more strongly associated with it.For CORA-aware rewriting, the reported correlations are r(∆move, RD) = 0.020, r(TAL, RD) = 0.546, and r(TBMD, RD) = 0.683.Source: M Query Preprocessing · p. 22

N.1 Query Prompt Definitions

  • The query prompts create five forms: yes/no questions, direct commands, polite indirect requests, noun-phrase key phrases, and complete descriptive statements.The prompts instruct each form to preserve important audible content.Source: §N.1 · p. 23

N.2 Real-User Query Classification Prompt

  • The real-user classification prompt classifies queries by linguistic form and intent, not by factuality..Source: §N.2 · p. 24

N.3 Query Preprocessing Prompts

  • The general rewriting prompt asks for clearer audio-search queries that preserve intent and audible content, while the CORA-aware prompt also supplies query type and requests a caption-style rewrite.Source: §N.3 · p. 25
Loading 2609.33433v1…