PaperlayerView exact revision on arXiv

Source-linked AI summary

TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models

Zhaolu Kang, Yidi Wang, Yile Li, Siqi Zeng, Yingjie He, Richeng Xuan, Zhichao hu

arXiv:2610.30841v1cs.CL

TL;DR

Existing multimodal benchmarks largely test perception, recognition, or already determinate inputs, leaving open whether omni-modal models can resolve ambiguity distributed across text, image, and audio. TRIAD builds a checked benchmark of triplets whose answer is unique only with all three modalities. Models show a substantial scene-level gap from humans, especially without answer options.

  • Problem

    Existing benchmarks rarely test ambiguity that remains unresolved unless text, image, and audio are considered together.

    Source: §1 · p. 1

  • Method

    TRIAD uses a taxonomy and checked text–image–audio queries grouped into scenes, each requiring all three modalities for a unique answer.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    71.2 group-wise accuracy for the best model trails the human baseline of 91.6, while its query accuracy is 90.4; without options, models fall to low single digits.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TRIAD evaluates whether models jointly constrain interpretation rather than rely on a single cue.

    Source: §6 · p. 9

    TRIAD is small, underrepresents fully three-way ambiguous items, has narrower language and audio coverage than real-world use, and relies on human or judge-model decisions for checks and grading.

    Source: §6 · p. 9

Abstract

from arXiv · show

Ambiguity is a central feature of natural communication: a sentence, image, or sound may remain underspecified until evidence from the other channels is considered. Yet current multimodal benchmarks mostly test perception, recognition, or reasoning over already determinate inputs, leaving unclear whether omni-modal large language models can resolve ambiguity distributed across text, vision, and audio. We introduce TRIAD, a benchmark for omni-modal ambiguity resolution. Each item is a text--image--audio triplet paired with a question and a set of answer options, and is constructed so that the full triplet determines a unique answer while removing any one modality makes the set of consistent options non-unique. TRIAD contains $327$ hand-written queries in $86$ scene groups and covers $18$ ambiguity categories across the three modalities. Evaluating $14$ omni-modal systems reveals that strong item-level performance does not translate into robust scene-level disambiguation: the best model still trails humans by over $20$ points in group accuracy. Removing answer options widens the gap further, driving model group accuracy to the single digits while humans remain robust. Leave-one-out and transcript/description counterfactuals further show that current systems often underuse audio cues when text and image are present. TRIAD reframes omni-modal evaluation around disambiguation rather than recognition, and provides a taxonomy, benchmark, and diagnostic protocol for measuring this capability.

1 Introduction

TRIAD addresses whether omni-modal models can resolve interpretations that remain ambiguous until text, image, and audio are combined. It makes this capability measurable with checked triplets and scene groups, then evaluates 14 systems against humans.

1 Introduction

  • Existing benchmarks emphasize visual or text–image ambiguity, audio-visual recognition, or broad omni-modal tasks rather than ambiguity that requires all three modalities.The intended answer in TRIAD becomes unique only when text, image, and audio are considered together.Source: §1 · p. 1
  • Each TRIAD item pairs text, image, and audio with a query and options, and is constructed so every two-modality view remains ambiguous.Related queries form scene groups that perturb one modality, testing whether a model tracks the changing cue rather than reuses a scene prior.Source: §1 · pp. 1–2
  • TRIAD combines an 18-category taxonomy, 327 queries in 86 scene groups, and evaluations of 14 systems across prompts and modality conditions.Its contributions also include human and random baselines, category and tier breakdowns, and transcript/description counterfactuals.Source: §1 · p. 2
  • 71.2 group-wise accuracy for the best model trails the human baseline of 91.6, despite 90.4 query accuracy; removing options pushes every model to low single digits.Humans remain substantially more robust without options.Source: §1 · p. 2

2 Related work

Prior work studies ambiguity across visual, textual, and audio-visual settings, but does not center the tri-modal mutual-disambiguation problem targeted by TRIAD. TRIAD treats audio cues as evidence alongside language and vision.

2 Related work

  • HallusionBench [1], CODIS [2], MUCAR [3], 3AM [4], and UNPIE make ambiguity explicit, but focus on visual, textual, or text–image settings.Their setup differs from TRIAD, where no two-modality subset suffices to determine the intended answer.Source: §2 · pp. 2–3
  • Audio-visual benchmarks such as AVQA and AVHBench usually test recognition, while broad omni-modal evaluations include OmniBench and OmniMMI.TRIAD instead asks whether three incomplete cues combine into one unique interpretation.Source: §2 · p. 3
  • Prosody, speaker information, phonetic form, environmental sound, and non-speech symbols can signal intent, affect, reference, or scene context.TRIAD places these audio cues in a mutual-disambiguation task with language and vision rather than treating them as standalone labels.Source: §2 · p. 3

3 The TRIAD benchmark

TRIAD is a diagnostic benchmark for interpretations that remain ambiguous unless text, image, and audio are combined. Its task, item checks, taxonomy, scene groups, and intended-use guidance make cross-modal disambiguation the focus rather than recognition.

3.1 Task formulation

  • Each query combines textual context, an image, audio, a question, and answer options; the answer becomes unique only after all three modalities are combined.No two-modality view is sufficient, and each channel narrows the plausible answers.Source: §3.1 · p. 3

3.2 Mutual-disambiguation criterion

  • A query is well posed when the full triplet supports exactly one answer while every two-modality view remains non-unique.This rules out both underdetermined examples and examples reducible to only two modalities.Source: §3.2 · p. 3

3.3 Ambiguity taxonomy

  • The 18-category taxonomy assigns six ambiguity labels to each modality and supports diagnosis of where ambiguity enters an item.Each query receives one text, one image, and one audio label; these labels describe the evidence each channel contributes, not equal degrees of ambiguity.Source: §3.3 · p. 4

3.4 Scene groups and metrics

  • Scene groups contain sister queries that share a scenario but perturb a modality so the correct answer changes with the cue.Accg counts a group correct only when every sister is correct, making it the primary metric for whether a model follows the perturbation.Source: §3.4 · p. 4

3.5 Dataset overview

  • The release contains 327 queries in 86 groups across English and Chinese; its 230, 59, and 38 items span one-, two-, and three-modality ambiguity tiers.Every item still satisfies the well-posedness criterion, regardless of tier.Source: §3.5 · p. 4

3.6 Intended uses and non-uses

  • TRIAD is intended for diagnostic error analysis, not as a training set, general recognition or transcription benchmark, or replacement for broad multimodal evaluation.The paper recommends Accg as primary, Accq as secondary, and open-ended scores as a lower bound on free-response disambiguation.Source: §3.6 · pp. 4–5

4 Construction pipeline

TRIAD’s construction combines three complementary ambiguity-seeding routes with review procedures designed to ensure that only the full text–image–audio triplet resolves each item. The pipeline also uses annotator review and case examples to document its quality checks and ambiguity tiers.

4.1 Seeding strategy

  • Scene groups are seeded from linguistic, visual, or audio ambiguity, with the other modalities added to create contexts that disambiguate each case.Text-based construction extends ambiguous sentences with audio cues; image-based construction adds textual and prosodic information; audio-based construction builds textual and visual contexts around ambiguous clips.Source: §4.1 · p. 5

4.2 Per-modality content acquisition

  • Images are mainly licensed real photographs, while audio prioritizes natural recordings; AI-generated images and text-to-speech are used only under specified screening or editing conditions.Two annotators screen AI-generated images for common artefacts; text-to-speech prosody must be human-edited and confirmed natural.Source: §4.2 · p. 5

4.3 Well-posedness review

  • Five annotators check every candidate against modality subsets, requiring at least two consistent options for each strict subset and exactly one for the full triplet.The checklist targets non-unique full-triplet answers and items where one modality is decorative.Source: §4.3 · p. 5
  • Approximately 40 percent of candidate items are admitted after authoring, exclusion checks, well-posedness review, independent answer checking, and group integration.A Python reference checker for well-posedness verification is provided in the supplementary material.Source: §4.3 · p. 5

4.4 Annotator pool and inter-annotator agreement

  • Five annotators contributed; each query was reviewed by the other four, with held-out agreement of κ = 0.84 on answers and κ = 0.79 on ambiguity tags.Admission also required a correct labelled answer, distinctness from existing groups, and passing exclusion and well-posedness checks.Source: §4.4 · pp. 5–6

4.5 Case examples

  • Figure 3 presents six cases spanning single-, multi-, and tri-ambiguity, illustrating that only the full triplet uniquely determines each answer.The tiers correspond to one, two, or all three modalities being ambiguous.Source: §4.5 · p. 6, Figure 3 · p. 6

5 Experiments and Analysis

The experiments test tri-modal ambiguity resolution, consistency across related queries, and sensitivity to intended modality evidence. Across prompt formats and modality conditions, results show a gap between strong isolated answers and stable scene-level disambiguation.

5.1 Setup

  • The evaluation covers 14 omni-modal systems across three prompt formats and full, empty, single-modality, and leave-one-out input conditions.Accq measures isolated answers; Accg requires every sister query in a scene group to be correct and is the primary metric.Source: §5.1 · pp. 6–7, §5 · p. 6

5.2 Main diagnostic results

  • 90.4 Accq versus 95.1 for the human reference contrasts with 71.2 Accg versus 91.6, widening the reference gap from 4.7 to 20.4 points.The group metric exposes unstable scene interpretations that isolated query accuracy can hide.Source: Table 1 · p. 7, §5.2 · p. 7
  • 18.4 Accq and 4.6 Accg for the strongest model under open-ended answering fall from 90.4 and 71.2 under multiple choice, while human reference scores remain 88.4 and 79.2.The option list scaffolds selecting among interpretations, but removing it exposes weak free-response resolution.Source: §5.2 · pp. 7–8

5.3 What explains the failures?

  • As group size grows from 2 to 8, per-query accuracy shifts only from 70.5 to 68.0, while group accuracy falls from 51.8 to 8.9.The pattern points to inconsistent application of a disambiguation rule across controlled sister queries, not simply worse performance on larger groups.Source: Figure 4 · p. 8, §5.3 · p. 9
  • For every model in Table 2, accuracy is lowest on 3-ambiguous items; Gemini 3.1 Pro falls from 95.4 on 1-ambiguous items to 87.8 on 3-ambiguous items.Humans remain above 93 in all tiers; performance worsens when uncertainty is distributed across modalities rather than one dominant cue being available.Source: Table 2 · p. 8, §5.3 · p. 9
  • Gemini 3.1 Pro retains 57.2 Accg with no image and 51.8 with no audio, although TRIAD requires all three modalities, indicating shortcut dependence.The results suggest text often anchors an answer while image and audio serve as confirmation rather than evidence that can overturn it.Source: §5.2 · p. 8, §5.3 · pp. 8–9
  • Textualizing image or audio recovers only part of Full-input performance, and error analysis finds premature commitment across transcript-only listening, image anchoring, option priors, and cross-modal contradiction.Models may notice cues but commit before using all modalities to narrow the answer.Source: §5.3 · p. 9

5.4 Takeaway

  • Current systems often behave more like candidate selectors than stable tri-modal reasoners, struggling to maintain and update one interpretation as controlled modality cues change.The conclusion links the query/group gap, closed-to-open collapse, high ablated scores, and weak sister-group consistency to failures of joint cross-modal disambiguation.Source: §5.4 · p. 9

6 Conclusion

TRIAD provides a controlled test of tri-modal ambiguity, and its results show that models often fail to combine incomplete cues as consistently as humans. Its taxonomy spans text, image, and audio ambiguity, with audio collection rules intended to preserve cues beyond transcripts.

6 Conclusion

  • Models lose sister-group consistency and often retain high scores under ablations, while humans perform better with closed options and open-ended answers expose a wider gap.The conclusion characterizes this pattern as premature commitment rather than stable tri-modal reasoning.Source: §6 · p. 9
  • TRIAD’s scope is constrained by underrepresented fully three-way ambiguous items, narrower language and audio coverage, and checks that depend on human or judge-model decisions.The authors call for expansion across languages, cultures, and noisier audio settings, alongside stronger human-centered validation.Source: §6 · p. 9

A.1 Text ambiguity (6 categories)

  • The text taxonomy covers anaphoric, syntactic, logical, pragmatic, temporal, and lexical ambiguity.These categories include ambiguity in reference, parsing, scope, speech acts, event timing, and word senses.Source: §A.1 · p. 16

A.2 Image ambiguity (6 categories)

  • Image ambiguity includes semiotic, grounding, spatial, kinematic, observational, and contextual cases.The categories cover alternative sign readings, unclear object targets or layouts, motion, viewpoint, and scene context.Source: §A.2 · pp. 16–17

A.3 Audio ambiguity (6 categories)

  • Audio ambiguity includes referential, prosodic, indexical, phonetic, environmental, and symbolic cases.Examples include speaker-dependent references, stress or intonation, voice traits, confusable sounds, background context, and auditory symbols.Source: §A.3 · p. 17

A.4 Worked tri-modal example

  • The worked example pairs sarcastic or sincere speech, facial expression, and prosody so that only the full triplet resolves each sister’s answer.Removing any one modality leaves a relevant part of the interpretation unresolved.Source: §A.4 · p. 17

A.5 Per-leaf item counts

  • The released benchmark contains 327 queries, with per-leaf counts reported separately for each of its three modality taxonomies.The audio counts range from 98 referential items to 20 symbolic items.Source: §A.5 · p. 17

B.1 Hard rules; B.2 Sources

  • TRIAD audio must contribute information beyond the transcript; natural recordings and voice acting are preferred, with edited TTS used only when those sources fail.About a third of the audio comes from natural recordings, about three fifths from voice acting, and fewer than five percent from manually edited TTS.Source: §B.1 · p. 18, §B.2 · p. 18

B.3 Recording specification

The audio protocol specifies recording and review controls, then applies an admission workflow that checks modality necessity, answerability, and consistency across scene-group sisters.

B.3 Recording specification

  • Voice-acted clips use 48 kHz, 16-bit mono WAV audio, normalized to −23 LUFS and recorded in a treated room; clips remain as short as the cue permits.The protocol does not impose a fixed six-second cap because some cues require longer context.Source: §B.3 · p. 18

B.4 Naturalness check

  • Two independent listeners rate naturalness, prosodic clarity, and artefacts; clips averaging below 3.5 on any axis are re-recorded.Ratings use a five-point Likert scale.Source: §B.4 · p. 18

B.5 Sister-clip control within audio-perturbed groups

  • Audio-perturbed sisters are recorded by the same actor, in one session, on the same equipment, with amplitude matching to limit unrelated acoustic differences.The control is documented in per-group metadata.Source: §B.5 · p. 19

C.1 Annotator pool

  • Five annotators contribute to construction, including three NLP graduate students and two annotators with formal linguistics training.Annotators are familiar with CODIS and MUCAR; each is paid a market-rate stipend per authored query.Source: §C.1 · p. 19

C.2 Author–reviewer flow

  • Queries pass authoring, grammar, audio, subset-consistency, blind-answer, and group-integration checks; about 40% of candidates are admitted.Roughly a quarter fail at the modality-subset checklist stage and are revised or discarded.Source: §C.2 · p. 19

C.3 Inter-annotator agreement

TRIAD reports annotator agreement and formalizes both its ambiguity checks and a probe-based checker. The checker supports reproducible screening, but human annotation determines the labelled answer.

C.3 Inter-annotator agreement

  • Per-modality ambiguity-tier tagging reached κ = 0.79 on a held-out fifty-query audit, with disagreements concentrated at temporal/logical and prosodic/indexical boundaries.Per-leaf agreement and the disagreement matrix are released with the dataset.Source: §C.3 · p. 19

C.4 Reviewer guidance for degenerate ambiguities

  • Reviewers reject reducible items, replace manifestly false trick options, and reword option strings that leak modality-specific evidence.The guidance seeks to ensure that the remaining modalities and option wording do not give away the answer.Source: §C.4 · p. 20

D.1 Formal restatement

  • A query is well-posed when all modalities together leave one consistent option, but removing any single modality leaves more than one.The human checklist also inspects every strict subset of modalities to catch annotation mistakes.Source: §D.1 · p. 20

D.2 Reference well-posedness checker

  • The reference checker estimates option-consistency sets by querying models under forced choice and aggregating repeated samples; its default flags suspect items using two-modality views and leave-one-out checks.The defaults use K = 5 for two-modality views, K = 10 for the full triplet, and τ = 0.20.Source: §D.2 · p. 20

D.3 Intended uses of the checker

  • The checker can screen contributed items, characterize other benchmarks, and provide a reproducible second opinion, but it approximates consistency rather than defining correctness.The labelled answer remains determined by human annotation.Source: §D.3 · p. 20

E Supplementary experimental results

The appendix documents supplementary diagnostics and how scores are computed, including the shared query and scene-group sets and the distinction between query and group accuracy.

E Supplementary experimental results

  • The appendix collects detailed ablations, counterfactuals, robustness checks, secondary stratifications, and grading reliability that support the main experimental section.The main paper retains the full diagnostic matrix and a compact visual summary.Source: E Supplementary experimental results · p. 21

E.1 Result provenance and aggregation

  • Unless stated otherwise, scores use the same 327 query identifiers and 86 scene-group identifiers; group accuracy requires every sister query in a group to be correct.Query accuracy averages correctness across queries, while the open-ended mean-judge group score is the product of per-sister mean judge scores.Source: §E.1 · p. 21
  • The eight Standard-prompt conditions include Full, None, three single-modality inputs, and three leave-one-out inputs; Table 1 reports group accuracy and Table 3 reports query accuracy.Table 4 is a separate counterfactual run on the same queries for six representative model endpoints and the human row shown there.Source: §E.1 · p. 21

E.2 Detailed modality ablations

Table 3 presents the full query-accuracy matrix for the Standard prompt across input conditions, separating option-prior, single-modality, and leave-one-out behavior.

E.2 Detailed modality ablations

  • Table 3 reports query accuracy across the Standard-prompt input conditions, complementing the group-accuracy subset in Table 1.The matrix covers Full, None, single-modality, and leave-one-out conditions.Source: §E.2 · p. 21
  • The None condition isolates option-prior behavior, while single-modality conditions show performance with only one input channel.These conditions are distinct from leave-one-out tests, which retain two modalities.Source: §E.2 · p. 21
  • Leave-one-out conditions measure query accuracy when one modality is removed from the full input.The passage describes the matrix scope but does not provide its cell values.Source: §E.2 · p. 21

E.3 Description-ablation protocol

These appendices specify how modality descriptions and reference comparisons are constructed, then document robustness, grading, and secondary analyses. They also state how uncertainty and scope affect interpretation.

E.3 Description-ablation protocol

  • A+ audio notes include transcript, disambiguating prosody, speaker or voice properties, and non-speech events; I+ captions describe visible scene content.Descriptions are written once per item and reused unchanged; they test preservation of modality cues, not optimized prompting.Source: §E.3 · pp. 21–22, Table 4 · p. 22
  • Table 4 compares Standard-prompt Accq on the same 327 queries after replacing audio or images with human-written descriptions.Its parenthetical values are deltas from each model’s Full score; Table 5 details group-size, per-query, and group-wise accuracy.Source: Table 4 · p. 22, Table 5 · p. 22

E.4 Ambiguity tiers and sister consistency

  • Table 2 stratifies accuracy by the number of individually ambiguous modalities and includes selected ablation probes; Table 5 supplies group-size detail.The supplied passages identify these tables’ scope but do not give their cell values.Source: §E.4 · p. 22

E.6 Confidence intervals and significance testing

  • Confidence intervals use 10,000 group-level bootstrap resamples, with paired group-level comparisons corrected by Holm–Bonferroni.Accg differences of at least three points are significant at p < 0.05 after correction; smaller differences are treated as not significant.Source: §E.6 · p. 22

E.5 Benchmark comparison details

  • Table 6 compares prior benchmarks on induced ambiguity, tri-modal coverage, two-modality irreducibility, and perturbation grouping.The passages describe comparison dimensions, not benchmark-specific table entries.Source: §E.5 · p. 22, Table 6 · p. 23

E.7 Human and random reference points

  • Table 7 contrasts human and random reference rows with the best model, using empirical option counts and a cost-controlled human subset.Human results use 100 stratified queries, while model and random rows generally use all 327 queries; the comparison is not paired across the full benchmark.Source: Table 7 · p. 23, §E.7 · p. 23

E.8 Prompt robustness

  • Standard and Chain-of-thought answers are relatively stable across prompts compared with the larger multiple-choice-to-open-ended collapse.Table 8 also reports agreement and score spread across prompt variants.Source: §E.8 · p. 23, Table 8 · p. 24

E.9 Open-ended grading reliability

  • Open-ended accuracy is judged by two independent APIs, with strict scores requiring both judges to mark an answer correct.Judge-agreement statistics use parseable short answers only, whereas full-denominator scores count empty, refused, and malformed responses as incorrect; open-ended scores are treated as a lower-bound diagnostic.Source: Table 9 · p. 24, §E.9 · p. 24, Table 10 · p. 25

E.11 Manual error-analysis protocol

The analyses characterize modality dependence, group consistency, language variation, and representative failure cases. They also document the evaluation prompts and audio-input routes used in the benchmark.

E.11 Manual error-analysis protocol

  • The error analysis labels 200 sampled incorrect predictions by their earliest commitment leading to a wrong answer; percentages are descriptive diagnostics.Two annotators assign primary labels independently, resolving disagreements through discussion.Source: §E.11 · p. 25

E.12 Modality-importance and flip-rate analyses

  • Across models, text has the largest modality-importance score, vision is intermediate, and audio is small but positive on the strongest systems.The score is computed on the same 327 items; audio-perturbed groups also show a larger model–gold flip-rate gap than image-perturbed groups for every evaluated model.Source: §E.12 · p. 26

E.13 Language breakdown

  • For all 14 models, English–Chinese Accg differs by no more than two points and preserves model ranking.Chinese items are slightly harder for open-source omni models, while proprietary systems are roughly symmetric.Source: §E.13 · p. 26

F.1 Text-primary, syntactic leaf (extends MUCAR); F.2 Image-primary, kinematic leaf (extends CODIS)

  • The text-primary example changes its answer when prosody shifts from neutral delivery to contrastive stress, despite shared text and image.In the image-primary example, the frame is bistable and the audio prosody distinguishes upward from downward motion.Source: §F.1 · p. 27, §F.2 · p. 27

F.3 Audio-primary, prosodic leaf; F.4–F.8 Audio-primary leaves

  • Audio-primary examples vary prosody, speaker identity, referential stress, phonetic juncture, environmental alarms, and symbolic sounds to change the intended reading.The examples include a stress-shift sentence and contrasts in kinship interpretation, pronoun reference, “ice cream” versus “I scream,” alarms, and phone versus doorbell sounds.Source: §F.3 · p. 27, §F.4 · p. 27, §F.5 · p. 28

F.9 Failure-mode walkthroughs; F.10 Rendered case figures with model responses

  • A walkthrough illustrates cross-modal contradiction: a model follows sincere lexical content while disregarding sarcastic prosody and selects the wrong option.The figures provide rendered tri-modal cases, gold interpretations, and selected model responses as illustrations rather than quantitative evaluation.Source: §F.9 · p. 28, §F.10 · p. 28

G.1–G.5 Prompt templates

  • The Standard prompt uses closed options, Chain-of-thought requests stepwise reasoning, and Open-ended asks for a short answer without showing options.The two-modality checker substitutes a placeholder for the missing input; the three-modality checker samples full-triplet responses, while human reviewers make admission decisions.Source: §G.1 · p. 31, §G.2 · p. 31, §G.3 · p. 31

G.6 Notes on audio handling

  • Models without native audio receive a Whisper transcript alongside the audio path; native-audio models receive normalized 48 kHz mono WAV.The passage states that the transcript alone is insufficient to determine the answer.Source: §G.6 · p. 32

H Datasheet for datasets

TRIAD is designed as a held-out diagnostic of omni-modal ambiguity resolution, with controlled scene groups and tagged items for analysis. The datasheet describes its composition, intended use, and release practices.

H.1 Motivation

  • TRIAD was created to test whether omni-modal models integrate text, image, and audio when only the full triplet determines a unique answer.Each modality is intended to contribute non-decorative evidence.Source: §H.1 · p. 32

H.2 Composition

  • The dataset contains 327 queries in 86 scene groups, with 184 English and 143 Chinese items.Groups have sizes 2, 3, 4, or 8; the mean group size is 3.8.Source: §H.2 · p. 33

H.3 Collection process

  • Images come from named image sources under research-friendly licenses, and audio is human-recorded or human-curated.Collection details are referred to Sections 4 and Appendices B and C.Source: §H.3 · p. 33

H.4 Pre-processing

  • Audio is normalized to −23 LUFS at 48 kHz mono, while images are re-encoded as JPEGs with a 1024-pixel maximum edge.A small number of generated images and edited text-to-speech clips are retained when real sources do not capture the target ambiguity, and documented in metadata.Source: §H.4 · p. 33

H.5 Uses

  • The benchmark is recommended for held-out diagnostics, not training, because it is small and its construction favors diagnostic rather than representative items.Source: §H.5 · p. 33

H.6 Distribution

  • Images retain their original licenses with attribution, while audio and metadata are under CC-BY 4.0.The datasheet says hosting is planned through GitHub and Hugging Face Datasets at the submission-to-camera-ready transition.Source: §H.6 · p. 33

H.7 Maintenance

  • The authors commit to maintaining TRIAD for at least 24 months, including versioning, a leaderboard, errata responses, and reviewed community extensions.Extensions must satisfy the well-posedness checklist and pass five-reviewer cross-review.Source: §H.7 · p. 33

H.8 Ethical and legal

  • The datasheet excludes violence, privacy violations, and sensitive personal information, and says the benchmark should not be used to identify or impersonate real people.Voice actors signed distribution releases, and images carry licenses permitting research redistribution.Source: §H.8 · p. 33

I Reproducibility checklist

The checklist documents release materials and evaluation details, while recording reproducibility caveats, benchmark limitations, and the use of LLM-assisted content.

I Reproducibility checklist

  • Open-source evaluations used eight H100 80 GB GPUs and took about 40 hours per full pass; proprietary evaluations cost roughly 800 USD per pass.The checklist also reports evaluated models, input formatting, prompting, decoding, metrics, and dataset statistics.Source: I Reproducibility checklist · pp. 33–34, NeurIPS Paper Checklist · pp. 36–37
  • Evaluations use temperature 0 and a fixed seed; the well-posedness checker defaults to temperature 0.7 with K = 5 or 10 and threshold τ = 0.20.K = 5 applies to two-modality views and K = 10 to the full triplet; users can override the checker defaults.Source: I Reproducibility checklist · p. 34

NeurIPS Paper Checklist

  • The authors identify limited language and audio-source diversity, a small fully internally ambiguous tier, human-judgment dependence, and occasional audio truncation as constraints.All clips are at most six seconds, and reported truncation rates are below one percent for every evaluated model.Source: I Reproducibility checklist · p. 34, NeurIPS Paper Checklist · p. 35
  • The release provides anonymized data, metadata, prompts, evaluation scripts, and instructions, with dataset documentation covering its format, taxonomy, and intended use.The dataset and scripts are to be made public after acceptance.Source: I Reproducibility checklist · p. 33, NeurIPS Paper Checklist · pp. 36, 39
  • Statistical uncertainty is estimated with 95% bootstrap confidence intervals over scene groups using 10,000 resamples, with Holm–Bonferroni correction for paired group-level comparisons.The group-level unit reflects that sister queries within a scene group are not independent.Source: I Reproducibility checklist · p. 34, NeurIPS Paper Checklist · p. 37
  • LLMs assisted with some scenario descriptions, captions, answer options, and ambiguity annotations; authors manually reviewed, corrected, and validated that content.Source: NeurIPS Paper Checklist · p. 40
Loading 2610.30841v1…