PaperlayerView exact revision on arXiv

Source-linked AI summary

MOSAIC: An Omni-Modal Benchmark for Long-Term Memory and Cross-Modal Reasoning in Conversational Agents

Hong Ting Tsang, Boyi Zhang, Haoyu Huang, Jiaxin Bai, Yufei Li, Yisen Gao, Zhongwei Xie, Hongyu Luo, Jeff Z. Pan, Yangqiu Song

arXiv:2610.29249v1cs.CLcs.AI

TL;DR

Existing conversational memory benchmarks largely lack four-modality, multi-session evidence chaining and do not isolate where evidence is lost. MOSAIC introduces targeted benchmark questions and a matched stage-diagnostic protocol; its results show modality-dependent representation losses, substantial residual reading error, and weak abstention.

  • Problem

    Existing long-term conversational memory benchmarks omit video and audio, treat images as optional, or stop at text–image co-reference, leaving four-modality multi-session chaining unrepresented.

    Source: §1 · p. 1

  • Method

    MOSAIC pairs persona-grounded questions with targeted modality signals and matched interventions that diagnose WRITE, RETRIEVE, DELIVER, and READ.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    ∼9 points are lost to textualization on spatial-heavy items, retrieval costs ∼8 points and packing ∼4, while ∼15 points remain under gold symbolic facts and abstention is weak.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark supports diagnosing stage-associated losses separately rather than relying only on an end-to-end score.

    Source: §1 · p. 2

    Personas, evidence sessions, and media are synthetic, and stage contrasts are matched-protocol associations rather than fully isolated mechanisms.

    Source: §8 · p. 9, §9 · p. 9

Abstract

from arXiv · show

Omni-modal assistants must remember not only what users say, but also what they show and convey. Existing long-term memory benchmarks are mostly text-and-image or report a single end-to-end score, so they cannot locate whether evidence is lost when it is represented, retrieved, delivered, or read. We introduce MOSAIC, a benchmark of 923 questions over persona-grounded histories of 11-24 sessions spanning text, image, video, and audio. MOSAIC constructs targeted spatial, temporal, and prosodic signals rather than decorative media, with the strongest causal validation for spatial information and selector-level validation for annotated prosody, and pairs them with a matched diagnostic protocol that freezes the question, reader, prompt, and budget while intervening on representation, retrieval, and delivery. Matched decomposition shows that representation loss is modality-dependent, residual reading dominates even under gold evidence, and abstention remains weak on these controls: strong answerable scores coexist with failures to refuse. MOSAIC thus turns omni-modal memory evaluation from a leaderboard into a diagnosis of write, retrieve, deliver, and read failures.

1 Introduction

MOSAIC addresses the lack of conversational memory benchmarks that combine four modalities with multi-session evidence chaining and stage-level diagnosis. It introduces targeted questions and matched interventions, finding substantial losses across representation, retrieval, delivery, and reading, alongside weak abstention.

1 Introduction

  • Prior conversational memory benchmarks omit video and audio, make images optional, or stop at text–image co-reference, despite assistants’ multi-session use cases.Answering may require chaining temporally scattered evidence across modalities.Source: §1 · p. 1
  • Its matched diagnostic protocol separates WRITE, RETRIEVE, DELIVER, and READ under a frozen reader, prompt, budget, and judge.The protocol distinguishes retrieving facts from delivering them in the reader’s request.Source: §1 · pp. 1–2
  • ∼9 points are lost to textualization on spatial-heavy items, while retrieval costs ∼8 points, production packing ∼4, and ∼15 remain under gold symbolic facts; abstention is weak.Under the matched protocol, each paired drop is associated with the intervened stage.Source: §1 · p. 2
  • MOSAIC contributes 923 persona-grounded questions with targeted spatial, temporal, and prosodic signals, cross-modal multi-hop chains, and 139 refusal controls.Spatial information has the strongest causal validation, while annotated prosody has selector-level validation.Source: §1 · p. 2
  • The benchmark also provides a validated public pipeline with human audits, media, filtering dispositions, manifests, and run configurations.Source: §1 · p. 2

2 Related Work

Prior memory and reasoning benchmarks cover parts of multimodal or multi-session evaluation, but leave gaps in four-modality conversational chaining and controlled stage diagnosis. MOSAIC combines those settings and makes delivery separately measurable from retrieval.

2 Related Work

  • MOSAIC requires retrieving and chaining evidence across all four primary modalities in multi-session conversations, a setting prior multimodal extensions do not cover together.Prior suites variously stop at text+image, aggregate personal evidence outside dialogue sessions, or add image/video reasoning without multi-session structure.Source: §2 · pp. 2–3
  • MOSAIC combines multi-hop reasoning with session-scattered multimodal retrieval, unlike static-corpus long-context suites and text-image QA benchmarks.The cited multi-hop works include HotpotQA, MuSiQue, MultiModalQA, WebQA, and CRIT (Yang et al., 2018; Trivedi et al., 2022; Talmor et al., 2021; Chang et al., 2022; Sung et al., 2026).Source: §2 · p. 3
  • MOSAIC substitutes stages under a frozen reader, prompt, packer, budget, and judge, separating DELIVER from RETRIEVE and enabling carrier-level signal attribution.Failure localization itself is not novel; the claimed distinction is a substitutable four-stage protocol for omni-modal conversational memory.Source: §2 · p. 3

3 Preliminaries

MOSAIC defines items through question type, modality gap, and reasoning-chain structure. Its answerable items require a deletion-tested path of linked facts, including at least one challenge-bearing fact tied to a non-text modality.

3 Preliminaries

  • Each item is a persona-grounded multimodal query in a chronological session haystack, organized by question type, modality gap, and reasoning chain.Source: §3 · p. 3

3.1 Question Types

  • The six question types include information extraction, cross-session selection and bridging, knowledge updates, temporal-plan disruptions, and refusal after evidence removal.Questions require one to three facts.Source: §3.1 · pp. 3–4

3.2 Modality Gaps

  • A modality gap is information preserved by its native modality but lost to text-only or caption-based agents, including spatial image detail, video action or state cues, and audio prosody.Annotated audio prosody is the most frequent strategy: 450 facts across 399 of 784 answerable items.Source: §3.2 · p. 4

3.3 Reasoning Chains and Fact Roles

  • Answerable items require a deletion-tested, bridge-entity-linked chain of NARROWER, SELECTOR, and ANSWER facts, with at least one required challenge-bearing fact.Removing any required fact makes the question unanswerable; the final answer value is a short text-expressible string.Source: §3.3 · p. 4, §3.2 · p. 4

4 The MOSAIC Generation Pipeline

MOSAIC is generated through a config-driven pipeline that designs and validates persona-grounded chains, creates chronological evidence and filler sessions, and synthesizes media. Filtering and human auditing yield a 923-item release with documented scale and modality coverage.

4 The MOSAIC Generation Pipeline

  • The automated pipeline designs persona-grounded chains, applies programmatic checks and per-fact deletion tests, builds chronological sessions, and synthesizes media.Unrepaired chains are discarded, and refusal items have required facts removed after auditing.Source: §4 · p. 4
  • 923 items remain after filtering and audit: 784 answerable questions and 139 refusal questions.The three-annotator audit covered 100 items, found 94.3% well-posed, and rescued 44 probe false negatives.Source: §4 · p. 4
  • The release spans 70 personas and 11–24 sessions each, with 10,765 multimodal artefacts; 85.2% of answerable items need at least two facts and 75.0% at least two modalities.Required evidence appears in image, audio, video, and text for 99.5%, 54.1%, 23.5%, and 35.3% of answerable items, respectively; these categories overlap.Source: §4 · p. 4

5 Diagnostic Evaluation Protocol

MOSAIC makes write, retrieve, deliver, and read stages separately substitutable under a frozen reader, prompt, and budget. Matched contrasts and modality interventions distinguish stage losses from within-item carrier effects, while two architecture tracks separate fixed-reader backend comparisons from descriptive end-to-end capability.

5 Diagnostic Evaluation Protocol

  • The protocol models memory as writing, retrieval, packing evidence for delivery, and reading it to answer.Retrieved artefacts count as delivered only when presented to the actual provider.Source: §5 · pp. 4–5

5.1 Matched Stage Decomposition

  • Seven matched conditions on the same 784 answerable questions change representation, retrieval, or delivery while keeping the reader, prompt, packer, and judge fixed.Contrasts compare symbolic with native evidence, structured with generic text, gold native with retrieved-all-delivered, and all-delivered with production-capped requests.Source: §5.1 · p. 5

5.2 Within-Item Modality Interventions

  • Within-item interventions change one required carrier while holding the question, evidence order, and other hops fixed.Contrasts cover image, video, and audio variants; retained items are answerable from native evidence, and leak audits check that non-intervened hops remain unchanged.Source: §5.2 · p. 5

5.3 Two Architecture Tracks

  • Track A compares backends through one fixed reader, whereas Track B measures intended end-to-end systems descriptively because multiple components change together.Track B includes adaptive ReAct only after a one-search parity gate against the corresponding Naive RAG run.Source: §5.3 · pp. 5–6

5.4 Metrics and Uncertainty

  • Answer quality uses a 0–1 semantic-judge score on 784 answerable items, paired with correct- and false-refusal rates; retrieval and delivery track required facts and complete chains.Paired QID comparisons use item-bootstrap 95% intervals; intervals are unadjusted across tests, so exploratory challenge-level nulls warrant caution.Source: §5.4 · p. 6

6 Results and Analysis

Under matched conditions, residual error remains largest under gold symbolic evidence, while losses elsewhere vary by modality and pipeline. Fixed-reader results also show that delivery constraints and answer–abstention trade-offs shape system performance.

6.1 Overall Stage Decomposition (RQ1–RQ3)

  • 0.846 gold-symbolic accuracy leaves 15.4 points of residual reading/grading error; retrieval costs 7.6 points and production packing costs 3.8 on identical ranked hits.Generic captions/ASR score 0.690 versus native evidence at 0.783, while the structured writer recovers only 1.4 points, indistinguishable from zero.Source: §6.1 · p. 6, Figure 3 · p. 7
  • 17.7 points are lost when reduced top-k and one-media-per-modality packing are combined, with complete-chain delivery falling from 0.675 to 0.184.The loss reflects the combined interventions, not the media cap alone; the Figure 4 sweep holds Recall@50 constant at 0.871 while changing packer allowance.Source: §6.1 · p. 7, Figure 4 · p. 7

6.2 Representation Loss by Modality (RQ1)

  • On spatial-configuration items, raw images outperform OCR-only text by 32.0 points, and explicit spatial descriptions beat object-only text by 15.4 points.OCR-only text also loses 18.5 points to raw images on fine-detail and background items; naming objects alone misses relational arrangement.Source: §6.2 · p. 7, Figure 5 · p. 8
  • Temporal and raw-audio contrasts remain inconclusive, but swapping gold prosody tags reduces scores by 4.6 points across 399 prosody questions.Video strata are small (n=56–64), and the reported power analysis gives minimum detectable effects of 8.9–11.2 points; the swapped-tag result indicates selector-level sensitivity under gold delivery.Source: §6.2 · pp. 7–8

6.3 Fixed-Reader Comparison (RQ2)

  • With the reader fixed (Table 4), A-MEM beats same-representation flat RAG by 16.2 points on caption text and 16.4 points on omni text.A-MEM delivers the complete required chain on 0.787 of items, versus 0.698 for flat caption and 0.466 for the native index.Source: §6.3 · p. 8, Table 4 · p. 8

6.4 End-to-End Systems & Abstention (RQ4)

  • Native ReAct with MiMo-V2.5 scores 0.732 on answerable items but correctly refuses only 0.410, while no end-to-end system occupies Figure 6’s upper-right region.MemoryOS exceeds 0.88 correct refusal but scores below 0.26 on answerable items and falsely refuses over 0.61; refusal controls retain question-form cues.Source: §6.4 · p. 8, Figure 6 · p. 8

7 Conclusion

MOSAIC frames omni-modal memory failures as write, retrieve, deliver, and read problems, using matched stage contrasts to diagnose them. Its results identify modality-specific evidence loss and persistent abstention weaknesses, with conclusions bounded by the protocol and control set.

7 Conclusion

  • MOSAIC separates omni-modal memory failures into WRITE, RETRIEVE, DELIVER, and READ stages.Under the matched protocol, reading has the largest residual loss under gold facts; retrieval and delivery constrain native pipelines, while textualization constrains text-derived ones.Source: §7 · p. 9
  • Spatial structure has the clearest causal validation and no adequate text surrogate, while annotated prosody tags show selector-level sensitivity.Temporal and raw-audio versus ASR contrasts remain inconclusive.Source: §7 · p. 9
  • No evaluated system both answers and refuses reliably; the paper concludes that progress requires preserving modality-specific signal, delivering complete chains, and reading them faithfully.The stage findings are protocol associations, not universal bottlenecks.Source: §7 · p. 9

8 Limitations

The benchmark and its conclusions have explicit scope limits: the histories and media are synthetic, stage contrasts are not fully isolated mechanisms, and several findings remain constrained by power or control design.

8 Limitations

  • MOSAIC uses synthetic personas, evidence sessions, and media; evaluation on organically collected multimodal histories remains future work.Source: §8 · p. 9
  • Matched stage contrasts are protocol associations rather than fully isolated mechanisms, despite paired conditions and a reader held out from filtering.Source: §8 · p. 9
  • Video null results are underpowered, and the raw-audio versus ASR gap is inconclusive for the fixed reader.Textualized audio can retain paralinguistic lexical cues, while other gold hops may already expose the answer value.Source: §8 · p. 9
  • RQ4 uses length-matched refusal controls that retain residual question-form cues, so its abstention findings are bounded by that control design.Source: §8 · p. 9

9 Ethical Considerations

MOSAIC uses synthetic personas, evidence sessions, and media rather than private user logs; real-deployment consent and retention controls are outside the benchmark’s scope. Its appendices document how the benchmark is generated, filtered, and evaluated through multiple system architectures and matched diagnostics.

9 Ethical Considerations

  • Personas, evidence sessions, and media are synthetic, and the authors release no private user logs; deployment consent and retention controls are outside scope.The human audit used short voluntary academic annotation.Source: §9 · p. 9

A.2 Modality Gap Strategies

  • A modality gap is information preserved by its native channel but lost to text-only or caption-based agents; strategies target spatial, temporal, and prosodic signals.Examples include image background detail, video action order, and voice tone conveyed despite neutral transcript wording.Source: §A.2 · p. 13

A.3 Fact Roles and Chain Structure

  • Answerable items use ordered NARROWER, SELECTOR, and ANSWER facts; a chain is cross-modal when required facts span at least two modalities.The final answer is text-expressible, while a required fact carrying it may itself be multimodal.Source: §A.3 · pp. 13–14, Table 6 · p. 14

B.1 Stages 1–3 in Detail

  • Chains are designed question-first, deletion-tested against twelve failure modes, and realized as chronological evidence sessions with safeguards against answer leakage.Unrepaired chains are discarded; filler sessions are interleaved chronologically with evidence sessions.Source: §B.1 · p. 14, §B.3 · p. 15, §B.5 · p. 16

B.2 Stage 4 Filtering Lineage and Release Statistics

  • Filtering and human review produce 923 items: 784 answerable and 139 refusal, after 44 probe-rejected answerable candidates are rescued.The 784/864 (90.7%) figure is a filtering retention rate, not a human answerability ceiling.Source: §B.2 · p. 14

C.1 System Architectures

  • Caption-Text converts media to indexed captions, Omni-Text preserves prosody in captions, and Native-MM indexes raw media for multimodal reasoning.The three architectures differ in how multimodal evidence is preprocessed and indexed.Source: Figure 8 · p. 16, §C.1 · pp. 16–17

C.4 Diagnostic Conditions

  • Matched diagnostics hold the reader, prompt, packer, budget, and judge fixed while varying evidence representation, retrieval, delivery, and end-to-end ranking.Gold symbolic facts bound residual reading and grading error; production-cap delivery uses top-20 units, three media per modality, and a 100 MB cap.Source: §C.4 · p. 17

C.6 Metrics Definitions

  • Evaluation pairs answerable-item judge scores with correct- and false-refusal rates, and measures retrieval and delivery using required-fact and complete-chain Recall@k.Controlled effects use within-item differences, item-bootstrap intervals, and a persona-clustered dependence check.Source: §C.6 · p. 18

D Dataset Statistics

MOSAIC-Eval contains 923 items across 70 personas, with long multimodal histories and a substantial share of multi-hop and non-text evidence. Its reported distributions describe the benchmark’s challenge mix, while the filtering lineage and video power analysis qualify how to interpret the release and intervention results.

D Dataset Statistics

  • The final MOSAIC-Eval set contains 923 items across 70 personas, averaging 13.2 items per persona and 17.5 sessions per item (range 11–24); its haystack comprises approximately 18.8M tokens, averaging ∼269K tokens per persona.Source: D Dataset Statistics · p. 18
  • 65.6% of the benchmark consists of multi-hop reasoning questions, while Answer Refusal items comprise 15.1%.Knowledge Update (20.9%) and Temporal Reasoning (20.5%) are the largest categories; Information Extraction accounts for 19.3%.Source: D Dataset Statistics · p. 18, Table 12 · p. 19
  • 85.1% of required facts are non-text, with image evidence accounting for 49.0%, audio 25.6%, and video 10.5% of required facts.These modality incidences are reported across facts, and MOSAIC’s challenge distribution emphasizes targeted prosodic, spatial, and temporal evidence.Source: D Dataset Statistics · p. 18, Table 13 · p. 19
  • 450 required-fact occurrences across 399 answerable items use tagged audio prosody, while video gaps are balanced across action ordering (67), repetition count (64), and state transition (64).Fine-grained attributes are the largest image gap, with 388 required-fact occurrences over 333 items.Source: D Dataset Statistics · p. 19, Table 14 · p. 19
  • At ∼269K tokens per persona, MOSAIC is about 1.7× LongMemEval’s ∼158K-token budget and about 16× LoCoMo’s ∼17K-token budget per conversation.MOSAIC averages 17.5 sessions per item, and its scale comes from a large multimodal haystack around sparse required facts.Source: D Dataset Statistics · p. 20, Table 15 · p. 20

E Pipeline Filtering and Quality Assurance

  • Human review restores 44 probe-rejected answerable items, yielding 923 unique items (784 answerable + 139 refusal) from the 1,004-item filtered set.The automated post-review set had 879 items; the final dataset is a 44-item superset of it.Source: E Pipeline Filtering and Quality… · p. 20, Table 16 · p. 20

F Statistical Power of Modality Interventions

G Extended Limitations

The appendix qualifies MOSAIC’s stage claims and describes limits on video and audio conclusions, refusal-surface diagnostics, annotation procedures, and release-label estimates.

G Extended Limitations

  • Stage labels are protocol associations, not isolated causal mechanisms, because gold and retrieved contexts differ in distractors and composition.The matched contrasts freeze the reader, prompt, packer, and judge.Source: G Extended Limitations · p. 21
  • Video strata are small (n=56–64 unique QIDs), so ordered-versus-shuffled and related nulls do not establish lossless temporal compression.At 80% power, minimum detectable effects are 8.9–11.2 points.Source: G Extended Limitations · p. 21
  • Gold-delivery audio nulls do not show that readers recover sentiment from plain transcripts or raw audio, because textualized carriers can retain paralinguistic cues.A correct-versus-swapped symbolic-tag intervention shows selector-level sensitivity; human-audited lexically clean subsets remain future work.Source: G Extended Limitations · p. 21

H Refusal Surface-Structure Diagnostics

  • Privileged construction fields perfectly identify refusal items (AUROC = 1.00), but are downstream of construction and must not be given to systems under test.Observable question-text cues remain partially predictive, so refusal calibration is not fully form-blind.Source: H Refusal Surface-Structure Diagnostics · pp. 21–22

I.2 Release Label Quality (Primary)

  • Reweighting the audit yields approximately 97.8% gold-answer correctness and 96.6% evidence sufficiency on the released answerable set, rather than the NO-heavy sample’s pooled rates.The first audit oversampled probe-NO items; Table 21 estimates the 784-item release set.Source: §I.2 · p. 22, Table 21 · p. 22

I.3 Stratified Audit Sample (Secondary)

These appendices report audit findings and additional matched results, including stage-specific losses, budget effects, intervention contrasts, retrieval and delivery behavior, and per-challenge scores.

K.1 Matched Stage Ladder: Paired Contrasts

  • Gold symbolic facts score 0.846, while the full production pipeline reaches 0.653; reading/grading error, perception, retrieval, and delivery each contribute to the matched stage ladder.The production score is 1.7 points below the original-query budgeted condition; gold symbolic facts exceed it by 19.2 points.Source: §K.1 · pp. 24–25

K.3 Delivery Budget Sweep

  • One-media-per-modality packing with reduced top-k costs 17.7 points and lowers complete-chain delivery from 0.675 to 0.184.The combined effect is not attributable to the media cap alone.Source: §K.3 · p. 25

K.4 Within-Item Modality Intervention Contrasts

  • Correct prosody tags outperform happy↔unhappy-swapped tags by +0.046 [0.017, 0.077] on the unique-QID mean, showing selector-level sensitivity rather than raw-audio superiority.The condition means are 0.824 (correct) and 0.782 (swapped).Source: §K.4 · p. 25

K.5 Track A Fixed-Reader Paired Contrasts

  • A-MEM beats same-representation flat RAG by 16.2 points on caption text and 16.4 on omni text, while MemoryOS trails flat RAG by 25.5 and 22.1 points, respectively.These are fixed-reader paired contrasts.Source: §K.5 · p. 25

K.7 Per-Question-Type Diagnostic Ladder

  • Knowledge-update items lose most when symbolic facts become native evidence (0.898 to 0.668), whereas MSR-BRIDGE degrades mainly at retrieval and delivery (0.790 to 0.507 end-to-end).Information extraction is nearly flat across the ladder.Source: §K.7 · pp. 25–26

K.8 Retrieval and Delivery Curves

  • Complete-chain recall remains far below required-fact recall; after packing, complete chains reach 0.675 with all feasible hits, 0.491 under production caps, and 0.464 end-to-end.The authors argue chain-level accounting is the relevant retrieval metric for multi-hop memory because every chain member must arrive.Source: §K.8 · p. 26, Table 30 · p. 26

K.9 Why Delivery Fails

  • All 9,338 artefacts dropped under production caps are attributed to per-modality media-count limits, not token limits, serialization failures, or unsupported formats.The drops comprise 5,559 images, 593 videos, and 3,186 audio clips.Source: §K.9 · p. 26, Table 31 · p. 26

K.10 Per-Challenge Accuracy

  • Repetition count is the weakest per-challenge score overall (0.666), while background detail is the weakest image challenge (0.670).Fine-grained attributes score 0.801; on prosody, ReAct Native-MM scores 0.740 versus Caption-Text Naive RAG at 0.361.Source: §K.10 · p. 26, Table 32 · p. 26

K.11 Method Family Ranking

This section contrasts method-family rankings across tracks and presents system and question-type results, cross-modal examples, and evaluation and retrieval prompts. It also describes a four-check review of benchmark-question quality, covering atomicity, grounding, multimodal necessity, and whether evidence gaps contribute to answering.

K.11 Method Family Ranking

  • Text-memory families differ from Native-MM in both media ingestion and retrieval substrate, while the fixed-reader track partly inverts rankings and motivates reporting both tracks.Table 33 groups twelve Track B configurations into architecture families.Source: §K.11 · pp. 26–27, Table 33 · p. 27

L.1 Example Item: Multi-Hop Cross-Modal Chain (MSR Bridge)

  • A multi-hop question chains a background sign in an image, action ordering in video, and happy tone in audio; captions lose the spatial and prosodic discriminators.The expected answer is “bundle-wrap”; the text transcript names both candidate methods neutrally, leaving prosody to distinguish them.Source: §L.1 · p. 27

L.2 Example of Model Success

  • The RAG MM MiMo model correctly retrieves an image-only brand name, answering “Lorraine” with a score of 1.0.The brand is stamped on a vintage cast-iron fondue set and never appears in the chat text.Source: §L.2 · p. 27

K.12 Per-System and Per-Question-Type Visualizations

  • Native-multimodal configurations rank highest and MemoryOS configurations lowest on answerable items, while correct-refusal rankings reverse the answerable ordering.Knowledge Update is the hardest answerable type for essentially every system.Source: Figure 9 · p. 28, Figure 12 · p. 28, Figure 11 · p. 30

L.3 Case Study: Audio Prosody Gap (MSR Selection)

  • Caption-Text Naive RAG selects the wrong binocular pair and predicts $159.99, whereas Native-MM uses labeled tone and the receipt image to answer $189.99.This is an imperfect selector-failure illustration: “surprised me” identifies the reacted-to pair but does not explicitly encode positive polarity.Source: §L.3 · pp. 28–29

M.1 Evaluation Prompts

  • The prompts require answers to use only retrieved evidence and say exactly “I don’t know” when it is insufficient; the retrieval agent answers immediately if possible and otherwise searches first.Source: §M.1 · pp. 31–32

M.2 Pipeline and Retrieval Prompts

  • Pipeline prompts extract keyword-rich search terms from questions and clean outputs into short answers, including “from X to Y” for change or update questions.The output-cleanup prompt returns “I don’t know” when the text indicates the answer is unknown.Source: §M.2 · p. 33
  • The LLM quality-judge prompt checks question atomicity, answer grounding in evidence facts, multimodal necessity, and whether required-fact gap strategies contribute to answering.It requests a PASS or FAIL and a short reason for each check, returned as JSON.Source: §M.2 · p. 33

Related papers

Loading 2610.29249v1…