PaperlayerView exact revision on arXiv

Source-linked AI summary

When History Helps and Hurts: Selective History Use across Multimodal Turns

Shuoyang Sun, Kerui Gu, Hao Fang, Shaoli Huang, Bin Chen

arXiv:2610.11948v1cs.CL

TL;DR

Multimodal history-use demands are difficult to distinguish from underlying question difficulty, even though conversation may require an earlier question or historical evidence. ReTurn evaluates these demands with matched task pairs and single-turn references; across 13 models, median OpenQA accuracy falls from 93.7% with direct input to 72.3% in conversation.

  • Problem

    Existing multi-turn evaluations rarely separate history-use demands from underlying question difficulty, despite history sometimes supplying a still-relevant question or necessary evidence.

    Source: §1 · p. 1

  • Method

    ReTurn uses 7,000 visual- and audio-evidence tasks with matched task-carrying and evidence-carrying history conditions, conflict variations, and single-turn answerability references.

    Source: §1 · p. 2

  • Results

    93.7% to 72.3%: median model-level OpenQA accuracy falls from Single-turn to Multi-turn across 13 omni-modal, vision-language, and audio-language models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Relevant history can be recovered without being correctly applied or selected over competing evidence, and mitigation yields only partial gains.

    Source: §1 · p. 2, §7 · p. 10

    Short/Long results are a configuration comparison rather than an isolated length effect because history composition and extent both differ.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

Reliable multimodal interaction depends on selective use of conversational history: an earlier question may remain relevant while its previous answer is outdated, whereas a current request may depend on historical evidence despite conflicting new observations. Existing multi-turn evaluations rarely separate these history-use demands from underlying question difficulty. To address this gap, we introduce ReTurn, a benchmark of 7,000 base tasks spanning visual and audio evidence for evaluating selective history use. For task-carrying history, Reconfirm/Reground require applying a historical question to current media while varying historical agreement; for evidence-carrying history, Retrieve/Rebind require answering a current question using historical media while varying current-media competition. Each pair preserves the target question, media, and answer. Tasks support open-ended and multiple-choice evaluation, with matched single-turn counterparts serving as answerability references. Across 13 omni-modal, vision-language, and audio-language models, median model-level open-ended accuracy falls from 93.7% with direct input to 72.3% in conversation. Behavioral probes show that high question recall can coexist with weaker task application, while competing media can redirect answers away from historical targets. Supervised adaptation yields only partial gains. ReTurn provides a controlled framework for assessing whether multimodal models select and use the historical information required by each request.

1 INTRODUCTION

Multimodal conversation requires selective history use: earlier questions may remain relevant after their answers become outdated, while current requests may depend on historical evidence. ReTurn evaluates these demands with matched tasks and single-turn references, finding that performance drops in conversation and that history recall does not guarantee correct application.

1 INTRODUCTION

  • ReTurn’s 7,000 visual- and audio-evidence tasks distinguish task-carrying from evidence-carrying history using matched pairs and single-turn answerability references.Reconfirm/Reground vary whether historical media supports the same answer; Retrieve/Rebind vary competition from current non-target media.Source: §1 · p. 2
  • 93.7% to 72.3%: median model-level OpenQA accuracy falls from Single-turn to Multi-turn across 13 omni-modal, vision-language, and audio-language models.Long conversations expose difficulty applying an earlier question when its answer is outdated and preserving a historical target amid current competitors.Source: §1 · p. 2
  • Behavioral probes find that high question recall can coexist with weaker task application, while competing media can redirect answers from historical targets.Inference-time assistance and supervised adaptation yield only partial gains.Source: §1 · p. 2
  • The benchmark treats history as both essential information and a potential source of interference, and investigates failures through behavioral diagnostics and mitigation studies.Its evaluation spans omni-modal models, LVLMs, and LALMs.Source: §1 · p. 2

2 RELATED WORK

Multimodal systems support interaction across visual and audio inputs, making earlier questions and observations relevant to later turns. Prior benchmarks cover multi-turn dialogue, memory, and updating, while ReTurn focuses on history as both necessary information and potential interference.

2.1 MULTIMODAL MODELS FOR INTERACTIVE SYSTEMS

  • Vision-language, audio-language, and omni-modal systems support successive requests that build on previously supplied media rather than isolated inputs.As interaction extends across turns, models must reuse earlier questions and observations according to the current request.Source: §2.1 · p. 3

2.2 MULTI-TURN EVALUATION AND HISTORY USE

  • Prior benchmarks study multi-turn dialogue, memory, and information updating; ReTurn varies historical agreement or current-media competition while preserving the dependency and target in matched pairs.Its single-turn references assess target answerability; benchmark counts use native units and are not directly comparable.Source: §2.2 · p. 3, Table 1 · p. 3

3 RETURN: BENCHMARKING SELECTIVE HISTORY USE

ReTurn evaluates selective history use with matched conversations built from verified media–question pairs. Its task design separates historical questions applied to current media from current questions answered using historical media, with controlled variations in agreement or competition.

3 RETURN: BENCHMARKING SELECTIVE HISTORY USE

  • ReTurn builds matched conversations from verified media–question pairs, varying contextual agreement or competition while preserving the information needed for the final request.Figure 2 depicts task-carrying and evidence-carrying conditions, supported-answer annotations, model-generated histories, and matched single-turn references.Source: §3 · p. 3, Figure 2 · p. 4

3.1 TASK DESIGN

  • Reconfirm/Reground keep the final request and current target fixed while varying whether historical media supports the same answer; Retrieve/Rebind require historical target media amid weaker or plausible-answer competitors.Each pair fixes the target question and media, answer, options, historical dependency, and horizon; Retrieve reduces but does not guarantee the absence of competition.Source: §3.1 · pp. 3–4

3.2 DATASET CONSTRUCTION

  • Construction uses localized clips and answer verification against visual or acoustic content; the benchmark contains 7,000 tasks in 3,500 matched pairs with balanced visual/audio evidence and Short/Long horizons.OpenQA and four-option MCQ provide two views per task; shared media and development exposure limit unseen-source generalization.Source: §3.2 · p. 4

3.3 EVALUATION PROTOCOL

  • Single-turn inputs provide the question and target media directly, while multi-turn inputs end after model-generated historical replies; final-answer accuracy is scored for OpenQA and MCQ.Rejected responses receive blinded GPT-5.6 Sol review, uncertain judgments count as incorrect, and conversational degradation is the single-turn minus multi-turn accuracy in percentage points.Source: §3.3 · pp. 4–5, Table 2 · p. 5

4 EVALUATING SELECTIVE HISTORY USE

The evaluation compares selective-history performance across 13 models and examines overall, modality, and answer-interface differences using paired conditions.

4.1 OVERALL RESULTS AND THE ACCESS–BINDING HYPOTHESIS

  • Median ∆conv rises from 6.8 to 34.5 pp on OpenQA and from 19.0 to 36.9 pp on MCQ between Short and Long configurations.Because the configurations differ in both history composition and extent, this is not an isolated length effect.Source: §4.1 · pp. 5–6
  • 93.7% Single-turn OpenQA accuracy falls to 72.3% in Multi-turn, with median ∆conv = 20.1 pp across models.The contrast shows history-use difficulty beyond direct answerability.Source: §4.1 · p. 5, Figure 3 · p. 6

4.2 MODALITY AND INTERFACE EXPOSE DISTINCT WEAKNESSES

  • All six omni-modal models have larger Rebind ∆conv on audio than visual tasks, while five have larger Reground ∆conv on visual tasks.The reversal across operations motivates separate diagnosis of the two history roles.Source: Figure 3 · p. 6, §4.2 · p. 6
  • The median absolute OpenQA–MCQ multi-turn difference is 2.5 pp overall but 24.4 pp on Reground.Eleven of 13 configurations score lower on Reground MCQ than OpenQA; task-carrying MCQ inherits both the question and option mapping.Source: Figure 3 · p. 6, §4.2 · p. 6, Table 3 · p. 7

4.1 OVERALL RESULTS AND THE ACCESS–BINDING HYPOTHESIS

  • In Long OpenQA, median accuracy falls from 79.5% in Reconfirm to 48.5% in Reground, and from 78.9% in Retrieve to 19.9% in Rebind.The paired ladders contrast direct answerability, historical dependency, and added interference; reference conditions also require binding, and Reconfirm permits answer reuse.Source: §4.1 · p. 6

5 DIAGNOSING SELECTIVE HISTORY FAILURES

Separate probes examine question recall and request restoration on the task side, and historical-source access and competitor changes on the evidence side; these provide behavioral evidence, not an error decomposition.

5.1 TASK-SIDE: ACCESS AND APPLICATION

  • Qwen3-Omni and Qwen3.8-Omni-Flash recall the Reground question on 96.8% and 99.8% of probes but score 81.8% and 84.1% on the task.Question restatement raises their scores to 95.2% and 96.4%; the separate-request probe cannot identify which original errors involved failed recall.Source: §5.1 · p. 7
  • Question restatement repairs 12.7–35.3% of valid Reground OpenQA pairs while harming at most 1.0%; answer neutralization reduces accuracy for two of three models.Neutralizing an earlier answer does not reproduce the benefit of making the inherited task explicit, limiting a stale-answer-only explanation.Source: §5.1 · pp. 7–8
  • Options-only restatement improves Reground by 41.4/62.5 pp for Qwen3-Omni/MiniCPM-o 4.5, versus 14.8/12.5 pp for questions alone.Task inheritance includes the answer interface; restoring both components adds no gain over options alone for Qwen3-Omni but further improves MiniCPM-o 4.5.Source: §5.1 · p. 8

5.2 EVIDENCE-SIDE: ACCESS AND SELECTION

  • Historical-source OpenQA accuracy reaches 66.9–85.7%, versus 90.4–92.1% with direct input.The models retain substantial source information, though direct access remains easier and historical text may also provide cues.Source: §5.2 · p. 8
  • Across 12 configurations, target accuracy changes by −3.6 to +6.0 pp after competitor replacement, while replacement-answer matching increases throughout.For Qwen3-Omni, target accuracy changes from 52.6% to 51.7%, while matching rises from 0.1% to 36.5%.Source: §5.2 · p. 8

6 MITIGATING SELECTIVE HISTORY FAILURES

The paper tests inference-time assistance for task and evidence selection, then supervised adaptation on native conversational histories.

6.2 LEARNING FROM NATIVE CONVERSATIONAL HISTORIES

  • SFT raises multi-turn accuracy by 3.42 pp on OpenQA and 2.36 pp on MCQ while retaining single-turn accuracy; Long OpenQA gains 4.36 pp despite Short-only training.Improvements extend beyond tasks sharing training sources: excluding input-parent overlap retains a 3.14 pp OpenQA gain.Source: Figure 5 · p. 9, §6.2 · p. 9

6.1 ASSISTING SELECTION AT INFERENCE TIME

  • Reweighting yields +1.8/−1.2 pp on OpenQA/MCQ for Qwen3-Omni and +1.6/+0.6 pp for MiniCPM-o 4.5; routing gains over restatement are mixed.The routed policy combines restatement for historical-question requests with reweighting when the request contains a complete question, relying on explicit benchmark references.Source: §6.1 · pp. 8–9

6.2 LEARNING FROM NATIVE CONVERSATIONAL HISTORIES

  • After SFT, Long Rebind reaches 26.2% OpenQA and 29.9% MCQ accuracy, while Long MCQ Reconfirm declines by 1.3 pp.The OpenQA net gain combines 261 repaired answers with 90 newly incorrect answers.Source: §6.2 · p. 9
  • SFT+Inv adds only 0.02 pp on OpenQA and 0.20 pp on MCQ over matched SFT, with sensitivity intervals spanning zero.In this single-model, single-seed study, no clear additional benefit from the consistency term is observed.Source: §6.2 · p. 9

7 CONCLUSION

ReTurn offers a controlled evaluation of selective history use, while finding that questions answerable in isolation become harder in conversation and mitigation only partially closes the gaps. Its construction and exposure analyses qualify how the benchmark was verified, partitioned, and interpreted.

7 CONCLUSION

  • ReTurn finds that questions readily answerable in isolation become substantially harder in conversation, while mitigation only partially closes the gaps.The diagnostics show that retrieving relevant history can coexist with failures to apply it or select the intended source.Source: §7 · p. 10

A CONSTRUCTION, PROVENANCE, AND EXPOSURE

  • Questions and answers are checked against audiovisual content, but human annotation agreement was not measured.Source annotations and captions support localization rather than serving as answer evidence.Source: A CONSTRUCTION, PROVENANCE, AND EXPOSURE · p. 13
  • Reconfirm/Reground vary whether historical media supports the current target answer, while Retrieve/Rebind vary competition from current non-target media.These are local matched constructions rather than a complete factorial crossing; correctness on Reconfirm alone does not prove current-media reinspection.Source: A CONSTRUCTION, PROVENANCE, AND EXPOSURE · p. 13
  • The benchmark contains 7,000 base tasks in 3,500 matched pairs, with OpenQA and four-option MCQ views and separate short and long horizons.The 14,000 task/interface views are not two independent datasets, and short and long tasks are not universally item-matched.Source: A CONSTRUCTION, PROVENANCE, AND EXPOSURE · p. 13
  • The 5,000-task Test split has no exact adaptation task IDs, but repartitioning does not remove prior research exposure.Its source-exposure indicators overlap, and task counts are not counts of independent source observations.Source: A CONSTRUCTION, PROVENANCE, AND EXPOSURE · p. 13
  • Requiring every input parent to be disjoint from development retains 1,928 tasks but removes all Long Retrieve/Rebind tasks.Source-filtered aggregates therefore reflect different exposure and task-composition profiles.Source: A CONSTRUCTION, PROVENANCE, AND EXPOSURE · p. 13, Table 5 · p. 14

B MODELS AND EVALUATION PROTOCOL

Models generate their own chronological histories, while model-specific input and runtime configurations shape the evaluation. Single-turn scores provide an empirical answerability reference, not a ceiling or a multi-turn prompt with history removed.

B MODELS AND EVALUATION PROTOCOL

  • Each model recursively generates its own assistant replies from the chronological prefix, with earlier errors retained rather than replaced by gold replies.LVLMs receive visual-only media, LALMs receive companion audio without ASR substitution, and hosted omni-modal configurations receive synchronized audio/video.Source: B MODELS AND EVALUATION PROTOCOL · p. 14
  • Evaluated configurations differ in media processing, decoding, and execution backends, so cross-model comparisons also reflect these differences.Nemotron-3-Nano-Omni uses SGLang for single-turn and vLLM for multi-turn execution, so its gap may include backend effects.Source: B MODELS AND EVALUATION PROTOCOL · p. 14, Table 6 · p. 15
  • The single-turn input supplies target media and the complete question as an empirical answerability reference, not a mathematical ceiling or a multi-turn prompt with history deleted.The 95,000 expected model/task/interface views yield 94,984 valid pairs; three model configurations have exclusions.Source: B MODELS AND EVALUATION PROTOCOL · p. 14

C FULL INTERFACE-SPECIFIC RESULTS AND AGGREGATION

The section defines how conversational accuracy and paired harm/rescue are measured and aggregated across models, interfaces, operations, and horizons. Tables 7–10 report full Test accuracy for OpenQA and MCQ across short and long settings.

C FULL INTERFACE-SPECIFIC RESULTS AND AGGREGATION

  • The reported Test cells use common valid response pairs, with operation-specific sample counts varying by model family and occasional model-operation exceptions.The marginal accuracies alone do not recover the common intersection used for operation-pair contrasts.Source: C FULL INTERFACE-SPECIFIC RESULTS AND… · pp. 15–17
  • Conversational harm counts single-turn-correct/multi-turn-incorrect pairs, while rescue counts the reverse; ∆conv is 100 times net harm divided by n.Intervention rescue and harm instead compare two multi-turn conditions.Source: C FULL INTERFACE-SPECIFIC RESULTS AND… · p. 15
  • Figure 3a pools horizons within models and weights the 13 models equally, while operation-pair contrasts use their common valid-response intersection.Counts pool both modalities for omni-modal models, visual tasks for LVLMs, and audio tasks for LALMs; interface averages use each interface's own valid denominator.Source: C FULL INTERFACE-SPECIFIC RESULTS AND… · p. 15
  • Tables 7–10 report full Test single-turn and multi-turn accuracy and conversational degradation for OpenQA and MCQ across short and long settings.Accuracy is reported as a percentage, and ∆conv is ST − MT in percentage points.Source: Table 7 · p. 16, Table 8 · p. 16, Table 9 · p. 17

D SCORING RULES AND AUTOMATED SENSITIVITY AUDITS

The evaluation combines frozen answer and interface rules with blind review of rule-rejected responses, then audits sensitivity to scoring choices and media-grounded evidence. The audits are limited by joint judge-and-rubric changes, sampling uncertainty, and non-independent model checks.

D SCORING RULES AND AUTOMATED SENSITIVITY AUDITS

  • Rule-rejected answers receive blind GPT-5.6 Sol review, and confirmed semantic correctness contributes to content scores while uncertain judgments receive no confirmed-correct credit.The rubric assesses equivalence at the requested granularity, and rule scores, content scores, format compliance, uncertainty, and review reasons are retained separately.Source: D SCORING RULES AND AUTOMATED… · p. 18, Table 11 · p. 18
  • The alternative scoring audit retains positive conversational degradation for all three sampled models, with smaller weighted ∆conv estimates than the original scoring.Original/alternative estimates are 17.34/10.82 pp for Qwen3-Omni, 32.46/29.23 pp for MiniCPM-o 4.5, and 7.62/5.61 pp for Qwen3.8-Omni-Flash.Source: D SCORING RULES AND AUTOMATED… · p. 18, Table 12 · p. 18
  • Because both judge and rubric change in the alternative audit, it measures sensitivity to their joint choice rather than isolating a judge effect.The sampled estimates are not adjudicated scoring-error rates, and the single-group standard error excludes rubric uncertainty.Source: D SCORING RULES AND AUTOMATED… · pp. 18–19
  • A 32-pair media-grounded sample found 30 of 31 necessary-evidence questions matched the gold answer, but did not estimate full-dataset gold-answer error.The audit had no human review, and the construction and evaluation model checks are not independent ground truth.Source: D SCORING RULES AND AUTOMATED… · p. 19

E.1 TASK RECALL, RESTATEMENT, AND HISTORICAL ANSWER TEXT

The probes separate historical-question recall and application from the effects of restatement, competing media, and reference wording. Results show that accuracy alone can conceal shifts toward a current competitor, while interventions and wording changes do not provide uniform repairs.

E.1 TASK RECALL, RESTATEMENT, AND HISTORICAL ANSWER TEXT

  • Qwen3-Omni and Qwen3.8-Omni-Flash showed high Reground question recall despite lower task accuracy, whereas MiniCPM-o 4.5 struggled with recall itself.Elicited recall measures whether the historical question can be produced, not whether recall failed in each original error.Source: §E.1 · p. 19, Table 13 · p. 19
  • Qwen3-Omni Reground accuracy fell by 2.64 pp on OpenQA and 13.20 pp on MCQ after historical-answer neutralization.The interventions were compared on common valid responses, and neutralization was not a consistent repair.Source: §E.1 · p. 19, Table 14 · p. 20

E.2 SEPARATING QUESTION AND OPTION RESTATEMENT

  • On Reground, question/option restatement effects were 7.03/33.59 pp for Qwen3-Omni and 8.20/58.20 pp for MiniCPM-o 4.5.The descriptive interactions were −15.62 and −8.59 pp; the format-matched Q0O0 arm was a diagnostic baseline, not the main-table native prompt.Source: §E.2 · pp. 19–20, Table 15 · p. 20

E.3 COMPETING MEDIA AND SOURCE-FACT ACCESS

  • 539/1,024 to 529/1,024 Qwen3-Omni OpenQA target accuracy changed after the Rebind swap, while replacement-answer matches rose from 1 to 374.The swap changed multiple perceptual attributes, and accuracy can conceal changes in wrong-answer content.Source: §E.3 · p. 20, Table 16 · p. 21
  • The answerability study queried the target, original competitor, and replacement separately with the same question; its post-hoc all-correct subset is a sensitivity analysis.The full sample comprised 128 Rebind OpenQA tasks.Source: §E.3 · p. 21, Table 17 · p. 21
  • Historical source-fact probes asked a distinct question about referenced media, and other textual cues may remain after literal gold strings are excluded.The probes complement task accuracy rather than decomposing individual errors.Source: §E.3 · p. 21

E.4 HISTORICAL SLOTS AND EXPLICIT REFERENCE WORDING

  • Slot interventions co-varied content, input load, replies, and sometimes position, so they do not isolate historical distance.The wording study found heterogeneous changes and lower overall accuracy; endpoint drift and residual ambiguity preclude causal interpretation.Source: §E.4 · p. 21

F ADAPTATION PROTOCOL AND ADDITIONAL RESULTS

Adaptation improved some Qwen3-Omni outcomes, but consistency regularization added no clear gain over SFT, and source-overlap filtering retained positive gains with qualifications. Inference-time routing helped relative to restatement on Test, while runtime-matched validation showed mixed action effects; the paper bounds these findings to controlled tasks and a limited adaptation study.

F ADAPTATION PROTOCOL AND ADDITIONAL RESULTS

  • Adaptation used 1,018 Short OpenQA views from 509 pairs and a candidate-set consistency objective for eligible answer-preserving pairs.The candidate set was used during training, not supplied at test time; SFT and SFT+Inv shared initialization, data order, and optimization budget.Source: F ADAPTATION PROTOCOL AND ADDITIONAL… · pp. 21–22
  • Historical text already contained the relevant gold in 74/134 first-target views of the target-switching set, so those views did not universally require media re-access.The counts were 59/73 audio views and 15/61 visual views.Source: F ADAPTATION PROTOCOL AND ADDITIONAL… · p. 22
  • SFT improved some Test outcomes, but consistency regularization showed no clear incremental gain over SFT.JS added one correct OpenQA answer and ten MCQ answers over SFT, with intervals spanning zero; the study used one model and seed.Source: F ADAPTATION PROTOCOL AND ADDITIONAL… · p. 22, Table 22 · p. 23
  • After excluding Test tasks sharing input-media parents with gradient-training inputs, SFT gains remained 3.14 pp on OpenQA and 2.50 pp on MCQ.Filtering changed task composition and did not remove all development exposure; the all-input Dev-disjoint subset contained no Long Retrieve/Rebind tasks.Source: Table 23 · p. 23, F ADAPTATION PROTOCOL AND ADDITIONAL… · p. 23

G INFERENCE-TIME ASSISTANCE: TEST AND DEV-VAL PROTOCOLS

  • On 5,000 Test tasks per interface, routing rescued/harmed 140/13 OpenQA answers (+2.54 pp) and 21/10 MCQ answers (+0.22 pp) relative to restatement.Routing reused selected outputs without another forward pass; its comparison with the formal native baseline cannot isolate intervention effects because runtime differed.Source: Table 24 · p. 24, G INFERENCE-TIME ASSISTANCE: TEST AND… · p. 24
  • On 500 Dev-Val tasks per model/interface, reweighting changed OpenQA/MCQ accuracy by +1.8/−1.2 pp for Qwen3-Omni and +1.6/+0.6 pp for MiniCPM-o 4.5.This runtime-matched neutral comparison covered Dev-Val, not full Test.Source: Table 25 · p. 24, G INFERENCE-TIME ASSISTANCE: TEST AND… · p. 24

H SCOPE AND LIMITATIONS

  • ReTurn measures controlled task constructions but does not exhaust open-ended conversation; its histories are short, references explicit, and source media can recur across partitions.Broader histories, independently held-out media, and replicated adaptation remain proposed tests of generalization.Source: H SCOPE AND LIMITATIONS · p. 24
Loading 2610.11948v1…