PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges

Syeda Faiza Ahmed, Zien Sheikh Ali, Hunzalah Hassan Bhatti, Firoj Alam, Shammur Absar Chowdhury

arXiv:2608.17605v1cs.CLcs.AIcs.SD

TL;DR

Sustained multimodal dialogue requires systems to preserve context across changing goals, evidence, and modalities, but session-level competence remains limited. This survey organizes research across data, models, training, and evaluation, finding that modality support has advanced faster than coherent interaction across sessions.

  • Problem

    Sustained multi-turn interaction remains unresolved because systems struggle to preserve context, revise assumptions, ground responses, coordinate tools, and maintain quality across sessions.

  • Method

    This survey synthesizes datasets, benchmarks, models, training strategies, and evaluation methods for session-level interaction across text, speech, vision, video, tools, and culture.

  • Results

    Modality support has advanced faster than session-level competence, while systems still struggle with memory, grounding, revision, tool state, spoken timing, and cultural adaptation.

  • Takeaways & Limitations

    Conversational systems should be evaluated by their ability to sustain coherent, grounded, safe, and culturally appropriate interaction across turns, not isolated response quality.

  • Takeaways & Limitations

    The survey may omit new or concurrent resources, and its broad categories may obscure differences in architectures, training data, deployment settings, and evaluations.

Abstract

from arXiv · show

Conversational AI is moving beyond isolated text prompts toward sustained, multimodal interaction. In real conversations, users clarify goals, revise requests, interrupt responses, switch topics, and introduce new evidence while expecting systems to preserve context across turns. This makes multi-turn dialogue a distinct challenge requiring systems to maintain and update memory, ground responses across modalities, tools, and external knowledge, and adapt across languages and cultures. This study reviews multi-turn conversational AI across text-only dialogue, AudioLLMs and speech-native systems, multimodal and omni-modal systems, and tool-augmented agents. We organize the literature around datasets and benchmarks, modeling paradigms, training strategies, evaluation setups, and cross-cutting challenges. Our analysis shows that support for multiple modalities has advanced faster than the ability to sustain coherent interaction across a session. Despite stronger capabilities to perceive, speak, and act across modalities, current systems still struggle with persistent memory, cross-turn grounding, full-duplex interaction, robust evaluation, and cultural alignment. We conclude with a research agenda for systems that can remember, revise, ground, speak, listen, act, and adapt across turns, modalities, and cultures. (https://github.com/faiza-sfa/multiturn-conversational-ai-survey)

1 Introduction

Session-level multi-turn dialogue requires models to preserve and update context across dependent exchanges, making it a distinct modeling and evaluation problem. This survey unifies text, speech, vision, video, agentic, and culturally grounded interaction while identifying persistent capability, resource, and evaluation gaps.

  • Motivation: Users clarify goals, revise requests, switch topics, introduce evidence, and revisit earlier points across extended conversations.The survey defines session-level multi-turn interaction as a complete conversation with more than one exchange.
  • Problem: Multi-turn dialogue requires preserving context, resolving cross-turn references, updating assumptions, and maintaining coherence across a session.Later turns may depend on earlier requests, responses, evidence, actions, or state.
  • Current limitations: Despite broader modality support, frontier models still underuse relevant context, degrade on tasks distributed across turns, and remain below human-level reliability on realistic multi-turn instruction following.These findings show that sustained interaction remains difficult even when information is available within the context window.
  • Survey scope: The survey spans interaction depth, modality complexity from text through speech to omni-modal systems, and cultural-linguistic diversity across multilingual, dialectal, and culturally grounded settings.Its coverage includes text and spoken dialogue, AudioLLMs, vision-language and video dialogue, omni-modal systems, conversational retrieval, tool use, and cultural grounding.
  • Findings: Current systems face capability, resource, and evaluation gaps involving memory, grounding, assumption revision, tool state, spoken timing, cultural adaptation, and limited multilingual, speech, video, omni-modal, and culturally grounded data.Existing datasets remain dominated by English, text-only, and text-image interaction, while benchmarks leave important session-level abilities insufficiently evaluated.

2 Multi-turn Dialogue

Multi-turn dialogue is defined as a session of more than one exchange in which later turns may depend on earlier requests, responses, evidence, actions, or state. Its analysis requires tracking multimodal inputs, dialogue context, external state, and consistency and usefulness across turns and sessions.

  • Multi-turn Dialogue: A multi-turn conversation is a session with T > 1 exchanges, making the full session the natural unit of analysis rather than a single response.Later turns may depend on earlier requests, responses, evidence, actions, or state.
  • Modality and Context: Dialogue context captures what has happened, whereas external state may persist or change through memory, retrieved evidence, tool outputs, task state, or user profiles.The model must decide what to retain, update, or ignore at each turn because these signals carry different information.
  • Modality and Context: Inputs and responses may use text, speech, images, video, or combinations, while context records modality-specific information such as acoustic cues, visual content, and video events.The multimodal context may be empty in text-only settings.
  • Evaluation Focus: Evaluation covers turn-level correctness and usefulness and session-level consistency, grounding, and usefulness as earlier turns shape later responses.Persistent memory, tool use, and spoken interaction add requirements beyond a single session or response.

3 Datasets and Benchmarks

Datasets and benchmarks span text-only, spoken, multimodal, video, and culturally grounded interaction, but English and text remain dominant. Modality coverage is expanding unevenly, with relatively limited spoken and video resources and many multimodal datasets relying on static images.

  • Coverage and organization: Datasets and benchmarks cover text-only, spoken, multimodal, video, and culturally grounded interaction, organized by modality and evaluation focus.Text resources emphasize context tracking and instruction retention, while spoken and multimodal resources add acoustic, temporal, visual, and cultural grounding challenges.
  • Coverage and organization: 43 of 52 dataset entries are English-only, while 32 of 53 benchmarks focus on text-only datasets.This pattern indicates that English-centered and text-based evaluation remains predominant outside the cultural and cross-lingual block.
  • Coverage and organization: Table 2 lists 18 text-image resources, compared with only 8 spoken and 6 video-oriented datasets.Modality coverage is expanding unevenly across resource types.
  • Coverage and organization: Many multimodal resources still use static images rather than streaming, temporal, or spoken interaction.This limits representation of the dynamic interaction conditions present in spoken, video, and full-duplex dialogue.

4 Modeling Paradigm

Conversational modeling has progressed from modular text pipelines to LLMs, AudioLLMs, omni-modal systems, and tool-augmented agents, broadening interaction beyond text. However, most systems still represent dialogue history as flat context, with limited explicit support for memory, cross-turn grounding, and long-session reasoning.

  • Architectural evolution: Modeling has evolved from modular dialogue pipelines to general-purpose LLMs, AudioLLMs, omni-modal systems, and tool-augmented agents, while multi-turn challenges have intensified.Classical systems separated language understanding, state tracking, policy learning, and response generation; newer models consolidate these functions through pretraining.
  • Architectural evolution: Most systems still treat dialogue history as a flat sequence rather than explicitly modeling memory, state updates, selective grounding, and efficient long-session reasoning.This limitation persists across the broader shift toward more capable multimodal and agentic architectures.
  • Audio-native interaction: AudioLLMs support streaming generation, controllable voice, emotion, timbre, and full-duplex dialogue, while textless models preserve turn-taking and paralinguistic cues from raw audio.Despite these capabilities, most AudioLLMs have limited explicit modeling of long-horizon spoken memory, interruptions, and cross-turn acoustic grounding.
  • Omni-modal interaction: Omni-modal models jointly handle text, speech, vision, and sometimes video through streaming duplex interaction, multimodal tokenization, modality-specific encoders, and separate reasoning and speech modules.These designs aim to unify perception and generation across modalities, but most still treat dialogue history as a flat sequence.
  • Audio-visual dialogue: Audio-visual dialogue uses visual cues for speaker tracking, turn-taking, and grounding, with AV-Dialog handling noise and competing speech and MAViD synchronizing audio-video response generation.MAViD has not been evaluated for sustained multi-turn interaction.

5 Training Strategies

Multi-turn dialogue training must address cross-turn dependencies, delayed rewards, changing user intent, and context-sensitive response quality. The survey groups existing approaches into five families, including supervised fine-tuning and conversational retrieval-augmented training.

  • 5 Training Strategies: Training must account for cross-turn dependencies, delayed rewards, changing user intent, and context-sensitive response quality.
  • 5 Training Strategies: Existing approaches are grouped into five families: supervised fine-tuning, reinforcement learning and preference optimization, multi-task learning, synthetic data generation, and conversational retrieval-augmented training.
  • Supervised fine-tuning: Supervised fine-tuning requires conversations featuring follow-up questions, anaphora, ellipsis, topic shifts, and safety escalation.UltraChat and WildChat provide large-scale conversational data for this training approach.
  • Conversational RAG: Conversational RAG trains models to retrieve, filter, and use evidence across turns, supporting persona, external knowledge, knowledge graphs, query reformulation, and history selection.Related work also explores joint retrieval-generation training, efficient evidence integration, and citation-aware conversational RAG.

6 Evaluation of Multi-turn Dialogue

Multi-turn dialogue evaluation must extend beyond single-response quality to assess session-level consistency, memory, state tracking, grounding, interaction success, and robustness. Current evaluation remains fragmented, especially for agentic, spoken, full-duplex, multilingual, dialectal, and culturally grounded interaction.

  • Evaluation requirements: Multi-turn evaluation must measure local response quality alongside cross-turn consistency, memory, state tracking, grounding, interaction success, and robustness.Single responses can appear fluent and factually correct while still ignoring earlier instructions, losing preferences, misusing retrieved evidence, mishandling tool state, or breaking multimodal grounding.
  • Evaluation approaches: Existing methods combine surface-form and task-oriented metrics with session-level measures of instruction retention, memory recall, constraint satisfaction, feedback integration, and dialogue-level hallucination.Surface-form and task-oriented metrics remain useful but provide limited insight into session-level coherence.
  • Evaluation gaps: Agentic dialogue lacks unified evaluation for hidden state, tool-side effects, source attribution, and repeated-trial reliability.These gaps limit evaluation of agentic interaction beyond task or response-level performance.
  • Evaluation gaps: Spoken and full-duplex systems lack integrated session-level evaluation of semantic correctness, audio quality, timing, interruptions, and paralinguistic behaviour.Multi-turn evaluation also remains concentrated in English and high-resource settings, while multilingual, dialectal, and culturally grounded evaluation is split across separate resources.

7 Challenges, and Future Directions

Sustained multi-turn competence remains unresolved despite progress in datasets, modeling, training, and evaluation. Future systems must preserve and update state, ground responses across turns and modalities, support speech and revision, evaluate sessions robustly, and adapt across cultures.

  • Core challenge: Stronger single-turn models do not ensure multi-turn competence, which requires preserving context, revising assumptions, grounding responses, coordinating tools, and maintaining quality across diverse sessions.The paper identifies memory, cross-turn grounding, speech-native interaction, robust revision, and session-level evaluation as central needs.
  • Persistent memory and state management: Persistent memory systems should separate dialogue state from task-specific working memory and support retrieving, updating, forgetting, and applying information across turns and sessions.Flat context windows remain insufficient as preferences, facts, and task states change over time.
  • Cross-turn grounding: Selective grounding should identify relevant history, evidence, and modality streams because retrieval noise and cross-turn decay weaken responses grounded in dialogue, external evidence, visuals, acoustics, and tools.Structured dialogue state is needed instead of treating full history as a flat input sequence.
  • Speech-native and full-duplex interaction: Speech-native full-duplex systems must handle timing, prosody, interruptions, repairs, noise, paralinguistic cues, simultaneous speech, clarification, and voice-grounded tool use.The paper argues that speech should be treated as an interaction medium rather than only an input modality.
  • Robustness and revision: Session-level robustness requires detecting uncertainty, recovering from mistakes, revising early assumptions after corrections, resisting adversarial escalation, and maintaining safety throughout interaction.The challenge includes failures caused by accumulating individually benign turns.
  • Evaluation and interaction scope: Evaluation should shift from turn-level scoring to unified session-level assessment spanning memory, grounding, tools, speech, safety, satisfaction, multilingual culture, and continuous adaptive interaction.The paper also calls for culturally grounded multilingual suites and interaction models supporting proactive, streaming, emotion-aware, and mixed-initiative exchange.

8 Conclusion

Multi-turn conversational AI is moving toward sustained interaction across modalities, tools, and culturally diverse contexts, but session-level competence lags behind modality support. Current systems increasingly perceive, speak, and act while still struggling with memory, grounding, external state, spoken timing, and cross-cultural adaptation.

  • Conclusion: Modality support has advanced faster than session-level competence in multi-turn conversational AI.The shift encompasses speech, vision, video, tools, and culturally diverse contexts.
  • Conclusion: Current systems increasingly perceive, speak, and act across sustained interactions.These capabilities span speech, vision, video, and tools.
  • Conclusion: Persistent challenges include maintaining memory, revising assumptions, preserving grounding, coordinating external state, handling spoken timing, and adapting across languages and cultures.The conclusion identifies these as continuing limitations despite advances in multimodal interaction.

Limitations

The study reviews recent multi-turn conversational AI research across data, models, training, and evaluation, but its coverage and categorization remain necessarily broad. Rapidly expanding literature may leave concurrent resources unincluded, while broad system categories can obscure important differences.

  • The review covers recent multi-turn conversational AI work across data, models, training, and evaluation.
  • Rapidly growing literature means some new or concurrent resources may not be included.
  • Broad system categories may hide finer differences in architecture, training data, deployment setting, or evaluation setups.

Societal/Broader Impact

The study clarifies the capabilities, gaps, and risks of multi-turn conversational AI while organizing prior work to guide systems supporting long-horizon, multilingual, speech-based, multimodal, and culturally grounded interaction.

  • Societal/Broader Impact: The review organizes prior work across datasets, models, training, and evaluation to clarify capabilities, gaps, and risks in multi-turn conversational AI.Its intended impact is to help researchers design systems for sustained interaction and broader access.
  • Societal/Broader Impact: The study supports research on long-horizon interaction, multilingual access, speech-based interfaces, multimodal assistance, and culturally grounded communication.These areas are presented as target benefits of organizing the existing literature.

Ethical Considerations … A.3 Details on Evaluation

The survey organizes multi-turn conversational AI around resources, modeling and training paradigms, and evaluation, while emphasizing persistent memory, cross-turn grounding, multimodal interaction, tool use, and cultural alignment. Across modalities and settings, benchmarks increasingly test realistic multi-turn capabilities, but explicit long-horizon memory and grounding remain underdeveloped.

  • Ethical Considerations / A.3 Details on Evaluation: The study does not introduce new datasets or conduct experiments with human participants, and it summarizes representative evaluation metrics and frameworks in Tables 9 and 10.The evaluation discussion compares evaluation families by focus, suitable settings, and limitations.
  • A.3 Details on Evaluation: Evaluation spans fine-grained multi-turn capabilities, long-term memory, spoken interaction, multimodal reasoning, cultural alignment, and tool-augmented behavior.Benchmarks test recollection, refinement, instruction retention, versioned editing, emotional-state changes, visual error propagation, cultural consistency, and stateful interaction.
  • A.2 Modeling Paradigms and Training Strategies: Multi-turn systems evolved from modular and Transformer dialogue architectures toward LLM-based, audio-native, omni-modal, and agentic interaction.Classical systems separated understanding, state tracking, policy, and generation, whereas newer systems integrate speech, vision, external actions, and general-purpose instruction following.
  • A.2.4 AudioLLMs: AudioLLMs improve speech-native and real-time interaction, but most still rely on the underlying language model for dialogue history and have limited long-horizon spoken memory and acoustic grounding.Full-duplex systems address low latency, interruption handling, turn timing, and controllable barge-in, while FD-Bench measures simultaneous bidirectional speech behavior.
  • A.2.5 Omni-modal Models: Omni-modal models combine vision, text, speech, and sometimes video, while newer methods add explicit note-taking, memory blocks, and memory-perception-planning-execution loops.Despite these mechanisms, explicit cross-turn memory and long-horizon grounding remain less developed than the broader multimodal interaction capabilities.
  • A.2.6 Agentic and Tool-augmented Dialogue: Agentic dialogue systems must track external state across actions, replan as goals or environments change, and maintain memory over extended interactions.ReAct, Reflexion, MetaGPT, ToolSandbox, xLAM-2, WebLINX, and SAPIENT illustrate reasoning, feedback, stateful tool use, function calling, navigation, and recommendation.
  • A.2.7–A.2.11 Training Strategies: Training strategies extend beyond isolated-response supervision to cross-turn preference optimization, multi-task learning, synthetic simulation, and conversational retrieval.These methods target delayed credit assignment, shared dialogue objectives, controllable multi-turn behavior, evidence retrieval, history selection, and evolving conversational context.

A.3.1 Evaluation Families · A.4 Search Keywords · B Comparison with Related Surveys

The survey organizes multi-turn evaluation into complementary families spanning surface and task metrics, session-level behavior, agentic state, speech interaction, and human or judge-based assessment, while noting that no single method captures complete session competence. Its literature search covers six keyword groups across modalities, methods, evaluation, and cultural-linguistic settings, and frames the survey around complete multi-turn sessions across modalities, training, evaluation, and cultural-linguistic contexts.

  • A.3.1 Evaluation Families: Surface-form metrics score individual responses, whereas task-oriented metrics track dialogue states and outcomes but remain task-specific and often reduce success to binary results.BLEU, ROUGE, and BERTScore provide useful baselines; joint goal accuracy, Slot-F1, Inform, and Success better reflect task completion.
  • A.3.1 Evaluation Families: Session-level metrics assess cross-turn behaviors including memory recall, instruction retention, constraint satisfaction, feedback integration, and dialogue-level hallucination.Representative approaches include APR, ARS, PWC, MMRC, MT-Eval, WCSR, and IHEval.
  • A.3.1 Evaluation Families: Agentic and retrieval-grounded evaluation measures responses and actions, making external state, side effects, and repeated-trial reliability central concerns.CORAL evaluates retrieval, generation, and citation attribution; τ-Bench, ToolSandbox, and AgentBoard assess repeated-trial success, stateful interactions, and subgoal progress.
  • A.3.1 Evaluation Families: Speech-native evaluation combines semantic content with interaction dynamics, including spoken clarification, vocal and ambient cues, speech–text matching, and first-packet latency.ADU-Bench, MTalk-Bench, ASK-QA, and URO-Bench represent these evaluation emphases.
  • A.3.1 Evaluation Families: LLM-as-a-judge and human evaluation address difficult-to-automate dimensions such as open-ended coherence, persona consistency, multimodal grounding, and human-likeness, but no single approach fully measures session-level competence.Judge-based approaches include MT-Bench and GPTScore, while ABC-Eval, MMDU, MTalk-Bench, and WildBench include human or hybrid assessment.
  • A.4 Search Keywords: The literature search used six query groups covering core multi-turn dialogue, spoken and audio interaction, multimodal and omni-modal dialogue, training and alignment, evaluation, and cultural and multilingual settings.Within groups, OR connected alternatives and AND combined complementary concepts to broaden coverage across modalities, methods, benchmarks, and language settings.
  • A.4 Search Keywords: Search terms covered dialogue concepts, speech and audio, multimodal and omni-modal systems, training and alignment, retrieval and memory, evaluation frameworks, and cultural or multilingual grounding.The keyword inventory included representative model families, optimization methods, evaluation terms, and language- or culture-specific concepts.
  • B Comparison with Related Surveys: Compared with related surveys, this survey centers the complete multi-turn session and analyzes session-level competence across modalities, training, evaluation, and cultural–linguistic settings.The comparison examines conceptual framing, modality coverage, and key distinctions among closely related surveys.
Loading 2608.17605v1…