PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompted LLMs as Chatbot Modules for Long Open-domain Conversation

Gibbeum Lee, Volker Hartmann, Jongho Park, Dimitris Papailiopoulos, Kangwook Lee

arXiv:2305.04533v1cs.CLcs.AIcs.LG

TL;DR

Fine-tuning LLMs for open-domain chatbots can be computationally burdensome and require scarce dialogue data. MPC uses pre-trained LLM modules with prompting and memory, and human evaluations found it outperformed the fine-tuned and vanilla baselines. Its findings are limited to English conversations, and modularity may add inference overhead or error accumulation.

  • Problem

    Fine-tuning LLMs for open-domain chatbots is computationally burdensome and challenging when representative dialogue data is scarce.

    Source: §1 · p. 1

  • Method

    MPC combines pre-trained LLMs in prompted modules with a memory pool that retrieves and condenses relevant information for response generation and later use.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    With a 9% SCE-p gap, MPCOPT-30B scores higher on all metrics than BB3-30B in human evaluation.

    Source: §5 · p. 4

  • Takeaways & Limitations

    Human evaluations found MPC effective for long-term open-domain chatbots without further fine-tuning, with better incorporation of persona and dialogue history than non-modular baselines.

    Source: §6 · p. 5

    The study evaluates long-term English conversations; whether the findings apply to other languages remains to be confirmed.

    Source: Limitations · p. 5

Abstract

from arXiv · show

In this paper, we propose MPC (Modular Prompted Chatbot), a new approach for creating high-quality conversational agents without the need for fine-tuning. Our method utilizes pre-trained large language models (LLMs) as individual modules for long-term consistency and flexibility, by using techniques such as few-shot prompting, chain-of-thought (CoT), and external memory. Our human evaluation results show that MPC is on par with fine-tuned chatbot models in open-domain conversations, making it an effective solution for creating consistent and engaging chatbots.

1 Introduction

Fine-tuning LLMs for open-domain chatbots is computationally burdensome, data-hungry, and can restrict versatility. MPC instead uses prompted pre-trained models without fine-tuning, and human evaluations found it on par with or preferred over fine-tuned chatbots.

1 Introduction

  • Fine-tuning billion-parameter LLMs for open-domain chatbots is computationally burdensome, data-scarce, and can reduce versatility by restricting domains or losing pre-training knowledge.Multi-task training can address versatility but still requires data for each skill and makes it difficult to determine which skills are needed.Source: §1 · p. 1
  • In-context learning lets an LLM perform a task from instructions and examples without additional fine-tuning.A prompt with a dialogue task and examples can guide generation of conversation-relevant information.Source: §1 · p. 1
  • MPC uses pre-trained language models, modular design, few-shot prompting, and Chain-of-Thought to support conversational capabilities and long-term consistency.Human evaluation found MPC on par with or preferred over fine-tuned LLM chatbots in open-domain conversation.Source: §1 · p. 1

2 Related Work

Prior work applies prompting and modularization to reasoning, conversation generation, and dialogue systems. MPC builds on this line of work while emphasizing reasoning-based prompts for contextual understanding and long-term memory.

Modular Prompting

Open-domain Chatbots

3 Modular Prompted Chatbot

MPC uses prompted LLM modules and a memory pool to support persona and engagement in long-term conversations. Its pipeline clarifies user messages, retrieves and condenses relevant memories, generates a response, and periodically stores dialogue summaries.

Modular Prompted Chatbot

  • MPC uses prompt-based LLMs to maintain persona and engagement across long-term conversations.A pre-defined persona is stored in the memory pool at the start, and dialogue summaries are added every few turns for future use.Source: §3 · p. 2

Dialogue Summarizer

  • Every few turns, a few-shot summarizer extracts important dialogue and user details and stores them in the memory pool for later conversations.The summarizer prompt is designed to capture specific details.Source: §3 · pp. 2–3

Utterance Clarifier

  • The clarifier uses recent dialogue to resolve ambiguous user messages and provide a more information-dense query for memory retrieval and processing.For example, it can rephrase “Do you like working there?” as “Does Sarah like working at ZYX company?”.Source: §3 · p. 3

Memory Processor

  • The memory processor uses prompted LLM reasoning, with Chain-of-Thought examples, to select and synthesize relevant memories.DPR retrieves the top-k memories using the clarified query, which the processor condenses into one refined memory; models unable to use Chain-of-Thought receive examples without reasoning text.Source: §3 · p. 3

Utterance Generator

  • The utterance generator produces the chatbot’s final response from recent dialogue, condensed memory, and a generation instruction.For some models, placing the instruction after the dialogue improves its effect.Source: §3 · p. 3

4 Experimental Setup

The experiments evaluate core skills for long-term conversation using human ratings and pairwise comparisons. They include single-model assessment and controlled comparisons of MPC with fine-tuned and vanilla baselines, including an implicit-persona setting.

4 Experimental Setup

  • Evaluators assessed five personas with 12 PersonaChat facts each, contributing 20 turns apiece and at least 500 turns across Mechanical Turk and university-student groups.The setup compared fine-tuned chatbots, vanilla prompted LMs, MPC, and MPC with the full persona prepended.Source: §4 · p. 3

4.1 Single Model Evaluation

  • Single-model evaluations rated sensibleness, consistency, engagingness, and overall quality across prompted LMs and fine-tuned BB3 models.The evaluation also reported SCE-p, requiring all component metrics to be positive, and weighted SCE-w.Source: §4.1 · p. 3

4.2 Pairwise Models Evaluation

  • Pairwise tests randomized responses A and B, controlled dialogue history, and compared MPCOPT-30B with BB3-30B and MPCtd2 with vanilla td2.The BB3 internet search feature was disabled for the direct consistency comparison; a separate implicit-persona experiment tested six new turns after a ten-turn dialogue, with context shorter than that dialogue.Source: §4.2 · pp. 3–4

5 Results

Human evaluations compare MPC with fine-tuned and vanilla chatbots, finding strong overall results alongside differences tied to persona setup, model size, and instruction tuning.

Pre-trained vs. Fine-tuned

  • A 9% SCE-p gap favors MPCOPT-30B over BB3-30B, with MPCOPT-30B scoring higher on all metrics.The majority of MPC models also outperform BB3-30B.Source: §5 · p. 4, Table 1 · p. 4
  • MPCOPT-30B remains on par with BB3-30B when repeated-utterance dialogues are excluded, and scores higher on sensibleness and preference while similar on consistency and interestingness.The reported repetition issue concerns consecutive utterances from BB3-30B.Source: §5 · p. 4

Modular vs. Non-modular

  • MPCtd2 scores significantly higher than td2 without persona in the implicit-persona experiment; with full persona, its consistency is on par, while engagingness, SCE-p, and rating are best overall.Without full persona, MPCtd2 has lower consistency than td2 with full persona, although sensibleness and consistency ties are frequent.Source: §5 · p. 4, Table 3 · p. 4, Table 4 · p. 5

Effect of Size

  • MPCOPT-66B is superior to MPCOPT-30B across all metrics, but model size alone does not explain evaluations: MPCBLOOM, one of the largest models, scores lowest.The comparison between OPT-66B and OPT-30B is especially informative because their base models were trained nearly identically.Source: §5 · p. 4

Effect of Instruction-tuning

  • Instruction-tuned models support modular dialogue: MPCtd2 outperforms MPCdavinci, while smaller MPCGPT-JT achieves high sensibleness and consistency.MPCdavinci underperforms davinci with full persona, which the authors attribute presumably to error propagation while not ruling out better prompts.Source: §5 · pp. 4–5

6 Conclusion

The paper presents MPC as an effective modular approach for long-term open-domain chatbots without further fine-tuning. Human evaluations find superior performance to fine-tuned and vanilla LM baselines, and more effective incorporation of persona and dialogue history for consistency.

6 Conclusion

  • MPC uses LLMs without further fine-tuning and outperforms fine-tuned and vanilla LM baselines in human evaluation.The modular system also incorporates persona and dialogue history more effectively than non-modular systems according to the consistency evaluation.Source: §6 · p. 5

Limitations

The paper’s findings are scoped to long-term English conversations, and the authors identify system, evaluation, subgroup-analysis, and hardware constraints. These boundaries limit how broadly the results can be applied.

Language scope

  • The study investigates long-term English conversations; applicability to other languages remains to be confirmed.Although BLOOM is multilingual, MPCBLOOM was implemented and evaluated only in English.Source: Limitations · p. 5

System design

  • Modular systems can add inference overhead or error accumulation, and selecting the best language model for each module remains future work.The authors say this would require evaluating every module.Source: Limitations · p. 5

Evaluation

  • Human evaluation has no definitive method for identifying the best chatbot, and results may reflect annotator subjectivity, instruction bias, and crowdworker conditions.The authors characterize these as challenges shared with prior interactive-conversation evaluations.Source: Limitations · p. 5

Subgroup analysis

  • Using MTurk annotators and university students may reduce cultural bias and platform homogeneity, but can complicate comparisons across subgroups.The authors note that controlling for MPC’s performance on different subgroups may be more difficult.Source: Limitations · p. 5

Hardware requirements

  • MPC requires at least as much memory as its underlying language model, making models such as BLOOM-176B infeasible to load on a single node.The authors state this hardware boundary specifically for heavy models.Source: Limitations · p. 5

Ethics Statement

The ethics statement notes safety concerns associated with publicly available pretrained language models and describes safeguards for data collection and crowdworker participation.

Safety

  • Because pretrained language models can produce toxic or biased text, the authors identify safe deployment and interaction as necessary.MPC uses publicly available pretrained language models to generate chatbot utterances.Source: Ethics Statement · p. 6

Data collection

  • Crowdworkers could give feedback, and outputs were manually checked for offensive or controversial content.Workers were instructed not to share private information and asked to consent to anonymous research data collection.Source: Ethics Statement · p. 6

Worker protections

  • Workers were informed of the study purpose before participating and received approximately $12–16 per hour.The authors describe the compensation as a competitive hourly rate.Source: Ethics Statement · p. 6

A.1 Metrics

The appendix specifies the human-evaluation metrics and model configurations, then describes the full single-model evaluation and related analysis. It also reports implementation details and evaluation constraints.

A.1 Metrics

  • The evaluation combines single-model and pairwise modes, with the latter comparing responses and the former rating individual chatbot responses.The single-model overall-conversation question uses a scale from 1 (very bad) to 5 (very good).Source: §A.1 · p. 8
  • Pairwise evaluation assesses sensibleness, persona consistency, and interestingness, while single-model evaluation assesses sensibleness, preference for a long conversation, and engagingness based on whether users want to continue.Source: §A.1 · p. 8

A.2 Models

  • Single-model tests include OPT, GPT-3, GPT-JT, and BLOOM pretrained LLMs alongside fine-tuned Blenderbot models; pairwise tests compare MPCOPT-30B with BB3-30B and test the full pipeline against a fixed-persona utterance generator.The authors note that BB3-175B comparisons are not fair because persona control and other factors cannot be matched.Source: §A.2 · p. 8

A.3 Dense Passage Retriever

  • The memory module uses a DPR model fine-tuned for MultiDoc2Dial, which the authors observed performed slightly better than the original DPR model.The original DPR model is attributed to Karpukhin et al. (2020).Source: §A.3 · p. 8

B Experimental Results

  • Section B reports experimental results and miscellaneous analysis.The supplied section introduction does not specify further findings.Source: B Experimental Results · p. 8

B.1 Full Single Model Evaluation

  • The full single-model evaluation includes latency and utterance-length details, and defines SCE-w using adjusted metrics averaged into a final score.SCE-w requires sensibleness for consistency and both sensibleness and consistency for engagingness; the appendix also notes latency variability in OpenAI API calls and a distinct BB3-175B evaluation procedure.Source: §B.1 · pp. 8–9
  • Table 5 presents full single-model evaluation results, including BB3-30B results with repeated-utterance conversations excluded.An asterisk marks experiments run only on MTurk.Source: Table 5 · p. 9

B.2 Model Configurations

  • Model configurations include decoding parameters, such as sampling method and temperature, and variations in utterance-generator prompts.The authors provide code and model configurations in an anonymous repository.Source: §B.2 · p. 9

B.3 Subgroup Analysis

The subgroup analysis finds MTurk and student ratings broadly similar, with sensibleness showing the greatest difference and BB3-30B as a notable exception. The appendices provide few-shot prompt and utterance-generator examples.

B.3 Subgroup Analysis

  • Students and MTurk workers showed no significant rating differences on most metrics, but BB3-30B differed significantly across all metrics, especially sensibleness (>20%).Students generally scored models slightly more harshly; the authors’ observation that students were less patient with BB3-30B repetition or topic shifts is qualitative.Source: §B.3 · p. 9, Figure 2 · p. 10

C Few-shot Prompts

  • Some few-shot examples ask the model to rephrase a user’s question in third person before answering it.Examples cover questions about earlier events, identities, and conversational state.Source: C Few-shot Prompts · p. 10
  • The Ashley memory example shows a CoT prompt reasoning over numbered facts to select relevant information and answer a question.The non-CoT example answers from the same facts without the step-by-step reasoning.Source: C Few-shot Prompts · p. 11

D Utterance Generator Prompt Examples

  • MPC utterance-generator examples combine persona facts, dialogue history, and an instruction to produce a friendly, sensible response ending with a question.The examples include both persona information and a response to the latest conversational turn.Source: D Utterance Generator Prompt Examples · p. 12

E Data Collection Details

The appendix describes evaluation materials, participant recruitment, and worker instructions. It reports that MTurk Master workers produced higher-quality evaluation data and documents a student group used to cross-validate MTurk results.

E Data Collection Details

E.1 MTurk Setting

  • The study recruited at least 15 qualified MTurk workers per task, requiring Master status, a HIT approval rate above 99%, over 100 approved HITs, and residence in specified countries.Eligible countries were the US, UK, Canada, Australia, and New Zealand.Source: §E.1 · p. 13
  • MTurk Master workers produced significantly better-quality data, with more fluent messages and fewer random or generic responses than non-Master workers.Non-Master workers sometimes averaged 99% scores despite clear repetition and contradiction problems in chatbot dialogue.Source: §E.1 · p. 13

E.2 Student Participants

  • For cross-validation, the authors gathered 49 English-proficient undergraduate or graduate students from several universities.Minimum proficiency thresholds were IELTS 7.0, TOEFL iBT 95, or TOEIC 900.Source: §E.2 · p. 13

E.3 Crowdworker Instruction

  • Workers continued chats for 20 turns, evaluated each response, and were instructed to write more than three words on average and avoid generic or meaningless replies.They were also asked to evaluate honestly and protect private information.Source: §E.3 · p. 13
Loading 2305.04533v1…