Source-linked AI summary
Chain-of-Discussion: A Multi-Model Framework for Complex Evidence-Based Question Answering
Mingxu Tao, Dongyan Zhao, Yansong Feng
TL;DR
Complex open-ended QA remains challenging because models must select reliable evidence and analyze questions in depth while producing well-supported, comprehensive answers. Chain-of-Discussion has multiple open-source LLMs summarize, criticize, and revise one another’s analyses. The paper reports that this framework improves response quality for small open-source models on a challenging legal consultation dataset.
Problem
Complex open-ended QA remains challenging even with retrieval, as models must analyze questions and select evidence to produce correct, well-reasoned, comprehensive answers.
Source: §1 · p. 1Method
Results
CoD improves open-source LLM evaluation results over baseline CoT for all four tested models: Baichuan2-7B +0.340, Deepseek-7B +0.115, Qwen-7B +0.120, and Xverse-7B +0.110.
Source: §1 · p. 2, §5.3 · p. 6, §7 · p. 9Takeaways & Limitations
On a challenging legal consultation dataset, discussions and criticism help open-source LLMs improve response quality in correctness and comprehensiveness.
Source: §1 · p. 2, §7 · p. 9The small open-source models may lack sufficient legal or background knowledge and reasoning ability, and may fail to follow instructions or integrate corrections to evidence analyses.
Source: §6.3 · p. 8
Abstract
from arXiv · showhide
Open-ended question answering requires models to find appropriate evidence to form wellreasoned, comprehensive and helpful answers. In practical applications, models also need to engage in extended discussions on potential scenarios closely relevant to the question. With augmentation of retrieval module, open-source Large Language Models (LLMs) can produce coherent answers often with different focuses, but are still sub-optimal in terms of reliable evidence selection and in-depth question analysis. In this paper, we propose a novel Chain-ofDiscussion framework to leverage the synergy among multiple open-source LLMs aiming to provide more correct and more comprehensive answers for open-ended QA, although they are not strong enough individually. Our experiments show that discussions among multiple LLMs play a vital role in enhancing the quality of answers.
1 Introduction
Complex open-ended QA requires evidence-grounded, well-reasoned answers that also address helpful related scenarios, yet retrieval-augmented LLMs can struggle with noisy evidence and in-depth analysis. The paper proposes Chain-of-Discussion, where multiple open-source LLMs summarize, criticize, and revise one another, and introduces a Chinese legal consultation dataset.
1 Introduction
- Complex open-ended QA remains challenging even with retrieval, requiring models to analyze questions, select evidence, and produce supported, detailed answers.Noisy retrieval can lead to incomplete analysis and misleading answers, while models may miss useful potential scenarios.Source: §1 · p. 1
- The authors collect 200 carefully annotated Chinese legal consultation questions in marriage and family affairs for complex evidence-based QA.The dataset includes annotated evidence and answers.Source: §1 · pp. 1–2
- Chain-of-Discussion has multiple LLMs summarize, criticize, and revise one another’s outputs to produce well-supported, helpful responses.The authors motivate this design by models’ differing knowledge and reasoning, and by the value of gathering diverse perspectives.Source: §1 · p. 2
- The paper studies collaboration among small open-source LLMs and reports improvements in correctness and comprehensiveness using GPT-4-based and evidence-centric evaluations.The authors contrast this focus with prior multi-model interaction work using strong closed-source LLMs.Source: §1 · p. 2
2 Related Works
The paper situates its approach against retrieval-augmented generation and chain-of-thought work. It emphasizes that complex QA requires both filtering irrelevant evidence and retaining material useful for related scenarios, and studies multi-model rather than single-model checking.
Retrieval-Augmented Generation
- Retrieval-augmented generation retrieves evidence before generation, but complex QA also requires retaining evidence useful for potential scenarios, even when it does not directly answer the question.The paper notes that deciding which evidence supports such scenarios can be difficult even for humans.Source: §2 · p. 3
Chain-of-Thought
- Chain-of-thought decomposes complex questions and has been used for single-model self-checking, whereas this paper studies interactive checking and regeneration across multiple models.The cited examples include Chain-of-Verification, where a model generates queries and revises its answer based on verification results.Source: §2 · p. 3
3 Preliminaries
The paper defines complex evidence-based QA as analyzing a question, finding relevant evidence, and generating a response. It expects answers to be both correct and comprehensive, and uses a multi-step CoT baseline for this task.
Task Definition
- In complex evidence-based QA, a model analyzes a user question, selects necessary evidence documents, and generates a paragraph as its response.The legal consultation example involves finding supporting law articles or prior legal cases to answer the user’s situation.Source: §3 · p. 3
- A correct response relies on supporting evidence without irrelevant evidence or out-of-context interpretation, while a comprehensive response discusses helpful potential scenarios.Those scenarios may not be explicitly mentioned in the question.Source: §3 · p. 3
- Models must filter irrelevant retrieved evidence while retaining evidence that can inform potential scenarios, even when it does not directly answer the question.The pilot study found that one small-scale LLM could generate fluent answers but often produced incomplete analysis or wrong reasoning paths.Source: §3 · p. 3
Baseline Framework: CoT
- The CoT baseline first analyzes the question and then judges whether each evidence document can support answering it.Question analysis identifies user roles and explicit or implicit needs, as well as the types of evidence required.Source: §3 · p. 3
4 CoD: Summarize, Criticize, and Revise
Chain-of-Discussion uses multiple LLMs to address limitations in individual models through discussion. Its stages gather question analyses, critique evidence analyses, revise when critiques reach a threshold, and generate a response from the resulting analyses.
4 CoD: Summarize, Criticize, and Revise
- CoD uses interactive discussions among multiple LLMs to address shortcomings in individual models’ intrinsic knowledge.The two-stage pipeline summarizes other models’ question analyses, gathers critiques of evidence analyses, and uses the resulting analyses for response generation.Source: §4 · p. 4
4.3 Response Generation
- The target model generates the final response using the summarized question analysis and revised evidence analysis, with prompts similar to the baseline’s for comparison.The response is generated from the evidence analyses and summarized question analysis.Source: §4 · p. 4, §4.3 · p. 4
4.1 Stage 1: Question Analysis
- In question analysis, models consider question facts, user needs, and potential scenarios, using retrieved evidence as a cue for scenarios absent from the question.The target model summarizes the models’ analyses, considering both consistency across models and reasoned minority viewpoints for relevance and coherence.Source: §4.1 · p. 4
4.2 Stage 2: Evidence Analysis
- In evidence analysis, other LLMs criticize the target model’s assessment of each document, and the target model revises when opposing critiques exceed threshold δ.The authors assume revision critiques are reliable only when a majority of critic models reach consensus; otherwise, the original analysis is retained.Source: §4.2 · p. 4
5 Experiments
The experiments evaluate Chain-of-Discussion on a newly collected Chinese legal-consultation CEBQA dataset using open- and closed-source LLMs. They report gains from multi-model discussion, with greater improvement when both stages are used, while model-capability differences remain.
5 Experiments
- With no existing CEBQA dataset, the authors collected the first one for legal consultation and used Chinese legal articles and interpretations as its evidence store.The paper situates this choice in China's civil-law system, where legal activities are based on law articles or judicial interpretations.Source: §5 · p. 5
5.1 Data Collection
- From 609 real-user questions, an expert selected 200 semantically distinct questions spanning marriage, family affairs, and inheritance disputes.Examples include divorce, custody, contracts, and property disputes.Source: §5.1 · p. 5
5.2 Experimental Setup
- The open-source experiments target four fine-tuned 7B LLMs, with the other three supplying diverse question analyses and critiques of the target model's evidence analysis.The tested models are Baichuan2-7B, Deepseek-7B, Qwen-7B, and Xverse-7B2.Source: §5.2 · p. 5
- The comparison includes single-model, Stage 1-only, Stage 2-only, and two-stage settings; GPT-4 scores responses for correctness and comprehensiveness.The authors explain that F1 and exact match are unsuitable for these long, knowledge-intensive responses with facts and causal relations to verify.Source: §5.2 · pp. 5–6
5.3 Main Results
- Against BS, S1S2 raised GPT-4 scores by +0.340 for Baichuan2-7B, +0.115 for Deepseek-7B, +0.120 for Qwen-7B, and + 0.110 for Xverse-7B.Table 1 reports average scores for each target LLM and setting, separately for open- and closed-source groups.Source: §5.3 · p. 6, Table 1 · p. 6
- Using discussion at both stages brought more improvement than using it at only one stage.The two stages are question-analysis discussion and article-analysis criticism.Source: §5.2 · p. 5, §5.3 · p. 6
- CoD-augmented Baichuan2-7B, Qwen-7B, and Xverse-7B still trailed baseline Deepseek-7B by around 0.5.The authors suggest response quality may depend primarily on a model's inherent ability to analyze context and generate responses.Source: §5.3 · p. 6
- CoD also consistently improved powerful closed-source LLMs, although their gains were small compared with those for open-source models.Manual checks found that powerful closed-source models could still produce erroneous question analyses that led to incorrect responses.Source: §5.3 · p. 6
6 Discussions
CoD improves necessary- and optional-evidence accuracy across the evaluated models, while human evaluation also finds response improvements. The discussion framework remains constrained by 7B models’ difficulty handling legal nuance and revising analyses.
6.1 Evidence-Centric Evaluation
- Simple majority voting reached 77.62% N-Acc; question-analysis discussion contributed more to comprehensiveness, while evidence-stage discussion improved correctness by filtering irrelevant evidence.The authors report that discussions at a single stage can improve correctness and comprehensiveness.Source: §6.1 · pp. 6–7
- Around 2% improvement on both N-Acc and O-Acc for Baichuan2-7B, Qwen-7B, and Xverse-7B; Deepseek-7B gained 0.86% N-Acc and 0.53% O-Acc.The table reports macro-average results by target LLM and setting.Source: Table 2 · p. 7, §6.1 · p. 7
6.2 Human Evaluation
- In human evaluation of 50 randomly sampled examples, 40% of Xverse+CoD responses surpassed baseline; after CoD, 16% of Baichuan-7B and 12% of Xverse responses deteriorated.Two human experts compared CoD-enhanced model responses with baseline responses.Source: §6.2 · p. 7, Figure 2 · p. 7
- In 30 manually examined responses, errors included misunderstanding legal terms (60%), discussing not-required articles (15%), and inconsistent explanations (10%); CoD corrected improper articles in 6 out of 30 cases.The authors note that inconsistent explanations often stemmed from term misunderstanding.Source: §6.2 · p. 7
6.3 Limitations of Open-Source LLMs
- The authors identify limited legal or background knowledge, reasoning, and instruction-following as difficulties for 7B models discerning others’ errors and revising evidence analysis.An example describes Baichuan2-7B incorporating Deepseek-7B’s inaccurate, irrelevant analysis into a summary.Source: §6.3 · p. 8
- Shallow semantic similarity can blur legally distinct concepts, while differences between question wording and law-article style can also hinder evidence-analysis revision.Examples include confusing parent support with child-rearing and conflating 复婚 with 再婚.Source: §6.3 · p. 8
6.3.1 Model Preference of Revising
- With Baichuan2-7B analyses, Baichuan2-7B revised 96.5% versus Qwen-7B’s 56.1%; with Qwen-7B analyses, the rates were 92.5% and 67.2%, respectively.Deepseek-7B and Xverse-7B served as critics in both comparisons.Source: §6.3.1 · pp. 8–9
- A model’s preference to refuse revisions may prevent better evidence analysis based on critiques and limit the improvement CoD can deliver.The authors note that supervised fine-tuning and reward modeling can affect revision preferences.Source: §6.3.1 · p. 9
7 Conclusions
The paper concludes that Chain-of-Discussion uses discussions among multiple LLMs to improve complex evidence-based question answering. Experiments on a challenging legal consultation dataset show improved open-source LLM performance through discussion and criticism.
7 Conclusions
- Experiments on a challenging legal consultation dataset show that Chain-of-Discussion can improve open-source LLM performance by encouraging models to discuss and criticize.Source: §7 · p. 9
- Chain-of-Discussion involves multiple LLMs discussing responses to complex evidence-based questions.Source: §7 · p. 9
- The framework aims to produce more correct and comprehensive responses with less hallucination and more supportive evidence.Source: §7 · p. 9
Limitations
The authors caution that Chain-of-Discussion cannot guarantee completely correct legal-advisory responses. They also note limits in the expert accuracy of dataset annotations and the multi-round framework’s time and model scope.
Limitations
- The framework cannot guarantee that all legal-advisory responses are completely correct, so further applications require caution.Source: Limitations · p. 9
- The dataset annotations may not be fully accurate from civil-law experts’ professional perspectives, and the data are for research purposes only.Source: Limitations · p. 9
- CoD uses multiple LLMs across several rounds, while directly using commercial APIs may produce more promising results and take less time.Source: Limitations · p. 9
- The framework’s stated aim is to validate exploiting synergy among small open-source LLMs without relying on larger LLMs.Source: Limitations · p. 9
Ethics Statement
The paper describes publicly accessible web-sourced consultation data and privacy screening, while the generation details specify sampling settings and in-context examples.
Ethics Statement
- Consultation question-answer pairs were collected through web search from publicly accessible websites that allowed search-engine crawlers.Source: Ethics Statement · p. 9
- Instances that could identify users by name, address, or phone number were removed, while gender, age, and family composition were retained as non-identifying judicial context.Source: Ethics Statement · p. 9
A Details of Generation
- Generation used temperature 0.3, repetition penalty 1.05, top-p 0.8, and two human-annotated in-context examples per step.Source: A Details of Generation · p. 12
B Details of Evidence Annotation
Evidence annotation combines classifier retrieval with legal-background annotators and a three-category article scheme. In baseline comparisons, real-vanilla CoT performs much worse across four LLMs, with question-analysis and self-criticism weaknesses reported for 7B models.
B Details of Evidence Annotation
- Evidence annotation fine-tunes RoBERTa-large on 80K examples, retains the top ten classifier-ranked articles, and uses six legal-background annotators to supplement missing articles.Source: B Details of Evidence Annotation · p. 12
- Annotators label articles as necessary, optional, or not required, corresponding to scores of 2, 1, and 0; average scores above 1.66 mark necessary articles, below 0.67 not required, and the remainder optional.Source: B Details of Evidence Annotation · p. 12
- To imitate retrieval-augmented generation, each question receives five articles, prioritizing all necessary and optional articles before classifier-ranked not-required articles.Source: B Details of Evidence Annotation · p. 12
C.1 Results of Other Baselines
- Real-vanilla CoT performs much worse across all four LLMs; without question analysis, 7B models fail to select correct articles, resulting in erroneous responses.Source: Table 5 · p. 12, §C.1 · p. 12
- The self-criticism baseline asks the target LLM to review its own evidence analysis, and the authors report that 7B models may fail to offer useful revision insights.Source: §C.1 · p. 12
C.2 Results under More Evidences
With more candidate articles, response quality falls slightly, though Chain-of-Discussion brings greater improvement for Baichuan2-7B and Deepseek-7B. The section also specifies its article-reference test, checks response-length bias, and illustrates a commonsense reasoning failure.
C.2 Results under More Evidences
- As candidate articles rise from 5 to 7 or 10, both systems score worse, while CoD's improvement grows for Baichuan2-7B and Deepseek-7B.The authors attribute the decline to noise from more irrelevant articles.Source: §C.2 · pp. 12–13, Table 6 · p. 13
D Rules to Identify the Article Reference
- An article counts as referenced when its number appears in a response or a sentence's longest common subsequence exceeds one-third of the article; otherwise it is not counted.Source: D Rules to Identify the Article… · p. 13
E.1 Lengths of Generated Responses
- Response lengths can be similar across settings, and the authors find no evaluator preference for longer responses based on correlations with GPT-4 scores.Source: §E.1 · p. 13, Table 7 · p. 13
E.2 Examples required Reasoning with Commonsense Knowledge
- LLMs may focus on literal wording differences and miss the commonsense that only courts have authority to revoke guardianship.Source: §E.2 · p. 13, Table 8 · p. 13
E.3 Significant Test
This section specifies GPT-4’s rubric for rating assistant responses against reference answers and reports a five-run standard-error check on a 20-example subset, noting the difficulty of repeating the full evaluation because of its cost.
E.3 Significant Test
- 0.0245 (baseline) and 0.0510 (CoD) were the standard errors of average GPT-4 scores across five runs, each smaller than CoD’s improvement.The runs used Baichuan2-7B and the same randomly sampled 20 examples; a full evaluation across settings could cost $150 per run, making repeated evaluation cost thousands of dollars.Source: §E.3 · pp. 13–14
F.1 Prompt of Question Analysis
- The question-analysis prompt separates analysis of the consultant’s question from article analysis, and generation stops after the Article Analysis heading to prevent the model from producing both together.It asks the model to identify the question’s key points and assess whether each reference article is relevant.Source: §F.1 · p. 14
F.2 Prompt of Summary
- The summary prompt asks a legal expert to assess multiple lawyers’ analyses, weigh common points as more likely correct, scrutinize minority claims, and critique errors before writing a final analysis.The final summary should match the lawyers’ tone and style.Source: §F.2 · p. 14
F.3 Prompt of Evidence Analysis
- The evidence-analysis prompt evaluates each retrieved legal article individually for its relevance to the question, including the rights, obligations, or prohibited conduct it addresses.The prompt cautions that retrieved articles may not help answer the question and says not to judge other articles while assessing one.Source: §F.3 · p. 15
F.4 Prompt of Critique
- The critique prompt asks a legal expert to judge whether a lawyer’s analysis of an article’s usefulness for a consultation question misconstrues the article.It supplies the question, article, and lawyer’s analysis as inputs.Source: §F.4 · p. 15
F.5 Prompt of Revising
- The revision prompt asks a lawyer to rewrite an article analysis using expert critiques and then assess whether the article helps address the question.The inputs include the question, articles, the lawyer’s question and article analyses, and expert critiques.Source: §F.5 · p. 15
F.6 Prompt of Response
- The response prompt directs a legal expert to answer from relevant retrieved articles, determine each article’s relevance, and avoid relying on outside articles or unrelated scenarios.The answer should identify legal rights, obligations, or prohibited actions and focus on the consultant’s question.Source: §F.6 · p. 16
G Scoring Prompt of GPT-4
- GPT-4 is instructed to score responses from 1 to 10 against human-written answers and reference articles, considering logic, helpfulness, relevance, accuracy, depth, and use of irrelevant articles.The rubric describes score bands from 1–2 for failed answers through 7–8 for good answers; the supplied passage does not include the remaining band’s full mapping.Source: G Scoring Prompt of GPT-4 · p. 16
- Scores of 9–10 indicate the response outperforms the reference in most aspects, fully meets the instruction, and offers more detailed analysis or unique advantages.The prompt advises scores around 7 when no clear advantages or disadvantages are found.Source: G Scoring Prompt of GPT-4 · p. 16
- The evaluator must avoid order, response-length, and assistant-name biases, explain the rating in Chinese, and use the format “Rating: [[Number]].”The prompt cautions against extreme scores without sound explanations.Source: G Scoring Prompt of GPT-4 · p. 16
- The scoring prompt supplies the question, reference articles, reference answer, and assistant response as evaluation inputs.Source: G Scoring Prompt of GPT-4 · p. 16