Chain of Evidences and Evidence to Generate: Prompting for Context Grounded and Retrieval Augmented Reasoning

1 Introduction

• Existing reasoning methods do not address context-grounded or retrieval-augmented generation, and long retrieved text can mislead models despite valid reasoning steps.Multi-step methods may also require rigorous intermediate-step verification and annotated exemplars.Source: §1 · pp. 1–2
• Chain of Evidences (COE) prompts models to produce reasoning steps explicitly stated in the context, treating those extracted rationales as evidence.This contrasts with prompting that merely asks models to think step by step.Source: §1 · p. 2
• Evidence to Generate (E2G) first generates an answer with evidence, then uses that evidence to produce the final answer.The second step can use evidence alone or evidence together with the original context.Source: Figure 2 · p. 2, §1 · p. 3
• Across eight context-driven tasks, the prompts consistently outperform existing approaches, and the authors release prompts and outputs as an instruction-tuning dataset.The tasks include natural QA, multi-hop and long-form QA, fact checking, dialogue generation, and reading comprehension.Source: §1 · p. 3

2.1 Prompting LLMs

• Prior prompting research includes search-based, program-aided, self-generated-prompt, self-evaluation, multi-step, and domain-specific approaches.The paper presents this as a non-exhaustive overview.Source: §2.1 · p. 3

2.2 Chain-of-Thoughts (CoT) Prompting

• Chain-of-thought prompting guides LLMs to produce intermediate reasoning steps toward a final answer.The original approach uses question–reasoning–answer exemplars, while a zero-shot version emerged to avoid labeled exemplars.Source: §2.2 · p. 3
• Zero-shot CoT uses a general instruction such as “think step by step” to elicit intermediate reasoning.The supplied passage continues this description across a sentence boundary.Source: §2.2 · p. 3

3 Our Prompting Framework

• The framework offers two exemplar-free prompting strategies: single-step Chain of Evidences and dual-step Evidence to Generate, which applies COE twice.It is designed for context-grounded and retrieval-augmented long-text reasoning.Source: §3 · p. 3

3.1 System/Objective Instruction

• The system/objective prompt is single-intent, addressing one target task at a time.Its template specifies a task agent, context and task input, and the requested task output.Source: §3.1 · p. 3

3.2 Chain of Evidences (COE)

• COE asks for an answer with evidence and explanation from the context, unlike zero-shot CoT’s instruction to think step by step.The paper defines COE-SHORT and COE-LONG alternatives.Source: §3.2 · p. 3
• Task-specific instructions can clarify how to generate answers and format outputs; the framework’s prompts include evidence-and-explanation instructions.Table 1 is described as summarizing recommended mono- or two-step prompts and contexts.Source: §3.2 · pp. 3–4, Table 1 · p. 4
• COE-LONG adds step-by-step reasoning and tends to be more accurate for commonsense, multi-step reasoning, or arithmetic, while COE-SHORT is more cost-effective.COE-LONG produces more verbose and expanded reasoning paths than COE-SHORT.Source: §3.2 · p. 4

3.3 Adaptation

• Single-step COE is generally sufficient, while E2G is used for longer contexts; COE-SHORT favors cost-effectiveness and COE-LONG offers more granular reasoning.The latter may enhance performance in context-less arithmetic and commonsense tasks.Source: §3.3 · p. 4
• For G-step queries with multiple sub-queries or answers, the context should combine E-step evidence with the original context when evidence alone may be partial.Otherwise, the G-step context is typically derived from the E-step evidence.Source: §3.3 · p. 4
• The approach’s worst-case runtime is approximately twice CoT’s, but shorter evidence reduces runtime; single-step COE can serve tighter inference-time constraints.The passage reports 1.5 seconds versus CoT’s 1 second on average.Source: §3.3 · p. 4

4 Experimental Setup

• The experiments evaluate eight context-intensive tasks with ChatGPT, GPT-4, and PaLM-2 (540B) through APIs.The tasks require reasoning over given contexts, including distractor documents and retrieval augmentation.Source: §4 · p. 4
• Because of dataset size, evaluation uses sampling and development splits, compares with CoT and other literature baselines, and reproduces zero-shot CoT where needed.Source: §4 · p. 5
• Retrieval tasks use datasets with DPR-retrieved top-five Wikipedia documents as context.The benchmark summary table describes datasets, average context token length, and ChatGPT/PaLM-2 columns.Source: §4 · p. 5, Table 2 · p. 5
• By default, LogiQA and DROP use single-step COE-LONG, while other tasks use two-step E2G with COE-SHORT and evidence-sourced G-step contexts.The setup notes that these choices can be overridden.Source: §4 · p. 5

Arithmetic/Logical Context Reasoning

• 9% and 11% higher Acc: GPT-4 COE-LONG beats CR and ToT, respectively, on LogiQA; on DROP, COE-LONG scores 83.3 F1 versus 82.7 for COE-SHORT.The paper also reports PaLM-2 with COE-LONG outperforming Gemini Ultra's few-shot F1 on DROP.Source: Table 3 · p. 5, §5 · p. 5, Table 5 · p. 6
• COE-LONG exceeds COE-SHORT by around 2% on LogiQA and 0.6% on DROP, while the method performs as well as CoT on simple context-free math tasks.The authors report that COE-LONG is more context-driven and modular, while COE-SHORT is more cost-effective.Source: §5 · p. 5

Multi-hop QA with Distracting Contexts

• 4% gains in both EM and F1 over CoT with ChatGPT, and 2% with PaLM-2, on HotpotQA; a three-agent alternative trails by 10 EM points.HotpotQA queries contain two relevant and eight irrelevant documents, with only 2–5 distant sentences serving as rationales.Source: Table 6 · p. 6, §5 · p. 6
• With distracting context, CoT can override contextual facts, while E2G emphasizes evidence to support factually verified reasoning.Table 7 illustrates this contrast.Source: §5 · pp. 6–7, Table 7 · p. 7

Open-ended Long Form Generation

• On open-ended generation, E2G shows a slight gain over CoT in WoW, and human evaluation favored its responses 71%; both benchmarks lag recent supervised SOTA models.The human evaluation considered factual correctness, similarity to gold responses, and naturalness.Source: Table 8 · p. 7, §5 · p. 8

Retrieval Augmented Generation

• On FEVER, E2G outperforms strong baselines by more than 4% across both LLMs, with gaps over 2% against CoT-SC.The authors attribute the advantage to evidence capturing rationales for claim evaluation and to global problem solving.Source: §5 · p. 7
• For multi-answer questions, using Evidence plus the original context in the G-step gives the model another chance to answer fully; an adaptive context choice improves performance further.The heuristic uses both contexts for well-formed questions and Evidence alone for ill-formed ones.Source: §5 · p. 7, Figure 3 · p. 7, Table 9 · p. 8

6 Qualitative Case Study: Why and How Our Methods Work?

• The 50-example HotpotQA case study identified four CoT error types, including challenging long-context questions and reasoning absent from context.The other types were questions that are very hard to reason through and relevant text located in the middle or bottom of retrieved context.Source: §6 · p. 8
• CoE shows an 8% lower error rate for cases where reasoning is not mentioned in the context.This category accounted for 23% of erroneous wh questions and 25% of erroneous yes/no questions.Source: §6 · p. 8
• 23% of erroneous wh questions contained the gold answer in the reasoning, while 75% of erroneous yes/no rationales supported the opposite answer.The authors cite these observations as support for using reasoning to answer questions with a two-step E2G prompt.Source: §6 · p. 8

7 Error Analysis and Challenges

• Overemphasizing context grounding can hinder common-sense, world-knowledge, arithmetic, and logic use, sometimes producing “unknown” or “cannot be determined.”The paper also notes persisted hallucination to some extent.Source: §7 · p. 9
• On ELI5, responses remain short for a verbose-answer task, which the authors suggest may contribute to the performance drop.The authors also report lower retrieval recalls for WoW and ELI5 than for their other RAG tasks.Source: §7 · p. 9
• The authors identify persisted hallucination to some extent as a limitation of the framework.They report this alongside overemphasis in grounding and low performance in long-form generation.Source: §7 · p. 9

8 Conclusion

• The framework introduces two prompting methods that identify evidence in context and generate answers based on that evidence.The conclusion frames these methods as addressing limitations in prompting for context-aware and retrieval-augmented reasoning.Source: §8 · p. 9
• The authors identify LLM instruction fine-tuning using their prompted outputs as future work.Source: §8 · p. 9
• Future work involves instruction fine-tuning LLMs using the prompted outputs.Source: §8 · p. 9

9 Limitations

• The reported gains may not hold in domains such as biomedical data or in low-resource languages, under different evaluation metrics, or with sufficient computational resources or LLMs.Source: §9 · p. 9
• RAG evaluations use the top 5 retrieved documents without context filtering, and the authors do not fine-tune models for the tasks.Different retrieval accuracy or these changed settings may produce different results.Source: §9 · p. 9
• Results may vary with a different number of test instances in the benchmark datasets.Source: §9 · p. 9

• Participants were informed about the survey’s voluntary nature, estimated time, and feedback criteria.Source: Ethics · p. 9

A.1 Additional Experiments on Generalizability

• The appendix proposes tests on 100 randomly sampled LogiQA and DROP examples with Gemini-Pro, and on ReCLOR with ChatGPT.These experiments are presented as a discussion of generalizability to new models and benchmark datasets.Source: §A.1 · p. 12

A.3 Qualitative Examples

• The appendix includes LogiQA examples comparing prompting approaches, examples of E2G versus CoT model inputs and outputs, and correct and incorrect outcomes.Source: Figure 5 · p. 13, Figure 6 · p. 14, Figure 7 · p. 15

A.2 Self-Reflection: Why COE Works?

• LLM reflections describe evidence-focused prompting as encouraging structured reasoning and reducing unsupported statements.Source: §A.2 · p. 13
• Reflections say asking for evidence and explanation encourages context-grounded answers based on relevant facts, rather than imaginary or unsupported claims.Source: §A.2 · p. 13
• The reflections say evidence prompting encourages reliance on known information or reasonable inferences and makes models accountable for answer accuracy and reliability.Source: §A.2 · p. 13