Source-linked AI summary
Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
Yihe Deng, Weitong Zhang, Zixiang Chen, Quanquan Gu
TL;DR
The paper addresses cases where LLMs misinterpret questions that appear clear to humans, making prompt quality and ambiguity important concerns. It introduces Rephrase and Respond, which rephrases questions before answering, and reports improved performance across diverse tasks, including compatibility with Chain-of-Thought prompting.
Problem
Humans and LLMs can interpret the same question differently, and benchmark ambiguities can lead models to answer unintended queries.
Method
Rephrase and Respond prompts an LLM to rearticulate and expand a question before responding, with a two-step variant transferring rephrased questions between models.
Results
Across 10 diverse tasks, RaR consistently improves GPT-4 performance, and the paper reports that its benefits transfer across models and complement CoT.
Takeaways & Limitations
RaR provides a training-free, unsupervised, zero-shot prompting method for improving responses and clarifying human-crafted evaluation questions.
Abstract
from arXiv · showhide
Misunderstandings arise not only in interpersonal communication but also between humans and Large Language Models (LLMs). Such discrepancies can make LLMs interpret seemingly unambiguous questions in unexpected ways, yielding incorrect responses. While it is widely acknowledged that the quality of a prompt, such as a question, significantly impacts the quality of the response provided by LLMs, a systematic method for crafting questions that LLMs can better comprehend is still underdeveloped. In this paper, we present a method named `Rephrase and Respond' (RaR), which allows LLMs to rephrase and expand questions posed by humans and provide responses in a single prompt. This approach serves as a simple yet effective prompting method for improving performance. We also introduce a two-step variant of RaR, where a rephrasing LLM first rephrases the question and then passes the original and rephrased questions together to a different responding LLM. This facilitates the effective utilization of rephrased questions generated by one LLM with another. Our experiments demonstrate that our methods significantly improve the performance of different models across a wide range to tasks. We further provide a comprehensive comparison between RaR and the popular Chain-of-Thought (CoT) methods, both theoretically and empirically. We show that RaR is complementary to CoT and can be combined with CoT to achieve even better performance. Our work not only contributes to enhancing LLM performance efficiently and effectively but also sheds light on a fair evaluation of LLM capabilities. Data and codes are available at https://github.com/uclaml/Rephrase-and-Respond.
1 Introduction
The paper argues that humans and LLMs can apply different interpretive frames to questions, causing models to answer unintended queries. It proposes self-rephrasing as a way to clarify questions and improve LLM performance.
- Motivation: LLMs are sensitive to prompt quality, while human and model thought frames can diverge during question interpretation.This disparity is presented as a source of unexpected or incorrect responses.
- Motivating Example: LLMs may misinterpret apparently clear questions, such as treating an even month as one with an even number of days.The motivating example describes several possible interpretations of the even/odd month query.
- Motivation: Benchmark ambiguities that are imperceptible to humans can cause LLMs to answer unintended queries incorrectly.The paper identifies such ambiguities across commonly used evaluation datasets.
- Our Approach: The proposed approach asks an LLM to rephrase and expand a question before answering, improving semantic clarity and resolving inherent ambiguity.The method is named Rephrase and Respond (RaR) and produces the rephrasing and response in one prompt.
- Our Approach: RaR is presented as unsupervised, training-free, economical, and applicable to zero-shot questions, while also complementing Chain-of-Thought prompting.The paper also proposes transferring rephrased questions from stronger models to weaker responding models.
2 Related Work
Related work covers prompt-engineering guidance, Chain-of-Thought prompting, post-hoc self-refinement, and concurrent uses of LLM rephrasing. The paper positions RaR as a zero-shot method that complements these prompting techniques rather than replacing them.
- Prompting Methods for LLMs: Prompt-engineering research includes manual guidelines emphasizing specificity, detail, and precision in user queries.The paper notes that users often find it difficult to refine their own questions.
- Prompting Methods for LLMs: Zero-shot and few-shot Chain-of-Thought prompting improve reasoning without evaluation or iterative prompt selection, but reliability limitations have also been reported.The paper identifies zero-shot CoT as the prompting method most closely aligned with its approach.
- Prompting Methods for LLMs: RaR is described as complementary to the prompting techniques reviewed in this section and combinable with them.This positions the method as an additional prompting strategy rather than an exclusive alternative.
- Self-correction Methods for LLMs: Post-hoc prompting uses LLMs to refine their own responses through self-feedback, critic models, trained models, or external references.Related studies question whether LLMs can reliably self-correct their reasoning processes.
- Concurrent Works: Concurrent work also studies LLM rephrasing, including rephrasing for synthetic-data generation, fine-tuning, and vision-language applications.The paper describes its work as among the first to examine misunderstandings between humans and LLMs through input rephrasing.
3 Aligning the Frames with Rephrase and Respond
Rephrase and Respond (RaR) asks an LLM to rearticulate and expand a question before responding, reducing ambiguity and aligning the query with the model’s interpretive frame. The method is concise, supports prompt variants, and preserves the original query’s context.
- Method: RaR systematically reduces prompt ambiguity by asking the LLM to rephrase and contextualize a question before answering.The method is framed as self-clarification that aligns the question with the LLM’s existing thought frame.
- Method: The core RaR prompt is “{question}” followed by “Rephrase and expand the question, and respond.”This single-query formulation combines rephrasing and answering.
- Prompt Variants: Alternative prompts preserve the same intention by requesting rewording, elaboration, reframing, or modification for clarity before answering.The paper states that these variations also improve performance in later experiments.
- Method Properties: RaR maintains the original query’s context, aligns human intent with the LLM’s preferred question style, and adds a question-rephrasing step before response generation.The paper also presents this process as a way for humans to inspect how LLMs understand questions.
4 RaR Effectively Improves LLM Responses
RaR is evaluated as a simple zero-shot prompting method across diverse benchmark tasks and models. It consistently improves GPT-4 performance, remains effective across wording variations, and benefits models to different degrees.
- Evaluation setup: RaR is assessed across 10 zero-shot tasks spanning different aspects of LLM capabilities.The evaluation includes knowledge, commonsense, date, and other benchmark tasks.
- Main results: RaR consistently improves GPT-4 accuracy across diverse tasks, with especially large gains on questions that GPT-4 initially finds difficult.On last-letter concatenation, accuracy improves to almost 100%.
- Prompt variations: Changing the RaR prompt wording causes minor fluctuations but preserves enhancement over baseline performance across all tested variants.The comparison uses four prompt variations and reports their average deviations from the main prompt.
- Performance across models: All examined models benefit from rephrasing, although GPT-4 gains the most while Vicuna achieves more modest improvements.GPT-3.5 and Vicuna show slightly reduced performance on some tasks, such as Sports.
5 Let Stronger LLMs Rephrase for Weaker LLMs to Respond
Two-step RaR separates question rephrasing from responding so a stronger LLM can clarify prompts for a weaker one. The approach uses both original and rephrased questions when generating the response.
- Method: Two-step RaR separates rephrasing from responding, allowing a stronger LLM such as GPT-4 to generate clarifying questions for another model.The rephrasing and responding LLMs may be the same or different models.
- Method: The rephrasing prompt asks the model to expand the question while maintaining all information in the original.The resulting question is then paired with the original question for response generation.
- Results: GPT-4-rephrased questions remarkably enhance Vicuna’s responses compared with Vicuna’s self-rephrased questions.The Even Year task is omitted because Vicuna’s rephrasing is already as clear as GPT-4’s on that task.
6 Discussion with Chain-of-Thought
RaR addresses weaknesses in both zero-shot and few-shot CoT by clarifying questions and correcting flawed examples. The methods are complementary, but CoT can hallucinate or follow erroneous user-provided logic.
- Combining RaR and CoT: RaR complements zero-shot CoT and can be combined with the instruction “Let’s think step by step.”The paper contrasts RaR’s consistent effectiveness with examples where zero-shot CoT fails or diminishes performance.
- Zero-shot CoT: Zero-shot CoT can reduce Chinese Idiom accuracy to 31.43% because hallucinated intermediate reasoning steps worsen performance.On StereoSet, RaR raises GPT-4’s Language Modeling Score to 97.73% and achieves the best Fair Score.
- Zero-shot CoT: RaR improves GPT-4’s StereoSet performance where zero-shot CoT fails to improve the Language Modeling Score.Zero-shot CoT improves fair-response percentage, while RaR achieves the best performance overall.
- Few-shot CoT: Few-shot CoT depends critically on the quality of human-crafted examples, which can adversely influence model responses when flawed.The method guides models to follow supplied human logic rather than aligning questions with what they best understand.
- Few-shot CoT: A single flawed few-shot example degrades one-shot CoT, while four flawed examples reduce four-word Last Letter Concatenation accuracy to 52.27%.GPT-4 follows the modified first-letter logic but produces an arbitrary final answer; RaR corrects these logical pitfalls.
7 Conclusion
The paper argues that human-clear questions can remain ambiguous to LLMs and introduces RaR to rephrase and clarify them before answering. Across benchmark datasets, rephrasing improves performance, transfers across models, and complements CoT.
- Conclusion: RaR prompts an LLM to rephrase and clarify a question before answering it.The approach is presented as a response to misunderstandings between humans and LLMs.
- Conclusion: Across benchmark datasets, empirical evaluations confirm that the proposed methods are effective.The conclusion reports enhanced performance through question rephrasing across models.
- Conclusion: Two-step RaR transfers rephrased-question improvements from a rephrasing LLM to a different responding LLM.More sophisticated models exhibit more substantial improvements, and the enhancement in question quality is transferable across models.
- Conclusion: RaR is complementary to CoT and can provide additional performance gains when combined with it.The paper supports this conclusion through mathematical and empirical comparisons with CoT methods.
A Experiment Details
The experiments evaluate LLM accuracy across multiple task datasets using exact matching followed by manual verification. They include GPT-4 variants, GPT-3.5-turbo, and Vicuna, with task-specific evaluation details and prompt-formatting materials.
- Evaluation: Accuracy is evaluated by exact matching generated answers to correct words, followed by manual inspection to verify and correct calculations.The Dates Understanding dataset uses all data, while other tasks use randomly drawn subsets of 220 questions.
- Models: The experiments use publicly available GPT-4, GPT-4-0613, GPT-3.5-turbo-0613, and Vicuna-13B-v1.5.These models provide the main model coverage for the reported experiments.
- Experimental materials: The experiment materials include task descriptions, formatting prompts, and improved few-shot examples for Last Letter Concatenation.These materials are organized in Tables 5, 6, 7, and 8.
- Task-specific evaluation: StereoSet evaluation includes Language Modeling Score and Stereotype Score, while the Chinese Idiom task uses zero-shot accuracy with automated and manual checking.The Language Modeling Score measures preference for related over unrelated options, and the Stereotype Score measures preference for stereotypical choices over anti-stereotypical ones.
B Additional Experiment Results
Additional results compare self-rephrasing across models and show that stronger models generally produce more useful clarifications. Weaker models may preserve ambiguity, alter question intent, or return a rephrased question instead of an answer.
- Rephrasings from Different LLMs: GPT-4 shows larger accuracy improvements from self-rephrased questions than GPT-3.5 and Vicuna across various tasks.Figure 7 compares original and self-rephrased questions for the three models under Two-step RaR.
B.2 Multiple Rephrasings: Will the Questions Converge?
Iterative self-rephrasing with Two-step RaR can make GPT-4’s clarification of ambiguous concepts converge, while progressively elaborating the question.
- By the third rephrasing, GPT-4 converges to a consistent explanation of “even day” after sometimes missing the clarification initially.The intended meaning is whether the day of the month is even.
- The “even day” example exposes ambiguity between an even day of the month, week, or year.Humans typically interpret the phrase as referring to the day of the month, whereas LLMs may interpret it differently.
- Repeated rephrasings make the question increasingly elaborate while producing more consistent clarification by the third round.
C Input/Output Examples
The paper provides input/output examples comparing original questions with one-step and two-step RaR across several reasoning tasks, including Even day, Coin flip, Last letter concatenation, and Sports.
- Even day: Examples compare GPT-4’s correct and incorrect responses to original questions and RaR variants for the Even day task.The examples include original questions, one-step RaR, and repeated table entries for one-step RaR.
- Coin flip: Coin flip examples present GPT-4 responses for original questions, one-step RaR, and two-step RaR.
- Last letter concatenation (2): Last letter concatenation examples use the question “Take the last letters of the words in “Beth Joel” and concatenate them,” with an incorrect original response of “hel”.Separate examples cover one-step and two-step RaR.
- Sports: Sports examples compare GPT-4’s responses to original questions with one-step and two-step RaR.