PaperlayerView exact revision on arXiv

Source-linked AI summary

MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback

Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, Heng Ji

arXiv:2309.10691v3cs.CLcs.AIcs.LG

TL;DR

Many LLM evaluations emphasize single-turn exchanges and overlook user language feedback, despite the importance of multi-turn interaction. MINT evaluates tool use and natural-language feedback, finding that models generally benefit from both, while stronger single-turn performance does not guarantee stronger multi-turn performance.

  • Problem

    Existing LLM evaluations predominantly emphasize single-turn inputs and often overlook user-provided natural-language feedback, despite multi-turn interaction’s importance in real-world applications.

    Source: §1 · p. 1

  • Method

    MINT repurposes established datasets into a compact benchmark for tool-augmented problem solving and natural-language feedback, with feedback simulated by GPT-4.

    Source: §1 · p. 2

  • Results

    LLMs generally benefit from tools (1–8% absolute gains per additional tool-use turn) and language feedback (2–17%), while single-turn performance does not guarantee multi-turn performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MINT can help track progress and incentivize research on multi-turn task-solving, particularly in open-source communities where human evaluation can be less accessible.

    Source: §1 · p. 2

    Some evaluated LLMs failed to produce parsable output as instructed in some cases.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

To solve complex tasks, large language models (LLMs) often require multiple rounds of interactions with the user, sometimes assisted by external tools. However, current evaluation protocols often emphasize benchmark performance with single-turn exchanges, neglecting the nuanced interactions among the user, LLMs, and external tools, while also underestimating the importance of natural language feedback from users. These oversights contribute to discrepancies between research benchmark evaluations and real-world use cases. We introduce MINT, a benchmark that evaluates LLMs' ability to solve tasks with multi-turn interactions by (1) using tools and (2) leveraging natural language feedback. To ensure reproducibility, we provide an evaluation framework where LLMs can access tools by executing Python code and receive users' natural language feedback simulated by GPT-4. We repurpose a diverse set of established evaluation datasets focusing on reasoning, coding, and decision-making and carefully curate them into a compact subset for efficient evaluation. Our analysis of 20 open- and closed-source LLMs offers intriguing findings. (a) LLMs generally benefit from tools and language feedback, with performance gains (absolute, same below) of 1-8% for each turn of tool use and 2-17% with natural language feedback. (b) Better single-turn performance does not guarantee better multi-turn performance. (c) Surprisingly, on the LLMs evaluated, supervised instruction-finetuning (SIFT) and reinforcement learning from human feedback (RLHF) generally hurt multi-turn capabilities. We expect MINT can help measure progress and incentivize research in improving LLMs' capabilities in multi-turn interactions, especially for open-source communities where multi-turn human evaluation can be less accessible compared to commercial LLMs with a larger user base.

1 INTRODUCTION

MINT evaluates LLMs’ multi-turn task-solving with tools and natural-language feedback, addressing the gap between real-world interactions and evaluations that focus mainly on single-turn exchanges. Its findings cover performance gains from interaction, model-training effects, and feedback-providing ability.

1 INTRODUCTION

  • About 73% of 94,000 ShareGPT human–ChatGPT conversations contained multiple turns, while prevailing evaluations predominantly emphasized single-turn exchanges and often overlooked natural-language feedback.The introduction presents this mismatch as the motivation for evaluating multi-turn interaction in realistic tool-assisted settings.Source: §1 · p. 1
  • MINT uses Python-executed tools and GPT-4-simulated natural-language feedback to evaluate 20 models on 586 curated instances from eight established datasets.The instances were selected as challenging and representative of multi-turn tasks from 29,307 collected examples spanning reasoning, code generation, and decision-making.Source: §1 · p. 2
  • Models gained 1–8% absolute performance per additional tool-use turn and 2–17% with natural-language feedback.The benchmark also found that stronger single-turn performance did not necessarily translate into stronger multi-turn performance.Source: §1 · p. 2
  • With language feedback, the best open-source model, Lemur-70b-chat-v1, trailed the best closed-source model by 8.7% in absolute success rate.Claude-1 benefited more from interaction than Claude-2 and performed better with more than two turns, despite Claude-2’s stronger single-turn result.Source: §1 · p. 2
  • SIFT and RLHF generally hurt multi-turn performance, though SIFT improved interaction for Vicuna-7B and Lemur-70b-chat-v1.SIFT reduced Codellama-34B performance by 11.1% and 15.4% with feedback; RLHF reduced LLaMA-2-70B performance by 8.5% and 8.7%, respectively.Source: §1 · p. 2
  • Feedback-providing ability could be orthogonal to task-solving ability, GPT-4-simulated feedback was as helpful as human-written feedback, and MINT revealed artifacts in ShareGPT data.CodeLLaMA-34B-Instruct performed worst at task-solving yet provided feedback that improved the stronger GPT-3.5.Source: §1 · p. 2

MINT

MINT evaluates LLMs’ multi-turn problem-solving with tools and natural-language feedback, using a compact set of challenging tasks repurposed from established datasets.

2.1 INTERACTION FRAMEWORK

  • LLMs can reason, execute Python code through a unified tool interface, or propose solutions across turns, with standardized prompts intended to support fair, reproducible comparisons.The framework measures interactions at increasing levels of complexity, from minimal user guidance to tool use and informative language feedback.Source: §2.1 · p. 3
  • In the lazy-user setting, the simulated user checks proposed solutions against ground truth and gives at most one binary correction; tool-use ability is assessed across interaction limits k = 1–5.Models can use tools or propose solutions on each turn, with reminders to propose a solution by the final turn; performance gain across k and absolute performance at k = 5 measure tool-augmented problem-solving.Source: §2.1 · pp. 3–4
  • For informative user feedback, GPT-4 simulates suggestions, and MINT compares this tool-augmented setting at k = 5 with lazy-user interaction to assess feedback use.The authors validate the simulated feedback through human evaluation.Source: §2.1 · p. 4

2.2 REPURPOSING EXISTING DATASETS FOR MINT

  • To make iterative evaluation more accessible, MINT excludes easy or guessable instances, then stratifies the remaining data into 586 examples preserving the original correct-to-incorrect ratio.The initial set has 29,307 instances; instances solved by gpt-3.5-turbo-0613 within two turns are excluded before subsampling.Source: §2.2 · pp. 4–5
  • MINT repurposes eight datasets spanning reasoning, code generation, and ALFWorld decision-making, adapting initially single-round reasoning and coding tasks for interaction.Reasoning includes math, multi-hop question answering, and knowledge problem-solving; code generation uses HumanEval and MBPP.Source: §2.2 · pp. 4–5

3 EXPERIMENTS

The experiments assess multi-turn tool use, language-feedback use and provision, model failure patterns, and the quality of GPT-4-generated feedback against human feedback.

3.1 SETUP

  • The setup evaluated 20 models—4 closed-source and 16 open-source—using success rate SRk from scratch at interaction limits k = 1–5.The five-turn maximum was chosen to fit within most modern LLMs’ 4,096-token context window.Source: §3.1 · p. 5

3.2 MEASURING LLM’S TOOL-AUGMENTED TASK-SOLVING IN MULTI-TURN INTERACTION

  • All open-source models trail the best commercial models in SR5 and ∆tools, while claude-instant-1 overtakes claude-2 by k = 3 and finishes at 45.9% vs. 39.9% SR5.Across CodeLLaMA and LLaMA-2 variants, scaling generally raises SR5 and ∆tools; LLaMA-2 RLHF reduces ∆tools by −0.7% to −2.6% and SR5 by −2.4% to −8.5%. Vicuna-v1.5 (7B, SIFT) is an exception, exceeding LLaMA-2 (7B, Base/RLHF) on ∆tools (+3.4% vs. +2.2% / +1.5%) and SR5 (12.6% vs. 9.7% / 7.3%); the authors suggest ShareGPT multi-turn fine-tuning may explain gains for Vicuna and Lemur, but find no similar gains for most CodeLLaMA SIFT models.Source: §3.2 · pp. 5–6, Table 2 · p. 6

3.5 MINT CAN HELP DETECT FAILURE PATTERNS OF EVALUATED LLMS

  • MINT surfaced formatting failures and likely training-data artifacts, including escaped underscores in Vicuna outputs and CodeLLaMA-Instruct’s disregard of requested <execute> tags on code-generation tasks.Vicuna outputs caused code-execution syntax errors; the paper found escaped-underscore artifacts in at least 15% of examined ShareGPT examples. CodeLLaMA-Instruct (34B) used [PYTHON] wrapping on 100% of code-generation tasks and 0% of other tasks.Source: §3.5 · p. 7

3.3 MEASURING LLM’S ABILITY TO LEVERAGE NATURAL LANGUAGE FEEDBACK

  • Feedback gains were +1.7–+17.2% for open-source and +6.5–+15.2% for closed-source models, but the best open model trailed claude-instant-1 by 8.7% in SRfeedback5.CodeLLaMA-34B-base achieved performance comparable to GPT-4 on decision-making tasks when using its language feedback. SIFT/RLHF generally reduced feedback gains and SRfeedback5 versus base variants, with Vicuna-v1.5 (7B) and Lemur-v1 (70B) as exceptions.Source: §3.3 · p. 7, Table 3 · p. 8

3.4 MEASURING THE EFFICACY OF DIFFERENT LLM’S ABILITY TO PROVIDE FEEDBACK

  • Task-solving ability did not guarantee feedback-providing ability: GPT-3.5 (16k) feedback degraded GPT-3.5 by −10.4%, while CodeLLaMA-34B-Instruct improved the stronger GPT-3.5.The evaluation fixed GPT-3.5 as the task-solving model and compared seven feedback providers.Source: §3.4 · p. 7, Table 4 · p. 9

3.6 CAN GPT-4 GENERATE HUMAN-LEVEL NATURAL LANGUAGE FEEDBACK?

  • Annotators rated 91.2% of GPT-4 feedback as helpful as or better than human feedback and human-like in 92% of cases; during gpt-3.5-turbo-0613 problem-solving, SRfeedback5 was 32.7% with human versus 33.6% with GPT-4 feedback.Source: §3.6 · pp. 7, 9

4 RELATED WORK

Prior work studies user collaboration and tool use, while MINT addresses an interaction-evaluation gap by spanning diverse tasks and combining tools with user feedback.

4.1 LLM IN INTERACTION

4.2 EVALUATING INTERACTION

  • Existing evaluations often target one task or dimension and focus on humans or environments separately, whereas MINT measures multi-turn interaction across diverse tasks with tools and user feedback.MINT is designed to better align interaction evaluation with real-world applications.Source: §4.2 · p. 9

5 CONCLUSION

MINT evaluates LLM task-solving in multi-turn interactions using tools and natural-language feedback simulated by GPT-4. The benchmark is intended to help track progress and encourage research on multi-turn capabilities.

5 CONCLUSION

  • MINT is an evaluation benchmark for LLM task-solving through multi-turn interaction using tools and natural-language feedback.The feedback is simulated using GPT-4.Source: §5 · p. 9
  • MINT is intended to help track progress in improving LLM multi-turn task-solving capabilities.Source: §5 · p. 9
  • The authors hope MINT will incentivize future research on improving LLM multi-turn task-solving capabilities.Source: §5 · p. 9

A LIMITATIONS AND FUTURE WORK

The authors identify limitations in simulated feedback, interaction-quality metrics, and benchmark coverage, and propose future work on feedback-based training and continual benchmark expansion.

Limitations and Future Work

  • GPT-4-simulated feedback resembled human-written feedback in a human experiment but may miss real-user responses, especially nuanced value judgments in tasks such as policy-making.The authors note that their experiment does not establish that GPT-4 can simulate every aspect of human feedback.Source: A LIMITATIONS AND FUTURE WORK · p. 16
  • MINT evaluates in-context multi-turn interaction but leaves direct use of language feedback for model training and improvement, as in Wang et al. (2023a), to future work.Source: A LIMITATIONS AND FUTURE WORK · p. 16
  • The authors caution that outcome-focused metrics may miss interaction quality and should penalize repetitive guessing used to raise scores.Source: A LIMITATIONS AND FUTURE WORK · p. 16
  • MINT does not cover all tools or real-world uses, including web-browsing (Deng et al. (2023b)) and operating systems (Liu et al. (2023d)), and the authors propose continually adding challenging tasks and tools.The authors cite Qin et al. (2023c) in noting the wide range of tools not covered.Source: A LIMITATIONS AND FUTURE WORK · p. 16

B DETAILS OF HUMAN EVALUATION

The human evaluation used two annotation stages: annotators first wrote feedback for sampled interaction trajectories, then different annotators compared human and GPT-4 feedback for helpfulness and human-likeness.

First-stage feedback annotation

  • Two annotators supplied feedback for 113 trajectories sampled across 8 evaluated LLMs (2 per task), truncated at a random turn; annotation averaged 96 seconds ($90 per 100 turns).Using Potato (Pei et al., 2022), annotators A and B were assigned random halves; instances succeeding on turn one were excluded, and annotators responded to prior interaction history.Source: B DETAILS OF HUMAN EVALUATION · p. 16

Second-stage feedback comparison

  • Two different annotators compared human-written and GPT-4 feedback for helpfulness and human-likeness, choosing which was better or whether both were equally good.Helpfulness meant whether feedback helped the LLM succeed at the task; human-likeness concerned similarity to literal human usage.Source: B DETAILS OF HUMAN EVALUATION · p. 16

C ABLATION STUDY · C.1 HOW DO FEEDBACK VARIATIONS IMPACT FEEDBACK QUALITY ∆FEEDBACK?

The ablation examines how feedback informativeness and frequency affect language-model performance.

C.1 HOW DO FEEDBACK VARIATIONS IMPACT FEEDBACK QUALITY ∆FEEDBACK?

  • The feedback ablation varies informativeness and frequency; its implementation is detailed in §F.4.2 and its results are presented in Tab. A.6.The passage does not provide the results from Tab. A.6.Source: §C.1 · p. 16

C.1.1 INFORMATIVENESS · C.1.2 FREQUENCY

Feedback quality depends on its information and timing: ground-truth-conditioned feedback can help reasoning and coding but hurt decision-making, while binary or sparse feedback can still improve on no feedback. More informative textual feedback and dense feedback outperform their respective alternatives.

C.1.1 INFORMATIVENESS

  • Informativeness varies by whether feedback is conditioned on the ground-truth solution and whether it is textual or binary.The default settings are feedback without ground-truth conditioning and textual feedback.Source: §C.1.1 · p. 16

C.1.2 FREQUENCY

  • Sparse feedback lowers performance by 11.02 on average versus dense feedback, yet improves performance over no feedback on every task.Under sparse feedback, MINT is equivalent to Reflexion feedback (Shinn et al., 2023).Source: Table A.6 · p. 17, §C.1.2 · p. 17

C.1.1 INFORMATIVENESS

  • Ground-truth-conditioned feedback improves reasoning and code generation but reduces decision-making performance by 8.95% versus no feedback.Decision-making solutions are action trajectories, so comparing early actions against the ground-truth trajectory may confuse the feedback provider and yield suboptimal feedback.Source: §C.1.1 · pp. 16–17, Table A.6 · p. 17
  • Binary feedback hurts performance on all tasks versus textual feedback, but alone improves over no feedback, especially for decision-making (+5.97).Step-wise binary feedback can help agents terminate bad initial actions and backtrack when early actions strongly affect task success.Source: Table A.6 · p. 17, §C.1.1 · p. 17

C.1.2 FREQUENCY

  • Feedback frequency compares dense feedback at every step with sparse feedback only when the agent proposes a solution.Sparse feedback is provided when the LLM believes it has finished the task.Source: §C.1.2 · p. 17

D DATASET FILTERING AND DOWN-SAMPLING · E ISSUES

The appendix describes a three-step curation process: collect and format test data, retain tasks requiring multi-turn interaction, and stratify the cleaned data for down-sampling. It also identifies format-compliance failures and summarizes tools by task type.

D DATASET FILTERING AND DOWN-SAMPLING

  • The authors collect test-set data from eight datasets and format prompts with a task label, task description, and solution-range field.HotpotQA uses its first 500 instances; the range is integer for GSM8K, instance-specific for TheoremQA, and empty for other datasets.Source: D DATASET FILTERING AND DOWN-SAMPLING · p. 17
  • Table A.7 tracks interaction turns lost to invalid actions for each LLM under k = 5 with no feedback, bolding models whose invalid-action share exceeds 20%.Invalid actions are defined as failing to produce the instructed <execute> or <solution> tag.Source: Table A.7 · p. 18
  • The curation process retains instances requiring interaction, screening out low-interaction classes and then excluding instances completed within two turns under a five-turn, no-feedback run.After removing multiple-choice tasks with fewer than four options, the authors screen MATH and MMLU classes with gpt-3.5-turbo-0613; the screening uses N = 100 and k = 3.5 for MATH and N = 20 and k = 2.5 for MMLU.Source: D DATASET FILTERING AND DOWN-SAMPLING · p. 18
  • After cleaning, instances are stratified by dataset and whether gpt-3.5-turbo-0613 completed them, creating 16 strata with dataset-specific preservation proportions.The proportions are 1 for PALFWorld; 0.5 for MBPP and HumanEval; 0.2 for GSM8K, HotpotQA, and TheoremQA; 0.1 for MMLU; and 0.05 for MATH.Source: D DATASET FILTERING AND DOWN-SAMPLING · p. 18
  • Table A.8 summarizes tools by task type.Source: Table A.8 · p. 19

E.1 VICUNA-V1.5 ESCAPE UNDERSCORE ISSUE

The appendix identifies escaped underscores in generated code as a recurring failure, especially for Vicuna and SIFT Lemur-v1. It suggests artifacts in instruction-tuning data may explain the pattern.

E.1 VICUNA-V1.5 ESCAPE UNDERSCORE ISSUE

  • In k=5 runs without language feedback, escaped underscores caused syntax errors and were a particular issue for Vicuna; the prose also identifies SIFT Lemur-v1.The table caption says percentages above 2% are bolded, but the supplied passage does not include the table’s cell values.Source: §E.1 · p. 19, Table A.9 · p. 20
  • About 15% (14,110) of 94,145 checked ShareGPT conversations contained backslash-underscore patterns, which the authors suggest may explain the code-generation artifacts.The source reports checking 94,145 instances and finding the pattern in about 15% (14,110) of conversations.Source: §E.1 · p. 19
  • In the example, expert feedback identified escaped backslashes in a function name, but the model repeated the syntax error across subsequent attempts.The execution output reports a SyntaxError, and later feedback again points to the backslashes as the cause.Source: §E.1 · p. 21

E.2 CODELLAMA-INSTRUCT PRODUCES [PYTHON] TAG

CodeLLaMA-Instruct produced [PYTHON] tags on coding tasks despite instructions to use <execute> or <solution> tags. The tags were associated with code generation, appeared most often in the 34B model, and were absent from the smallest versions; the authors suspect instruction-tuning artifacts explain the pattern.

E.2 CODELLAMA-INSTRUCT PRODUCES [PYTHON] TAG

  • Despite explicit instructions to use <execute> or <solution> tags, CodeLLaMA-34b-Instruct wrapped its coding-task output in [PYTHON] tags.The paper illustrates this behavior in a random coding-task trajectory.Source: §E.2 · p. 23
  • [PYTHON] tags appeared on code-generation tasks but not other task types, were most frequent in the 34B model, and did not appear in the smallest versions.Table A.10 reports the percentage of task instances with at least one [PYTHON] tag under k = 5 without language feedback.Source: §E.2 · p. 23, Table A.10 · p. 24
  • The authors hypothesize that instruction-tuning exposure to [PYTHON] tags caused the model to ignore other coding-task tags, and that larger models memorize those examples more often.They connect this explanation to Code Llama’s self-instruct process, which prompted code generation between [PYTHON] and [/PYTHON] tags.Source: §E.2 · p. 23

F PROMPTS AND EXAMPLES · F.2 FORMATTING ERROR MESSAGE

This appendix documents MINT’s task-specific prompts, tool descriptions, and example interactions across code generation, reasoning, and household decision-making. It illustrates how evaluation prompts structure tool use and language feedback, including required response tags, expert feedback, and multi-step actions.

F.1 TOOL DESCRIPTIONS FOR EACH TASK TYPE

  • MINT’s tools vary by task: code generation uses the Python interpreter, reasoning includes Wikipedia search, and ALFWorld provides actions for navigating and manipulating objects.The appendix presents these resources alongside the prompts and examples designed to elicit interaction behavior.Source: §F.1 · pp. 26–27, F PROMPTS AND EXAMPLES · p. 26

F.2 FORMATTING ERROR MESSAGE

  • When the agent’s instructed format is missing, the error message asks it to use <execute> for code or <solution> for an answer, with a tagged answer example.The supplied example uses <solution> 42 </solution>.Source: §F.2 · pp. 27–28

F.3 EXAMPLE TASK TRAJECTORY

  • In a MATH trajectory, GPT-3.5’s initial answer is corrected after GPT-4 feedback identifies a Vieta-formula error, and the agent then submits the corrected product.The example uses GPT-4 as a simulated human advisor and GPT-3.5 as the agent.Source: §F.3 · pp. 28–29

F.4.1 EVALUATE LLM

  • The evaluation prompt defaults to five total steps and two solution proposals, asking the LLM to provide tagged thoughts and then either execute Python code or submit a tagged solution.Prompt templates substitute task-specific instructions, tools, and in-context examples.Source: §F.4 · p. 29, §F.4.1 · p. 29

F.4.2 SIMULATE LANGUAGE FEEDBACK

  • The feedback-generation prompt uses the agent trajectory and supplies the ground-truth solution only when available, directing the advisor to give concise guidance without revealing the answer.When no ground truth is provided, the template substitutes “NOT GIVEN.”Source: §F.4.2 · p. 30

F.5 IN-CONTEXT EXAMPLE

  • MINT uses one-shot in-context examples with and without feedback, curating code-generation and reasoning examples and constructing separate examples for ALFWorld task subtypes.Without-feedback examples remove the expert-feedback text; the appendix includes only a subset of prompts.Source: §F.5 · p. 30

F.5.1 DECISION-MAKING

  • The ALFWorld example demonstrates searching for a lettuce, cleaning it at a sinkbasin, and putting it on a dining table.The example is adapted from Yao et al. (2022) and is identified as a clean-type task.Source: §F.5.1 · pp. 30–31

F.5.2 CODE GENERATION

  • In the code-generation example, an incorrect set union fails the assertion, while replacing it with intersection executes successfully after feedback.The example is curated from an MBPP in-context example and reused for HumanEval with a format adjustment.Source: §F.5.2 · pp. 31–32
  • The code-generation example ends by submitting a `similar_elements` function that returns the intersection of two tuples as a tuple.The trace first states that there is no more AssertionError.Source: §F.5.2 · p. 32

F.5.3 REASONING TASK

  • The reasoning subset’s in-context examples are manually curated to cover both multi-hop QA-style and mathematical reasoning.The passage names HotpotQA as an example of the former and MATH and TheoremQA as examples of the latter.Source: §F.5.3 · p. 33
  • The calculus example asks for a double integral over a circle whose squared radius is Li Bai’s birth year, with the result rounded to three decimal places using half-up rounding.The worked interaction searches for Li Bai’s birth year before proceeding with the integral.Source: §F.5.3 · p. 33
  • The worked reasoning trace uses a search result and expert feedback to confirm the retrieved birth year, then evaluates the integral with SymPy and reports 771890.886.The intermediate symbolic result is 245700.5*pi; the expert feedback confirms retrieval and successful conversion of the integral.Source: §F.5.3 · pp. 33–34

F.6 EXAMPLE GROUND-TRUTH FOR DECISION-MAKING TASK

  • The Alfworld ground-truth example asks the agent to put two peppershakers in a drawer and demonstrates locating and transferring them into drawer 1.The trace takes one peppershaker from countertop 2 and another from cabinet 1, then puts both in drawer 1.Source: §F.6 · pp. 34–35

Related papers

Loading 2309.10691v3…