PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning

Murong Yue, Jie Zhao, Min Zhang, Liang Du, Ziyu Yao

arXiv:2310.03094v3cs.CLcs.AIcs.LG

TL;DR

The paper addresses the high cost of using strong LLMs for reasoning while preserving task performance. It builds cascades that use weaker-model answer consistency, including mixed Chain-of-Thought and Program-of-Thought sampling, to route questions. Across six reasoning datasets, the cascades achieve comparable performance to GPT-4 at approximately 40% of its cost.

  • Problem

    The paper asks how to reduce the monetary cost of LLM reasoning without sacrificing task performance.

  • Method

    The paper routes questions using the weaker LLM's answer consistency and samples answers through Chain-of-Thought and Program-of-Thought representations.

  • Results

    Across six reasoning datasets, mixture-of-thought cascades achieve comparable task performance to fully using GPT-4 while requiring approximately 40% of its cost.

  • Takeaways & Limitations

    Answer consistency and diverse thought representations provide an economical cascade for dynamically deciding when to use stronger LLMs in reasoning tasks.

  • Takeaways & Limitations

    The weaker LLM must be sufficiently capable for the task; LLAMA2-13B performs poorly on GSM8k and CREPE because it fails to answer complex questions consistently.

Abstract

from arXiv · show

Large language models (LLMs) such as GPT-4 have exhibited remarkable performance in a variety of tasks, but this strong performance often comes with the high expense of using paid API services. In this paper, we are motivated to study building an LLM cascade to save the cost of using LLMs, particularly for performing reasoning (e.g., mathematical, causal) tasks. Our cascade pipeline follows the intuition that simpler questions can be addressed by a weaker but more affordable LLM, whereas only the challenging questions necessitate the stronger and more expensive LLM. To realize this decision-making, we consider the "answer consistency" of the weaker LLM as a signal of the question difficulty and propose several methods for the answer sampling and consistency checking, including one leveraging a mixture of two thought representations (i.e., Chain-of-Thought and Program-of-Thought). Through experiments on six reasoning benchmark datasets, with GPT-3.5-turbo and GPT-4 being the weaker and stronger LLMs, respectively, we demonstrate that our proposed LLM cascades can achieve performance comparable to using solely the stronger LLM but require only 40% of its cost.

1 INTRODUCTION

The paper studies cost-efficient reasoning with LLM cascades that route easy questions to weaker models and difficult questions to stronger ones. It uses answer consistency, including mixed Chain-of-Thought and Program-of-Thought sampling, to guide routing decisions.

  • LLMs with stronger capabilities typically cost more, motivating cascades that use weaker models for simple questions and stronger models only for difficult ones.The paper notes that GPT-4 is 30 times more expensive than GPT-3.5-turbo for output tokens.
  • The proposed decision-maker uses answer consistency from the weaker LLM because easy questions tend to produce consistent samples while hard questions produce inconsistent ones.The paper develops vote-based and verification-based consistency checks.
  • The methods evaluate sampling from single or multiple demonstration sets and mix Chain-of-Thought with Program-of-Thought representations for more diverse answers.The mixture of thought representations is intended to provide diverse perspectives on the same question.
  • Across six mathematical, symbolic, and causal reasoning datasets, the cascades generally match or exceed fully using the stronger LLM at half or less of its cost.Mixture-based approaches achieve comparable performance with only 40% of GPT-4's cost.

2 LLM CASCADES FOR COST-EFFICIENT REASONING

The cascade routes questions through a weaker, cheaper LLM first and uses answer consistency to decide whether to accept its answer or invoke a stronger model. It combines voting and verification across sampling sources, including Chain-of-Thought and Program-of-Thought representations.

  • Cascade overview: The cascade first obtains an answer from the weaker LLM, then decides whether to accept it or route the question to the stronger LLM.The decision maker uses the weaker model’s output and metadata before making the routing decision.
  • Consistency signal: Answer consistency is treated as a difficulty signal because easy questions tend to yield consistent weaker-LLM answers, whereas hard questions tend to yield inconsistent answers.The paper considers multiple sampling paths and sources for measuring consistency.
  • Sampling configurations: The methods use in-distribution samples, different demonstration sets, and different thought representations, including CoT, PoT, and their mixture.The paper instantiates six vote-based and four verification-based approaches, including MoT variants combining thought representations.
  • Vote-based decision-making: Vote-based decision-making selects the most frequently sampled answer and accepts it when its agreement score reaches a predefined threshold.The threshold controls whether the weaker answer is accepted or rejected and therefore affects cascade cost.
  • Verification-based decision-making: Verification-based decision-making compares the most consistent answers from two prompts and accepts the weaker answer only when those answers match.The prompts can differ in demonstrations or thought representations.
  • Method comparison: Vote-based methods support threshold tuning under cost constraints, whereas verification-based methods avoid threshold tuning but offer less flexibility.The approaches are compared under cost-comparable sample-size configurations.

3 EXPERIMENT

Across six reasoning datasets, the proposed cascades preserve performance close to GPT-4 while substantially reducing cost. Mixture-of-thought approaches improve consistency-based routing, although threshold choice and weaker-model capability affect results.

  • Main results: ∼0.929 average accuracy matches GPT-4-CoT-SC at 0.931 while MoT variants use only 40% of its cost.On CREPE, MoT variants reach 0.885 versus 0.871 for GPT-4-CoT-SC at 47% of its cost.
  • Main results: 1.4% absolute accuracy separates CoT-2D-Vote from CoT-1D-Vote at the same 0.4 relative cost on average.Sampling from diverse prompt settings improves the distinction between easy and hard questions.
  • Main results: Mixing Chain-of-Thought and Program-of-Thought generally outperforms using either representation alone, except on DATE.The mixed variants provide more diverse weaker-LLM opinions, while one versus two demonstration sets makes little obvious difference for MoT.
  • Main results: Thresholds around 0.5–0.6 typically balance cost and accuracy, while excessively high vote thresholds can route easy questions unnecessarily.Verification tolerates a few hallucinations better than voting, and LLAMA2-13B performs poorly when questions are too complex for consistent answers.
  • Analysis on mixture of thought representations: Higher consistency scores on easy than hard questions make the consistency gap useful for cascade routing.MoT can have lower absolute consistency than CoT yet achieve a larger easy–hard gap, particularly on Navigate.
  • Analysis on mixture of thought representations: CoT demonstrations often repeat the same hard-question mistakes, whereas PoT tends to produce different mistakes, lowering MoT consistency.This contrast explains why mixing thought representations can distinguish hard questions more effectively.

4 RELATED WORK

Prior work improves LLM cost efficiency through model-compression and decoding techniques, API cascades, and uncertainty estimation. This paper builds on these directions while emphasizing that uncertainty methods had not explored different intermediate-step representations.

  • Cost-efficient LLM use: Cost-efficiency research includes quantization, pruning, decoding methods, and API-efficient LLM cascades.One cascade sequentially queries increasingly capable LLMs when earlier answers are deemed unacceptable.
  • Reasoning with LLMs: Reasoning research established Chain-of-Thought prompting, which asks LLMs to generate step-by-step reasoning for improved problem solving.The passage situates CoT within the progression from direct answer generation to explicit reasoning processes.
  • Uncertainty of LLMs: LLM uncertainty studies use voting scores to evaluate answer uncertainty, but had dismissed different representations of intermediate reasoning steps.This gap motivates considering multiple thought representations when measuring answer consistency.

5 CONCLUSION

The paper introduces an economical cascade that dynamically routes reasoning questions among LLMs using answer consistency. Its mixture of thought representations achieves comparable results to fully using the stronger LLM at approximately 40% of the expense.

  • 5 CONCLUSION: The pipeline dynamically decides which LLM handles each reasoning task to save token costs.The approach checks answer consistency from the weaker LLM as the routing signal.
  • 5 CONCLUSION: Approximately 40% of the stronger LLM’s expense is required to achieve a comparable result.The paper identifies a mixture of thought representations in the weaker LLM as its best-performing approach.

6 ETHICS STATEMENT

The paper frames cost-efficient LLM reasoning as potentially valuable for organizations with limited financial resources. It also connects large-throughput cost savings with environmental sustainability.

  • 6 ETHICS STATEMENT: Cost-effective reasoning could help local businesses, educational institutions, and non-profit organizations with limited financial resources use LLMs cheaply.The stated significance is improved access to LLM reasoning for resource-constrained entities.
  • 6 ETHICS STATEMENT: For industry giants handling extremely large throughputs, cost savings may also be substantial and may reduce the carbon footprint of large-scale computation.The passage links optimized cost-efficiency with both economic savings and environmental sustainability.

7 REPRODUCIBILITY STATEMENT

The pipeline is described as simple to implement and reproducible, with experimental details documented in the main text and appendices. Prompt examples are provided despite omitting complete prompts because of their length.

  • 7 REPRODUCIBILITY STATEMENT: Experimental details are documented in the main text and appendix sections to support reproducibility.The authors also state that implementations and prompt scripts are provided, though the passage truncates before completing that description.
  • 7 REPRODUCIBILITY STATEMENT: Prompt examples are included in Appendix L because the complete prompts are too long to provide.These examples are intended to help readers understand the style of prompting used.

A COST ANALYSIS OF LLM CASCADE APPROACHES

The cost analysis configures cascade approaches to make weaker-LLM usage comparable while accounting for representation-dependent token usage and preserving the demonstration count.

  • Token usage: Different thought representations can induce different token usage, making their cost differences difficult to quantify and dependent on the reasoning task.For GSM8k, Program-of-Thought representations are typically shorter than Chain-of-Thought representations.
  • Experimental controls: The number of task demonstrations M is held constant because LLMs are very sensitive to this configuration.
  • Cost comparability: Cost analysis guides configurations that keep approaches cost-comparable by adjusting their total sample sizes and accounting for input and output token prices.The analysis sets K_MoT = K_2D = K_1D − M × C_w^o under the stated simplifying assumptions.
  • Implementation: The experiments run each approach twice, average the results, and use Python 3.10 to execute generated Program-of-Thought code.DATE and Navigate use randomly sampled and annotated demonstrations because they lack training sets.
  • Metrics: Performance is evaluated using task accuracy and cost efficiency, where higher accuracy and lower cost indicate better performance.String answers use exact matching, while numerical answers use a 0.001 tolerance.

C MAIN RESULT TABLES

The main results cover six reasoning datasets and illustrate how Chain-of-Thought and Program-of-Thought can produce complementary errors, motivating their mixture in cascade reasoning.

  • Dataset results: The experiments report exact results for GSM8k, ASDIV, TabMWP, DATE, Navigate, CREPE, and their average.
  • Mixture of thoughts: 43% of cases where Chain-of-Thought makes a mistake are solved by Program-of-Thought, while 57% produce a different mistake.
  • Mixture of thoughts: Program-of-Thought avoids Chain-of-Thought’s repeated mistakes by providing different reasoning paths or avoiding computation errors.
  • Case studies: In the lemonade example, Program-of-Thought returns the correct lemon cost of 15, whereas both Chain-of-Thought samples return 7.5.
  • Case studies: In the bookshelf example, both Chain-of-Thought samples obtain 200, while the Program-of-Thought example returns 100.
  • Failure cases: Program-of-Thought can make different mistakes from Chain-of-Thought, such as grounding an incorrect value or total number of tests.
  • Failure cases: For the test-score example, both Chain-of-Thought samples return 120, while the Program-of-Thought execution returns 191.

E IMPLEMENTATION DETAILS OF EXTERNAL VERIFIERS

The external-verifier experiments compare question-only and question-and-answer inputs, including LLM and fine-tuned RoBERTa verifiers, while measuring accuracy and relative cost.

  • Setup: The weaker model remains GPT-3.5-turbo, and experiments use only Chain-of-Thought because the fine-tuned model cannot understand Python code.
  • Verifier inputs: The verifier can receive only the question or the question concatenated with the weaker model’s answer and intermediate thoughts.
  • Verifier variants: LLM-Q predicts question difficulty, whereas LLM-QA rethinks the solution and evaluates whether the weaker model’s answer is correct.
  • Fine-tuned verifier: Fine-tuned RoBERTa-base is tested with question-only and question-and-answer inputs using benchmark training data, except for DATE, which lacks a training split.
  • Results: External verifiers can improve performance, but the improvement is not significant enough, and LLM-QA costs less than LLM-Q.Final costs are mainly determined by the number of cases transferred to the stronger LLM.

F LLAMA2-13B AS THE WEAKER LLM

Using LLAMA2-13B as the weaker model does not reproduce the cascade’s favorable behavior on GSM8k and CREPE, although it performs well on DATE.

  • Setup: LLAMA2-13B is evaluated as the weaker LLM alongside GPT-4 as the stronger LLM.
  • Performance: With LLAMA2-13B, accuracy and cost change approximately linearly on GSM8k and CREPE, preventing comparable performance at lower cost, while DATE shows a favorable curve.
  • Answer consistency: Only 16.3% of GSM8k cases exceed the 0.6 agreement threshold with LLAMA2-13B, compared with 77.3% for GPT-3.5-turbo.
  • Scope boundary: LLAMA2-13B is not suitable as the weaker LLM for particular tasks because it provides consistent answers on fewer easy questions.
  • Progressive hints: Progressive hints provide only a slight improvement on DATE and diminishing returns on GSM8k and CREPE, partly because GPT-4 can be misled by incorrect weaker-model answers.
  • Batch prompting: Batch prompting reduces costs further but slightly affects the accuracy of both weaker and stronger LLMs, indicating that it is orthogonal to the cascade method.

I CALIBRATION ANALYSIS

The calibration analysis evaluates confidence-based cascade decisions, finding that confidence generally tracks answer correctness and that MoT performs strongly in direct effectiveness comparisons. The analysis also cautions that calibration curves alone omit bin-size information relevant to cascade performance.

  • Calibration results: LLM-QA T=2 is the least calibrated decision maker, while the other approaches show no significant calibration differences and outperform LLM-QA.The comparison uses expected calibration error and calibration curves.
  • Calibration results: All decision-making methods produce monotone calibration curves, so higher confidence generally corresponds to a higher likelihood that an answer is true.For vote-based approaches, this supports the hypothesis that consistency across weaker-LLM samples signals easier questions.
  • Interpretation and caveat: Calibration plots omit bin sizes, even though the number of instances in each confidence range can substantially affect practical cascade performance.Perfect calibration is therefore unnecessary when accepted answers are highly accurate, but curve shape alone does not reveal bin-size effects.
  • Direct effectiveness comparison: MoT ranks higher than other variants and baselines in subset accuracy as the confidence threshold increases.This analysis uses answers satisfying n/K greater than the confidence score and reflects the cascade’s actual operating procedure.

J CAN OUR METHOD GENERALIZE TO FACTUALITY-BASED TASKS?

The paper tests MoT-based consistency checking on factual reasoning with StrategyQA and finds that combining CoT and PoT remains effective. Diverse reasoning representations can expose unreliable answers even when neither representation is uniformly superior.

  • Experimental generalization: MoT remains effective on StrategyQA, a factual question-answering task requiring implicit reasoning over facts.The experiment uses open-domain questions without supplying context to the LLM.
  • Mixture of thought representations: Combining CoT and PoT generates diverse thoughts and answers, reducing factual-reasoning errors when the two representations agree.The paper states that PoT is not better than CoT overall, but their combination produces complementary reasoning paths.
  • Mixture of thought representations: For the curling-iron example, most CoT answers hallucinate a yes response, whereas PoT lists curling equipment and correctly rejects the necessity of a curling iron.MoT-1D-Vote uses the disagreement and agreement between the two representations to identify the unreliable answer.
  • Limitations: The approach is limited to questions with clear answers, may increase latency when stronger-LLM calls are required, and can fail when the weaker LLM is overconfidently wrong.It also assumes that intermediate reasoning steps can be expressed through different representations.
Loading 2310.03094v3…