PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Self-Consistency Improves Language Model Agents

Han Wang, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal

arXiv:2402.13212v2cs.CLcs.AIcs.LG

TL;DR

In interactive tasks, many distinct valid actions make exact-match majority voting less effective and costly across long sequences. SOFT-SC uses continuous likelihood-based scores for selection, and improves success rates over SC across the evaluated domains, with reported gains at a fixed sample count.

  • Problem

    In interactive tasks, diverse valid actions make exact-match voting less effective and require many samples across lengthy action sequences.

    Source: §1 · p. 1

  • Method

    SOFT-SC replaces exact-match voting with continuous action scores computed from model likelihoods.

    Source: §1 · p. 1

  • Results

    At k=10, SOFT-SC outperforms SC in success rate by 1.3% on Bash, 6.6% on WebShop, and 4.7% on ALFWorld.

    Source: §3 · p. 3

  • Takeaways & Limitations

    SOFT-SC can let smaller models replace larger same-class models and can rerank black-box outputs with fewer total calls than SC in a reported comparison.

    Source: §3 · p. 4

    SOFT-SC still requires multiple samples, and like other sample-and-select methods costs more than greedy decoding.

    Source: §6 · p. 5

Abstract

from arXiv · show

Generations from large language models (LLMs) can be improved by sampling and scoring multiple solutions to select a final answer. Current "sample and select" methods such as self-consistency (SC) rely on majority voting to score answers. However, when tasks have many distinct and valid answers, selection by voting requires a large number of samples. This makes SC prohibitively expensive for interactive tasks that involve generating multiple actions (answers) sequentially. After establishing that majority voting fails to provide consistent gains on such tasks, we demonstrate how to increase success rates by softening the scoring criterion. We introduce Soft Self-Consistency (SOFT-SC), which replaces SC's discontinuous scoring with a continuous score computed from model likelihoods, allowing for selection even when actions are sparsely distributed. SOFT-SC improves both performance and efficiency on long-horizon interactive tasks, requiring half as many samples as SC for comparable or better performance. For a fixed number of samples, SOFT-SC leads to a 1.3% increase over SC in absolute success rate on writing bash programs, a 6.6% increase on online shopping (WebShop), and a 4.7% increase for an interactive household game (ALFWorld). Finally, we show that SOFT-SC can be applied to both open-source and black-box models.

1 Introduction

Majority-vote self-consistency (SC) can be ineffective and costly for interactive agents, where many distinct valid actions make exact-match agreement rare. SOFT-SC replaces vote matching with continuous likelihood scoring and is reported to improve performance and sample efficiency, including when reranking black-box model outputs.

1 Introduction

  • In interactive tasks, exact-match voting loses effectiveness when actions have many distinct valid forms, making SC costly across long action sequences.For Bash prediction with five samples, SC fails to produce a single majority action 86% of the time.Source: §1 · p. 1
  • SOFT-SC replaces exact-match votes with continuous scores from model likelihoods, allowing selection without a unique majority answer.For white-box models, it uses probabilities already produced by the model, with no additional cost or external tests or metrics.Source: §1 · p. 1
  • SOFT-SC can rerank outputs from closed-source models such as GPT-4 (OpenAI, 2023) using a smaller language model to score generations.The paper reports gains of up to 4% on WebShop.Source: §1 · p. 2
  • At the same sample count, SOFT-SC improves over SC by up to 6.6% on WebShop using CodeLlama-34B.The paper also reports better sample efficiency than SC.Source: §1 · p. 2
  • From 7B to 70B, SOFT-SC improves Bash performance by 8.8%, compared with 5.8% for SC.The paper also reports that SOFT-SC can outperform SC when scoring closed-source models, by up to 4% on WebShop.Source: §1 · p. 2

2 Methodology

The methodology introduces likelihood-based action selection and an adaptive stopping rule, then evaluates them on three interactive datasets against greedy, self-consistency, and adaptive-consistency baselines.

2.1 Soft Self-Consistency (SOFT-SC)

  • SOFT-SC scores each candidate action by aggregating the probabilities of its tokens, then selects the action with the highest score.The aggregation function is chosen using development-set performance: mean probability for Bash and ALFWorld, and minimum probability for WebShop.Source: §2.1 · p. 2

2.2 Adaptive Soft Self-Consistency

  • Adaptive Soft Self-Consistency samples one action at a time and stops when aggregated minimum probabilities reach a development-set threshold.This replaces adaptive-consistency’s approximation of the final vote tally with a likelihood-score threshold.Source: §2.2 · p. 2

2.3 Datasets

  • Evaluation covers Bash command tasks, WebShop product selection, and ALFWorld household chores, using success rate based on task reward.WebShop also reports a product-match score; ALFWorld includes long action sequences with thousands of valid actions at some steps.Source: §2.3 · pp. 2–3

2.4 Baselines

  • Baselines include single-sample greedy decoding, exact-match majority-vote SC, and adaptive-consistency, which reduces samples using a Beta estimator.Greedy decoding is equivalent to SC and SOFT-SC when k = 1.Source: §2.4 · p. 3

3 Results and Discussion

SOFT-SC outperforms self-consistency (SC) at the same sample count and generally achieves better performance with fewer samples. The results also report gains across model sizes and black-box settings, while showing that verbalized confidence scores and calibration are not prerequisites for strong SOFT-SC performance.

For the same number of samples k, SOFT-SC outperforms SC

  • 1.3%, 6.6%, and 4.7% success-rate gains over SC at k=10 on Bash, WebShop, and ALFWorld, respectively, favor SOFT-SC.The comparison reports average performance across three random seeds for each test set.Source: §3 · p. 3
  • SOFT-SC outperforms adaptive-consistency by 4% on WebShop and 3.1% on ALFWorld while generally using fewer samples.Source: §3 · p. 3

SOFT-SC exhibits better scaling with k

  • On Bash, increasing k from 5 to 20 yields SC only a 1 percentage point success-rate improvement, while SOFT-SC gains approximately 3 percentage points.Source: §3 · p. 3
  • On ALFWorld, SOFT-SC with k=5 improves 2.2% over SC with k=10.Source: §3 · pp. 3–4

SOFT-SC effectively scales with model size

  • At fixed k=10, SOFT-SC beats SC across model sizes; CodeLlama-13B with SOFT-SC also outperforms CodeLlama-34B with SC.Source: Figure 2 · p. 4, §3 · p. 4

SOFT-SC improves black-box models more than SC

  • With black-box generations, SOFT-SC improves success rate over SC by 4% for GPT-4 on WebShop and 1.8% on Bash.SOFT-SC with k=5 uses 10 total calls and outperforms SC with k=15, which uses 15 black-box calls.Source: Figure 3 · p. 4, §3 · p. 4

Logit-based score outperforms verbalized confidence score

Calibration is not required for strong SOFT-SC performance

  • SOFT-SC performance has no significant correlation with ECE, while AUROC has a moderate negative correlation on Bash and no significant correlation on WebShop.For Bash, the reported AUROC correlation is r = −0.55.Source: §3 · p. 4

4 Related Work

Prior sample-and-select work applies ensemble voting to LLMs, while related approaches address consistency judgments, reranking, and agent workflows. The paper distinguishes SOFT-SC from methods requiring external tests or model-based scoring and presents it as complementary to several multi-step agent improvements.

Sample and Select Methods for LLMs

  • Ensembling aggregates or votes over outputs, and Wang et al. (2023) introduced self-consistency for LLM reasoning tasks.Source: §4 · pp. 4–5
  • Majority voting can be poorly suited to agent domains with large action spaces; other consistency or reranking methods use model judgments, external tests, or model-based metrics [Chen et al., 2023b; Chen et al., 2022; Li and Xie, 2024; Golovneva et al., 2023; Prasad et al., 2023b].The cited passage notes that model consistency judgments also struggle in partially observable interactive domains [Ruan et al., 2023].Source: §4 · p. 5

LLM-Agents

  • SOFT-SC is complementary to work improving multi-step agent pipelines through feedback, task decomposition, or trajectory search, and focuses on one agent interacting with an external environment.The passage also says this single-agent setup is compatible with work on discussion among multiple LLM agents.Source: §4 · p. 5

5 Conclusion

SOFT-SC improves performance and efficiency across three interactive benchmarks and is compatible with white-box and black-box models. The conclusion also reports that logits outperform verbalized confidence and that good calibration is not required.

5 Conclusion

  • SOFT-SC improves performance and efficiency on three commonly used interactive benchmarks.The conclusion also says it can be integrated into a more efficient adaptive variant of self-consistency.Source: §5 · p. 5
  • SOFT-SC is compatible with both white-box and black-box models.The paper also reports integration into a more efficient adaptive self-consistency variant.Source: §5 · p. 5
  • A well-calibrated model is not required for SOFT-SC to work well, and logits outperform verbalized confidence scores.Source: §5 · p. 5

6 Limitations and Broader Impacts

SOFT-SC remains a sample-and-select method, so it costs more than greedy decoding and gains little from identical samples. Its agent evaluation is simulated, while real-world agent errors could have negative outcomes.

6 Limitations and Broader Impacts

  • SOFT-SC still requires multiple LLM samples, so it costs more than greedy decoding.Source: §6 · p. 5
  • If a model generates k identical samples, the result is no better than generating one.Source: §6 · p. 5
  • Direct SOFT-SC use without a smaller scoring model depends on users gaining access to the generation model's logits.Source: §6 · p. 5
  • Agent errors could harm users, but the evaluated agents operate in simulations and cannot affect the world.Source: §6 · p. 5

A Method and Dataset Details

The appendix reports decoding and adaptive-selection settings, then identifies model checkpoints, inference details, and licensing.

A.1 Hyperparameters

  • Adaptive thresholds were selected on development data and differ by dataset and method.For Adaptive-Consistency, thresholds were 0.8, 0.85, and 0.8 for Bash, WebShop, and ALFWorld; Adaptive SOFT-SC used 0.95, 3.0, and 3.5, respectively.Source: §A.1 · p. 8
  • Sampling used dataset-specific temperatures, with top-p 0.9 and top-k 40 for open-source models.The passage also specifies a 100-token maximum; greedy decoding used temperature 0.7 across datasets.Source: §A.1 · p. 8

A.2 Model Checkpoints and Licenses

  • The appendix names the CodeLlama and Mistral checkpoints and notes that WebShop, Bash, and ALFWorld have MIT licenses.CodeLlama uses a custom permissive license, while Mistral uses Apache License 2.0.Source: §A.2 · p. 8
  • Open-source models were run through Huggingface Transformers with int-8 quantization on Nvidia 40GB A100 GPUs.OpenAI inference used gpt-3.5-turbo-0613 and gpt-4; the authors say they will release code under an MIT license.Source: §A.2 · p. 8

A.3 Bash

The appendix describes task environments and selection procedures for Bash, WebShop, and ALFWorld, including how actions or trajectories are scored and selected.

A.3 Bash

  • Bash uses 200 user instructions split into 50 development and 150 test queries, with success measured by task reward.The setup samples commands in a single turn because environment observations and oracle rewards rarely help generate correct commands, and real-world oracle rewards are unavailable.Source: §A.3 · pp. 8–9

A.4 WebShop

  • WebShop simulates shopping across 1.18 million products and measures both reward-based score and perfect-score success rate.The setup generates k shopping trajectories, votes over product IDs, then uses a separate prompt to buy the selected product and choose its options.Source: §A.4 · p. 9
  • Because exact-match voting cannot select multi-step product options, SOFT-SC scores full shopping trajectories while SC follows a greedy trajectory after product selection.The passage explains that greedy follow-up choices can lead to suboptimal products, such as selecting a default option.Source: §A.4 · p. 9

A.5 ALFWorld

  • ALFWorld evaluates household tasks with action-level sampling and majority voting for SC, while both methods score actions rather than generated thoughts.The appendix reports 134 unseen tasks across six types, but Figure 1 uses a 30-task subset for computational reasons; success is measured by task completion.Source: §A.5 · pp. 9–10

A.6 Aggregation Methods

SOFT-SC scores sampled actions by aggregating token probabilities, with the aggregation choice matched to each task’s evaluation level. The section also describes greedy decoding and exact-match majority voting as baselines.

A.6 Aggregation Methods

  • Given a task description and a sampled action made of tokens, aggregation methods compute a score used to select the highest-scoring action.Source: §A.6 · p. 10
  • For Bash and ALFWorld, selection scores actions by mean token probability; for WebShop, it scores trajectories with minimum probability because success depends on a sequence of actions.The WebShop trajectory selects a suitable product by navigating webpages.Source: §A.6 · p. 10
  • Table 3 reports dev-set success rates for one seed across aggregation methods, using CodeLlama-34B for Bash and WebShop and Mistral-7B for ALFWorld.Source: Table 3 · p. 10

A.7 Baselines

  • Greedy decoding samples one trajectory and is equivalent to SC or SOFT-SC when k = 1, because no selection among samples is needed.Source: §A.7 · p. 10
  • SC tallies votes only when the model generates the exact same response multiple times.Source: §A.7 · p. 10

A.8 Adaptive SOFT-SC

Adaptive SOFT-SC stops sampling when accumulated likelihood scores reach a threshold, rather than estimating a stopping point from sampled vote distributions. The threshold comparison uses minimum token probabilities because average probabilities are poorly calibrated.

A.8 Adaptive SOFT-SC

  • Adaptive-consistency instead estimates whether more samples are needed by sampling from a discrete distribution over vote distributions parameterized by current vote counts.It terminates when its stopping criterion is satisfied or the maximum number of generations is reached.Source: §A.8 · p. 10
  • Adaptive SOFT-SC samples actions one at a time and stops when accumulated minimum token probabilities reach a score threshold τ.The threshold is selected on a dev set and is domain-specific: 0.95 for Bash, 3.0 for WebShop, and 3.5 for ALFWorld.Source: §A.8 · p. 11
  • Average probability is poorly calibrated for the threshold comparison, so SOFT-SC uses minimum token probability instead.Source: §A.8 · p. 11

B Calibration

The calibration analysis finds that well-calibrated models are not a prerequisite for SOFT-SC, possibly because the calibration metrics do not measure ranking performance. The appendix also identifies the prompts and in-context examples used for Bash and WebShop, and notes that ALFWorld uses prompts from prior work.

B Calibration

  • ECE measures the difference between average confidence and accuracy within confidence bins; adaptive binning adjusts bin sizes to reduce bias in confidence estimates.ECE can be sensitive to the number of bins used.Source: B Calibration · p. 11
  • AUROC assesses how well estimated confidence distinguishes correct from incorrect samples, with higher values indicating better discrimination.Source: B Calibration · p. 11
  • Bash shows a moderate negative correlation between AUROC and SOFT-SC performance, while neither ECE nor AUROC significantly correlates with WebShop performance.ECE has no significant correlation with SOFT-SC performance on Bash either.Source: B Calibration · p. 11

C Prompts

Loading 2402.13212v2…