PaperlayerView exact revision on arXiv

Source-linked AI summary

Program-Aided Reasoners (better) Know What They Know

Anubha Kabra, Sanketh Rangreji, Yash Mathur, Aman Madaan, Emmy Liu, Graham Neubig

arXiv:2311.09553v1cs.AI

TL;DR

Reliable confidence estimates are important alongside accuracy, yet the calibration of program-aided reasoning relative to text-based chain-of-thought prompting is the question this paper examines. It compares PAL and COT across five datasets and two model families, finding better calibration for PAL in most settings. The analysis also relates generation similarity and answer entropy to calibration and tests temperature changes.

  • Problem

    Reliable confidence estimates matter for applications that may need abstention, human intervention, or confidence passed to a downstream model.

    Source: §1 · p. 1

  • Method

    The study evaluates PAL and COT on five reasoning datasets using OpenAI and LLaMA models, with confidence estimated through self-consistency majority voting over answers.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    PAL has better calibration than COT in 75% of settings overall; across models and varied temperatures, it generally has higher calibration and accuracy in 82.5% of cases.

    Source: §4.1 · p. 4, §6 · p. 8

  • Takeaways & Limitations

    Greater generation similarity accompanies lower answer entropy and lower ECE, while reducing generation diversity improves calibration only at certain temperatures.

    Source: §6 · p. 8

Abstract

from arXiv · show

Prior work shows that program-aided reasoning, in which large language models (LLMs) are combined with programs written in programming languages such as Python, can significantly improve accuracy on various reasoning tasks. However, while accuracy is essential, it is also important for such reasoners to "know what they know", which can be quantified through the calibration of the model. In this paper, we compare the calibration of Program Aided Language Models (PAL) and text-based Chain-of-thought (COT) prompting techniques over 5 datasets and 2 model types: LLaMA models and OpenAI models. Our results indicate that PAL leads to improved calibration in 75% of the instances. Our analysis uncovers that prompting styles that produce lesser diversity in generations also have more calibrated results, and thus we also experiment with inducing lower generation diversity using temperature scaling and find that for certain temperatures, PAL is not only more accurate but is also more calibrated than COT. Overall, we demonstrate that, in the majority of cases, program-aided reasoners better know what they know than text-based counterparts.

1 Introduction

The paper motivates confidence calibration as a reliability property beyond accuracy, then compares PAL and COT across datasets and model families. It reports stronger calibration for PAL in most settings, with differences across model families.

1 Introduction

  • Prior work reports that code-based PAL prompts can surpass COT accuracy on some tasks [Madaan et al., 2022; Lyu et al., 2023; Zhang et al., 2023a,b].The paper positions its study as extending this accuracy-focused work to calibration.Source: §1 · p. 1
  • Reliable confidence estimates matter because they can support abstention, human intervention, or confidence passed to a downstream model.The paper describes calibration as alignment between confidence and real outcomes.Source: §1 · p. 1
  • The study compares PAL and COT on five reasoning datasets using OpenAI and LLaMA models, evaluating accuracy and calibration.Its questions include whether calibration differs by prompting style or model family, and whether generation consistency relates to calibration.Source: §1 · p. 2
  • About 50% calibration enhancement over COT accompanies higher accuracy for OpenAI PAL; LLaMA shows no consistent gain unless sampling temperature is adjusted.The authors also report a correlation between generation similarity and calibration.Source: §1 · p. 2

2 Preliminaries and Mathematical Formulation

The preliminaries define calibration and its bucketed ECE estimate, then explain how the study derives confidence and analyzes generation similarity and answer-space entropy.

2.1 Measuring Calibration

  • Calibration means predicted probabilities align with actual correctness; the study estimates it across confidence buckets, using each bucket's average accuracy and confidence.The text illustrates perfect calibration with predictions at confidence 0.6 having expected accuracy 0.6.Source: §2.1 · p. 2
  • Expected calibration error (ECE) is the sample-weighted average of the absolute accuracy–confidence gap across buckets.The bucket score is weighted by its share of all samples.Source: §2.1 · p. 2

2.2 Self-consistency as a measure of confidence

  • Self-consistency generates multiple reasoning paths and uses majority voting over their answers to estimate confidence.The study uses this as a proxy for calibration, including for closed models without per-token log probabilities.Source: §2.2 · pp. 2–3
  • K controls the number of generations; increasing K improves the approximation of each sample's answer probability.Figure 2 illustrates confidence estimation by majority voting over answers.Source: §2.2 · p. 3, Figure 2 · p. 3

2.3 Similarity and Answer Entropy

  • The qualitative analysis measures average similarity among generated reasoning chains or programs using sentence embeddings.The authors treat these generations as the latent variable Z.Source: §2.3 · p. 3
  • Answer-space entropy H(A) is computed from the probabilities of answers among K extracted or executed outputs, to examine its relationship with generation similarity.The measures are used to investigate whether similarity in latent space accompanies lower answer entropy.Source: §2.3 · p. 3

3 Experimental Design

The experimental design compares CoT and PaL on equal numbers of open- and closed-source models across five reasoning datasets, using specified sampling settings and adapted prompts.

3.1 Models

  • The comparison covers LLaMA2-13B and LLaMA2-70B as open-source models, and gpt-3.5-turbo and text-davinci-003 as closed-source models.All models had some form of code supervision during pre-training, in addition to being primarily trained on text.Source: §3.1 · p. 3

3.2 Hyperparameters

  • Experiments use temperature 1.0 and nucleus-sampling probability 1.0, with 10 generations per sample and a 1,024-token maximum per generation.The token limit includes input and output.Source: §3.2 · p. 3

3.3 Tasks

  • Five datasets span arithmetic (GSM8K and GSM8K Hard), algorithmic (Object-Counting and Repeat-Copy), and symbolic (Date-Understanding) reasoning tasks.The datasets are listed in Table 1.Source: §3.3 · p. 4, Table 1 · p. 4

3.4 Prompt Design

  • CoT prompts use their original form when available; otherwise they are adapted to the task, while PaL prompts adapt code prompts from Gao et al. (2022).The paper points to Appendix Section A for the prompts.Source: §3.4 · p. 4

4 Results

Across OpenAI and LLaMA models, the results compare PAL and COT on accuracy and calibration, then examine how generation diversity relates to calibration and whether temperature changes help. PAL’s calibration advantage varies by model type and generation setting.

4.1 Effect of prompting style on Calibration

  • PAL yields better calibration than COT in 75% of settings; for OpenAI models, it reduces calibration error by about 50% and improves average accuracy by 18.42%.For LLaMA models, PAL improves accuracy by 14.83% but shows better calibration in only half of settings.Source: §4.1 · p. 4, Table 2 · p. 5
  • With accuracy controlled, PAL has a negative, statistically significant ECE coefficient for OpenAI models, but no statistically significant effect for LLaMA models.Across both model families, PAL has a significant correlation of -0.067 with ECE (p=0.02).Source: §4.1 · pp. 4–5, Table 3 · p. 5

4.2 Effect of generation diversity on calibration

  • PAL generations follow a three-part code structure—initialization, problem-solving logic, and answer return—with most variation concentrated in the logic.The initialization and answer-return parts vary little across generations.Source: §4.2 · p. 5
  • Across most settings, prompting with more similar generations and lower answer entropy is also more calibrated, although the pattern is not consistent across all LLaMA datasets.For OpenAI models, PAL has higher generation similarity and lower answer entropy than COT.Source: §4.2 · pp. 5–6

Better calibration observed for PAL when inducing similarity in generations for LLaMA2-70B

5 Related Work

Prior work studies reasoning prompts that generate intermediate reasoning steps or programs, and separate work examines calibration in language models. This paper’s related-work discussion distinguishes its focus on code-prompt calibration and its use of self-consistency to measure calibration.

5.1 Prompting Strategies for Reasoning

  • Earlier code-prompt studies report reasoning-performance gains over chain-of-thought prompting, while this paper focuses on code prompts’ effect on calibration.The discussion identifies CoT, PaL, and Program-of-thoughts as approaches that generate reasoning steps or programs for an interpreter.Source: §5.1 · p. 7

5.2 Calibration in Language Models

  • Calibration studies span structured prediction, language understanding, and language generation; this work adopts self-consistency rather than verbalized probability estimates.The cited comparison describes self-consistency methods as outperforming verbalized methods.Source: §5.2 · p. 7

6 Conclusion

The paper compares PAL and COT calibration across five reasoning datasets and OpenAI and LLaMA models. It reports that PAL generally has higher calibration and accuracy, while temperature scaling can improve both for certain settings.

6 Conclusion

  • The evaluation uses self-consistency across five reasoning datasets and measures ECE, accuracy, generation similarity, and answer entropy.Source: §6 · pp. 7–8
  • 82.5% of cases showed PAL generally having higher calibration and accuracy across OpenAI and LLaMA models for varied temperatures.Source: §6 · p. 8
  • OpenAI models were generally better calibrated than LLaMA models, with up to 19% improvement in ECE score.Source: §6 · p. 8
  • PAL generations showed up to 7% greater similarity than text generations; the authors hypothesize code structure may contribute, while greater similarity accompanies lower answer entropy and ECE.Source: §6 · p. 8
  • Temperature scaling improved calibration only over a limited range: lower temperatures such as 0.1 and 0.3 had negative effects, while 0.5 and 0.7 were optimal across datasets.Source: §6 · p. 8, Figure 5 · p. 8
  • The authors hope this study encourages further research into prompts across task domains and dimensions beyond accuracy.Source: §6 · p. 8

A Prompts

The appendix presents one example of the few-shot prompts used for each dataset and prompting style.

A Prompts

  • The appendix provides one few-shot prompt example for each dataset across the prompting styles.Source: A Prompts · p. 11

A.1 PAL Prompts

The PAL appendix illustrates code-based prompts for arithmetic, object counting, date understanding, and repeat-copy tasks.

A.1.1 GSM8K/GSM8K-Hard

  • For GSM8K/GSM8K-Hard, the PAL example sets up a calculation of Olivia’s remaining money from an initial $23 and five bagels at $3 each.Source: §A.1.1 · p. 11

A.1.2 Object Counting

  • The object-counting example asks for vegetables among potatoes, cauliflower, lettuce, cabbage, and onions, while excluding furniture and fridges.Source: §A.1.2 · p. 11
  • The PAL code stores the vegetable counts as 2, 1, 1, 1, and 2, then sums them.Source: §A.1.2 · p. 11

A.1.3 Date Understanding

  • The date-understanding code treats “2015 is coming in 36 hours” as a date offset, then adds one week and formats the result as MM/DD/YYYY.Source: §A.1.3 · p. 11

A.1.4 Repeat Copy

  • The repeat-copy PAL example loops through four “duck” outputs and inserts “quack” when the loop index reaches 2.Source: §A.1.4 · p. 11
  • The repeat-copy code joins the generated sequence into a single returned string.Source: §A.1.4 · p. 11

A.2 COT Prompts

The COT appendix gives natural-language worked examples for arithmetic, object counting, date understanding, and repeat-copy tasks.

A.2.1 GSM8K/GSM8K-Hard

  • In the grove problem, there were 15 trees originally and 21 after planting; subtracting 15 from 21 gives 6 trees planted, and the answer is 6.Source: §A.2.1 · p. 12

A.2.2 Object Counting

  • The object-counting explanation excludes the chair, tables, and fridges because they are not vegetables.Source: §A.2.2 · p. 12
  • It counts two potatoes, one cauliflower, one lettuce head, one cabbage, and two onions, totaling 7 vegetables.Source: §A.2.2 · p. 12

A.2.3 Date Understanding

  • The date explanation interprets 36 hours as two days and places today on 12/30/2014, two days before 01/01/2015.Source: §A.2.3 · p. 12
  • Adding one week to 12/30/2014 yields 01/05/2015 in the worked example.Source: §A.2.3 · p. 12

A.2.4 Repeat Copy

  • For repeat-copy, the explanation places “quack” between the second and third “duck,” producing “duck duck quack duck duck.”Source: §A.2.4 · p. 12

B Temperature Scaling Experiments - Line Plots

Figure 6 presents temperature-scaling trends for GSM8K Hard, Date Understanding, and Repeat Copy.

B Temperature Scaling Experiments - Line Plots

  • Figure 6 presents temperature-scaling trends for GSM8K Hard, Date Understanding, and Repeat Copy.The caption does not specify the plotted outcomes or comparisons.Source: Figure 6 · p. 13

Related papers

Loading 2311.09553v1…