PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards A Unified View of Answer Calibration for Multi-Step Reasoning

Shumin Deng, Ningyu Zhang, Nay Oo, Bryan Hooi

arXiv:2311.09101v3cs.CLcs.AIcs.IRcs.LG

TL;DR

Systematic analysis of answer calibration strategies for multi-step reasoning remains limited. The paper unifies step- and path-level calibration in a framework and evaluates them across tasks and metrics. It finds that integrating both forms of dominance typically achieves optimal performance.

  • Problem

    The conditions behind answer calibration’s effectiveness, the relative strengths of step- and path-level strategies, and their robustness and generalizability remain underexplored.

    Source: §1 · pp. 1–2

  • Method

    The paper defines step- and path-level dominance in a unified framework and evaluates calibration strategies across multiple paths, tasks, and metrics.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Combining step-level and path-level dominance typically achieves optimal performance; path-level calibration benefits accuracy more, while step-level calibration better addresses low-quality prompting.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The findings identify integration of both dominance strategies as a promising route to optimal answer calibration performance.

    Source: §5 · p. 8

    The study does not analyze more calibration strategies, varying numbers of steps or paths, other path-generation models, or more tasks.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) employing Chain-of-Thought (CoT) prompting have broadened the scope for improving multi-step reasoning capabilities. We generally divide multi-step reasoning into two phases: path generation to generate the reasoning path(s); and answer calibration post-processing the reasoning path(s) to obtain a final answer. However, the existing literature lacks systematic analysis on different answer calibration approaches. In this paper, we summarize the taxonomy of recent answer calibration techniques and break them down into step-level and path-level strategies. We then conduct a thorough evaluation on these strategies from a unified view, systematically scrutinizing step-level and path-level answer calibration across multiple paths. Experimental results reveal that integrating the dominance of both strategies tends to derive optimal outcomes. Our study holds the potential to illuminate key insights for optimizing multi-step reasoning with answer calibration.

1 Introduction

The introduction frames multi-step reasoning as path generation followed by answer calibration, then motivates a systematic comparison of calibration strategies. The study evaluates step- and path-level approaches and reports that combining their dominance typically yields the best performance.

  • Answer calibration post-processes generated reasoning paths, with step- and path-level strategies applicable to single or multiple paths.Source: §1 · p. 1
  • The study asks when calibration boosts performance, how step- and path-level strategies compare, and how robust and generalizable they are.Source: §1 · pp. 1–2
  • The unified framework evaluates step- and path-level strategies across accuracy, faithfulness, informativeness, consistency, and perplexity on five arithmetic and commonsense tasks.Source: §1 · p. 2
  • Combining step-level and path-level dominance typically achieves optimal performance; path-level calibration benefits accuracy more, while step-level calibration better addresses low-quality prompting.Source: §1 · p. 2
  • Calibration improves accuracy more noticeably in zero-shot settings and less on stronger backbone models, while its effects on other metrics vary by task.Source: §1 · p. 2

2 Related Work

Related work covers methods for generating or polishing reasoning paths and methods that calibrate answers after paths are generated. The latter literature is organized around step- and path-level approaches.

  • Path-generation work refines prompts or questions and polishes rationales through step-aware training, path-level optimization, and deliberate reasoning structures.Source: §2 · p. 2
  • Step-level answer calibration checks or revises intermediate steps, including through external knowledge, interaction, forward reasoning, backward verification, and code-based verification.Source: §2 · pp. 2–3
  • Path-level answer calibration includes iteratively revising rationales or outputs, enriching paths with knowledge, and reasoning over or voting across multiple paths.Source: §2 · p. 3

3 Comprehensive Analysis of Answer Calibration

The section defines step- and path-level calibration for single or multiple reasoning paths, then combines path-answer frequency and intermediate-step correctness in a unified score. It describes dominance settings and evaluates calibration with accuracy and reasoning-quality metrics.

3.1 Formulation of Answer Calibration

  • For a single path, step-level calibration rectifies erroneous intermediate answers, while path-level calibration revises a wrong final answer; across paths, they select or aggregate steps or choose the most consistent answer.Self-verification is identified as an approach to multi-path step-level calibration, and self-consistency as a multi-path path-level technique.Source: §3.1 · p. 3

3.2 Unified View of Answer Calibration

  • The unified score D_i combines answer frequency n_i with the number of correct intermediate steps m_i, with α controlling their relative weight.The selected answer is the answer associated with the path having the highest score.Source: §3.2 · p. 3
  • Step-level dominance makes m_i decisive, while path-level dominance makes n_i decisive; the other score serves as a tie-breaker in each setting.The framework defines these as particular choices of α.Source: §3.2 · pp. 3–4

3.3 Evaluation of Answer Calibration

  • Evaluation uses accuracy and representative ROSCOE measures spanning faithfulness, informativeness, consistency, and language coherence, with higher scores indicating better results.The described measures assess alignment with a source rationale, grounding, contradictions within steps or against source context, and path perplexity.Source: §3.3 · p. 4

4 Experiments

Across five arithmetic and commonsense benchmarks, answer calibration generally improves accuracy, with larger gains in zero-shot settings, but its effects on other reasoning metrics vary by task. Integrated step- and path-level calibration peaks between their dominance thresholds in most scenarios, while backbone and prompting conditions shape which strategies help.

4.1 Setup

  • The evaluation covers five arithmetic and commonsense datasets and measures answer accuracy alongside rationale-quality metrics.The datasets are GSM8K, SVAMP, MultiArith, MathQA, and CSQA; selected ROSCOE metrics evaluate rationale quality beyond final-answer correctness.Source: §4.1 · p. 5

4.2 Analysis on Step-Level and Path-Level Answer Calibration Strategies

  • Answer calibration improves accuracy on almost all tasks, with stronger gains for zero-shot than few-shot CoT, but consistency, faithfulness, informativeness, and perplexity effects vary.It improves consistency on arithmetic tasks while weakening faithfulness, informativeness, and perplexity on both arithmetic and commonsense tasks.Source: §4.2 · p. 6

4.3 Analysis on Unified Answer Calibration Strategies

  • In 8 of 10 cases, accuracy peaks between the step-level and path-level dominance thresholds, supporting a balance of both calibration strategies.The two exceptions have lower peak performance than the initial SV accuracy at α = 0.Source: §4.3 · p. 7

4.4 Effects of Backbone Models

  • Path-level calibration is generally more accurate and less computationally costly than step-level calibration, while integrated SV+SC approaches the better individual strategy.SV and SC improve accuracy on GPT-3 and Instruct-GPT, but their gains are weaker on GPT-3.5; SV slightly lowers CSQA accuracy there.Source: §4.4 · p. 7

4.5 Effects of Prompting

  • No-relevance prompts hurt performance more than prompts lacking coherence, while CoT+SV tends to outperform CoT+SC under no-relevance prompting.CoT+SV and CoT+SC perform comparably with standard or incoherent prompts.Source: §4.5 · p. 8

4.6 Analysis on Tasks

  • Calibration generally helps arithmetic tasks more than CSQA, where it can worsen consistency and perplexity despite correct intermediate statements.The paper suggests that correct commonsense steps may be irrelevant or redundant, whereas correct arithmetic intermediate answers contribute to a correct final answer.Source: §4.6 · p. 8

5 Conclusion and Future Work

The paper evaluates answer calibration through a unified view of step- and path-level strategies. It concludes that their strengths differ by setting and that integrating both can yield optimal performance.

5 Conclusion and Future Work

  • The paper presents a unified evaluation of answer calibration, separating multi-step reasoning into path generation and answer calibration.It identifies path-level calibration as particularly potent for accuracy and step-level calibration as more suitable for low-quality prompting.Source: §5 · p. 8
  • Integrating step- and path-level dominance is promising for optimal performance, with two thresholds defining their dominance.The paper reports that calibration gains are less significant on stronger backbones and more pronounced in zero-shot scenarios.Source: §5 · p. 8
  • The authors propose developing more sophisticated multi-step reasoning models informed by this study’s findings.They describe answer calibration as versatile and integrable into various models.Source: §5 · p. 8

Limitations

The paper’s limitations include the range of strategies, configurations, models, and tasks examined.

Limitations

  • The study does not analyze additional calibration strategies, including single-path step- or path-level methods.It also leaves the number of steps and paths in unified strategies unvaried.Source: Limitations · p. 9
  • The evaluation is limited to CoT-based path-generation models, although calibration could also be applied to other path-generation models.Source: Limitations · p. 9
  • The authors identify evaluation on more tasks as needed to make the results more sufficient.Source: Limitations · p. 9

Broader Impact

The study unifies disparate answer-calibration approaches in an empirical framework and reports accuracy, robustness, and other metric outcomes. Its findings distinguish where path-level and step-level calibration are beneficial and identify trade-offs across metrics.

Broader Impact

  • The paper presents a unified framework to study how step- and path-level calibration for single or multiple paths can be integrated.Source: Broader Impact · p. 9
  • Answer calibration improves accuracy more noticeably in zero-shot settings and less on stronger backbone models.Source: Broader Impact · p. 9
  • The unified strategy typically performs best when it synthesizes step-level and path-level dominance.Source: Broader Impact · p. 9
  • Path-level calibration is more beneficial for accuracy, while step-level calibration is more effective under low-quality prompting.Source: Broader Impact · p. 9
  • Calibration can improve consistency on arithmetic tasks but weaken faithfulness, informativeness, and perplexity on arithmetic and commonsense tasks.Source: Broader Impact · p. 9

A Terminology Clarification of Answer Calibration and Model Calibration

The paper distinguishes its use of “answer calibration” from model calibration. It defines answer calibration by its post-processing role and divides methods according to whether they operate on individual reasoning steps or whole paths.

A Terminology Clarification of Answer Calibration and Model Calibration

B Model Choice Justification

The authors justify GPT-3.5 by its relevance and accessibility for their research objectives. They describe the backbone as pluggable, report releasing code and generated data anonymously, and identify more transparent models as future work.

B Model Choice Justification

C Cases of Low-Quality Prompts

Table 3 presents examples of prompts used in the experiments, covering standard prompts and prompts with no coherence or no relevance.

C Cases of Low-Quality Prompts

Loading 2311.09101v3…