Source-linked AI summary
ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning
Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, Asli Celikyilmaz
TL;DR
LLMs can improve reasoning-task performance by generating step-by-step rationales, but assessing whether those steps are correct and support final answers remains difficult without reliable automatic evaluation. ROSCOE introduces interpretable, unsupervised metrics and a reasoning-error taxonomy, outperforming prior semantic and lexical baselines across diverse reasoning datasets.
Problem
Assessing whether step-by-step rationales are correct and support final predictions remains difficult, despite their benefits for reasoning-task performance and interpretability.
Method
ROSCOE combines 18 fine-grained metrics across semantic alignment, logical inference, semantic similarity, and language coherence, evaluated using synthetic perturbations and expert judgments.
Results
ROSCOE outperforms prior semantic and lexical baseline metrics across synthetic and human-labeled datasets spanning commonsense, arithmetic, and logical reasoning.
Takeaways & Limitations
ROSCOE provides interpretable evaluation of step-by-step reasoning without requiring gold reasoning references and captures factual and logical errors in step-wise decisions.
Takeaways & Limitations
Dataset-agnostic fixed thresholds for distinguishing reasoning errors are not currently possible, and the metrics do not monitor explanations for integrity or bias.
Abstract
from arXiv · showhide
Large language models show improved downstream task performance when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness (independent of the final answer) is difficult without reliable methods for automatic evaluation. We simply do not know how often the stated reasoning steps actually support the final end task predictions. In this work, we present ROSCOE, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics. To evaluate ROSCOE against baseline metrics, we design a typology of reasoning errors and collect synthetic and human evaluation scores on commonly used reasoning datasets. In contrast with existing metrics, ROSCOE can measure semantic consistency, logicality, informativeness, fluency, and factuality - among other traits - by leveraging properties of step-by-step rationales. We empirically verify the strength of our metrics on five human annotated and six programmatically perturbed diagnostics datasets - covering a diverse set of tasks that require reasoning skills and show that ROSCOE can consistently outperform baseline metrics.
1 INTRODUCTION
ROSCOE addresses the difficulty of objectively evaluating whether step-by-step rationales support language-model predictions. It introduces interpretable metrics and evaluates them across diverse reasoning tasks against existing metrics.
- Step-by-step rationales improve reasoning-task performance but remain difficult to evaluate independently of final answers.
- ROSCOE provides fine-grained metrics for semantic alignment, logical inference, semantic similarity, and language coherence.
- ROSCOE is evaluated using a reasoning-error taxonomy, synthetic data, and human evaluations on commonly used reasoning datasets.
- ROSCOE demonstrates performance gains over baseline evaluation metrics on all tasks requiring reasoning over context.
- The work contributes a reasoning-error taxonomy, sequence- and step-level metrics, and comparative analysis across 11 complex reasoning datasets.
2 RELATED WORK
Prior explanation and text-generation metrics are useful but do not directly assess logical inconsistencies or individual reasoning steps. ROSCOE extends interpretable evaluation toward fine-grained reasoning analysis.
- Automatic NLG metrics are fast and cost-effective but are not equipped to measure logical inconsistencies or information gain in thinking steps.
- Existing automatic metrics commonly evaluate whole-text alignment using n-gram, regression, embedding, paraphrasing, or generation-based methods.
- The paper organizes reasoning errors through a taxonomy covering overall-chain and fine-grained step-level inconsistencies.
- ROSCOE differs from CTC and BARTScore by interpreting reasoning steps, detecting contradictions, and identifying ordering issues.
3 REASONING ERROR TAXONOMY AND DATASETS CONSTRUCTION
ROSCOE formulates rationale scoring without requiring gold reasoning references and constructs datasets that expose diverse reasoning errors for evaluation.
- The scoring problem assumes a source context and a generated hypothesis of reasoning steps ending in a final answer, without gold rationale references.
- The taxonomy identifies nine generic reasoning-error types centered on reasoning-chain quality and fine-grained step consistency.
- Evaluation datasets cover logical, arithmetic, and commonsense reasoning through synthetically perturbed diagnostic chains and expert-judged model outputs.
4 REASONING SCORER: ROSCOE
ROSCOE scores reasoning through four metric perspectives, using source contexts and generated hypotheses without requiring references for unsupervised evaluation. Its components assess grounding, semantic equivalence, logical contradictions, and language quality.
- ROSCOE metrics are organized into semantic alignment, semantic similarity, logical inference, and language coherence perspectives.
- Reference-free metrics use the source and hypothesis, whereas reference-based metrics compare the hypothesis with a reference.
- Semantic alignment: Semantic alignment computes step-level normalized cosine similarity to the most similar context sentence, measuring reasoning grounding in the source.
- Semantic similarity: Semantic similarity evaluates semantic equivalence between texts as wholes rather than through text-unit comparisons.
- Logical inference: Logical inference metrics use an NLI model to estimate contradiction probabilities between text pieces.
- Language coherence: Language coherence metrics use GPT2-Large perplexity and a grammatical-acceptability classifier without fine-tuning.
5 EXPERIMENTAL SETUP
The experiments combine synthetic perturbations and human judgments across diverse reasoning datasets to evaluate ROSCOE against established reference-free and text-generation metrics. Evaluation uses Somers’ D to measure ordinal association between metric scores and evaluation scores.
- Datasets: Synthetic diagnostics modify reference reasoning steps and label chains according to whether they were perturbed.The diagnostics cover seven language-understanding, entailment, arithmetic, commonsense, and question-answering datasets.
- Datasets: Human-judged datasets cover arithmetic, discrete, deductive, and commonsense reasoning tasks.The selected datasets are GSM8K, DROP, ESNLI, COSMOS-QA, and SemEVAL.
- ROSCOE training: ROSCOE obtains reasoning-step embeddings by fine-tuning supervised SimCSE on multi-step reasoning datasets.The model embeds both individual reasoning steps and complete reasoning chains after training on source context and hypothesis chains as wholes.
- Baselines: Baseline comparisons include n-gram, pretrained, embedding-based, and BARTScore-based text-generation metrics.The baselines include ROUGE-1, ROUGE-2, ROUGE-L, BLEURT, PRISM, BERTScore, BARTScore, and BARTScore+ variants.
- Meta-evaluation: Somers’ D evaluates ordinal association while handling ties that constrain Kendall’s τ and τ-b.The study uses correlations against synthetic and human scores for each metric.
6 EXPERIMENTAL RESULTS
ROSCOE is evaluated on diagnostic and human-judgment datasets using reference-free metric scores and Somers’ D correlations. Results report broad gains over baselines, while fine-tuning generally improves correlations with some repetition-score exceptions.
- Diagnostic datasets: Table 7 reports reference-free Somers’ D correlations across six diagnostic datasets, taking the maximum over perturbations.Non-significant correlations are omitted during aggregation, and the two highest correlations per dataset are highlighted.
- Fine-tuning effects: Fine-tuning improves average correlations across datasets for nearly all non-aggregated ROSCOE metrics.The reported average improvement reaches up to 0.556 for Informativeness-Chain, while repetition metrics are exceptions.
- Fine-tuning effects: Mean Repetition-Step correlation decreases by 0.005, whereas mean Repetition-Token correlation does not change.The authors report these as the principal repetition-score exceptions to the broader fine-tuning improvements.
- Human judgments: Table 8 reports reference-free Somers’ D correlations between metrics and human judgments across error types.The table highlights the two highest correlations in each column and omits non-significant correlations.
7 ANALYSIS
The analysis tests sensitivity to increasing reasoning errors and examines whether score thresholds transfer across datasets. ROSCOE semantic metrics behave consistently as errors increase, but dataset-agnostic thresholds remain unreliable.
- Error sensitivity: ROSCOE-SA and ROSCOE-SS show consistent behavior across MATH and EntailmentBank as injected error levels increase.The analysis measures how well metric values match human assessments of reasoning.
- Error sensitivity: Baseline metrics fluctuate at low correlations, although they perform better on EntailmentBank.On MATH, ROSCOE-LC and baseline metrics show minimal sensitivity to some perturbations.
- Cross-dataset analysis: Within a dataset, score distributions separate perturbed and original samples with little interquartile overlap.This pattern is examined for strongly correlated metric–perturbation pairs.
- Cross-dataset analysis: Fixed error-detection thresholds do not transfer reliably across datasets, affecting ROSCOE and baseline scores.For example, a Repetition threshold near 0.95 from EntailmentBank would classify nearly all EQASC values as perturbed.
- Error sensitivity: Figure 1 examines metric sensitivity on Somers’ D as error is injected into reasoning steps.The figure focuses on how selected metrics respond to different injected error levels.
8 CONCLUSION
The paper introduces ROSCOE as an interpretable, unsupervised suite for evaluating step-by-step reasoning without golden reference generations. Across synthetic and human-labeled datasets, it reports stronger performance than prior semantic and lexical baselines and improved analysis of factual and logical errors.
- Conclusion: ROSCOE evaluates step-by-step reasoning generations when no golden reference generation exists.The suite is unsupervised and interpretable.
- Conclusion: A taxonomy of reasoning errors supports both metric development and evaluation on synthetic and human-labeled datasets.The evaluated reasoning types include commonsense, arithmetic, and logical inference.
- Conclusion: ROSCOE outperforms prior semantic- and lexical-similarity-based baseline metrics across the evaluated datasets.The conclusion reports results spanning multiple reasoning types.
- Conclusion: The analysis indicates improved evaluation of nuanced factual and logical errors in step-wise decisions.This conclusion concerns errors within reasoning steps rather than only final answers.
ETHICS STATEMENT
The evaluation metrics do not monitor explanations for integrity or bias, although annotated generations were judged safe and unbiased by anecdotal consensus.
- ROSCOE does not monitor generated explanations from integrity or bias perspectives.The authors note that annotations covered logical, arithmetic, and general commonsense domains, where generations were considered safe and unbiased by anecdotal consensus.
A LIMITATIONS
The study is an initial investigation of evaluating LLM-generated step-by-step reasoning, with incomplete coverage of possible reasoning errors and downstream performance questions left open.
- The taxonomy does not cover all possible reasoning errors, although the metrics are designed to be generic for natural-language rationales.They consider alignment with both the input context and the generated explanation.
- The paper presents an initial step toward evaluating step-by-step reasoning produced by large language models.The authors position the study as a template that can be extended with additional reasoning errors, code, and datasets.
- The application of ROSCOE metrics to improving downstream task performance remains for future exploration.The authors deferred this question because thoroughly testing and communicating the metrics required extensive analysis.
C TAXONOMY OF REASONING ERRORS (CONT. FROM § 3)
The paper develops a taxonomy for reasoning errors and evaluates reasoning with chain- and step-level metrics, using multiple diagnostic and human-judged datasets across reasoning tasks.
- C TAXONOMY OF REASONING ERRORS (CONT. FROM § 3): The diagnostic datasets are constructed from sampled reasoning data and model-generated explanations, with perturbations informed by the reasoning-error taxonomy.The experiments use task-specific in-domain prompting and evaluate metrics on diagnostic datasets after embedding-model finetuning.
- C TAXONOMY OF REASONING ERRORS (CONT. FROM § 3): The taxonomy distinguishes chain-level errors from step-level errors in multi-step reasoning.Chain-level evaluation covers overall quality, coherence, consistency within the explanation, and consistency with context; step-level evaluation examines individual-step consistency with prior steps.
- D ROSCOE METRICS DETAILS (CONT. FROM §4): ROSCOE organizes metrics into semantic alignment, semantic similarity, logical inference, and logical coherence.Reasoning alignment measures how step-by-step reasoning is grounded in source text using token- and step-level similarities.
- RELATED METRICS: CTC and BARTScore provide related baselines based on information alignment or source-to-target generation probability.Unlike ROSCOE, the cited existing metrics are described as evaluating texts primarily as wholes rather than individual reasoning steps.
- DATASET USE: The dataset summary distinguishes reasoning-chain availability, diagnostic versus human-judged use, and train, validation, and test roles.StrategyQA is used only for finetuning the SimCSE embedding model, while the remaining datasets support evaluation and/or finetuning as specified.
- DATASETS: The study spans datasets for entailment, logical reasoning, mathematical problem solving, and multi-hop or commonsense question answering.Examples include EntailmentBank, ProofWriter, MATH, ASDIV, AQUA, EQASC, and StrategyQA.
G SENTENCE EMBEDDING MODEL TRAINING (CONT. FROM §6)
The embedding model is finetuned on perturbed reasoning chains, selected using similarity-based validation, and compared with multiple sentence-embedding models during inference.
- G SENTENCE EMBEDDING MODEL TRAINING (CONT. FROM §6): The SimCSE model is finetuned on training portions of perturbed diagnostic datasets and evaluated on their test portions against baseline metrics.Five hundred thousand samples with replacement are selected from each dataset to create uniform representation and reduce bias.
- HYPERPARAMETERS: Hyperparameter search varies batch size, learning rate, and maximum sequence length, with some combinations omitted because of memory limitations.The reported batch sizes range from 32 to 2048, learning rates from 5e-06 to 1e-04, and maximum sequence lengths from 64 to 512.
- VALIDATION: Checkpoint selection maximizes cosine similarity for positive pairs and minimizes it for hard-negative pairs within each batch.This replaces the original semantic-textual-similarity validation procedure with validation on perturbed reasoning chains.
- VALIDATION: The model is evaluated every 100 steps, and the best development-set checkpoint is used for inference.Other unspecified parameters remain as in the original SimCSE initialization.
- INFERENCE: ROSCOE scores are compared using finetuned SimCSE, sup-simcse-roberta-base SimCSE, and all-mpnet-base-v2 embeddings.Inference uses random seed 42 because embedding-based scores otherwise vary by about 0.01.
H.1 CONTROLLED EXPERIMENTS WITH DIAGNOSTICS DATASETS
ROSCOE’s diagnostics experiments evaluate how well fine-grained reference-free and reference-based scores detect synthetic reasoning errors across datasets. Results show strong task- and error-specific correlations, alongside important failure modes such as missing-step detection and false positives caused by superficially similar reasoning.
- Info-Chain covers 10 of 12 error types but does not consistently identify Remove Step or Semantic perturbations.
- Reference-based scores cover all synthetic errors, with Semantic Coverage Chain showing strong correlations across perturbation types.
- Some reference-based metrics achieve Somers’ D = 1.0, enabling perfect separation of perturbed and non-perturbed chains at a threshold in particular scenarios.
- Reference-free Repetition-* scores achieve perfect correlation on EQASC, but similar non-duplicated steps produce false negatives on other datasets.
- Self-Consistency correlates strongly with Repetition on GSM8K at 0.98, although only two repetition errors occur in that dataset.
- Faithfulness-Step shows no correlation with FACT errors in one example, while Source-Consistency reaches only 0.425 amid false positives and false negatives.
- Grammar scores overlap substantially between perturbed and unperturbed subsets, yielding a low Somers’ D correlation of 0.392.
- Info-Step obtains a 0.879 Somers’ D correlation for shuffled steps, but some unperturbed chains receive unusually low scores.