ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning

1 INTRODUCTION

• Prior work found that step-by-step rationales can improve reasoning, but existing evaluation often emphasizes end-task performance or scores generated text as a whole.The introduction notes that many generative metrics target summarization or machine translation rather than reasoning.Source: §1 · p. 1
• ROSCOE comprises 18 fine-grained metrics across semantic alignment, logical inference, semantic similarity, and language coherence.These perspectives assess grounding and coherence, consistency and fallacies, similarity that can reveal hallucinations or repetition, and natural flow across the chain.Source: §1 · p. 2
• ROSCOE gains on baseline evaluation metrics across tasks requiring reasoning over context, with sensitivity analysis finding greater robustness on logical and arithmetic reasoning.The comparative analysis covers 11 datasets of varied complex reasoning problems.Source: §1 · p. 2
• The authors devise a taxonomy of multi-step reasoning errors to collect human annotations and create synthetic datasets for studying reasoning inconsistencies and LLM weaknesses.The taxonomy supports both diagnostic analysis and evaluation of ROSCOE.Source: §1 · p. 2

Evaluating Explanations

• Human explanation judgments are reliable but expensive, domain-specific, and time-consuming, while automatic NLG metrics miss logical inconsistencies and information gain in reasoning steps.Prior work assessed explanation correctness through human labels (Camburu et al., 2018) and correlated metrics with judgments of clarity, relevance, and informativeness (Leiter et al., 2022; Welleck et al., 2022).Source: §2 · p. 2

Automatic Metrics

• NLG metrics based on n-gram match, regression, embeddings, paraphrasing, generation, or information alignment evaluate whole-text alignment rather than individual reasoning steps.Examples include Lin (2004), Sellam et al. (2020), Zhang et al. (2020), Thompson & Post (2020), Yuan et al. (2021), and Deng et al. (2021).Source: §2 · p. 2
• ROSCOE provides fine-grained interpretations of reasoning steps, detecting contradictions and ordering issues beyond CTC and BARTScore’s text-similarity metrics.CTC (Deng et al., 2021) and BARTScore (Yuan et al., 2021) are described as interpretable metrics for similarity between two texts.Source: §2 · pp. 2–3

Self-Consistency with LLMs

• Unlike self-consistency and related work on reasoning-path diversity, inference-step consistency, or finetuning, ROSCOE identifies error types such as commonsense or logical inconsistency.Self-consistency samples diverse reasoning paths and selects the most consistent final answer (Wang et al., 2022).Source: §2 · p. 3

Problem Formulation

• ROSCOE scores a model-generated hypothesis of reasoning steps, including its final answer, given source context, without assuming gold step-by-step references.The context consists of a problem statement followed by a question.Source: §3 · p. 3

Taxonomy

• The proposed taxonomy identifies nine reasoning-error types spanning chain-level quality, step-to-step consistency, contribution to the final decision, logical inference, and fluency.The taxonomy was developed through preliminary manual analysis of errors in five human-judged datasets.Source: §3 · p. 3

Datasets and Annotations

• Evaluation uses diverse reasoning tasks in diagnostic datasets with synthetically perturbed steps and human-judged datasets with expert assessments of model-generated reasoning.Perturbations introduce errors such as missing steps and mathematical errors.Source: §3 · p. 3

4.1 SEMANTIC ALIGNMENT METRICS (ROSCOE-SA)

• ROSCOE-SA matches each reasoning step to its most similar source sentence using normalized cosine similarity to measure grounding.Unlike CTC’s information alignment, which measures source-token grounding by hypothesis tokens, ROSCOE measures whether a hypothesized reasoning step supports the source context.Source: §4.1 · p. 4, Table 3 · p. 4

4.2 SEMANTIC SIMILARITY METRICS (ROSCOE-SS)

• ROSCOE-SS measures semantic equivalence between texts as wholes, rather than comparing individual text units.Its proposed metrics are summarized in Table 4.Source: §4.2 · p. 5, Table 4 · p. 5

4.3 LOGICAL INFERENCE METRICS (ROSCOE-LI)

• ROSCOE-LI uses an NLI model trained on hypothesis-context pairs to estimate contradiction probability, pcontr.The logical inference metrics measure logical errors between pieces of text.Source: §4.3 · p. 5, Table 5 · p. 5

4.4 LANGUAGE COHERENCE METRICS (ROSCOE-LC)

• ROSCOE-LC assesses language coherence with GPT2-Large perplexity and grammatical acceptability from Krishna et al. (2020), using both models without fine-tuning.These language coherence metrics are summarized in Table 6.Source: §4.4 · p. 5, Table 6 · p. 5

Diagnostics Datasets

• Seven reasoning datasets receive 12 taxonomy-based perturbation rules, with binary labels indicating whether half of reference reasoning steps were modified.The datasets cover entailment, logic, arithmetic, commonsense question answering, and implicit reasoning strategies.Source: §5 · pp. 5–6

Human Judged Datasets

• Five reasoning datasets are human-judged: five experts rate chains and individual steps, using Wei et al. (2022) GSM8K chains and GPT-3-generated chains elsewhere.Judges receive source text and reasoning steps separated individually; annotations use the paper’s reasoning-error taxonomy.Source: §5 · p. 6

ROSCOE Training

• ROSCOE’s SimCSE model trains contrastively on reference and perturbed reasoning chains, then embeds individual steps and whole chains for scoring.Initialized from sup-simcse-roberta-base, it is trained for five epochs on synthetic data; scores are also compared using sup-simcse-roberta-base and all-mpnet-base-v2 embeddings.Source: §5 · p. 6

Baseline Metrics

• Baselines include ROUGE, BLEURT, PRISM, BERTScore, BARTScore variants, and CTC variants, with BARTScore-P finetuned on the reasoning datasets.The study uses official metric code and also examines ROSCOE metrics that use reference reasoning steps against human judgments.Source: §5 · p. 6

Meta Evaluation

• Meta-evaluation uses Somers’ D to compare each scorer with synthetic and human scores, preferring it over Kendall’s τ(-b) because it handles ties in biased variables.Correlations use the perturbation or error-presence indicator X and compute D(Y | X) = τ(X, Y) / τ(X, X).Source: §5 · p. 7

Controlled Experiments with Diagnostics Datasets

• Table 7 reports reference-free Somers’ D correlations across six diagnostic datasets, while Table 8 reports correlations with human judgments.Table 7 takes the maximum score over perturbations within each dataset; Table 8 reports the maximum over error types. The supplied passages do not include the tables’ correlation values.Source: Table 7 · p. 7, §6 · p. 7, Table 8 · p. 8
• Up to 0.556 on Informativeness-Chain, finetuning improved average correlations across datasets except Repetition_* scores.Mean Repetition-Token was unchanged, while mean Repetition-Step decreased by 0.005. The authors hypothesize that finetuning on reasoning chains and context better captures step-by-step rationales but may degrade word- and sentence-level semantics.Source: §6 · p. 8
• On EQASC, Repetition* scores separated chains with duplicated steps from non-perturbed chains at a threshold, achieving perfect correlation scores.The perfect scores are reported in Appendix Table 20.Source: §6 · p. 8

• Metric sensitivity was measured by correlating Somers’ D with error levels 1–3 after injecting up to three random perturbations per instance.The analysis used diagnostic examples from MATH arithmetic and EntailmentBank deductive reasoning.Source: §7 · p. 8
• ROSCOE-SA and ROSCOE-SS behave consistently across MATH and EntailmentBank, whereas baselines fluctuate; baselines fare better on EntailmentBank.On MATH, ROSCOE-LC and baseline metrics show minimal impact, possibly because RandomOperation and ShuffleNumbers are harder for BARTScore and ROSCOE-LC to detect.Source: §7 · pp. 8–9, Figure 1 · p. 9
• Scores separated by perturbation use within each dataset but overlapped across datasets, showing dataset drift for ROSCOE and strong baselines.Source: Figure 2 · p. 9

8 CONCLUSION

• ROSCOE evaluates language-model step-by-step reasoning without requiring a gold reference generation.The paper also presents a taxonomy of reasoning errors used to generate and evaluate its metrics.Source: §8 · p. 9
• Across synthetic and human-labeled datasets covering commonsense, arithmetic, and logical inference, ROSCOE outperforms prior semantic and lexical text-generation metrics.Source: §8 · p. 9
• ROSCOE better evaluates reasoning nuances, including factual and logical errors in step-wise decisions.Source: §8 · p. 9

ETHICS STATEMENT

• The authors note that ROSCOE does not monitor explanations for integrity or bias.They identify this as a limitation of the evaluation metrics.Source: ETHICS STATEMENT · p. 10
• The work used five human expert annotators to rate model-generated candidate rationales.The authors acknowledge that model-generated explanations can potentially contain unsafe content.Source: ETHICS STATEMENT · p. 10
• Annotators evaluated datasets covering logical and arithmetic concepts and general commonsense knowledge.The authors report an anecdotal consensus that the generations were safe and did not include biased statements.Source: ETHICS STATEMENT · p. 10
• The authors argue that explainability builds user transparency and trust, eases bug-fixing, and shortens metric-improvement cycles.They also state that explainability will be required by laws or regulations for AI systems used in large-scale, high-stakes domains.Source: ETHICS STATEMENT · p. 10
• The authors hope their work will catalyze efforts on explainable evaluation metrics for language-model rationale generations.Source: ETHICS STATEMENT · p. 10

A LIMITATIONS

• The study is an initial effort with a taxonomy of reasoning errors, and its generic metrics assess natural-language rationales in relation to input context and generated explanations.The authors note that the taxonomy does not cover every possible reasoning error and offer the work, code, and datasets as templates for further research.Source: A LIMITATIONS · p. 15
• Applying the metrics to improve downstream task performance is left for future exploration.The authors defer this question because thoroughly testing and communicating the metrics’ ability to capture reasoning errors required extensive analysis.Source: A LIMITATIONS · p. 15

B FEW-SHOT PROMPTING EXAMPLES (CONT. FROM § 1)

• The appendix presents a 2-shot prompt for eliciting logical inferences and step-wise explanations from GPT-3.The examples are introduced as prompts used to generate explanations, with Table 9 illustrating the in-context examples.Source: B FEW-SHOT PROMPTING EXAMPLES (CONT.… · p. 15, Table 9 · p. 16
• In the other example, friends’ praise of Sam’s “Chopsticks” rendition is treated as sarcasm, leading the explanation to infer he likely did not major in piano at Juilliard.The explanation reasons that Chopsticks is simple and that a Juilliard piano major would be an expert pianist; it answers “no.”Source: B FEW-SHOT PROMPTING EXAMPLES (CONT.… · p. 15

C TAXONOMY OF REASONING ERRORS (CONT. FROM § 3)

• The taxonomy distinguishes chain-level errors in overall quality, coherence, and context consistency from step-level errors in support, factuality, and logical inference.The authors developed it by manually inspecting GPT-3 explanations and model-generated explanations shared in Wei et al. (2022), then used its categories to create perturbed diagnostics and human-judged datasets.Source: C TAXONOMY OF REASONING ERRORS (CONT.… · p. 17, Table 10 · p. 18

D ROSCOE METRICS DETAILS (CONT. FROM §4)

• ROSCOE groups metrics into semantic alignment, semantic similarity, logical inference, and logical coherence, with reasoning alignment measuring how well each step and token is grounded in the source.Figure 3 illustrates Faithfulness-Step and Faithfulness-Token scores using token- and step-level cosine similarities; the section also describes BARTScore and CTC Relevance and Consistency as comparison metrics.Source: D ROSCOE METRICS DETAILS (CONT. FROM §4) · p. 19, Figure 3 · p. 19

E.1 DIAGNOSTIC DATASETS

• The diagnostic collection spans logical inference, multi-step QA, math, and commonsense reasoning; StrategyQA is used only to fine-tune SimCSE, while the other six diagnostic datasets support fine-tuning and evaluation.Examples include EntailmentBank’s multi-premise entailment trees, ProofWriter’s natural-language rule proofs, MATH and ASDIV problems, and AQUA algebraic word problems; EQASC supplies multi-step factual explanations.Source: §E.1 · p. 20

E.2 HUMAN JUDGED DATASET CONSTRUCTION

• Human-judged datasets use GPT-3 few-shot prompts to generate step-by-step reasoning, which is labeled with taxonomy-based perspectives across varied QA, math, commonsense, and inference tasks.The datasets include DROP discrete reasoning, GSM8K grade-school math, CosmosQA commonsense reading comprehension, ESNLI with human-labeled explanations, and SemEval commonsense comprehension.Source: §E.2 · p. 22

E.3 SYNTHETIC DIAGNOSTICS DATASET GENERATION WITH PERTURBATION RULES

• Synthetic diagnostics perturb half the reasoning chains from six datasets, using grammar changes, semantic substitutions, and steps sampled from other chains to model extrinsic hallucinations.Math datasets receive additional perturbations targeting arithmetic, common-sense equation construction, and factuality, including shuffling or replacing numbers and operations.Source: §E.3 · pp. 22–23, Table 14 · p. 24

F HUMAN ANNOTATIONS (CONT. FROM § 5)

• Five expert annotators rated overall chain errors on a 1–5 Likert scale and marked taxonomy error types at the step level across five datasets.Annotators took an average of 754 seconds to complete a session of up to five examples; the authors report that crowd workers were difficult to rely on for accurate labels even with pay averaging over $30 per hour.Source: F HUMAN ANNOTATIONS (CONT. FROM § 5) · p. 25, Table 15 · p. 26, Table 16 · p. 26

G SENTENCE EMBEDDING MODEL TRAINING (CONT. FROM §6)

• Training uses perturbed-dataset train splits, validation selects the embedding model, and test splits evaluate ROSCOE against baseline metrics.Source: G SENTENCE EMBEDDING MODEL TRAINING… · p. 27
• To balance representation and reduce bias, the authors randomly sample 500,000 examples with replacement from each dataset.Source: G SENTENCE EMBEDDING MODEL TRAINING… · p. 27
• Fine-tuning hyperparameters are searched over batch sizes {32, 64, 256, 512, 1024, 2048}, learning rates {5e-06, 1e-05, 5e-05, 1e-04}, and sequence lengths {64, 128, 512}, with some combinations excluded by memory limits.Training uses NVIDIA Tesla V100 Volta GPU instances with 32GB graphics cards.Source: G SENTENCE EMBEDDING MODEL TRAINING… · p. 27
• Validation replaces the original semantic-textual-similarity procedure with a criterion that increases cosine similarity for positive pairs and decreases it for in-batch hard negatives.The hard-negative pairs are evaluated within a batch of size B.Source: G SENTENCE EMBEDDING MODEL TRAINING… · p. 27
• The model is evaluated on the development set every 100 steps, and the best checkpoint is used for inference; unspecified parameters remain as in the original SimCSE initialization.Source: G SENTENCE EMBEDDING MODEL TRAINING… · p. 27
• At inference, ROSCOE scores are compared using fine-tuned SimCSE, sup-simcse-roberta-base, and all-mpnet-base-v2 embeddings, with random seed 42 set for reproducibility.Without the seed, embedding-based scores varied by about 0.01.Source: G SENTENCE EMBEDDING MODEL TRAINING… · p. 27

H.1 CONTROLLED EXPERIMENTS WITH DIAGNOSTICS DATASETS

• Info-Chain covered 10 of 12 diagnostic error types, missing Remove Step and Semantic errors, while ROSCOE scores did not consistently detect missing steps.Other synthesized error types were covered by at least one score type.Source: §H.1 · p. 28, Figure 6 · p. 29
• Reference-based scores covered all synthetic errors; Semantic Coverage Chain correlated strongly across perturbations, while ROUGE excelled on wording-changing errors but underperformed on smaller changes.ROUGE performed strongly on Repetition, Hallucination, Remove Step, Shuffle Steps, Swap Steps, Negate Step, and Semantic perturbations, but less strongly on Random operation and Shuffle operations.Source: §H.1 · p. 28
• 1.0 Somers’ D correlation let Semantic Coverage-Chain separate perturbed from unperturbed EQASC chains for Hallucination using a score threshold of 1.0.The example had no false positives: all perturbed chains scored below 1.0, while unperturbed chains scored 1.0.Source: §H.1 · pp. 28–29, Table 26 · p. 38
• Reference-free Repetition-* scores correlated perfectly on EQASC, but other datasets had false negatives when similar, non-duplicate steps received low scores.EQASC’s fact-based explanation steps were especially dissimilar, helping similarity-based scores distinguish them.Source: §H.1 · p. 29, Figure 7 · p. 29

H.2 EXPERIMENTS WITH HUMAN JUDGEMENT DATASETS

• On human-judgment data, Self-Consistency covered 6 of 7 evaluation perspectives except Missing Step, and Repetition-* and Self-Consistency correlated most strongly with Redundancy.Repetition errors were excluded because they occurred at most three times per dataset.Source: §H.2 · p. 44, Figure 8 · p. 44
• On GSM8K, Self-Consistency had 0.98 Somers’ D for Repetition with only two such errors, while Repetition-Step had 0.869 Somers’ D amid three low-scored non-error chains out of 198.The latter illustrates false positives when non-duplicate steps sound similar.Source: §H.2 · p. 54
• Examples showed weak alignment in some cases: FACT error had no correlation with Faithfulness-Step (Somers’ D 0.014), while Source-Consistency had moderate correlation (0.425) amid false positives and false negatives.Source: §H.2 · p. 54
• Info-Chain scored 0.983 on a negated conclusion that the metric failed to detect.The negation occurred in the final step, which did not overlap in wording with the context.Source: §H.2 · p. 57
• 0.392 was the Grammar correlation, as perturbed and unperturbed score distributions overlapped significantly.The unperturbed subset had mean 0.909 and variance 0.005, versus 0.854 and 0.014 for perturbed data; the classifier also missed a dropped verb and gave low scores for simple chains.Source: §H.2 · p. 57
• 0.879 Somers’ D was the Info-Step correlation for shuffle detection, and the metric flagged most chains with shuffled steps.Info-Step scores ranged from 0.005 to 0.796; perturbed chains had mean 0.356 and variance 0.056, versus 0.705 and 0.004 for unperturbed chains.Source: §H.2 · p. 57
• Info-Step assigned a higher score to a chain with perturbed operations, while scoring an unperturbed chain on a lower spectrum.The passages describe these as examples of a perturbation receiving a higher score and a chain with no perturbation or other error receiving a lower-spectrum score.Source: §H.2 · p. 57