PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Assessment of Self-Consistency in LLM Explanations using Perturbation Strength

Phuong Q. Le, Kemal Kurniawan, Jey Han Lau

arXiv:2609.30849v1cs.CL

TL;DR

Prior work uses surface perturbations to assess self-consistency in LLM explanations, but does not explicitly measure and control perturbation strength. This paper introduces a unified LLM-judge measure and finds that input perturbations generally affect models more strongly than CoT perturbations, so comparisons are fair only within the same perturbation type.

  • Problem

    Prior self-consistency studies give limited attention to measuring perturbation strength, leaving unclear whether changes warrant an answer change or are comparable across perturbation types.

    Source: §1 · pp. 1–2

  • Method

    The paper develops an LLM-as-a-judge framework to measure surface perturbation strength across input and CoT perturbations and uses strength-controlled flip rates to assess responsive and robust self-consistency.

    Source: §1 · p. 2, §4.3 · p. 7, §4.4.2 · p. 7

  • Results

    The LLM measure outperforms embedding- and probability-based alternatives, more than 50% of perturbations in existing datasets are weak, and input perturbations generally affect models more strongly than CoT perturbations.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Judgments about a model’s self-consistency are fair only within the same perturbation type.

    Source: §5 · p. 8

    The analysis assumes the strength measure reflects human judgments perfectly, although intermediate strength levels can sometimes be misclassified as the strongest level.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Prior work has examined the self-consistency of LLM-generated explanations using surface-level perturbation methods. However, the strength of these perturbations is not explicitly measured and controlled. In this work, we propose an LLM-as-a-judge approach to measure perturbation strength in a unified manner across input and CoT perturbations. We then evaluate the self-consistency in explanations generated from various LLMs under controlled strength conditions, ensuring a fair comparison across perturbation types. Experiments show that our proposed LLM-based perturbation strength measure outperforms other embedding- and probability-based approaches and that input perturbations generally affect LLMs more strongly than CoT perturbations. Our work suggests that judgments about a model's self-consistency is fair only within the same perturbation type.

1 Introduction

The introduction argues that self-consistency comparisons need perturbations whose strength is measured and controlled, because weak changes may not interfere with reasoning and different perturbation types may otherwise be compared unfairly. It proposes an LLM judge for measuring strength and uses it to study responsive and robust self-consistency.

1 Introduction

  • Prior work gives less attention to measuring perturbation strength, risking false conclusions about self-consistency when changes are too weak to warrant an answer change.Unmatched perturbation strengths can also make comparisons across perturbation types unfair.Source: §1 · pp. 1–2
  • More than 50% of perturbations in existing self-consistency datasets are weak, questioning their suitability for self-consistency checks.The proposed measure also finds that input perturbations generally affect models more strongly than CoT perturbations.Source: §1 · p. 2
  • The paper proposes an LLM-as-a-judge approach to measure perturbation strength uniformly across input and CoT perturbations, regardless of how they are generated.It reports that this approach outperforms two alternatives in correlation with human judgments.Source: §1 · p. 2
  • The paper introduces responsive and robust self-consistency as complementary notions for assessing model behavior under perturbations.The framework measures whether answers change under strong perturbations and remain stable under weak ones.Source: §1 · p. 2
  • Input perturbations generally affect models more strongly than CoT perturbations.This result motivates comparing self-consistency only within the same perturbation type.Source: §1 · p. 2

2 Related Work

The related work distinguishes self-consistency from the broader question of whether explanations faithfully reveal internal reasoning. It reviews input and CoT perturbation approaches, noting that prior strength proxies are indirect and specific to particular methods.

2.1 Self-consistency

  • Self-consistency measures whether a model’s behavior aligns with the reasoning presented in its generated explanations.Even a self-consistent explanation may not reveal the model’s actual internal reasoning path; internal analysis is needed to assess that broader faithfulness question.Source: §2.1 · p. 2
  • This work focuses on model-agnostic self-consistency assessment using input and CoT perturbations, rather than claiming to establish explanation faithfulness.The paper notes that surface-level consistency cannot show whether the explanation reflects the model’s internal processes.Source: §2.1 · p. 2

2.2 Perturbation-based Methods

  • Input perturbations modify the input text, and this work adopts free-form edits rather than edits targeting predefined concepts.The cited prior approaches include concept-targeted and free-form input changes.Source: §2.2 · p. 2
  • CoT perturbations modify the reasoning rather than the input; this work uses Lanham et al. (2023)’s approach to evaluate strength across more diverse perturbations.Lanham et al. generate mistaken CoT steps, whereas Paul et al. (2024) use CoTs from counterfactual questions, which are likely to produce strong perturbations.Source: §2.2 · pp. 2–3
  • Earlier strength proxies, such as perturbed-step counts or changing important concepts, are indirect and specific to particular perturbation approaches.The work notes that equal numbers of perturbed steps need not yield comparable strength across tasks.Source: §2.2 · p. 3

3 Perturbation Strength Assessment

The section introduces a unified discrete measure of input and CoT perturbation strength, evaluates it against human judgments, and applies it to find that many existing CoT perturbations are weak.

3.1 Datasets

  • The datasets cover four multiple-choice QA tasks and four CoT-generating models; each perturbed CoT replaces one original step with a mistaken step while keeping the others unchanged.This yields 16 perturbed datasets, each with 230 questions.Source: §3.1 · pp. 3–4

3.2.1 Perturbation Strength Criteria

  • Perturbation strength is defined by interference with the original reasoning path and the shift of support from the original answer toward another option.Strong changes are expected to change the answer, whereas weak changes are not expected to do so.Source: §3.2.1 · p. 4, §3.2 · p. 4

3.2.2 Sampled Data for Annotation

  • The evaluation sample includes 96 questions, with CoT perturbations diversified by combining perturbed steps or paraphrasing them, and 96 generated input perturbations.The original CoT was supplied when generating question changes to make them more relevant to the reasoning.Source: §3.2.2 · p. 4

3.2.3 Annotation Procedure

  • 0.855 for input and 0.649 for CoT perturbations were the annotators’ Kappa scores; disagreements often concerned intermediate strength levels 1 and 2.The primary annotator labeled 96 CoT and 96 input perturbations using the strength criteria.Source: §3.2.3 · p. 4

3.3 Perturbation Strength Measures

  • The measures include embedding-based cosine distance, probability-based change in mean surprisal, and an LLM judge prompted with the original and perturbed text plus strength criteria.The LLM prompt also includes a strength-level example and justification for each dataset; larger surprisal changes indicate higher strength.Source: §3.3 · pp. 4–5

3.4 Evaluation and Results

  • The LLM judge outperforms cosine distance and change in surprisal, with consistently high correlations to human judgments for input and CoT perturbations.The authors use it as their measure in subsequent analyses.Source: §3.4 · pp. 5–6, Table 2 · p. 6
  • Cosine distance and surprisal change are mostly uncorrelated with human judgments; a large semantic shift may still leave the reasoning path intact.The authors attribute the limitation to surface-level measures that cannot incorporate task, reasoning context, or strength criteria.Source: §3.4 · p. 6

3.5 Applications of the Proposed Measure

  • Across datasets, more than half of the perturbed CoTs are weak; prior answer-change tests did not verify strength, so their results may not represent model behavior under CoT perturbations.For most CoT models, fewer than 10% of perturbed CoTs are strong.Source: §3.5 · p. 6

4 CoT vs Input Perturbations

The section compares input and CoT perturbations using flip rates stratified by perturbation strength, distinguishing expected responses to strong perturbations from stability under weak ones.

4.1 Metric

  • Flip rate at strength s is the proportion of strength-s instances where the model changes its answer after perturbation.The comparison uses the model’s answers before and after each perturbation.Source: §4.1 · p. 6

4.2 Data Construction

  • For input perturbations, the study constructs perturbed questions and obtains model answers; for CoT perturbations, it reuses answers and measured strengths from prior data.The input-answer extraction follows procedures consistent with those used for CoT perturbations.Source: §4.2 · p. 6

4.3 Self-consistency Measurements

  • The framework distinguishes responsive self-consistency, measured by FR(3), from robust self-consistency, measured by FR(0).Higher FR(3) indicates greater responsiveness to strong perturbations; lower FR(0) indicates greater robustness to weak perturbations.Source: §4.3 · p. 7

4.4.1 Responsive and Robust Self-consistency

  • Across both perturbation types, stronger changes generally produce higher flip rates, while input perturbations generally yield higher rates than CoT perturbations, including at zero strength.Under input perturbations, nearly all models have high FR(3), but FR(0) frequently exceeds 20%; under CoT perturbations, FR(0) remains tiny.Source: §4.4.1 · p. 7, Figure 6 · p. 8

4.4.2 Overall Self-consistency

  • Phi-3 leads on OpenBookQA and Mistral-2 on StrategyQA for both perturbation types; LLaMA-3B and LLaMA-8B show opposite strengths across types.The overall measure is FR(3) − FR(0), and models that perform well across perturbation types and datasets provide stronger evidence of self-consistency.Source: §4.4.2 · p. 7, Table 3 · p. 8

4.4.3 Discussion

  • Because input perturbations generally affect models more strongly, the authors recommend comparing self-consistency against other models within the same perturbation type.No established flip-rate ranges define good self-consistency, so the analysis relies on relative model performance.Source: §4.4.3 · p. 8
  • The strength-aware framework supports comparisons across perturbation types and helps evaluate self-consistency by controlling perturbation strength and checking existing perturbations.The section presents these as applications of a more nuanced interpretation of when a model should change its answer.Source: §4.4.3 · p. 8

5 Conclusion and Future Work

The conclusion presents an LLM-judge framework for measuring perturbation strength and uses it to distinguish self-consistency patterns across input and CoT perturbations. It identifies ambiguity in intermediate strength levels and a broader range of perturbation forms for future work.

5 Conclusion and Future Work

  • The LLM-judge framework measures perturbation strength, outperforms embedding- and probability-based approaches, and finds input perturbations generally affect models more strongly than CoT perturbations.The paper argues that self-consistency judgments are fair only within the same perturbation type.Source: §5 · p. 8
  • Controlling perturbation strength captures two complementary aspects of self-consistency: responsive and robust behavior.Source: §5 · p. 8
  • Future work could reduce ambiguity in intermediate strength levels and extend strength measurement beyond surface-level perturbations to parameter interventions and embedding perturbations.Source: §5 · p. 8

Limitations

The analysis assumes the perturbation-strength measure perfectly reflects human judgments, but intermediate levels can be misclassified. It also has sampling limitations, particularly for strength-3 CoT perturbations and the unequal numbers of CoT and input perturbations.

Limitations

  • The analysis assumes its perturbation-strength measure perfectly reflects human judgments, but intermediate level 2 can sometimes be misclassified as strongest level 3, potentially undermining the reliability of findings.The measure has the best correlation with human judgments among the alternatives, though it is not perfect.Source: Limitations · p. 9
  • Strength-3 CoT perturbations are substantially fewer than other levels, making FR(3) more uncertain and potentially higher-variance.The authors report that higher-strength CoT results fluctuate more, while generally retaining the same patterns: higher strength brings higher flip rates, and CoT perturbations have lower flip rates than input perturbations.Source: Limitations · p. 9
  • The comparison uses more perturbed CoTs than questions, and its 230 input questions support pattern-level conclusions rather than exact flip-rate estimates.The authors do not expect additional input samples to significantly change the patterns across 16 experiment sets.Source: Limitations · p. 9

A.1 Prior Perturbation Strength Controls

This section describes how perturbation strength can be used to assess prior controls, then details data-contamination context, perturbation generation, sampling, and annotation. The procedures cover both question-input and CoT edits and use a shared strength guide.

A.1 Prior Perturbation Strength Controls

  • The strength measure can verify, improve, or replace prior perturbation controls and relate answer-change rates to strength across tasks.For Lanham et al. (2023), it can check whether perturbation strengths are comparable across tasks or relate answer changes directly to measured strength.Source: §A.1 · p. 11

A.2 Data Contamination Risk

  • Golchin and Surdeanu (2025) found that models exposed to evaluation questions during training favored the exactly memorized option, motivating contamination awareness in perturbation studies.Without training exposure, the models retained their original positional biases rather than favoring the option from the training instance.Source: §A.2 · p. 11

B Perturb CoT and input

  • Four instruction-tuned CoT models generate original reasoning; gpt-4o-mini edits individual CoT steps or questions, with input edits informed by the full CoT.For CoT perturbations, the edited step replaces its original while the other steps remain unchanged.Source: B Perturb CoT and input · p. 12

C Sampling Strategy

  • 24 of 230 question IDs per dataset were sampled and divided among four CoT models, yielding 453 instances for 96 questions across four datasets.Each model was assigned one set of six question IDs per dataset.Source: C Sampling Strategy · p. 12

D Constructing Diverse Perturbed CoTs

  • For each sampled question, a uniformly sampled step count determines how many CoT steps are replaced; when it is zero, all steps are paraphrased.The procedure produces one perturbed CoT per question.Source: D Constructing Diverse Perturbed CoTs · p. 13

E.1 Guideline for Human Annotators

  • A primary annotator rated 96 CoT and 96 input perturbations using the strength criteria and the original question and CoT as context.The guide is shown in Figure 11.Source: §E.1 · p. 13

E.2 Inter-Annotator Agreement

A second annotator’s ratings were compared with the primary annotator’s to assess the strength guide’s reliability. Agreement was strongest at the extreme strength levels, with more confusion between intermediate CoT levels.

E.2 Inter-Annotator Agreement

  • A second annotator independently rated 18 examples of each perturbation type, and ordinal agreement was measured with Cohen’s linear Kappa.The second annotator received dataset-specific examples illustrating all four strength levels.Source: §E.2 · p. 13
  • 0.6489 for CoT perturbations indicates substantial agreement, while 0.8548 for input perturbations indicates near-perfect agreement.The authors conclude that the strength guide is reliable to use.Source: §E.2 · p. 13, Table 4 · p. 15
  • Disagreements were concentrated in CoT perturbations with true strength 1 or 2; extreme strengths 0 and 3 had very strong agreement across both types.The second annotator tended to rate true strength 1 as 2, and true strength 2 as 3.Source: §E.2 · p. 15

F Experimental Setup

The experimental setup describes embedding- and probability-based strength measures alongside an LLM judge. The judge uses a strength guide and is a different model from the one used to generate perturbations.

F.1 Cosine Distance

  • Original and perturbed texts are represented with 384-dimensional all-MiniLM-L6-v2 embeddings using mean pooling for cosine-distance assessment.The authors selected the lightweight sentence-encoding model for competitive performance and computational efficiency.Source: §F.1 · pp. 15–16

F.2 Surprisals

  • GPT-2 token probabilities provide the surprisals used by the probability-based measure.The authors describe GPT-2 as a lightweight, open-source research baseline for log-probabilities and surprisals.Source: §F.2 · p. 16

F.3 LLM-as-a-judge

  • Gemini 3 Flash Preview judges perturbation strength using input- and CoT-specific prompts and a 0–3 rating guide.The authors avoid gpt-4o-mini, which generated the perturbations, to avoid possible unintended effects from using the same model as judge.Source: §F.3 · pp. 16–17, Figure 15 · p. 16, Figure 16 · p. 16

G Evaluation of Perturbation Strength Measures

The evaluation excludes dataset-specific examples, leaving 80 of 96 questions and their perturbations. Change in surprisals correlates weakly and negatively with true CoT perturbation strengths, in part because paraphrasing changes many more tokens at zero strength.

G Evaluation of Perturbation Strength Measures

G.1 Change in surprisals is not effective for paraphrases

  • Change in surprisals has a weak negative correlation with true CoT perturbation strengths, and is especially high for zero-strength paraphrases.Paraphrasing entire reasoning steps changes more tokens than perturbing selected steps, so token-level changes accumulate in the surprisal measure.Source: §G.1 · p. 17
  • More token modifications from paraphrasing accumulate into a larger change in token probabilities and surprisal.Selected-step perturbations change only some tokens in the perturbed steps, unlike paraphrases that affect tokens across all steps.Source: §G.1 · p. 17

H Data for Input Perturbation

Because the cited datasets lacked input perturbations, the authors generated perturbed questions and measured their strengths with the same LLM-as-a-judge method used for CoT perturbations. They extracted answers directly to compare original and perturbed questions while minimizing extraneous variables.

H Data for Input Perturbation

  • The cited datasets lacked input perturbations, so the authors generated perturbed questions, their strengths, and post-perturbation answers to compute flip rates.The generated collection comprises 16 datasets, one per dataset–CoT-model pair, with 230 questions in each.Source: H Data for Input Perturbation · p. 17
  • GPT-4o-mini perturbed each question by adding a mistake, and an LLM-as-a-judge measured strength consistently with the CoT perturbation assessment.Each question was paired with its original CoT; Figure 18 shows the prompt used to obtain the CoT model’s answer.Source: H Data for Input Perturbation · p. 17, Figure 18 · p. 17

Model’s Answers After Input Perturbations

  • Direct prompting extracts the perturbed-question answer from first-token option probabilities, minimizing variables by comparing direct answers that differ only in the question.The highest-probability option letter is selected; the original CoT-based answers match direct-prompting answers in the cited setup.Source: Model’s Answers After Input… · p. 17

I Additional Results for CoT vs Input Perturbation

Across the reported comparisons, input perturbations generally have higher flip rates than CoT perturbations under either direct or CoT prompting. The examples also illustrate weak robustness to negligible input changes and weak responsiveness to strong CoT changes.

I.1 Flip Rates

I.2 Examples of Fail Cases

  • Examples show an answer change under negligible input perturbation and no answer change despite a strong CoT perturbation.In the CoT example, altered reasoning removes support for the original answer and can shift support to another option, yet the model retains its original answer.Source: §I.2 · p. 18, Figure 22 · p. 20, Figure 23 · p. 20

I.3 Direct Prompting vs CoT Prompting for Input Perturbation

  • Input perturbations generally have higher flip rates than CoT perturbations, and direct and CoT prompting give similar flip rates.This pattern holds across the four datasets and prompting comparisons; the authors focus subsequent analysis on direct prompting to minimize extraneous variables.Source: §I.3 · p. 18, Figure 24 · p. 21, Figure 25 · p. 21

Related papers

Loading 2609.30849v1…