PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Reasoning Representations Help Humans Evaluate LLM Outputs?

Jaewoo Lim, Sungbok Shin, Sanghyun Hong

arXiv:2609.09038v1cs.LGcs.HC

TL;DR

Reasoning representations are increasingly used as explanations, but model-centric evaluations do not establish whether they help people assess model responses. This study compares six formats through controlled human evaluation and finds that preference, verification performance, and trust calibration do not always align.

  • Problem

    Model-centric measures such as answer accuracy and faithfulness leave unclear whether people can use reasoning representations to evaluate model responses.

    Source: §1 · p. 1

  • Method

    A controlled study compares six reasoning formats under matched conditions, measuring human judgments and verification behavior using correct and error-injected traces.

    Source: §1 · p. 2

  • Results

    Participants preferred planning- and decomposition-based formats, but simpler chain-of-thought traces better supported error detection and localization, while preferred formats could bring calibration risks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Perceived usefulness does not guarantee effective verification, and more visible or structured reasoning does not uniformly improve human evaluation.

    Source: §6 · p. 9

    Because all traces came from GPT-5, the study does not establish whether its findings generalize across models.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reasoning representations are increasingly used as explanations for large language model outputs. Yet they are typically evaluated with model-centric criteria, such as answer accuracy and faithfulness, leaving it unclear whether they help people evaluate model responses. In this work, we study reasoning representations as human-facing interfaces rather than proxies for model reasoning ability. We conduct a controlled human study of six reasoning formats across tasks of varying complexity, supported by a web-based framework that randomizes task domains, problem instances, and representation order. The study collects fine-grained judgments of structural understanding, error detection and localization, and trust calibration. Our study shows a mismatch between perceived preference and support for human evaluation. Participants prefer planning- and decomposition-based representations, but simpler chain-of-thought traces better support verification, trust, and interpretability. Preferred representations also introduce calibration risks, with more false alarms on correct traces and high trust despite low willingness to verify.

1 Introduction

Reasoning representations are increasingly used to help people assess LLM outputs, but they are usually evaluated with model-centric criteria that do not establish whether humans can use them. This study evaluates six representations as human-facing interfaces, finding that preference, verification performance, and trust calibration do not always align.

1 Introduction

  • Model-centric measures such as accuracy, faithfulness, or coherence do not establish whether people can use a trace to evaluate a response.A trace may be accurate or faithful yet make errors, component roles, or trust difficult to assess.Source: §1 · p. 1
  • The study asks whether reasoning representations help humans evaluate LLM outputs through structural understanding, error detection and localization, and trust calibration.It tests whether structured formats improve evaluation or instead make reasoning appear more interpretable.Source: §1 · p. 1
  • The study compares six representations under matched task and problem conditions, separating perceived usefulness from verification behavior with correct and error-injected traces.A web-based framework randomizes task domains, model families, problem instances, error conditions, and representation order.Source: §1 · p. 2
  • Participants preferred planning- and decomposition-based formats, while simpler chain-of-thought traces better supported error detection and localization.Preferred formats also brought calibration risks, including false alarms on correct traces and high trust paired with low willingness to verify.Source: §1 · p. 2
  • The paper reframes reasoning representations as interfaces for human evaluation and reports distinct trade-offs among preference, verification performance, and trust calibration.It also contributes a controlled comparison protocol and a web-based framework for collecting representation-level judgments.Source: §1 · p. 2

2 Background and Related Work

Prior evaluations of reasoning representations focus largely on model-centric properties, while human-facing use raises separate questions about how people understand, verify, and trust outputs. This work compares structurally distinct formats using human-centered judgments alongside established model-centric criteria.

2 Background and Related Work

  • The paper distinguishes a reasoning trace—the intermediate tokens before an answer—from the user-visible representation in which that trace is organized.It does not treat traces as faithful records of the model’s internal computation.Source: §2 · p. 2
  • Model-centric evaluations of reasoning quality do not determine whether users can identify errors, understand their impact, or calibrate trust.The study addresses this gap by jointly measuring structural understanding, error detection and localization, trust calibration, and perceived usefulness.Source: §2 · pp. 2–3
  • Table 1 categorizes 25 reasoning methods into groups A–I by the structural forms they expose, including topology, scope, and decomposition.The table marks the six representations used in this study with gray rows.Source: Table 1 · p. 3
  • Prior human-centered explanation studies distinguish subjective preference from behavioral outcomes such as error detection and decision quality.They also report that explanations can increase reliance on incorrect outputs.Source: §2 · p. 3
  • The paper identifies few prior comparisons of structurally distinct representations under controlled conditions, with existing work often limited in coverage, control, or behavioral evaluation.It uses the taxonomy to select distinct formats for controlled comparison.Source: §2 · pp. 2–3

3 Controlled Human Evaluation

The study uses a controlled protocol to compare six user-visible reasoning formats across three benchmarks, measuring structural understanding, verification, and trust-related judgments. It holds final-answer correctness constant for representation comparisons and adds manually checked error injections to assess verification.

3 Controlled Human Evaluation

  • The protocol compares six representations across three benchmarks using measures of structural understanding, error detection and localization, and trust calibration.Traces reaching the same correct answer are compared, while error-injected traces provide ground truth for verification behavior.Source: §3 · pp. 3–4

3.1 Evaluation Dimensions

  • Structural understanding measures whether users identify component roles, reasoning shifts, and dependencies between steps.The study operationalizes this dimension through cues such as plan-execution, subproblem transitions, and template instantiations.Source: §3.1 · p. 4

3.2 Representation Selection

  • The six selected formats span direct answers, linear and aggregated chains, planning, decomposition, and template-grounded reasoning.Plan-and-Solve separates a plan from its stepwise execution; Least-to-Most solves ordered subproblems progressively; Buffer of Thoughts exposes a retrieved template and its instantiation.Source: §3.2 · pp. 4–5

3.3 Study Materials and Sampling

  • The materials cover GSM8K arithmetic, HotPotQA multi-hop factual questions, and BBH symbolic and logical reasoning, using only instances where all six formats answer correctly.The final pool contains 27 problems, with 9 per benchmark; 27 of 140 candidates met the all-correct filter.Source: §3.3 · p. 5

3.4 Controlled Error Injection

  • For error evaluation, researchers inject one manually verified localizable error into selected otherwise-correct traces and compare participants’ judgments with known error locations.The 42 errored traces include all six formats for selected GSM8K and BBH problems, but only Least-to-Most and Buffer of Thoughts for HotPotQA, so unsupported cross-benchmark contrasts are avoided.Source: §3.4 · p. 5

4 A Web-Based Evaluation Framework

A web application supports controlled assignment, randomized presentation, and representation-level logging for the human evaluation. Participants complete one benchmark’s problems, with each problem assigned to a distinct evaluation dimension, then report comparative judgments and an overall preference.

4 A Web-Based Evaluation Framework

  • The web framework assigns study cells, randomizes the six representation conditions, and records judgments and timing at the participant-representation level.It supports branching error-detection follow-ups and persistent storage of responses.Source: §4 · p. 6

4.1 Participant Workflow

  • The 50-person analytic sample each completed one online session covering one benchmark and three problems, one per evaluation dimension.Each problem presented six representations in randomized order; the session ended with comparative questions and an overall preference.Source: §4.1 · p. 6

4.2 Randomization and Assignment

  • Representation order was randomized within each problem, while balanced assignment rotated error conditions so formats could be evaluated in correct and errored conditions where supported.The scheme controls comparisons for task domain, problem instance, error condition, and presentation order.Source: §4.2 · pp. 6–7

4.3 Measures and Analysis

  • Measures covered perceived structural cues, error detection and localization, reliance, verification intention, interpretability, and post-study preference.Error detection and localization were checked against known injected-error labels; trust items used five-point Likert scales.Source: §4.3 · p. 7

5 Evaluation

The evaluation compares six reasoning representations across structural understanding, verification, trust, and preference. It finds that stronger perceived structure or preference does not consistently align with effective human evaluation.

5 Evaluation

  • The evaluation measures structural understanding, verification on error-injected traces, trust ratings, and post-survey preferences across the six representations.Table 3 reports shift recognition and key-/support-step agreement; Table 4 reports item-level selection counts.Source: §5 · p. 7, Table 4 · p. 8
  • Planning formats led shift recognition (60% Plan-and-Solve; 64% Buffer of Thoughts), but Standard I/O led support-step agreement (α = 0.48), while Least-to-Most was lowest (α=0.05).Across representations, key-step agreement was α = 0.11–0.37 and support-step agreement was α = 0.05–0.48; participants did not consistently agree on which steps mattered.Source: Table 3 · p. 7, §5 · pp. 7–8
  • Zero-shot CoT had the lowest false-alarm rate (7.7%) and highest localization accuracy (95.5%); Plan-and-Solve led overall preference (26% vs. 22% for Zero-shot CoT).Least-to-Most had the highest false-alarm rate (20.8%), while Plan-and-Solve had the second-highest (19.2%).Source: Table 5 · p. 8, §5 · pp. 8–9
  • Zero-shot CoT received the highest mean reliance (4.04) and interpretability (4.12) ratings.These ratings accompany its stronger verification results, despite Plan-and-Solve receiving more overall-preference votes.Source: §5 · p. 8

6 Conclusion

The conclusion treats reasoning representations as interfaces for human oversight, not just indicators of model ability. It reports that preference, verification, and trust calibration can diverge, and argues that representations should be evaluated for the human judgments they support.

6 Conclusion

  • The paper reframes reasoning representations as interfaces for human oversight rather than solely as indicators of model reasoning ability.It evaluates how people understand, verify, and calibrate trust in model outputs.Source: §6 · p. 9
  • Users favor planning- and decomposition-based representations, while simpler chain-of-thought traces better support error detection and localization.The conclusion also notes that preferred formats can produce miscalibrated judgments.Source: §6 · p. 9
  • The paper argues that user-facing reasoning representations should be designed and evaluated for human judgments, not only model-centric criteria such as accuracy or faithfulness.The conclusion challenges the assumption that more visible or structured reasoning uniformly improves human evaluation.Source: §6 · p. 9

Limitations

The study’s findings are bounded by its use of pre-generated outputs, constructed traces and errors, and a single trace-generating model. These design choices support controlled comparisons but leave interactive and cross-model settings untested.

Limitations

  • Pre-generated outputs limit interaction, excluding follow-up questions, alternative explanations, and dialogue-based revision of judgments.The study chose this design to enable matched comparisons and isolate representation-level effects; interactive settings are left for future work.Source: Limitations · p. 9
  • Trace wording, length, detail, and structure may reflect prompt design, while verification may depend on injected-error type and location.The localization procedure requires a localizable trace, limiting applicability to formats such as Standard I/O.Source: Limitations · p. 9
  • Using GPT-5 alone controls model-level variation but leaves whether the findings generalize across models unevaluated.The authors leave cross-model replication for future work.Source: Limitations · p. 9

Ethics Considerations

The study was determined exempt by Oregon State University’s Institutional Review Board; participation was voluntary, and responses were anonymized.

Ethics Considerations

  • The study was IRB-exempt, voluntary, collected no identifying information, and stored responses anonymously.Source: Ethics Considerations · p. 9

A Detailed Experimental Setup

The appendix describes the shared generation setup, prompt templates, participant survey, and recruitment and consent procedures.

A Detailed Experimental Setup

A.1 Model and Generation Setup

This section describes the model and study setup for comparing six reasoning formats, including trace generation and selection, prompt and retrieval procedures, survey measures, and participant recruitment and analysis.

A.1 Model and Generation Setup

  • 27 (19.3%) of 140 candidate problems satisfied the filter requiring correct final answers from all six representations, with 9 retained per benchmark.The study used this filtered set for evaluation.Source: §A.1 · p. 13
  • All traces used GPT-5 at temperature 1.0, top-p 1.0, and at most 8,192 completion tokens; Self-Consistency CoT used five samples and majority voting, while other formats used one.The endpoint constrained Self-Consistency CoT to temperature 1.0.Source: §A.1 · p. 13
  • Adaptations included benchmark-specific Plan-and-Solve answer triggers, a “Final Answer:” format for Buffer of Thoughts, and at most six Least-to-Most subquestions with prior answers supplied in the second pass.Buffer of Thoughts used text-embedding-3-large for retrieval; Least-to-Most prepended a short “We already know” hint before each answer slot.Source: §A.1 · p. 13

A.2 Prompt Templates

  • The prompt templates include direct question-to-answer Standard I/O and two-stage Zero-shot Chain-of-Thought; Self-Consistency CoT shares the few-shot CoT prompt but aggregates five samples by majority vote.The few-shot templates include 8-shot exemplars for GSM8K and HotPotQA and per-subtask exemplars for BBH.Source: §A.2 · p. 13
  • Plan-and-Solve uses a planning trigger followed by answer extraction, Least-to-Most decomposes problems before sequential solving with accumulated answers, and Buffer of Thoughts uses a three-step pipeline.The Buffer of Thoughts pipeline reproduces the official implementation with GPT-5 as its backbone and text-embedding-3-large for retrieval.Source: §A.2 · pp. 14–15
  • The pipeline distilled problem templates, used LightRAG hybrid graph-and-vector retrieval to select relevant templates for solution generation, and inserted a new template only when a judge found a fundamental difference in approach.The initial meta-buffer contained 21 math templates, all indexed by LightRAG using text-embedding-3-large (dim 3072).Source: §A.2 · pp. 15–16

A.3 Survey Instrument

  • Per-trace questions assessed structural understanding, error detection and localization, and trust, verification intention, and interpretability across the six representation conditions.Structural items asked participants to identify reasoning shifts, key steps, and supporting steps; error follow-ups were conditional on selecting a specific step.Source: §A.3 · pp. 16–17
  • Post-survey items compared formats on comprehension, error handling, reliability, decision confidence, critical evaluation, and overall usefulness, with format labels randomized for each participant.Participants selected among six formats or “No difference / Not sure.”Source: §A.3 · p. 18

A.4 Participants and Consent

  • Of 64 participants who began the survey, 50 completed responses were included in the analytic sample; participants were recruited from courses at the authors’ institution and consented electronically.Thirteen partial responses were excluded, and one completion after the data-collection window was excluded; participants were required to be at least 18 and comfortable reading English.Source: §A.4 · p. 18

B Additional Experimental Results

This appendix presents supplementary per-trace results by evaluation dimension, followed by post-survey results. Table 6 describes D2 verification metrics and the verification-depth scale.

B Additional Experimental Results

  • Supplementary per-trace results are organized by evaluation dimension, followed by post-survey results.The appendix describes one subsection per dimension before reporting post-survey results.Source: B Additional Experimental Results · p. 18
  • Table 6 reports D2 sensitivity, false alarms, localization, type accuracy, impact judgments, and verification depth across six representations.Localization is conditional on step-flagged error trials; type accuracy is conditional on correct localization.Source: B Additional Experimental Results · p. 19, Table 6
  • Verification depth on errored trials runs from L0 missed errors to L4 full verification, with mean depth defined as P4.The intermediate levels distinguish noticing, locating, diagnosing, and projecting an error.Source: B Additional Experimental Results · p. 19, Table 6

B.1 Structural Understanding (D1)

The additional-results section points to tables covering structural understanding, verification, trust calibration, and post-survey preference. These tables report full results or summarize item-level and overall preference counts.

B.1 Structural Understanding (D1)

  • Table 3 reports transition recognition and key- and support-step agreement for each representation.The agreement measures are Krippendorff’s α.Source: §B.1 · p. 19

B.2 Verification Performance (D2)

  • Table 6 presents the full verification-performance results.It covers the D2 results.Source: §B.2 · p. 19

B.3 Trust Calibration (D3)

  • Table 8 reports the mean and standard deviation of three trust items for each representation.The items concern reliance, verification intention, and interpretability.Source: §B.3 · p. 19, Table 8 · p. 19

B.4 Post-Survey Preference

  • Tables 4 and 7 report post-survey item-level selection counts and overall usefulness-preference counts, respectively.Table 7 counts each completed participant’s choice of the most helpful representation overall.Source: §B.4 · p. 19, Table 7 · p. 19

Related papers

Loading 2609.09038v1…