PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language

Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mrinmaya Sachan

arXiv:2610.01627v1cs.CLcs.AI

TL;DR

Difficulty predictors often return scores without explaining what makes a question hard. This paper generates and validates natural-language hypotheses from contrasting questions; across three reasoning datasets, the hypotheses predict difficulty, improve existing predictors, and support edits that shift measured difficulty as expected.

  • Problem

    Difficulty measurement and prediction methods yield item scores but do not reveal the underlying factors that make questions difficult.

    Source: §1 · p. 1

  • Method

    The approach estimates difficulty with IRT, prompts an LLM to propose hypotheses from contrasting easy and hard questions, then validates and selects them on held-out questions.

    Source: §1 · p. 2

  • Results

    Across three reasoning datasets, the hypotheses predict unseen-question difficulty as well as or better than black-box predictors, improve those predictors as features, and support edits that shift measured difficulty as expected.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The discovered hypotheses can guide question editing to design and iterate on benchmarks.

    Source: §1 · p. 2

    On Winogrande, human annotators agree with IRT-estimated difficulty on 73.4% of item pairs, implying an R2 ceiling of approximately 0.45 under a bivariate-normal assumption.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

Difficulty is one of the most fundamental properties of a question: it determines whether the question can meaningfully discriminate between models of differing ability. Although a variety of methods can now estimate or predict difficulty automatically, they yield only a single descriptive number, with no account of the underlying factors that make a question difficult in the first place. In this work, we propose a data-driven approach that automatically generates and validates natural-language hypotheses explaining what makes one question harder than another. We first estimate each item's difficulty from the responses of a large pool of LLMs using Item Response Theory. We then sample contrasting sets of easy and hard questions and prompt an LLM to propose candidate explanations of the difference, which are subsequently validated and selected on held-out questions. Experimental results across three datasets spanning mathematical, logical, and commonsense reasoning show that our method produces interpretable and predictive hypotheses. On their own, they predict the difficulty of unseen questions competitively with, or better than, advanced black-box difficulty regressors; used as additional features, they further improve those regressors, implying that they discover difficulty signals that existing models fail to capture. Moreover, we demonstrate that editing questions according to a hypothesis can shift their measured difficulty in the expected direction, indicating that the discovered hypotheses are causally valid difficulty factors rather than post-hoc descriptions. Our approach thus turns a purely descriptive difficulty score into actionable statements.

1 INTRODUCTION

Prior approaches measure or predict question difficulty but provide limited insight into the factors that make questions hard. This work generates natural-language hypotheses from contrasting questions and finds that they can predict difficulty and guide question edits.

1 INTRODUCTION

  • IRT and other measurement methods provide difficulty scores without revealing the factors that make an item hard.Hand-specified features are difficult to scale, while neural and pretrained-LLM predictors offer little interpretability.Source: §1 · pp. 1–2
  • The method estimates difficulty with IRT, then generates and validates hypotheses from contrasting easy and hard questions.Difficulty is latent, continuous, and shaped by interacting factors, making it harder to explain than observable text properties.Source: §1 · p. 2
  • Across mathematical, logical, and commonsense reasoning benchmarks, hypotheses predict unseen-question difficulty competitively with or better than black-box predictors and improve them as added features.Editing questions to satisfy or violate a hypothesis shifts measured difficulty in the expected direction.Source: §1 · p. 2

2 METHOD

The method measures item difficulty from LLM responses, generates hypotheses from contrasting questions, and selects hypotheses based on held-out question performance. Sampling and prompting are designed to discover factors at different granularities and reduce redundancy.

2 METHOD

  • The pipeline estimates difficulty, generates candidate explanations from contrasting questions, and selects hypotheses that explain held-out questions.Figure 1 summarizes these stages.Source: §2 · p. 2, Figure 1 · p. 3

2.1 QUESTION DIFFICULTY ESTIMATION

  • The method fits a one-parameter logistic IRT model to LLM response data, jointly learning item difficulty and model ability.The response matrix records whether each model answered each question correctly; the fitted measure is described as invariant and noise-aware.Source: §2.1 · p. 2

2.2 HYPOTHESIS GENERATION

  • Generation and verification use disjoint, difficulty-stratified sets, while repeated sampling creates a pool of candidate hypotheses.Group-wise sampling weights bucket combinations and questions to favor larger difficulty gaps and discourage repeated selections; pair-wise sampling also down-weights previously selected questions and pairs.Source: §2.2 · p. 3
  • Pair prompts seek explanations for individual gaps, while group prompts seek broader hypotheses; combining them targets different granularities.The group form uses larger sets than the pair form.Source: §2.2 · p. 3

2.3 HYPOTHESIS SELECTION

  • The method groups similar candidates for LLM refinement, matches refined hypotheses to questions, then selects informative hypotheses using LASSO selection frequency on validation subsamples.Each of 100 subsamples contains 75% of the validation set, with the penalty chosen by five-fold cross-validation within each subsample.Source: §2.3 · p. 4

3 EXPERIMENTAL SETUP

The experiments use three benchmarks for mathematical, logical, and commonsense reasoning. They assess hypothesis prediction and question-editing effects, comparing generation methods and difficulty predictors.

3.1 DATASETS

  • The evaluation spans GSM8K, BBH-structured, and Winogrande, using response records from 5,000 LLMs for GSM8K and Winogrande and 3,811 for BBH-structured.The datasets cover multi-step arithmetic, logical reasoning, and pronoun-resolution commonsense reasoning, respectively.Source: §3.1 · p. 4

3.2 EVALUATION

  • Predictive evaluation measures in-sample adjusted R2 and out-of-sample difficulty prediction on a test set.For the out-of-sample setting, OLS is fit on the generation and verification sets and evaluated on the test set.Source: §3.2 · p. 4
  • Causal evaluation edits only the attribute described by a hypothesis, then measures difficulty-shift magnitude and the fraction of edits moving difficulty in the predicted direction.Only questions that can be meaningfully edited according to the hypothesis are included.Source: §3.2 · pp. 4–5

3.3 BASELINES

  • Hypothesis-generation baselines include NLPARAM, HYPOTHESAES, HYPOGENIC, and EDISA; difficulty-prediction comparisons include fine-tuned RoBERTa, frozen-embedding regressors, few-shot LLMs, and hypothesis-based predictors.HYPOGENIC is adapted by discretizing IRT difficulty into three buckets; the predictor set includes Gemini and Claude few-shot prompting.Source: §3.3 · p. 5

4 HYPOTHESIS EVALUATION

Across three datasets, hypotheses predict unseen-question difficulty competitively and add signals to existing predictors; combining sampling strategies generally improves discovery. Sampling results also show that contrast strength should depend on the dataset, with a mixed strategy the more reliable option.

4.1 MAIN RESULTS

  • Most hypotheses capture cognitive operations or information manipulation rather than surface text; 13/15 have SEP 95% CIs with lower bounds above zero.Except on WinoGrande, 9/10 hypotheses on the other two datasets reach at least a medium effect, and half reach a large effect.Source: §4.1 · pp. 5–6, Table 1 · p. 6
  • Our hypotheses achieve the best adjusted R² fit at most budgets across three datasets, while baselines yield only 5–10 effective hypotheses before fit declines.The authors attribute the continued discovery of informative hypotheses to repeatedly contrasting difficult and easy questions.Source: §4.1 · p. 6, Figure 2 · p. 7
  • Hypothesis features rank first on GSM8K and WinoGrande and second on BBH-structured for unseen-question difficulty prediction.All models perform poorly on WinoGrande, but the hypotheses still extract signal there.Source: §4.1 · pp. 6–7, Table 2 · p. 7
  • Adding hypotheses gives the largest R² gains across settings, including +0.11 over fine-tuned RoBERTa on GSM8K and +0.008 over RoBERTa-base on BBH-structured.On BBH-structured, the hypotheses still add +0.05 to the two frozen embeddings.Source: §4.1 · p. 7, Figure 3 · p. 8
  • On WinoGrande, where baselines sit around zero, hypotheses lift each baseline to R² between 0.08 and 0.10.All baselines except CLAUDE-SONNET-5 attain R² values around or below zero.Source: §4.1 · pp. 7–8

4.2 IMPACT OF SAMPLING STRATEGY

  • On GSM8K and BBH-structured, mixing group and pair prompts outperforms either alone in most cases, scales better with budget, and reaches a higher upper bound.Group-only converges faster on GSM8K; pair-only reaches comparable or better performance at larger budgets, while the two perform similarly on BBH.Source: Figure 4 · p. 8, §4.2 · p. 8
  • High contrast performs better on GSM8K, low contrast on BBH-structured; the mixed strategy stays close to the better single strategy in most cases.High and low contrast use bucket gaps of 7–9 and 4–6, respectively.Source: Figure 5 · p. 8, §4.2 · p. 8
  • A suitable sampling and prompting strategy reaches good performance within 200 questions, and mixed-strategy curves have not fully converged.The authors treat question count as a proxy for annotation budget and note potential to scale further.Source: §4.2 · p. 9

5 CAUSAL ANALYSIS

The authors test whether changing only the property described by a generated hypothesis shifts question difficulty in the expected direction. Edits generally produce aligned accuracy changes across both datasets and most tested models, though evaluation uses accuracy as a proxy and is not a controlled comparison against baselines.

Counterfactual question editing

  • For 50 test questions from GSM8K and BBH-structured, an LLM edits only the property described by an applicable hypothesis and verifies the rewrite.WinoGrande is excluded because its difficulty signal is described as inherently weak.Source: §5 · p. 9

Evaluation protocol

  • Because re-estimating difficulty across thousands of LLMs was prohibitively costly, evaluation uses accuracy from a subset of models under 30B as a proxy.Under the 1PL model, mean accuracy within a fixed respondent set is strictly monotone in difficulty; the section does not make a controlled comparison with baselines.Source: §5 · p. 9

Results

  • Increase edits lower mean accuracy by 15.27 percentage points on GSM8K and 23.88 on BBH-structured; decrease edits raise it by 32.18 and 28.52 points, respectively.Table 4 reports mean accuracies and accuracy shifts averaged over pairs and models.Source: §5 · p. 9, Table 4 · p. 10
  • Question-level alignment is 70.9% and 92.2% for increase and decrease edits on GSM8K, and 77.6% and 85.5% on BBH-structured.The aggregate changes therefore align directionally across most edited questions, not only a few large shifts.Source: §5 · pp. 9–10, Table 4 · p. 10
  • Across models, 14/17 GSM8K and 15/16 BBH-structured models lose accuracy after increase edits, while 16/17 and 16/16 gain accuracy after decrease edits.The paper notes that all three models with reversed increase-direction changes started below 15% accuracy on the corresponding original questions.Source: §5 · p. 10

6 CONCLUSION

The paper presents a data-driven method for generating natural-language hypotheses about question difficulty. Across three datasets, the hypotheses predict difficulty competitively with black-box predictors and guide edits that shift difficulty in the intended direction.

Conclusion

  • The method estimates difficulty from thousands of LLM responses with IRT, then generates, validates, and selects natural-language hypotheses from contrasting easy and hard questions.The conclusion frames difficulty as a quantity to explain rather than only a number to estimate.Source: §6 · p. 10
  • Across three datasets, the hypotheses predict unseen-question difficulty as well as or better than black-box predictors and further improve them as added features.The paper reports that the hypotheses capture difficulty signals not captured by semantic encoders.Source: §6 · p. 10
  • Editing questions according to the hypotheses shifts their difficulty in a controlled direction, supporting their status as genuine difficulty factors rather than post-hoc correlations.The authors suggest this could support benchmarks whose difficulty can be understood and designed.Source: §6 · p. 10

A RELATED WORK

Related work measures or predicts question difficulty, explains model-specific errors, and uses LLMs to describe shared patterns in grouped data. The paper adapts hypothesis generation to latent difficulty inferred from model responses, with prompts designed to produce reusable, testable explanations.

A RELATED WORK

  • Explanatory item-response models make difficulty a function of item covariates, but these features are typically specified by experts; this method discovers covariates from data.Text-only prediction has also progressed from hand-designed features to neural encoders and LLM-based estimates.Source: A RELATED WORK · p. 17
  • Slice-discovery methods explain regions where a particular model errs, so their findings may reflect that model’s weaknesses rather than item difficulty transferable across models.The paper notes their value for model debugging but limited insight into dataset curation and iteration.Source: A RELATED WORK · p. 17
  • Prior LLM hypothesis-generation work describes patterns in grouped data that are often directly observable; this paper instead targets latent, continuous difficulty inferred from model responses.Its hypotheses are scored on held-out items by the variance they explain individually and jointly.Source: A RELATED WORK · p. 17

B PROMPTS

  • The group prompt asks for distinct, specific, checkable hypotheses grounded in harder and easier question groups, favoring recurring, generalizable patterns over item-specific observations.It instructs the model to use IRT values as graded evidence and avoid padding the hypothesis list.Source: B PROMPTS · p. 18
  • The pair prompt presents five random harder/easier pairs with a minimum IRT gap and asks for reusable hypotheses supported by pair numbers, without copying item-specific details.It permits narrower reusable subtypes when supported and allows up to ten non-redundant hypotheses.Source: B PROMPTS · p. 19
  • The match judge scores whether each candidate property applies to a question, returning one binary score per property and zero when uncertain.The prompt requests a fixed-order JSON score list.Source: B PROMPTS · p. 19

C.1 DATASET STATISTICS

Table 5 summarizes the dataset splits, the number of models used for IRT difficulty estimation, and item-difficulty distributions.

C.1 DATASET STATISTICS

  • Table 5 summarizes dataset splits, the number of models used for IRT estimation, and the resulting item-difficulty distributions.Source: §C.1 · p. 20

C.2 HYPOTHESIS DISCOVERY

Hypothesis discovery combines group- and pair-based prompting to generate difficulty explanations, then uses stability selection to reduce the candidate pool. The section also specifies the implementations and budget matching used for comparison baselines.

C.2 HYPOTHESIS DISCOVERY

  • The method runs 30 group-based and 30 pair-based generation rounds, then uses Lasso stability selection and refinement to reduce the hypothesis pools.The candidate pools contain 376, 235, and 171 hypotheses for GSM8K, BBH-structured, and Winogrande; after deduplication and refinement, they contain 233, 127, and 146, respectively.Source: §C.2 · p. 20
  • Group prompts use 40 questions and pair prompts use five hard/easy pairs; sampling settings include difficulty-gap thresholds and reuse penalties.Group samples contain 20 hard and 20 easy questions; pair prompts use 10 questions, and the minimum IRT gap varies by dataset.Source: §C.2 · p. 20
  • HYPOGENIC is adapted to classify three equal-frequency difficulty levels, with predicted levels mapped to mean difficulty for regression scoring.Its comparison uses a 20-hypothesis bank and a fixed question budget matched to the proposed method.Source: §C.2 · p. 20
  • NLPARAM uses INSTRUCTOR-XL question embeddings and iteratively refines 20 predicate directions through regression and candidate replacement.The procedure runs 10 alternations per refinement and repeats refinement 10 times, replacing the least useful predicate with the best of five candidates.Source: §C.2 · pp. 20–21
  • HYPOTHESAES uses pooled Gemma-Scope-25 autoencoder features and selects nonzero features with a cross-validated Lasso.Features active on fewer than 20 training questions are discarded.Source: §C.2 · p. 21
  • EDISA clusters question-answer embeddings with standardized IRT difficulty, then asks an LLM to write a predicate for each inferred slice.At inference, difficulty is marginalized out; the method shows the five highest-posterior slice members and the slice’s mean difficulty.Source: §C.2 · p. 21

C.3 FINE-TUNED DIFFICULTY PREDICTORS

This section describes trained and frozen-representation difficulty predictors, prompted judges, and the procedure for generating and checking hypothesis-guided question edits.

C.3 FINE-TUNED DIFFICULTY PREDICTORS

  • RoBERTa is fine-tuned with a scalar regression head, while Gemma-3-4B and Qwen3-Embedding-4B representations feed Lasso regressors evaluated on the test pool.The RoBERTa model uses question and answer text, squared-error training, and reports means over five seeds.Source: §C.3 · p. 21

C.4 PROMPTED JUDGES

  • Prompted judges receive the same fixed context of 10 labelled demonstrations, sampled across five difficulty buckets, and predict each test item’s IRT difficulty.Both judges use three demonstration draws; Table 6 lists the sampled questions in increasing measured difficulty.Source: §C.4 · p. 21, Table 6 · p. 22

C.5 QUESTION EDITING

  • Question editing first determines whether a hypothesis applies, then rewrites the item and reference answer, with two models checking solvability and preservation of the original setting.Interventions judged inapplicable are skipped, and only candidates accepted by both verifiers enter evaluation.Source: §C.5 · p. 22

D MORE RESULTS

The section reports counterfactual evaluation of hypothesis-guided edits on GSM8K and BBH-structured, with outcomes organized by edit direction and hypothesis.

D MORE RESULTS

  • Table 7 evaluates edits on GSM8K and BBH-structured, where a negative accuracy change is intended for difficulty increases and a positive change for decreases.The evaluation includes 141 increase and 51 decrease edits on GSM8K, and 67 increase and 55 decrease edits on BBH-structured.Source: Table 7 · p. 23
  • Table 8 defines hypothesis-level rewritten-minus-original accuracy change and success rate from pooled correct/incorrect transitions.Success rate includes unchanged outcomes and is calculated differently for increase- and decrease-difficulty edits.Source: Table 8 · p. 24
  • Table 8 pools transitions across 17 models for GSM8K and 16 for BBH-structured, and identifies original outcomes as historical RouterEval evaluations.Accepted rewrites are counted per hypothesis; a dash denotes no accepted rewrites.Source: Table 8 · p. 24

E ALL GENERATED HYPOTHESES

This appendix presents generated and selected difficulty hypotheses across GSM8K, BBH-structured, and Winogrande, including feature inventories from several methods. The examples cover reasoning patterns that distinguish harder questions requiring implicit, multi-step, or counterintuitive reasoning from easier ones answerable through direct lexical, physical, or causal cues.

HypotheSAEs

  • HypotheSAEs examples include a BBH-structured feature about sorted order and Winogrande features labeled as lacking a coherent item-level property or keyed to a final blank-period token.The accompanying tables describe 20 explained features per dataset; matching is defined by SAE activation greater than zero.Source: Table 12 · p. 28, Table 13 · p. 29, E ALL GENERATED HYPOTHESES · pp. 30, 32

Edisa – GSM8K

  • Edisa’s GSM8K examples describe arithmetic word problems, including vehicle distance, rate, or time and item costs involving promotions or discounts.Other listed hypotheses characterize multi-step arithmetic and quantity tracking.Source: E ALL GENERATED HYPOTHESES · p. 37

Edisa – BBH-structured

  • Edisa’s BBH-structured examples span table-based extraction or counting and logical deductions about the order of fruits, golfers, vehicles, and birds.One highlighted hypothesis describes determining three golfers’ finishing order from relative statements.Source: E ALL GENERATED HYPOTHESES · p. 39

Edisa – Winogrande

  • Edisa’s Winogrande hypotheses characterize choosing between two options to fill a sentence blank, often resolving an ambiguous reference or coreference.Source: E ALL GENERATED HYPOTHESES · p. 41

Ours – GSM8K

  • Selected GSM8K hypotheses identify implicit +1 counting and calculating an unasked-for intermediate value as item properties.The intermediate-value hypothesis specifies dependent calculations in which one result feeds the next.Source: E ALL GENERATED HYPOTHESES · p. 43

E ALL GENERATED HYPOTHESES

  • Difficulty increases when answers depend on inverse, negative, or counterintuitive physical and causal relationships that override typical expectations.Hypotheses include inverse quantitative relations, negative constraints, and comparisons involving the lesser or less prominent entity.Source: E ALL GENERATED HYPOTHESES · pp. 46–47
  • Harder questions often require combining stated clues with implicit goals, consequences, or background and domain knowledge rather than relying on explicit causal markers.Some hypotheses also describe inferring unstated causes of negative outcomes or the logical consequences of actions.Source: E ALL GENERATED HYPOTHESES · p. 46
  • Reference resolution is harder when it requires identifying an implicit social role or assigning a pronoun to a non-agent, non-primary, or otherwise unexpected entity.The hypotheses also contrast direct physical actions with abstract states and describe cases requiring careful separation of interpersonal cause and experiencer roles.Source: E ALL GENERATED HYPOTHESES · pp. 46–47
  • Other hypotheses identify contrast, comparative degree, and constraints on difficulty as reasoning demands that can separate harder questions from easier ones.These include indirect consequences or concessions and inferring how having less money makes a budget harder to follow.Source: E ALL GENERATED HYPOTHESES · pp. 46–47
  • Easier questions often provide direct lexical associations, physical constraints, conventional event scripts, or explicit causal conjunctions that point to the answer.Examples include strong predicate–entity associations and causal words such as “because,” “since,” and “as.”Source: E ALL GENERATED HYPOTHESES · pp. 46–47

Related papers

Loading 2610.01627v1…