PaperlayerView exact revision on arXiv

Source-linked AI summary

Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring

Wonjeong Jo, Gyuwan Kim, Ming Shan Hee

arXiv:2610.22187v1cs.CLcs.AIcs.NAmath.NA

TL;DR

Whether LLM-generated scores remain comparable across rating-scale designs is understudied, despite their use to measure social and psychological constructs. The paper tests sensitivity across scale designs, tasks, prompting strategies, and models, finding score and category changes when scale design changes.

  • Problem

    It is understudied whether LLM-generated scores remain comparable across rating scales, although studies use different designs to measure the same constructs.

    Source: §1 · p. 1

  • Method

    The study defines numerical and semantic scale sensitivity and tests format, range, and sign across tasks, prompting strategies, and models.

    Source: §1 · p. 2

  • Results

    Changing a scale-design element shifts normalized scores by about 0.10 and changes roughly 20% of texts’ categories; toxicity is most sensitive, and rubric-based prompting does not reduce sensitivity.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Researchers should test at least one alternative scale, match scales to construct structure, and assess agreement with suitable human labels when available.

    Source: §5 · pp. 8–9

    The findings’ generalizability is limited by the tasks and datasets examined; extension to broader cultural contexts, other languages, and additional tasks remains to be tested.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs. However, studies use different rating scales, and whether their scores remain comparable across scale designs is understudied. To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design. We distinguish numerical sensitivity, changes in normalized scores, from semantic sensitivity, changes in score interpretation. To assess both, our experimental design crosses 2 formats (continuous vs. ordinal), 4 ranges (3, 5, 11, and 21 points), and 2 signs (signed vs. unsigned) across 3 tasks (political leaning, sentiment, and toxicity), 2 prompting strategies (zero-shot and rubric-based), and 5 LLMs. We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts. Sensitivity is greatest for toxicity, the only unipolar construct examined. Also, rubric-based prompting does not reduce sensitivity compared with zero-shot prompting. We provide practical guidance for designing more reliable LLM-based text scoring and highlight the need for robustness checks across alternative scales.

1 Introduction

The paper addresses limited evidence about whether LLM-generated scores remain comparable across rating-scale designs. It defines and measures numerical and semantic scale sensitivity, finds both in experiments, and offers guidance for more reliable scoring.

1 Introduction

  • Different scale designs have been used to measure the same constructs, while the consistency of LLM outputs across those designs has received limited attention.The paper identifies cross-scale comparability as an important question for the reliability and validity of LLM-based measurement.Source: §1 · p. 1
  • Scale sensitivity is variation in LLM-generated scalar outputs across rating-scale designs, with numerical sensitivity tracking score changes and semantic sensitivity tracking interpretation changes.The paper examines format, range, and sign across tasks, prompting strategies, and models.Source: §1 · p. 2
  • Changing one scale-design element shifts normalized scores by about 0.10 and changes the category of roughly one in five texts; toxicity is most sensitive, and rubrics do not reduce sensitivity.The reported effects concern scale format, range, and sign across the examined scoring tasks.Source: §1 · p. 2
  • The authors recommend robustness checks across alternative scales, construct-aligned scale choices, and agreement checks against suitable human labels when available.They connect scale sensitivity to the comparability, replicability, and validity of LLM-scored measurements.Source: §1 · p. 2

2 Related Work

Prior measurement research finds that scale format, range, and sign can affect ratings, though evidence does not establish one universally preferable design. The paper asks whether scale design similarly affects LLM scores and whether task, prompting strategy, or model moderates that sensitivity.

2.1 Rating Scale Design in Measurement

  • Ordinal and continuous scales are both used for scalar constructs; continuous scales preserve finer-grained information, but neither format consistently yields higher reliability or validity.Prior comparisons find that the two formats can produce different ratings of the same stimuli.Source: §2.1 · p. 2
  • Evidence on scale range is inconclusive: more points may improve distributional properties and internal consistency, while overly fine-grained scales may reduce consistency in some contexts.Scale sign can also shift human responses and alter whether a low endpoint is read as absence or the opposite attribute.Source: §2.1 · pp. 2–3
  • Whether scale design changes LLM scores remains unclear, so the study holds prompts fixed within each strategy and varies the specified scale to isolate scale-design differences.It quantifies these differences as numerical and semantic sensitivity.Source: §2.1 · p. 3

2.2 Variances across Task, Prompting Strategy, and Model

  • The study compares political leaning, sentiment, and toxicity, which differ conceptually; political leaning and sentiment are bipolar, whereas toxicity is unipolar.The paper asks whether task moderates numerical and semantic sensitivity.Source: §2.2 · p. 3
  • Zero-shot prompting requests a direct rating, whereas rubric-based prompting supplies coding guidelines; the paper tests whether rubrics stabilize scores across scales.The stated hypothesis is that rubrics could reduce sensitivity if it stems from underspecified evaluation criteria, but not if it stems from mapping judgments onto numerical scales.Source: §2.2 · p. 3
  • The study examines model choice as a potential moderator and asks how much sensitivity variation is explained by scale design, task, prompting strategy, and model.Source: §2.2 · p. 3

3 Experimental Design

The study tests how rating-scale design affects LLM scores across three human-labeled tasks, five models, and two prompting strategies. It compares scores under scale conditions that differ in one design element, measuring both normalized-score movement and category changes.

3.1 Dataset

  • Three human-labeled datasets cover political leaning, sentiment, and toxicity.One dataset is used for each task.Source: §3.1 · p. 4

3.2 Rating Scale Conditions

  • The 16 scale conditions cross continuous or ordinal format, four ranges, and signed or unsigned scales.The ranges are 3, 5, 11, and 21 points.Source: §3.2 · p. 4, Table 2 · p. 4

3.3 Models

  • Five models are compared: GPT 5.6 Luna, Gemini 3.5 Flash Lite, DeepSeek V4 Flash, Qwen3.8 27B, and GLM 5.3 Flash.The three open models are self-hosted; the two proprietary models are accessed through provider APIs.Source: §3.3 · p. 4

3.4 Prompts

  • Zero-shot and rubric-based prompts share core scoring instructions, while rubric-based prompts add task-specific dimensions and ask for a holistic final score.Analyses use only the final score from each strategy to compare the same output type.Source: §3.4 · p. 4, §3.4.1 · p. 4, §3.4.2 · p. 4

3.5 Evaluation Framework

  • Raw scores are normalized to [0, 1], and minimal pairs isolate one scale element by holding the other two fixed.The design yields 40 pairs: eight for format, 24 for range, and eight for sign.Source: §3.5.1 · p. 5, §3.5.2 · p. 5

3.5.3 Numerical Scale Sensitivity

  • Numerical sensitivity is the mean absolute difference between normalized scores for the same items under a minimal pair.SSMAE ranges from 0 to 1; a value of 0.10 represents movement by one tenth of the full scale on average.Source: §3.5.3 · p. 5

3.5.4 Semantic Scale Sensitivity

  • Semantic sensitivity is the share of items that change category, using equal-width low, middle, and high bins with task-specific interpretations.The bins map to negative, neutral, and positive sentiment; left, center, and right political leaning; and not, moderately, and extremely toxic content.Source: §3.5.4 · p. 5

3.5.5 Analytical Framework

  • The analysis computes 1,200 estimates per metric and uses weighted least squares to model design-element, task, prompt, and model effects and their interactions.Estimates are averaged by design element with equal weight across tasks, prompts, models, and pairs; uncertainty uses 2,000 item-bootstrap resamples, and the framework also decomposes explained variation.Source: §3.5.5 · pp. 5–6

4 Results

Changing rating-scale design produces both normalized-score shifts and category changes, with sensitivity varying by scale interactions, task, prompting strategy, and model. Task explains more semantic sensitivity than scale design, while supplemental analyses find small reductions associated with higher reasoning effort.

4.1 Effects of Rating Scale Design

  • Changing format shifts normalized scores by 0.098 and changes 19.2% of texts’ categories; changing sign shifts scores by 0.085 and changes 20.3%.Across design elements, sensitivity is approximately one tenth of the normalized scale and affects about one in five texts; sign changes have the smallest numerical but largest semantic sensitivity.Source: §4.1 · p. 6
  • Scale interactions alter sensitivity: format shifts scores by 0.146 on 3-point scales versus 0.078 on 21-point scales, while category changes remain 18.2%–20.1%.Ordinal scales amplify range changes (0.128 versus 0.068 on continuous scales; 21.5% versus 14.9% category changes), while continuous scales amplify sign-related category changes (24.4% versus 16.3%).Source: §4.1 · pp. 6–7, Figure 3 · p. 6

4.2 Moderation by Task, Prompting Strategy, and Model

  • Toxicity is most sensitive, with up to 40% of texts changing category, compared with 13–15% for political leaning and 8–10% for sentiment.Rubric prompting remains within 0.007 of zero-shot sensitivity for every design element; model choice affects sensitivity more than prompting but much less than task.Source: §4.2 · pp. 7–8

4.3 Relative Importance of Each Design Choice

  • Scale design explains 40.3% of numerical sensitivity but 8.9% of semantic sensitivity, while task explains 29.7% and 58.9%, respectively.Model and prompting strategy explain little variation; the scale-design-by-task interaction accounts for 16.0% of semantic variation.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 Additional Analyses

  • In supplemental analyses, higher reasoning effort is associated with small reductions in scale sensitivity.The authors also assess agreement between LLM scores and the human labels available in each dataset.Source: §4.4 · p. 8

5 Conclusion

Changing rating-scale design shifts LLM scores and category assignments, with toxicity especially sensitive. The authors recommend robustness checks across scales, construct-aligned designs, and human-label validation when available.

5 Conclusion

  • About 0.10 average normalized-score shifts and category changes for roughly 20% of texts accompany changes to scale format, range, or sign across three tasks and five models.Toxicity, the only unipolar construct examined, is most sensitive; rubric-based prompting does not reduce sensitivity relative to zero-shot prompting, and model choice has a modest effect.Source: §5 · p. 8
  • Researchers should test whether main findings hold under at least one alternative rating scale to support replicability and reliability.When suitable human labels are available, compare scores across scales with labels from trained annotators using goal-aligned coding guidelines; agreement provides evidence of validity.Source: §5 · pp. 8–9
  • Unipolar constructs such as toxicity should use unsigned scales whose lowest endpoint represents absence of the attribute.On signed scales, models appeared to treat 0 as “not toxic,” assigning substantially more texts to the high-toxicity category than on unsigned scales.Source: §5 · pp. 8–9
  • When suitable reference labels are unavailable, robustness checks across alternative scales become especially important for assessing finding stability.The authors also recommend choosing scales that match the construct's structure.Source: §5 · p. 9

Limitations

The findings are limited by the tasks and datasets examined and by the two prompting approaches compared.

Limitations

  • The findings’ generalizability is limited by the tasks and datasets examined.The authors invite tests in broader cultural contexts, languages other than English, and additional tasks.Source: Limitations · p. 9
  • The prompting comparison covers only zero-shot and rubric-based approaches.Few-shot, chain-of-thought, and chain-of-verification prompting were excluded because demonstrations or intermediate reasoning steps could add confounds.Source: Limitations · p. 9
  • The authors leave evaluation of few-shot, chain-of-thought, and chain-of-verification strategies to future work.They note that chain-of-thought has shown limited gains on subjective tasks, while chain-of-verification is designed for factual claims and may be less applicable where constructs lack verifiable ground truth.Source: Limitations · p. 9

Statement on the Use of AI Assistants

The authors disclose using generative AI for proofreading and grammar checking, and GitHub Copilot for coding assistance.

Statement on the Use of AI Assistants

  • Generative AI was used for proofreading and grammar checking under instructions to avoid verbosity and extensive paraphrasing.The prompt asked the tool to edit grammar errors without paraphrasing much from the original sentences.Source: Statement on the Use of AI Assistants · p. 9
  • GitHub Copilot, using GPT-5-mini where available, assisted with coding, while human authors reviewed suggestions and made final edits.Source: Statement on the Use of AI Assistants · p. 9

A Dataset Details

The appendix describes three public datasets and the prompt designs used to score political leaning, sentiment, and toxicity. It outlines zero-shot and rubric-based prompts and how scale instructions specify format, range, and labels.

A Dataset Details

A.1 Political Leaning

  • The political-leaning dataset contains 1,700 MBIC sentences labeled for media bias and opinion, with left, center, and right labels indicating outlet leaning.Ten crowdworkers annotated each sentence; the outlet labels were based on AllSides and Ad Fontes Media ratings.Source: §A.1 · p. 12

A.2 Sentiment

  • The sentiment test set comprises 5,000 sampled movie reviews from VADER’s dataset of 10,605 reviews rated on a continuous −4 to 4 scale.Twenty independent human raters labeled the original reviews.Source: §A.2 · p. 12

A.3 Toxicity

  • The toxicity test set comprises 5,000 sampled comments from Jigsaw’s Unintended Bias in Toxicity Classification dataset.The training set contains 1,804,874 Civil Comments, with aggregated human toxicity scores from 0 to 1 and ratings from multiple annotators.Source: §A.3 · p. 12

B Prompts

  • The appendix provides prompts for the scoring tasks and points to full prompts for all tasks and scale conditions.Political leaning is used as the example task.Source: B Prompts · p. 12

B.1 Zero-Shot Prompt

  • The zero-shot prompt structure is illustrated with political leaning; sentiment and toxicity prompts adapt task descriptions, text types, and scale guidance.The toxicity instructions reflect its unipolar nature, assigning the lowest score to no toxicity and greater distance from that endpoint to greater toxicity.Source: §B.1 · p. 12, Figure 6 · p. 12

B.2 Rubric-Based Prompt

  • The rubric-based prompt extends the zero-shot prompt with a marking rubric, added scoring instructions, and separate output keys for rubric dimensions.The authors define each construct and its endpoints, then identify dimensions from established research, codebooks, and annotation frameworks; task-specific dimensions are adapted for political leaning, sentiment, and toxicity.Source: §B.2 · pp. 12–13

B.3 Scale Instructions

  • Scale instructions state the scale’s format, range, and labels; the appendix lists labels for each task and scale length.For example, the unsigned five-point political-leaning scale labels points from far left through neutral to far right.Source: §B.3 · p. 13

C LLM Parameter Settings

The section describes inference settings for the five models, using low reasoning for the main analyses and provider- or model-recommended settings where available.

C LLM Parameter Settings

  • The three open models used Hugging Face–recommended sampling settings when available, with unspecified parameters left at endpoint defaults.For GLM 5.3 Flash, the card had no general recommendation, so the study used its evaluation values and enabled clear_thinking.Source: C LLM Parameter Settings · p. 13
  • The main analyses used the lowest reasoning level for each open model; higher levels are reported in Appendix E.1.Source: C LLM Parameter Settings · p. 13
  • GPT 5.6 Luna and Gemini 3.5 Flash Lite ran through their providers’ batch APIs at the lowest reasoning levels that kept reasoning enabled.The study omitted sampling parameters because provider guidance said they were unsupported, ignored, or should be omitted.Source: C LLM Parameter Settings · p. 13

D LLM Scoring: Output Description

LLM score distributions differed across tasks and scales, while human labels served only as contextual references. Higher reasoning slightly reduced some sensitivity measures but did not remove scale sensitivity.

D LLM Scoring: Output Description

  • Human labels are contextual references rather than ground truth because annotation instructions may differ, political labels are categorical, and the study focuses on cross-scale consistency.Source: D LLM Scoring: Output Description · p. 14
  • Political leaning scores clustered in the middle, sentiment scores were polarized, and toxicity category shares varied strongly with scale.Political leaning placed 44%–67% of texts in the middle; sentiment placed 39%–48% low and 32%–44% high; continuous signed toxicity scales placed 47%–52% high versus 12%–22% on continuous unsigned scales.Source: D LLM Scoring: Output Description · pp. 14–15, Figure 8 · p. 15

E.1 Reasoning Effort

  • More reasoning slightly reduced numerical sensitivity across format, range, and sign, while semantic sensitivity fell for format and sign but not range.Numerical SSMAE fell by 0.009 for format, 0.006 for range, and 0.016 for sign; semantic sensitivity fell by 1.1 percentage points for format and 2.6 points for sign.Source: §E.1 · p. 15
  • At the highest reasoning level, sensitivity remains close to the Section 4 values; the reported change rises from 19.1% to 22.0%, so more reasoning does not remove scale sensitivity.Source: §E.1 · p. 15

E.2 Agreement with Human Labels

Agreement with human labels varied substantially by task and rating scale, with the appropriate metric depending on whether labels were continuous or categorical.

E.2 Agreement with Human Labels

  • The study used Spearman’s rank correlation for continuous sentiment and toxicity labels, and macro-F1 for categorical political leaning labels.Political scores were mapped to the three SSswitch categories before macro-F1 was calculated.Source: §E.2 · p. 16
  • Agreement varied substantially by task: Spearman’s correlation was 0.87–0.95 for sentiment and 0.24–0.38 for toxicity, while political leaning macro-F1 was 0.34–0.65.The authors note that toxicity labels capture the proportion of annotators marking toxicity, while political labels describe outlets rather than necessarily the sentences.Source: §E.2 · p. 16
  • Political leaning macro-F1 on ordinal scales fell from 0.58 with 3 points to 0.48 with 21 points, and the scale-wise range within model-prompt combinations was 0.09–0.23.Source: §E.2 · p. 16
  • Rubric-based prompting yielded lower political leaning macro-F1 than zero-shot prompting for every model and scale, averaging 0.51 versus 0.56.For sentiment and toxicity, the average correlation change between prompting strategies was less than 0.01.Source: §E.2 · p. 16

F Results Details

This section details the weighted least squares analysis and variance decomposition used to report the paper’s scale-sensitivity results.

F Results Details

  • Weighted least squares estimates weight sensitivity pairs so format, range, and sign receive equal total weight.Table 6 reports interaction coefficients comparing how sign or range sensitivity differs from format at moderator levels relative to reference levels.Source: F Results Details · p. 16, Table 6 · p. 17
  • The analyses used NumPy for weighted least squares, SciPy for Spearman correlations, and Matplotlib for figures.The reported software versions were Python 3.9.6, NumPy 2.0.2, pandas 2.3.3, SciPy 1.13.1, and Matplotlib 3.9.4.Source: F Results Details · p. 16
  • The variance decomposition covers all 40 minimal pairs and partitions the estimates into non-overlapping shares that sum to 100% for each metric.Table 7 gives the full decomposition illustrated in Figure 5; subcomponents may not sum exactly to totals because of rounding.Source: F Results Details · p. 16, Table 7 · p. 19

Related papers

Loading 2610.22187v1…