PaperlayerView exact revision on arXiv

Source-linked AI summary

A Contextual Measure of Ambiguity in Prescriptive Text

Sally Dong, Luke Thorburn, Andrew Konya, Michiel Bakker

arXiv:2610.35461v1cs.CL

TL;DR

Prescriptive text can be both useful and harmful when ambiguous, yet existing measures often assess how text is perceived rather than how it applies in context. The paper measures ambiguity through disagreement among raters applying policies to cases, with LLMs simulating cases and raters. Its preliminary analyses show correspondence with human disagreement and no broader support for ambiguous statements in the public deliberations examined.

  • Problem

    Existing ambiguity measures often assess text in isolation, while a general, principled way to measure how prescriptive texts are interpreted in context is lacking.

    Source: §1 · p. 1

  • Method

    DELTA averages disagreement among raters about whether policies apply to cases, and LLM-DELTA uses simulated cases and raters to scale this evaluation.

    Source: §1 · p. 1, §3.1 · p. 4

  • Results

    LLM-DELTA scores showed substantial positive correlation with human-DELTA in one domain, while ambiguous statements did not attract broader support in the public deliberations examined.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In the public deliberation settings examined, ambiguity did not reliably produce common ground.

    Source: §6 · p. 10

    LLM-DELTA's human-validation correlation is moderate and established in one domain with 20 policies; scores also cannot be compared directly across different case sets or rater distributions.

    Source: §7 · p. 10

Abstract

from arXiv · show

Ambiguity in prescriptive text such as laws, contracts, rules, policies, or recommendations can be both desirable (e.g., if it enables a law to adapt to changing context, such as new technologies) and undesirable (e.g., if it leads to arbitrary discretion or conflict over the law's interpretation). To help navigate these trade-offs, it would be valuable to have methods for quantifying ambiguity that are predictive of such outcomes, but existing measures capture how text is perceived in isolation rather than how it is interpreted in context. In this paper, we propose quantifying ambiguity as the degree of disagreement among a set of stakeholders ("raters") on how alternative prescriptive texts ("policies") apply in each of a set of relevant scenarios ("cases"). Using two policy domains, we present preliminary analysis suggesting that, even when using simulated raters, this approach (i) correlates with real human disagreement, and (ii) is robust to spurious text features. We then apply the measure to existing deliberation datasets to describe the association between policy ambiguity and support for those policies.

1 Introduction

Ambiguity in prescriptive text can impede understanding and enable selective enforcement, but flexibility can also accommodate unforeseen circumstances and help defer implementation disputes. DELTA measures ambiguity through disagreement over how policies apply to cases; preliminary analyses validate the approach and find no broader support for ambiguous statements in the public deliberations examined.

1 Introduction

  • Ambiguity can hinder citizens' understanding and give officials greater discretion, yet interpretive flexibility can accommodate unforeseen change and preserve decision-maker discretion.In political settings, ambiguity can also defer implementation disagreements among actors who agree on broad goals.Source: §1 · p. 1
  • A general, principled operationalization remains lacking, while existing clarity rubrics must be engineered for each domain.The paper notes prior evidence that a hand-built clarity rubric predicted which prediction-market contracts later ended up in dispute.Source: §1 · p. 1
  • DELTA measures ambiguity as disagreement among raters about how policies apply to cases, averaged across cases.The measure is relative to the chosen cases and rater population.Source: Figure 1 · p. 2
  • LLM-DELTA uses LLMs to generate challenging cases, construct rater personas, and simulate judgments as a scalable approximation.The framework also allows cases and raters to be human-provided.Source: Figure 1 · p. 2, §1 · p. 2
  • In a single domain, LLM-DELTA scores showed substantial positive correlation with human-DELTA, while the adversarial dataset distinction was recovered by DELTA but not direct LLM-as-a-judge ratings.The distinction was between consensus-inducing and disagreement-inducing policies.Source: §1 · p. 2
  • Across two public deliberation datasets, more ambiguous statements did not attract broader support and appeared to soften strong agreement rather than convert opposition.This contrasts with the intuition that ambiguity facilitates compromise.Source: §1 · p. 2

2 Related work

Prior work measures ambiguity through text labels or ratings, disagreement among interpreters, and analysis of rules or principles applied by different executors. The section positions DELTA within these strands and emphasizes that relatively few measures are tested against disagreements arising in practice.

2 Related work

  • Text-focused measures label texts as ambiguous or specific, or predict human ratings such as Likert-scale ambiguity and concreteness scores.The literature also varies in its use of terms such as “vagueness,” “specificity,” and “concreteness.”Source: §2 · p. 3
  • A second line treats disagreement between interpreters as part of ambiguity, with NLI and Judgment Sieve work distinguishing uncertainty within an individual from divergence across people.Matched interventions—adding context or deliberation—reduced the corresponding forms of uncertainty or disagreement, whereas uniform interventions tended to trade one off against the other.Source: §2 · p. 3
  • Work on AI rules measures variation in compliance judgments across interpreters and finds that executors can disagree when applying principles reconstructed from preference data.He et al. (2026) report mechanisms for reducing rule ambiguity; Clifford et al. (2026) report 73% agreement among executors of the same constitution.Source: §2 · p. 3
  • Most ambiguity measures are validated against judgments elicited for research, while Hall and Paschal (2026) report that rubric-based scores predicted later prediction-market disputes with roughly 0.75 out-of-sample AUC.Such rubrics require domain-specific engineering, and a general, principled operationalization remains absent.Source: §2 · p. 3
  • The automated pipeline draws on prior work in case generation, persona generation, case-based reasoning, and prediction of individual judgments, without aiming to advance those tasks' state of the art.The cited work spans synthetic cases, generated personas, and individual annotation or political judgments.Source: §2 · p. 3

3 Formalization

DELTA measures policy ambiguity through disagreement over whether concrete cases are consistent with a policy. Its score combines uncertainty within individual judgments and disagreement across raters, using generated cases and personas to make evaluation scalable.

3.1 The DELTA score

  • DELTA defines ambiguity as average disagreement across cases about whether raters consider each case consistent with, permitted by, or compatible with a policy.Scores range from 0 for perfect consensus to higher values for increasing division among raters.Source: §3.1 · p. 4, §3 · p. 4
  • DELTA scores are relative to the chosen cases and stakeholder distribution, so comparisons are meaningful only within a shared evaluation setup.Cases should be realistic and debate-inducing, while raters should be diverse and broadly representative of domain stakeholders.Source: §3.1 · p. 4
  • To capture uncertainty about whether a policy permits or prohibits a tangential case, DELTA averages the positive-framing answer with the complement of the negative-framing answer.This yields a revealed judgment interpreted as the probability that a rater would issue a positive verdict.Source: §3.1 · p. 4
  • The score combines intra-rater indeterminacy and inter-rater disagreement, treating both as symptoms of a policy failing to determine its consequences.Unanimous indeterminate judgments yield the maximum score through the intra-rater term alone.Source: §3.1 · pp. 4–5

3.2 Approximation via language models

  • LLMs generate challenging cases by contrasting policy pairs, aiming to expose nuanced differences that may prompt disagreement in interpretation.Generated batches are sampled to retain contextual variety while avoiding redundancy in the underlying cause of disagreement.Source: §3.2 · p. 5
  • To broaden simulated viewpoints, LLMs generate opposed stakeholder identities and expand each with reflections that anchor subsequent judgments in stated values and concerns.The judgments use both positive and negative framings of the question.Source: §3.2 · p. 5

3.3 Score convergence

  • Because repeated prompting yields diminishing gains in semantic diversity, empirical score convergence can guide when to stop generating cases and raters.The convergence check compares rankings from random subsets with rankings from the full evaluation design.Source: §3.3 · pp. 5–6
  • Subsampling correlations rise quickly toward the full-design rankings, supporting a practical stopping rule of typically 30 raters and at most 50–60 cases.This diagnostic establishes rank stability within a fixed generated pool, not convergence to the broader distribution of cases and personas.Source: §3.3 · p. 6

4 Methods

The methods test whether LLM-DELTA tracks human disagreement, resists adversarial text features, and relates ambiguity to consensus in deliberation datasets. They use four datasets and compare DELTA with lexical and direct LLM-judgment baselines.

4.1 Hypotheses

  • The authors hypothesize that LLM-DELTA will better correlate with human-DELTA than lexical and LLM-as-a-judge baselines, and prove more robust to adversarial examples.They compare rank correlations because DELTA measures relative rather than absolute ambiguity.Source: §4.1 · p. 6
  • The substantive hypothesis is that greater ambiguity will associate with broader consensus or bridging agreement by postponing disagreement to policy application.The proposed test looks for a positive relationship between ambiguity and consensus measures.Source: §4.1 · p. 6

4.2 Datasets

  • The four datasets cover university speech policies, an adversarial policy setup, public Pol.is discussions, and collective statements from the Habermas deliberation experiment.Pol.is statements are treated as directional policy proposals whose supported and ruled-out concrete actions can be compared.Source: §4.2 · pp. 6–7
  • The university-speech benchmark pairs 20 alternate policies with 20 paraphrased campus controversies, while the adversarial dataset scales a referendum-based setup to 20 policies, 32 cases, and 30 personas.The university policies span platitudes to a policy modeled on the legal test for incitement.Source: §4.2 · p. 6
  • The Habermas dataset aggregates collective statements across groups into eight topic-specific sets, each containing approximately 70 entries.Participants rated collective statements after deliberating in groups of four or five on one of eight UK political topics.Source: §4.2 · p. 7

4.3 Baselines

  • The human-DELTA benchmark uses politically balanced U.S. adults, with 30 raters per policy evenly distributed among Republicans, Independents, and Democrats.Each participant rated 20 cases against two policies; human raters did not evaluate all policies and cases as in the LLM-DELTA setup.Source: §4.3 · p. 7
  • The lexical baseline negates average dictionary concreteness scores so higher values represent greater ambiguity.It uses human-annotated 1–5 concreteness scores for dictionary words appearing consistently in more than 85% of each policy text.Source: §4.3 · p. 7
  • Direct LLM-as-a-judge baselines rate policy ambiguity on a 1–7 scale using either dictionary-style wording or a prompt about diverse viewpoints and plausible scenarios.The second prompt is designed to mirror DELTA’s scenario-based mechanism.Source: §4.3 · p. 7

5 Results

Across the four Pol.is datasets, ambiguity shows no stable positive association with bridging, and greater ambiguity is associated with less strong agreement rather than broader support. DELTA aligns positively with human judgments and recovers the intended distinction in an adversarial test where direct LLM ratings reverse its ranking.

5.1 Human-DELTA benchmark

  • LLM-DELTA and human-DELTA scores are positively associated, suggesting the simulated procedure captures a form of interpretive disagreement found in human judgments.Figure 4 compares raw scores and policy rankings; the human raters were balanced across three political groups, unlike the unspecifically constructed simulated personas.Source: Figure 4 · p. 8, §5.1 · p. 8
  • Dictionary-based ambiguity has little relationship with human-DELTA, while direct LLM ambiguity ratings remain weakly associated and abstract disagreement ratings perform poorly.The authors suggest direct ratings may respond more to apparent vagueness or rhetorical style than to disagreement about policy application; concrete cases and diverse judgments provide scaffolding absent from abstract prompting.Source: §5.1 · p. 8

5.2 Adversarial robustness

  • In the adversarial design, DELTA ranks all ten consensus-inducing policies as the ten least ambiguous.The test scales a minimal consensus-versus-disagreement construction to 20 policies, with two rater types and four case types.Source: §5.2 · p. 8
  • Both LLM-as-a-judge baselines rank adversarial policies nearly opposite to DELTA, failing to track realized disagreement despite prompts about disagreement.The setup deliberately separates apparent ambiguity from disagreement produced when policies are applied to cases.Source: §5.2 · pp. 8–9, Figure 5 · p. 8

5.3 Relationship with bridging

  • Bridging is measured as the lower of the two opinion groups’ agreement rates, so high scores require shared endorsement.The measure is conservative and distinguishes shared support from common rejection.Source: §5.3 · p. 9
  • Four Pol.is datasets show no stable positive association between ambiguity and bridging across tested thresholds.At the 20% engagement threshold, the association is not significant in any dataset.Source: §5.3 · p. 9, Figure 6 · p. 9
  • One interpretation is that bridging may depend more on concrete solutions both groups can endorse than on interpretive latitude.This is offered as an interpretation of the absent positive association, not as a demonstrated causal explanation.Source: §5.3 · p. 9
  • As DELTA increases, strong agreement falls, while slight agreement and neutral responses rise and disagreement changes only modestly.The pattern suggests ambiguity softens initially supportive participants’ enthusiasm without producing a bridging effect.Source: §5.3 · p. 10

6 Conclusion

The paper defines ambiguity in prescriptive text as divergence in interpretations of consequences and operationalizes it through explicit cases and stakeholder perspectives. In the public deliberation settings examined, ambiguity did not reliably produce common ground.

6 Conclusion

  • DELTA operationalizes ambiguity as divergence in interpretations of consequences, evaluated across explicit cases and stakeholder perspectives rather than lexical properties alone.The authors conclude that ambiguity did not reliably produce common ground in the public deliberation settings they examined.Source: §6 · p. 10

7 Limitations

The authors identify limits in LLM-rater validity, the relative nature of DELTA scores, the scope of human validation, and the adversarial and bridging analyses. They also caution against treating scores as objective measures or simulated raters as substitutes for stakeholder participation.

7 Limitations

  • LLM-simulated raters may inherit model biases, and their agreement with human-DELTA was moderate (Spearman ρ = 0.48) in one domain and 20 policies.Whether simulations reproduce interpretive disagreement in other domains, non-English settings, or for views misrepresented in training data remains open.Source: §7 · p. 10
  • DELTA scores are relative to a shared pool of cases and raters, so differing evaluation setups do not support direct score comparisons.Case and persona choices encode assumptions about which scenarios and viewpoints count; generated cases may miss ambiguity emerging across more policies or unanticipated cases.Source: §7 · p. 10
  • Human validation covered one U.S. university-speech context and one viewpoint-diversity axis, and each human rated only two policies rather than all 20.Confidence intervals hold LLM-DELTA scores fixed, and Pol.is and Habermas were not benchmarked against human-DELTA.Source: §7 · p. 10
  • The adversarial robustness result tests a dataset designed to expose LLM-as-a-judge weaknesses, not robustness against attacks designed for LLM-DELTA.The authors describe such adversarial constructions as plausible and worth investigating.Source: §7 · p. 10
  • The bridging analysis is observational and cannot distinguish causal effects from selection effects or features of online deliberation.It also does not account for possible confounds such as statement length, extremist rhetoric, or LLM-translation artifacts.Source: §7 · pp. 10–11
  • DELTA scores depend on evaluation design, so treating them as objective measures of policy quality or legitimacy risks misuse.The authors also caution that simulated raters could lend a veneer of consultation without actual stakeholder participation; they frame LLM-DELTA as support for human judgment, not a substitute for democratic participation.Source: §7 · p. 11

A.4 Score convergence

On the campus-speech dataset, LLM-DELTA rankings stabilized before the full evaluation design was reached. The convergence diagnostic compares scores from random subsets of cases and raters with scores from the full design.

A.4 Score convergence

  • 0.87 mean correlation was reached with 10 cases and 10 raters, and correlation exceeded 0.9 from roughly 15 cases and 8 raters.These correlations compare subsampled rankings with the full-design reference on the campus-speech dataset.Source: Figure A.5 · p. 16, §A.4 · p. 16
  • The reference design used 20 manually curated cases and 30 LLM raters on the campus-speech dataset.Each cell reports average Spearman correlation over 100 random subsamples.Source: Figure A.5 · p. 16
  • The diagnostic compares rankings computed from random subsets of i cases and j raters against scores from the full design.The reported metric is average Spearman correlation.Source: Figure A.5 · p. 16

B Procedure for generating adversarial examples

The adversarial-example procedure contrasts ambiguity perceived from policy wording alone with disagreement over how policies apply to cases. It constructs policies and cases around consensus and conflict dimensions to separate those assessments.

B Procedure for generating adversarial examples

  • The procedure targets a contrast between surface ambiguity judged from text alone and operational ambiguity measured as disagreement when policies are applied to cases.The adversarial examples are designed so these two conceptions yield different policy assessments.Source: B Procedure for generating adversarial… · p. 16

B.1 Overview

  • Each example pairs two policies, four cases, and two rater personas: a seemingly vague consensus policy and a concrete-sounding conflict policy.Surface-level judges are expected to rate the consensus policy as more ambiguous, while LLM-DELTA should rank the conflict policy as more ambiguous.Source: §B.1 · p. 16

B.2 Latent dimensions

  • The construction uses a consensus axis shared by both groups and a conflict axis interpreted differently across groups.Four cases form the cross-product of these axes: the consensus policy yields matching judgments, while the conflict policy yields split judgments.Source: §B.2 · p. 16

B.3 Construction procedure

The construction procedure creates policies and realistic cases that expose a shared consensus axis and a principled interpretive conflict. Rater personas, quality checks, and within-domain scaling preserve and test that distinction.

B.3 Construction procedure

  • The procedure constructs two policies and four realistic cases from a shared consensus axis and an interpretive conflict axis, with each case combining values on both axes.Policy A tracks consensus while using open-textured language; Policy B tracks conflict while using concrete-sounding cues. Each group’s judgments are checked with rationales against the target matrix.Source: §B.3 · p. 17

B.4 Rater personas

  • Rater personas represent each group’s values and interpretive stance, while additional personas vary role and vocabulary without changing that stance.With real respondents, groups can instead be derived from diagnostic anchor items and validated for within-group agreement, between-group disagreement, and assignment stability.Source: §B.4 · p. 17

B.5 Quality criteria

  • Examples must make Policy A appear open-textured and Policy B operational, produce the target agreement pattern, and use charitable, realistic cases well away from Policy A’s consensus boundary.Examples that depart from the standard target matrix are flagged and justified.Source: §B.3 · p. 17, §B.5 · p. 18

B.6 Scaling within a domain

  • The default design uses two policies, four cases, and two personas; scaling varies policy paraphrases, realistic cases within matrix cells, or personas within groups.Paraphrases test robustness to wording; added cases need not be evenly distributed because ecological realism takes priority over balanced cell counts.Source: §B.6 · p. 18

C Pol.is dataset details

The Pol.is analysis uses one Taiwan ride-hailing conversation and three Austrian climate-policy conversations as separate topic subsets. After filtering and translation, DELTA scores each retained statement against sampled cases.

C Pol.is dataset details

  • Researchers use statement texts and published group vote aggregates, treating statements as policy-relevant proposals whose possible implementations are evaluated as cases.Group-level vote tables are used directly rather than recomputing group assignments from participant–vote matrices.Source: C Pol.is dataset details · p. 23
  • The analysis uses four Pol.is conversations: one on ride-hailing regulation in Taiwan and three Klimarat topics on production and consumption, housing, and energy.Pol.is collects statements and votes in facilitated online deliberations; the Klimarat conversations gathered public input on climate-policy recommendations.Source: C Pol.is dataset details · p. 23
  • Only statements marked included by platform moderation are retained; the Traditional Chinese and German texts are translated into English with GPT-5 while preserving hedging, sentiment, and policy terminology.DELTA scores machine translations, not participants’ original wording; vTaiwan additionally excludes factual self-disclosures and descriptive Uber–taxi comparisons, while Klimarat curation removes spam and consultation-process comments. Each topic subset is scored separately, using 60 sampled cases for vTaiwan and 40 per Klimarat conversation.Source: C Pol.is dataset details · pp. 23–24

C.1 Engagement threshold

The bridging analysis filters out statements with low response rates in either opinion group because vote-based agreement estimates are noisy with little engagement. Across engagement cutoffs, no dataset shows a stable positive ambiguity–bridging correlation.

C.1 Engagement threshold

  • The analysis retains statements with responses from at least 20% of each group, although the cutoff is a judgment call and results are recomputed from 0 to 0.9.A response is an agree, disagree, or pass; vote-based agreement rates use Laplace smoothing and exclude passes. Figure C.6 truncates lines when fewer than 20 statements remain.Source: §C.1 · p. 24
  • No dataset shows a stable positive ambiguity–bridging correlation at any engagement cutoff; energy is negative throughout and significantly so from cutoffs 0.1 to 0.4.The energy result is suggestive rather than conclusive: it occurs in one of four datasets, and its Pearson correlations are weaker than its Spearman correlations.Source: §C.1 · p. 24, Figure C.6 · p. 25

D Habermas dataset details

The Habermas dataset follows a representative UK Virtual Citizens’ Assembly deliberating on nine policy questions, using the winning initial and revised statements for DELTA analysis. Cleaning yields 307 rounds and 612 winning statements, scored within question-specific sets.

D Habermas dataset details

  • The Virtual Citizens’ Assembly recruited a demographically representative UK sample and deliberated over the same nine public-policy questions, but omitted the fact-finding and expert-testimony phases typical of citizens’ assemblies.The nine questions were selected from a larger corpus of about 5,500 based on earlier participant ratings of topic importance.Source: D Habermas dataset details · p. 25
  • Each deliberation round elicited opinions, generated and ranked candidate group statements, gathered critiques of the top-ranked draft, then generated revised candidates for final evaluation.The initial top-ranked statement was selected by the Schulze method; participants rated personal agreement and statement quality on seven-point scales and ranked candidates.Source: D Habermas dataset details · p. 25

D.1 Extraction and cleaning

  • Cleaning excluded the Internet-evaluation question and incomplete rounds, leaving 307 rounds and 612 winning statements after winner recomputation and text truncation.The winning statements comprise 72–82 statements per question and were scored in separate question-specific runs using 30 raters and 60 cases; 2,350 of 3,060 participant ratings carried agreement values.Source: §D.1 · p. 26

D.2 The agreement-distribution figure

This material specifies the agreement-distribution analysis and documents how cases, simulated raters, and LLM-judge baselines are constructed.

D.2 The agreement-distribution figure

  • Figure 7 relates agreement with winning statements to within-question DELTA percentile; estimates use initially divided rounds and round-clustered bootstrap bands.The restriction keeps 203 of 307 rounds and 1,706 agreement ratings; cluster resampling accounts for ratings sharing a round and winning statement.Source: §D.2 · pp. 26–27

E.1 Case and rater generation

  • Cases come from 30 ordered policy pairs and are subsampled to 60 for vTaiwan and Habermas or 40 for Klimarat; 30 opposing-interest raters are expanded to five-sentence descriptions.Source: §E.1 · p. 27

E.2 Judgment simulation

  • Gemini-2.5-Flash records one binary answer per rater-policy-case-framing query, elicited under separate positive and negative framings.The authors choose a lightweight non-reasoning model partly because reasoning tends to collapse the across-rater answer spread used for DELTA.Source: §E.2 · p. 28

E.3 LLM-as-a-judge baselines

  • The direct baselines rate policy text alone, while the second rubric models consequential ambiguity as stakeholder disagreement on concrete actions; three judges’ ratings are averaged.The baselines are applied to campus speech and adversarial policies.Source: §E.3 · pp. 28–29

Related papers

Loading 2610.35461v1…