Source-linked AI summary
Activating the Annotator: Operationalising Lived Experience in Data Annotation
Sheza Munir, Ramaravind Kommiya Mothilal, Syed Ishtiaque Ahmed
TL;DR
Subjective annotation lacks an annotator-independent gold standard, while demographic records do not ensure that relevant positional knowledge is active during annotation. AXLE elicits lived experience before labelling; in a preliminary study, elicitation was associated with greater resonance and harm sensitivity and with more label plurality.
Problem
Subjective tasks lack an annotator-independent gold standard, and recruitment-time demographic records do not ensure that indexed knowledge is engaged when annotators read items.
Source: §1 · p. 1Method
AXLE places a vignette-anchored, axis-guided conversation and a synthesised first-person micro-narrative before annotation, comparing this elicitation condition with a matched control.
Source: §3.2 · p. 4, §3.3 · p. 4Results
AXLE raised identity resonance and harm sensitivity, shifted rationales toward situated knowledge, and increased rather than collapsed label plurality.
Source: §7 · pp. 8–9Takeaways & Limitations
The findings fit a state-based account in which elicitation activates positional reasoning rather than selecting annotators by identity.
Source: §6 · p. 8, §7 · pp. 8–9The preliminary study used 30 synthetic, deliberately hard items and a small, unbalanced sample, so effect sizes are best read as an upper bound and domain comparisons are underpowered.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Perspectivist NLP has established that annotator positionality shapes labels for subjective tasks, but the field still lacks methods for operationalising it. Positionality is typically operationalised as a demographic form completed at recruitment: a static attribute recorded once and treated as a covariate, on the assumption that the knowledge it indexes is active when the annotator reads an item. This paper treats that assumption as a design problem and asks whether lived experience can instead be primed by the task itself. We introduce AXLE, a pre-annotation stage in which a scenario vignette anchors a short, axis-guided conversation that is synthesised into a first-person micro-narrative and returned to the annotator before labelling begins. In a 2x2 between-subjects study (104 annotators, 30 items, 1,040 annotations) on ambiguous hate speech spanning immigration, religion and gender, elicitation was associated with higher identity resonance and higher sensitivity to harm. It also increased label entropy and lowered inter-annotator agreement: the stage appears to surface plurality rather than manufacture consensus, which distinguishes it from annotator training, whose success criterion is convergence.
1 Introduction
Perspectivist NLP preserves annotator variation after it is produced, but demographic proxies and classification-oriented interfaces leave open how positional variation arises. AXLE elicits lived experience before annotation, and the study reports greater resonance and harm sensitivity alongside increased label plurality.
1 Introduction
- Positionality is usually recorded once through recruitment-time demographics, which name a position without ensuring its indexed knowledge is active during annotation.The paper notes that demographic attributes predict individual behavior poorly and persona prompting recovers little real annotator variance.Source: §1 · p. 1
- Classification-oriented interfaces can treat annotators’ own histories as bias to suppress, although lived experience may distinguish readings of ambiguous statements.The passage connects this posture to fast rule application and crowdwork incentives.Source: §1 · p. 2
- AXLE places a vignette-anchored, axis-guided conversation before annotation and returns a first-person micro-narrative that remains available while annotators work.The elicitor probes themes matched to the later items and synthesises the conversation into the narrative.Source: §1 · p. 2
- The preliminary 2×2 study tested elicitation against a matched control and lived-experience against non-lived-experience screeners with 104 annotators and 1,040 annotations.The study covered 30 ambiguous hate-speech items across three scenarios.Source: §1 · p. 2
1.1 Contributions
- Elicitation raised identity resonance and harm sensitivity while increasing label entropy and lowering inter-annotator agreement.The paper presents this as surfacing plurality rather than manufacturing consensus, unlike annotator training, whose success criterion is convergence.Source: §1.1 · p. 2
- Rationale analysis found that elicitation shifted cited grounds from literal text toward situated knowledge, especially among participants not screened for the relevant experience.In a subsample of 24, invocation of a lived-experience axis rose by roughly 38 percentage points, often on an axis the screener had not tested.Source: §1.1 · p. 2
2 Related Work
Prior work preserves human label variation after annotation and documents identity effects, while demographic operationalisations leave a proxy gap. Methods for eliciting situated knowledge exist, but have not been used as annotation-time procedures in the ways described here.
Preserving label variation
- Variation-preserving NLP methods model annotators, use soft labels or multi-annotator architectures, and assess disagreement, but work post hoc on variation already produced.The paper positions its question upstream: what determines how much variation the annotation process generates.Source: §2 · pp. 2–3
The proxy problem
- Demographic operationalisation is limited because categories are coarse proxies and recording them does not ensure indexed knowledge is engaged during annotation.The paper identifies this as the gap addressed by its design.Source: §2 · p. 3
Eliciting lived experience
- Vignettes, interview progressions and fidelity-constrained micro-narratives can elicit situated knowledge, but cited annotation uses are recruitment or co-design practices rather than annotation-time procedures.The paper draws on these methods for its pre-annotation elicitation design.Source: §2 · p. 3
Reflexivity as an intervention
- Prior reflexive annotating uses a static form during labelling, whereas this study puts adaptive conversational elicitation before annotation and evaluates it against a matched control.The adaptive elicitor responds to participant answers and runs without researcher presence, which the paper connects to crowdsourcing-scale feasibility.Source: §2 · p. 3
3 Study Design
The study uses ambiguous, experience-divergent hate-speech items and compares an elicitation arm with a control that delays disclosure and removes elicitation stages. It measures annotation outcomes after a vignette-guided conversation and narrative synthesis, with participant recruitment, screeners and analysis procedures defining the study’s scope.
3.1 Task and Items
- Annotators rated harm on a five-point scale, selected a target, wrote a rationale and rated identity resonance on a 1–5 scale.Identity resonance refers to how relevant identity or personal experience felt to the item.Source: §3.1 · p. 3
- Items were synthetic ambiguous hate-speech cases screened for surface deniability, experiential divergence, discourse embeddedness and absence of explicit derogation.The experiential-divergence constraint excludes cases where disagreement follows only from lexical vagueness or toxicity is near-universal, concentrating the corpus on hard cases and making effect sizes an upper bound for a general corpus.Source: §3.1 · p. 3
3.2 Conditions
- The elicitation arm completed vignette, interview and synthesis stages before annotation; the control removed them and moved disclosure after annotation, while other annotation prompts and stages were shared.Both arms used descriptive-paradigm annotation, making the contrast latent versus activated positionality rather than descriptive versus prescriptive framing.Source: §3.2 · p. 4
3.3 The Elicitation Instrument
- AXLE used scenario-matched neutral vignettes, an adaptive conversation guided by seven lived-experience axes, and a fidelity-constrained first-person narrative returned for review.The interview bridged from each response toward uncovered axes, with a five-turn cap; the narrative was at least 120 words and remained accessible during annotation.Source: §3.3 · pp. 4–5
3.4 Participants
- The 104 participants were recruited through Prolific, and disclosure agreed with screener stratum for 97 participants (93.3%).Seven non-LE participants reported direct personal experience, concentrated in scenarios B and C, where screeners tested a narrower axis than the disclosure question.Source: §3.4 · p. 5
3.5 Measures and Analysis
- Participant-level tests were primary because annotations were nested; paired item-level tests were secondary, and agreement was reported with ordinal Krippendorff’s α.The primary measures were identity resonance and harm rating; secondary measures included time, label entropy and target attribution.Source: §3.5 · p. 5
4 Results
Elicitation increased identity resonance and harm ratings while also increasing label plurality and reducing agreement. Effects varied by annotator stratum and domain, and the added time was projected to amortize only beyond roughly 48 items.
4.1 Resonance and Harm Sensitivity
- Elicitation raised identity resonance (M=3.35 vs. 2.94) and harm ratings (M=3.12 vs. 2.78); the harm shift is also compatible with a demand characteristic.The authors identify entropy analysis as the primary evidence against the demand-characteristic reading.Source: §4.1 · pp. 5–6
4.2 Elicitation Increases Plurality
- 2.05 vs. 1.88 bits of harm-label entropy and α=.066 vs. .111 show greater plurality under elicitation, which raises engagement while lowering agreement rather than training annotators toward consensus.The α values are not reliability estimates; only their between-arm comparison at a fixed item set is interpreted.Source: §4.2 · p. 6
4.3 Effects Are Larger for Non-LE Annotators
- Elicitation's resonance and harm gains were larger for non-LE than LE annotators (0.51 vs. 0.31; 0.40 vs. 0.28), while elicited non-LE resonance matched unelicited LE (3.19 vs. 3.20).LE annotators reported higher resonance than non-LE annotators, but their harm ratings did not differ.Source: §4.3 · p. 6
4.4 Domain Effects
- For REL and INT items, mean harm rose while dispersion stayed roughly constant; for GEN items, resonance and entropy rose despite unchanged mean harm, a descriptive pattern from small domain cells.For GEN items, mean harm was 2.94 → 2.88, resonance 2.54 → 3.28, and entropy 1.74 → 2.11; domain cells contained 6–10 items.Source: §4.4 · p. 6
4.5 Cost
- Median sessions took 51.4 vs. 43.8 min and overhead 18.6 vs. 7.9 min; projected cumulative cost crosses near 48 items, an illustrative estimate that ignores fatigue.The projection relies on a per-item advantage that was not significant.Source: §4.5 · p. 6
5 Qualitative Analysis
A reflexive thematic analysis examined how elicitation changed annotators’ grounds for judgement and which lived-experience axes they invoked. It found more situated grounding, with axis invocation rising especially among non-LE annotators.
5.1 Sample and Protocol
- The analysis covered 276 units from 24 participants, with two coders independently coding units and reconciling disagreements through discussion.Agreement on whether a rationale invoked situated grounding was κ=.84.Source: §5.1 · pp. 6–7
- The figures show harm responses shifting toward both ends and per-item points above the diagonal indicating more dispersed labels under elicitation.The share rated “not at all relevant” to identity or experience fell by more than half.Source: Figure 3 · p. 7, Figure 4 · p. 7
5.2 Elicitation Shifts the Grounds of Judgement
- 50.8% of elicitation-arm rationales used situated grounding, versus 20.8% under control, while literal-only appeals fell from 40.8% to 12.5%.Situated grounding included personal episodes, group membership, or recognised discourse patterns.Source: §5.2 · p. 7
- Single-reading commitment remained high and acknowledged multiplicity stayed at 16.7% in both arms, so annotators diverged through confident readings rather than greater uncertainty.The arm-level grounding shift was bimodal: it appeared on items where the micro-narrative surfaced in the rationale, including within participants.Source: §5.2 · p. 7
5.3 What Non-LE Annotators Draw On
- Axis invocation among non-LE annotators rose from 10.0% under control to 48.3% under elicitation, compared with 36.7% to 56.7% among LE annotators.The gain was largely carried by sociocultural and geographic grounding, which rose from 4.2% to 25.8% of all rationales.Source: §5.3 · p. 7
- Screeners tested one axis per scenario, but rationales could invoke another, such as class and regional upbringing rather than gender in scenario C.The seven-axis instrument could locate whichever axis was live for an annotator, unlike a single-axis screener.Source: §5.3 · pp. 7–8
- Elicitation did not uniformly amplify harm: some annotators used unfamiliarity with a discourse to justify a lower harm rating.The qualitative account describes increased variance in readings, not increased mean severity.Source: §5.3 · p. 8
6 Discussion
The discussion frames positionality as an activatable state rather than only a stable demographic attribute. It argues that plurality, not agreement, is the relevant evaluation target and considers implications for annotation platforms and models.
Positionality as a state
- The results fit a state-based account in which positionality varies with whether an annotator’s experience is engaged, rather than only with recorded demographic attributes.The discussion connects this account to why annotator-history conditioning may outperform demographic conditioning and why persona prompts recover little annotator variance.Source: §6 · p. 8
Plurality as the evaluation criterion
- If the target is the population’s distribution of readings rather than its mode, increased agreement is the wrong success criterion for annotation interventions.The discussion contrasts plurality-oriented evaluation with conventional agreement-based assessment.Source: §6 · p. 8
Implications for crowdsourced annotation
- Elicitation is a fixed per-worker cost that can be amortised across later tasks, while whether its narrative is live for a given item matters more than completion alone.The discussion also identifies elicitation artefacts as annotator-level metadata for models and argues that agreement-based quality assurance would penalise perspectivist signal.Source: §6 · p. 8
7 Conclusion
The conclusion identifies a practical gap between knowing that positionality matters and deciding what to do at annotation time. AXLE raised resonance and harm sensitivity, shifted rationales toward situated knowledge, and increased label plurality, with especially large gains among annotators not screened for relevant experience.
7 Conclusion
- Knowing that positionality matters does not tell dataset builders what to do at annotation time.Source: §7 · p. 8
- AXLE raised identity resonance and harm sensitivity, shifted rationales toward situated knowledge, and increased rather than collapsed label plurality.Annotators not screened for relevant experience showed the largest gains in resonance and lived-experience-axis invocation, suggesting activation rather than identity selection.Source: §7 · pp. 8–9
Limitations
The authors describe this as a preliminary study with a small, deliberately hard sample and several limits on generalisation and interpretation.
Limitations
- The study uses 30 synthetic items and 104 participants, with unbalanced scenario cells and uncorrected secondary analyses, so effect sizes are best read as upper bounds.Participants were English-fluent North American residents, and the encoded discourse patterns were Anglophone; replication on naturally occurring multilingual data is proposed.Source: Limitations · p. 9
- Coarse screeners tested one axis per scenario, and identity resonance was self-reported by participants aware of the study’s focus, leaving possible demand effects.Elicitation is model-mediated, so narratives may align annotators to a model’s rendering of experience; fidelity coding bounds but cannot remove that risk.Source: Limitations · p. 9
Ethical considerations
AXLE’s elicitation invites disclosure of potentially distressing experiences, creating emotional burden for participants. The authors describe safeguards and warn that deployment should account for workers’ emotional labour and the risk of using framing to shape labels.
Ethical considerations
- Participants encountered plausibly hateful content and some disclosed discrimination, immigration precarity, or religious hostility during elicitation.The design intentionally incurs this burden in inviting experiences of marginalisation.Source: Ethical considerations · p. 9
- A hard turn cap, a wind-down turn, and no follow-up pressure on thin answers were used to mitigate the burden.Participants could also withdraw at any point.Source: Ethical considerations · p. 9
- Researchers deploying elicitation at scale should treat emotional labour as a cost borne by workers, not an externality.The authors connect this cost to the tension between reflexive engagement and affective demand.Source: Ethical considerations · p. 9
- Upstream framing could be used to manufacture a desired label distribution, although elicitation widened rather than narrowed the distribution in this study.The authors describe the entropy result as partial protection and say the risk applies to framing interventions placed upstream of labelling.Source: Ethical considerations · p. 9
A Prolific Screener Specifications
The appendix describes screener requirements, participant-demographic reporting, three scenario vignettes, AXLE prompt design, and the item corpus. The supplied passages specify screening constraints and prompt procedures, but do not provide the underlying screener or demographics table values.
A Prolific Screener Specifications
- All conditions required US or Canadian residence, English fluency, a 95–100 approval rate, and exclusion from other project studies; sex was restricted to male or female in scenarios A and B.The table caption identifies these shared conditions but does not give the stratum-specific screener entries.Source: Table 2 · p. 11
B Participant Demographics
- Table 3 reports participant demographics for N=104; religion and sexual-orientation measures were collected only in scenarios B and C, respectively.Those counts sum to 24, according to the caption.Source: Table 3 · p. 12
C Vignettes
- The three vignettes stage contested belonging through migration housing, religious-holiday backlash and solidarity, and disputes over gender identity, discrimination, and reverse bias.The religion scenario includes a community buy-out in support of the targeted bakery.Source: C Vignettes · p. 12
D AXLE Prompts
- AXLE frames the conversation as eliciting personal experience rather than opinions and uses seven lived-experience axes as invisible lenses.The axes include sociocultural context, language, socioeconomic status, race, gender and sexuality, disability, and epistemic proximity.Source: D AXLE Prompts · p. 12
- Prompts ask one specific question at a time, stay with a thread after thin answers, and broaden or deepen across turns before inviting integration.The final turn asks no question; narrative synthesis requires at least 120 words and three dimensions, while preserving the participant’s expressed content.Source: D AXLE Prompts · pp. 12–13
E Item Corpus
- Table 4 lists 30 items, all reported to satisfy the study’s C1–C4 ambiguity-codebook criteria.The caption maps scenario prefixes A, B, and C to immigration, religion, and gender/sexuality.Source: Table 4 · p. 14
F Additional Quantitative Detail
Additional results describe exploratory shifts in target attribution, a null difference in rationale length, and domain-specific effects on resonance and harm. The appendix also reports agreement context and a session-time projection, but the supplied passages do not provide their underlying numerical values.
Agreement
- Absolute ordinal Krippendorff’s α values on the five-point harm scale were near chance in both arms; only the between-arm comparison is interpretable.The caption attributes the near-chance absolute values to the corpus being selected for experiential divergence.Source: Table 7 · p. 15
Target attribution
- “No one” target attribution fell from 15.0% to 9.3%, while “multiple overlapping groups” rose from 17.7% to 23.8%; the item-level shift was exploratory.The item-level test was χ2(6)=25.45, p=.0003, but the participant-level difference did not reach significance (p=.38).Source: F Additional Quantitative Detail · p. 15
Rationale length
- Rationale length was 62.7 vs. 62.5 words, with no arm difference at participant or item level.Participant p=.89 and item p=.52; the authors report this null result because it rules out lower-effort responding as an explanation for dispersion.Source: F Additional Quantitative Detail · p. 15
Domain effects
- Harm rose for REL (2.53 →3.33) and INT (2.91 →3.27), but not GEN (2.94 →2.88); resonance gains were largest for GEN, INT, and REL.Resonance values were GEN 2.54 →3.28, INT 2.60 →3.27, and REL 2.89 →3.47; IMM had the smallest gain (3.26 →3.36).Source: F Additional Quantitative Detail · p. 15, Figure 6 · p. 15
Cost projection
- Figure 7 projects session time by items annotated, using median fixed overhead and median per-item time in each arm.The supplied caption gives the projection method, not projected values.Source: Figure 7 · p. 15
G Coder Agreement, Carryover and Fidelity
Coder analyses report varied links between elicited narratives and later rationales, alongside limits in narrative fidelity. Reflexivity responses also show that participants’ self-reports of positionality do not always match how their rationales operate.
G Coder Agreement, Carryover and Fidelity
- 44.2% of elicitation-arm rationales showed no detectable narrative relation, 36.7% applied a narrative theme, 2.5% reused wording near-verbatim, and 16.7% appended a fixed clause.Fixed-clause tagging came from 2 of 12 participants, each repeating the same sentence across all ten rationales.Source: G Coder Agreement, Carryover and… · pp. 15–16, Figure 9 · p. 16
- 72.7% vs. 30.2% situated grounding occurred with thematic carryover versus no detectable narrative relation; the latter resembled control (20.8%, p=.24).The difference between the two elicitation groups was Fisher p=3.9 × 10−5; ten of twelve participants produced both patterns, but recall-triggering items were not experimentally controlled.Source: G Coder Agreement, Carryover and… · p. 16
- Only 7 of 12 transcript–narrative pairs were faithful; 5 contained at least one unsupported move.Six conversations yielded mostly short, general answers, and two entered an unrecovered repair loop.Source: G Coder Agreement, Carryover and… · p. 16
- Before reconciliation, coders agreed most closely on the construct carrying §5’s central claim among constructs present in both schemes.The figure caption does not name that construct.Source: Figure 8 · p. 16
H Reflexivity Responses
- 13 of 24 sampled participants explained divergence by reference to their own position, including 8 of 12 in elicitation and 5 of 12 in control.An immigrant participant attributed their readings to education rather than immigration status, despite grounding rationales in discourse familiarity; self-report and positionality’s operation do not always coincide.Source: H Reflexivity Responses · p. 16
I Qualitative Codebook
- The codebook covers grounding, axis invocation, discourse recognition, epistemic stance, carryover, reflexivity type, chat quality, and narrative fidelity.Each dimension has distinct coding categories, including seven lived-experience axes plus none invoked.Source: I Qualitative Codebook · p. 16