PaperlayerView exact revision on arXiv

Source-linked AI summary

Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments

Amir Homayounirad, Enrico Liscio, Tong Wang, Catholijn M. Jonker, Luciano C. Siebert

arXiv:2509.06704v1cs.CL

TL;DR

Different annotators may interpret the values supporting an argument differently, while aggregation can obscure such disagreement. The paper compares inferring subjectivity from value predictions with directly classifying it. Direct identification improves performance, while contrastive loss does not improve prediction performance but reduces dependence on per-label subjectivity.

  • Problem

    Annotators may disagree about which values support an argument, and consensus or majority aggregation can obscure this subjectivity.

    Source: §1 · p. 1

  • Method

    The paper compares inferring subjectivity from annotator-specific value predictions with directly classifying whether an argument is subjective.

    Source: §1 · p. 2

  • Results

    Direct identification significantly enhances performance over inferred subjectivity; combining contrastive loss with BCE does not directly improve identification but reduces dependence on per-label subjectivity.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Identifying potentially subjective arguments may support additional annotations and moderator questions that foster reflection.

    Source: §1 · p. 2

    Evaluation is confined to Touché23-ValueEval, primarily because few value datasets include annotator-level annotations.

    Source: §8 · p. 9

Abstract

from arXiv · show

Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process.

1 Introduction

Value-laden arguments can invite different interpretations of the values that motivate them, yet annotation practices often consolidate judgments. The paper compares inferred and direct approaches to identifying subjectivity, finding a performance advantage for direct identification.

1 Introduction

  • The work studies value-laden arguments, where values motivate choices and actions and can reveal underlying motivations.The introduction situates value identification as an NLP research topic.Source: §1 · p. 1
  • Majority aggregation can obscure natural disagreement about which values support an argument, risking misinterpretation or annotation-process bias.For example, annotators may assign different values to the same argument.Source: §1 · p. 1
  • Identifying potentially subjective arguments may prompt additional annotations or moderator questions that support reflection during participatory deliberations.The paper presents these as possible applications.Source: §1 · p. 2
  • Direct subjectivity identification greatly improves performance over inferring subjectivity through value prediction on Touché23-ValueEval.The study also examines contrastive learning strategies.Source: §1 · p. 2
  • The paper proposes three methods that infer subjectivity from individual value annotations and three that identify subjectivity directly, with contrastive-learning variants.The approaches are evaluated on Touché23-ValueEval.Source: §1 · pp. 1–2

2 Related works

Related work spans value identification, annotation subjectivity, and approaches to measuring disagreement. This paper focuses on predicting whether value annotations for an argument will differ, rather than predicting specific annotators’ opinions.

2 Related works

  • The related-work review covers value identification, NLP subjectivity, and measuring disagreement around moral ambiguity.These are the three themes named in the review’s overview.Source: §2 · p. 2

2.1 Identifying Values in Text

2.2 Identifying Subjectivity

  • Prior NLP research models individual or group annotation behavior, while this paper predicts expected disagreement without relying on specific annotators or groups.Related approaches combine annotator and annotation embeddings, use multi-annotator architectures, or incorporate demographic information.Source: §2.2 · p. 2

2.3 Measuring disagreement on moral ambiguity

  • Disagreements in morally ambiguous cases can reflect plural values and moral intuitions, rather than mere annotation errors.The discussion also connects disagreement with cognitive biases and deliberative-democracy frameworks.Source: §2.3 · pp. 2–3

3 Methods

The methods compare inferring subjectivity from annotator-specific value predictions with directly classifying subjectivity. Both approaches operate on arguments and their value annotations, with contrastive-learning variants added to direct classification.

3 Methods

  • The paper proposes inferred subjectivity from annotator-specific value predictions and direct classification of whether an argument is subjective.The two approaches are described as IS and DS, respectively.Source: §3 · p. 3

3.1 Task Formalization

  • An argument is subjective when at least one annotator assigns it a different set of values than others.The task predicts agreement or disagreement from annotated text–value–annotator triples.Source: §3.1 · p. 3

3.2 Inferred Subjectivity Identification (IS)

  • IS predicts multi-label values for each annotator, then identifies subjectivity for a value when annotator predictions differ.The architectures vary how argument text is passed through sentence embeddings and classification heads.Source: §3.2 · p. 3, Figure 1 · p. 4

3.2.1 IS-each: A Dedicated Model for Each Annotator

  • IS-each trains a separate multi-label value-prediction model for every annotator, making it computationally expensive.Each model predicts values from the annotations provided by its annotator.Source: §3.2.1 · p. 3

3.2.2 IS-shared and 3.2.3 IS-single

  • IS-shared shares embeddings across annotators but keeps separate classification heads, while IS-single adds an annotator ID to the text input.The IS-single description notes that using the ID may not capture annotator nuances.Source: §3.2.2 · p. 3, §3.2.3 · p. 3

3.3 Direct Subjectivity Identification (DS) and 3.3.1 DS-simple

  • DS directly classifies subjectivity for each value label, comparing a dedicated model with supervised and unsupervised contrastive variants.The figure represents processing text to identify subjectivity for each value label.Source: §3.3 · p. 3, §3.3.1 · p. 3, Figure 2 · p. 4

3.3.2 Contrastive Loss Primer

  • The contrastive variants retain binary cross-entropy as the primary subjectivity-classification objective and add contrastive loss, weighted by λ.The contrastive objective is calculated in supervised or unsupervised fashion.Source: §3.3.2 · p. 4

3.3.3 DS-sup and 3.3.4 DS-unsup

  • DS-sup uses labeled same-class and different-class examples to shape embeddings; DS-unsup instead separates semantically similar and dissimilar samples without label-based shaping.The supervised triplet loss uses a margin to separate positive and negative representations; the unsupervised method uses contrastive tension loss.Source: §3.3.3 · pp. 4–5, §3.3.4 · p. 5

4 Experimental Setup

The experiments use multi-annotator value annotations to compare subjectivity-identification methods across two models. Evaluation measures per-value performance and its relationship to subjectivity levels.

4.1 Models

  • The experiments test BERT-base and two Llama-3.1-8B-Instruct variants, comparing proposed methods with a random subjectivity-prediction baseline.The authors describe the approach as model-agnostic, though experiments use only these two models.Source: §4.1 · p. 5

4.2 Dataset

  • Touché23-ValueEval contains 9,324 natural-language arguments annotated across 54 values; experiments select eight frequent level-2 values and four high-overlap annotators from 39.The study uses the annotators’ original labels before aggregation; the annotators are Western, and the dataset’s argument sources are almost exclusively Western.Source: §4.2 · p. 5
  • For direct subjectivity methods, the minority class is balanced by paraphrasing; this augmentation is not used for multi-label inferred-subjectivity methods.In the inferred approach, augmenting one value could affect other values annotated on the same sentence.Source: §4.2 · p. 5

4.3 Evaluation Metrics

  • The study reports test-set precision, recall, and F1 per value and averaged across values, plus Spearman correlation between per-value F1 and subjectivity ratio.The test set is fixed.Source: §4.3 · p. 6

5 Results

Direct subjectivity identification outperforms inferring subjectivity from value predictions. Results also show differences among methods, across value subjectivity levels, and between the tested models.

5.1 Comparison across Methods

  • Direct subjectivity identification consistently beats the random baseline and outperforms inferred subjectivity, which the random baseline mostly also beats.The stated exceptions are IS-each recall for Achievement, Power, and societal Security.Source: §5.1 · p. 6, Table 1 · p. 6
  • IS-each consistently outperforms other inferred methods, while IS-single beats IS-shared; DS-simple and DS-sup generally perform similarly and better than DS-unsup.The authors suggest dedicated annotator models better capture annotation tendencies; DS-unsup fares notably worse on less subjective values such as Benevolence: dependability.Source: §5.1 · p. 6

5.2 Comparison across Values

  • IS performance correlates with value subjectivity; among direct methods, only DS-unsup correlates with subjectivity, while DS-simple and DS-sup perform better on less subjective values.DS-sup is more balanced across subjectivity levels than DS-simple; the authors conjecture supervised contrastive loss mitigates overfitting by shaping embeddings more consistently.Source: §5.2 · p. 6

5.3 Comparison across Models

  • Using Llama for IS-single yields no improvement over BERT IS-single, whereas Llama DS-simple improves performance, particularly recall across values and precision for Power and Universalism.The authors conjecture that adding annotator IDs in IS-single is insufficient to distinguish annotators.Source: §5.3 · p. 7

6 Discussion

The discussion weighs direct subjectivity prediction against value-based inference and describes how their embeddings may support annotation and deliberation. It also relates value subjectivity to differences among values.

6.1 Comparison across Methods

  • Direct identification’s focused objective may explain its stronger performance, while inferred subjectivity can help examine annotators’ differing value interpretations.The authors suggest that additional information, such as annotator backgrounds, may help model the relationship between value prediction and subjectivity.Source: §6.1 · p. 7
  • Because argument subjectivity may vary across contexts and demographic groups, direct prediction could support updating models for new contexts and populations.The paper presents this as a pathway for active learning.Source: §6.1 · p. 8
  • DS-sup embeddings better separate subjective from non-subjective examples, while DS-unsup groups semantically similar samples; both may support active-learning annotation.The paper proposes selecting DS-sup examples near the decision boundary and using DS-unsup to retrieve semantically similar examples labeled differently; these patterns are reported for Ben-dep and observed across considered values.Source: §6.1 · p. 8, Figure 4 · p. 8

6.2 Comparison across Values

  • The discussion links higher subjectivity to values tied more closely to personal experiences and individual goals, such as Achievement, than to Universalism.It describes Universalism as grounded in broader ethical principles and identifies Security and Conformity as boundary values.Source: §6.2 · p. 8

7 Conclusion

The paper presents multiple approaches to identifying subjectivity in value-laden arguments and finds that direct identification improves performance over inferring subjectivity through value prediction. It also discusses combining value and subjectivity prediction as future work and notes that annotation differences can be confounded with noise.

7 Conclusion

  • Directly identifying subjectivity significantly enhances performance over inferring it through value prediction; combining contrastive and BCE losses does not improve performance but reduces dependence on per-label subjectivity.A state-of-the-art model improves direct, but not inferred, subjectivity identification.Source: §7 · p. 9
  • Combining value and subjectivity prediction is proposed as future work to capture annotators’ perspectives alongside their annotations, potentially improving subjectivity identification.Datasets with diverse demographics, lived experiences, and moral values are described as facilitating this integration.Source: §7 · p. 9
  • Annotation differences can be confounded with noise, and the proposed methods may help identify potentially subjective annotations for approaches addressing that issue.Prior approaches include worker deliberation and assessing the validity of annotators’ explanations.Source: §7 · p. 9

8 Limitations

The evaluation is limited to Touché23-ValueEval, and several methods face computational or scalability constraints. The study also cautions that its classification results may not generalize broadly and that useful deliberation-setting predictions require representative annotators.

8 Limitations

  • The evaluation is confined to Touché23-ValueEval, primarily because few value datasets include annotator-level annotations.The authors call for datasets collecting diverse annotators’ value annotations to enable more robust evaluation.Source: §8 · p. 9
  • ISV-each, ISV-shared, and DS-simple are computationally expensive because they rely on dedicated embeddings and classifiers, limiting practicality at large scale.The authors suggest optimizing these models or developing lightweight alternatives, including grouping annotators with shared perspectives and selecting informative examples.Source: §8 · p. 9
  • Because the study focuses exclusively on classification, its findings may not generalize to summarization, question-answering, or fields beyond NLP.Source: §8 · p. 9
  • For deliberation settings, annotators should represent participants’ varied moral values, beliefs, and backgrounds; users are encouraged to train or fine-tune with representative annotators.The study’s dataset was not designed for a specific deliberation setting, and the authors had no control over alignment between annotators and participants.Source: §8 · p. 9

A.1 Dataset details

The appendix documents the ValueEval data and annotator agreement, then describes paraphrasing and implementation details. It also reports that agreement varies across value categories.

A.1 Dataset details

  • The ValueEval dataset contains 5,270 arguments annotated for a taxonomy of 54 values, with abstract levels 2–4, Western-background annotators, and an approximately 78/22 train/test split.Ten percent of the training set was reserved for validation.Source: §A.1 · p. 14
  • The appendix examines the four workers with the most annotations and the eight most-annotated value categories.Figures A1 and A2 show unique arguments by worker and value-category distributions by worker, respectively.Source: §A.1 · p. 14, Figure A1 · p. 14, Figure A2 · p. 14

A.1.1 Fleiss Kappa score

  • Agreement ranges from fair for Value_0, Value_7, and Value_3 to moderate for Value_1, Value_2, and Value_5, and substantial for Value_6.The appendix reports these value-specific Fleiss’ kappa categories in connection with Figure A3.Source: §A.1.1 · p. 14, Figure A3 · p. 15

A.2 Paraphrasing

  • Paraphrases are generated with Vamsi, a T5-based transformer fine-tuned on PAWS.Table A2 lists paraphrasing hyperparameters.Source: §A.2 · p. 15, Table A2 · p. 15

A.3 Hyperparameters and Infrastructure

  • Llama-3.1-8B-Instruct is adapted for classification with a fully trainable head and LoRA adapters, using 4-bit quantization to lower computational cost.Experiments used an RTX 2080 Ti for BERT and an NVIDIA A40 with AMD EPYC processors for Llama.Source: §A.3 · p. 15

B Extended Results on value prediction and subjectivity prediction with STD

The appendix lists hyperparameters for the proposed method variants and provides tables summarizing value-prediction and per-value subjectivity-prediction metrics.

Hyperparameters

Value prediction

  • Table A8 reports average and standard deviation of precision, recall, and F-score for value predictions.Source: Table A8 · p. 16

Subjectivity prediction

  • Table A9 reports per-value subjectivity-prediction precision, recall, and F1, with standard deviations across five runs using a fixed test set.Source: Table A9 · p. 16

Related papers

Loading 2509.06704v1…