PaperlayerView exact revision on arXiv

Source-linked AI summary

More Context, Larger Models, or Moral Knowledge? A Systematic Study of Schwartz Value Detection in Political Texts

Víctor Yeste, Paolo Rosso

arXiv:2605.22641v3cs.CLcs.AIcs.LG

TL;DR

Implicit, overlapping values in political text often depend on surrounding arguments, making sentence-level Schwartz value detection difficult. This study systematically compares context, retrieved moral knowledge, model families, scale, and fusion strategies, finding that context and simple early-fusion knowledge help selectively while larger models do not automatically improve performance.

  • Problem

    Implicit, overlapping, and rare values can depend on surrounding political arguments, making fine-grained sentence-level classification difficult.

  • Method

    The study systematically compares context lengths, retrieval augmentation, supervised encoders, zero-shot LLMs, and encoder-side knowledge-fusion architectures on political value detection.

  • Results

    Full-document context benefits supervised encoders, early-fusion moral knowledge helps, and larger models or more complex fusion architectures do not consistently improve performance.

  • Takeaways & Limitations

    Value detection should use context deliberately, add inspectable early-fusion knowledge for ambiguous labels, and evaluate performance by individual value rather than aggregate macro-F1 alone.

  • Takeaways & Limitations

    The conclusions may not transfer directly beyond the English task and its political-text domains, languages, or communicative settings.

Abstract

from arXiv · show

Detecting Schwartz values in political text is difficult because implicit cues often depend on surrounding arguments and fine-grained distinctions between neighboring values. We study when context and explicit moral knowledge help sentence-level value detection. Using the ValuesML/Touché ValueEval format, we compare sentence, window, and full-document inputs; no-RAG and retrieval-augmented settings with a curated moral knowledge base; supervised DeBERTa-v3-base/large encoders; and zero-shot LLMs from 12B to 123B parameters. The results show that more context is not uniformly better: full-document context improves supervised DeBERTa encoders by 3.8-4.8 macro-F1 points over sentence-only input, but does not consistently help zero-shot LLMs. Retrieved moral knowledge is more consistently useful in matched comparisons, improving each tested model family and context condition under early fusion. However, scaling from DeBERTa-v3-base to large and from 12B to larger LLMs does not guarantee gains, and simple early fusion outperforms the tested late-fusion and cross-attention RAG variants for encoders. Per-value analyses show that context and retrieval help most for socially situated or conceptually confusable values. These findings suggest that value-sensitive NLP should evaluate context, knowledge, and model family jointly rather than treating longer inputs or larger models as universal improvements.

1 Introduction

The introduction frames sentence-level Schwartz value detection in political texts as difficult because values are implicit, fine-grained, overlapping, and context-dependent. It motivates a controlled study of document context, retrieved moral knowledge, model family and scale, fusion strategy, and value-specific effects.

  • Motivation: Political texts often express values indirectly, making fine-grained sentence-level classification difficult when values are implicit, overlapping, rare, or context-dependent.Schwartz’s refined 19-value taxonomy supports computational distinctions but increases classification difficulty.
  • Research problem: The unresolved methodological question is what information models should receive when deciding whether a sentence expresses a value.Sentence-only inputs may be insufficient, while surrounding argument context can clarify implicit appeals.
  • Research problem: Retrieved moral knowledge may reduce ambiguity by supplying concise definitions, annotation guidance, or contrasts among Schwartz values, but it may also add irrelevant material.The introduction highlights possible interactions between retrieval and long document contexts.
  • Study design: The study systematically compares sentence-only, local-window, and full-document inputs; no-retrieval and retrieval-augmented conditions; supervised DeBERTa-v3 encoders; zero-shot LLMs; and encoder fusion strategies.The comparisons are designed to separate effects of context, retrieval, model family, model scale, and fusion strategy.
  • Research questions: The paper’s research questions examine context, retrieved knowledge, model family and scale, fusion strategy, and which Schwartz values benefit most from these factors.Its stated contribution is a controlled analysis rather than a new value taxonomy or foundation model.
  • Central hypothesis: The introduction states that additional context and external knowledge can help value detection, but their usefulness depends on the model, input format, fusion strategy, and predicted value.This framing connects aggregate comparisons with per-value analysis.

2 Related Work · 3 Dataset and Task

The paper situates value detection within ValueEval, moral-language analysis, and context-aware retrieval research, then defines a multi-label political-text task over reconstructed sentence, window, and document contexts. It collapses attained and constrained annotations into value-presence labels and emphasizes sparse, imbalanced evaluation with macro-F1 and per-value analysis.

  • 2 Related Work: ValueEval operationalizes Schwartz’s refined taxonomy for identifying values in arguments and political texts.The study builds on shared-task systems rather than proposing another one.
  • 2 Related Work: The study uses ValueEval as a controlled testbed for target-sentence context, retrieved value knowledge, model family, and retrieval-fusion strategy.This isolates the effects of these factors instead of introducing another shared-task system.
  • 2 Related Work: Unlike work measuring models’ own values, this task tests whether zero-shot LLMs identify values expressed in external political sentences alongside supervised DeBERTa encoders.The framing connects value detection with moral-language analysis and instruction-based classification.
  • 2 Related Work: Because value cues may be distributed across sentences or obscured by distractors, the study explicitly compares sentence, window, and document inputs.The task labels one marked target sentence rather than the whole document.
  • 3 Dataset and Task: The dataset follows the ValuesML/Touché24-ValueEval format, organizing political documents into sentences with text_id and sent_id for reconstructing local and full-document context.The prediction unit is a single target sentence.
  • 3 Dataset and Task: The label space contains 19 refined Schwartz values, and attained or constrained annotations are collapsed into one binary value-presence label.The resulting task is multi-label classification: a sentence may express no value, one value, or several values.
  • 3 Dataset and Task: Roughly half of sentences have no positive value label and about 6% are multi-label, making the task sparse and highly skewed.The supplied passage identifies sparsity and imbalance as central evaluation considerations.
  • 3 Dataset and Task: Macro-F1 is the primary metric, while per-value analysis tests whether context and retrieved knowledge benefit frequent, rare, or conceptually subtle values.The motivation is the dataset’s sparsity, imbalance, and uneven value distribution.

4 Knowledge Base and Retrieval

This section describes a compact, manually curated moral knowledge base designed to provide conceptual evidence without labeled examples. Retrieval uses normalized sentence embeddings, FAISS nearest-neighbor search, and fixed top-k = 2, enabling controlled comparisons of fusion mechanisms.

  • Knowledge base: The KB contains 58 manually curated chunks: 19 value-definition, 25 operational-guideline, and 14 theory-level chunks.Definition and theory chunks follow the refined Schwartz taxonomy, while guideline chunks encode task-facing annotation distinctions.
  • Knowledge base: The KB contains no training or test instances and supplies concise conceptual evidence rather than additional labeled examples.Its purpose is to test whether explicit value knowledge helps classification beyond in-document context.
  • Retrieval: Retrieved text is provided without gold label information, keeping retrieval label-agnostic at inference time.Value metadata supports logging and qualitative analysis but does not filter retrieval in the main experiments.
  • Retrieval: Retrieval embeds chunks with all-MiniLM-L6-v2, indexes normalized vectors using FAISS IndexFlatL2, and retrieves nearest chunks with fixed top-k = 2.The query uses the same embedding encoder at inference time.
  • Retrieval: Early-fusion, late-fusion, and cross-attention RAG conditions share the same KB, embedding model, FAISS index, query construction, and top-k setting.Thus, their differences reflect knowledge-fusion mechanisms rather than retrieval-system changes.

5 Models and Input Conditions

The study varies the context supplied for each target sentence, compares supervised DeBERTa encoders with instruction-tuned LLMs across scales, and tests multiple retrieval-fusion conditions. All models perform the same 19-way multi-label Schwartz value prediction task, with LLMs evaluated zero-shot through prompts containing definitions and optional knowledge.

  • Input conditions: Each condition predicts labels for the same target sentence using sentence-only, a ±2-sentence window, or the reconstructed full document.Window inputs include up to two preceding and two following sentences, subject to document boundaries.
  • Encoder models: Supervised encoders use DeBERTa-v3-base and DeBERTa-v3-large as 19-way multi-label classifiers with sigmoid outputs for each Schwartz value.Training uses binary cross-entropy with logits, validation-selected checkpoints, and a fixed validation-selected threshold for test evaluation.
  • Retrieval conditions: Encoder experiments compare no-RAG, early-fusion, late-fusion, and cross-attention knowledge conditions.Early fusion concatenates retrieved knowledge with the selected text before encoding; late fusion encodes document context and knowledge separately before representation fusion.
  • LLM models: Zero-shot decoder models span approximately 12B, 72B, and 123B parameters: Gemma 3 12B IT, Qwen2.5-72B-Instruct, and Mistral-Large-Instruct-2407.The comparison tests direct use of label definitions, optional retrieved knowledge, and longer contexts rather than supervised fine-tuning.
  • LLM prompting: LLM prompts include the task description, 19 Schwartz value names with definitions, output instructions, optional retrieved snippets, and the target sentence with its selected context.Decoding is deterministic, and outputs are parsed from several list, object, and delimited-text formats.

6 Experimental Setup · 7 Results

The experiments vary model family, context, and retrieved moral knowledge, using macro-F1 as the primary metric. Results show that document context mainly benefits supervised encoders, retrieved knowledge consistently improves matched conditions, and task supervision outweighs scale under this protocol.

  • 6 Experimental Setup: The main experiment crosses model family, input context, and retrieved knowledge across supervised DeBERTa encoders and zero-shot LLMs.Encoder conditions span sentence, window, and document inputs with and without early-fusion RAG.
  • 6 Experimental Setup: Macro-F1 is the primary metric because label imbalance makes performance across all Schwartz values more important than frequent-label performance alone.Micro-F1 is secondary, with per-label precision, recall, and F1 used for value-level analysis.
  • 7.1 RQ1: Effects of Document Context: DeBERTa-v3-base reaches .285 macro-F1 with document context versus .237 with sentence-only input, while DeBERTa-v3-large reaches .280 versus .242.The local window helps base but hurts large, reaching .207 for DeBERTa-v3-large; document context improves both encoder scales in every seed.
  • 7.1 RQ1: Effects of Document Context: Longer prompts do not reliably help zero-shot LLMs: Gemma and Qwen score lower with full-document context than with sentence-only input.Thus, document context benefits supervised encoders but is not a reliable substitute for task-specific supervision in zero-shot settings.
  • 7.2 RQ2: Effects of Retrieved Moral Knowledge: Retrieved moral knowledge improves macro-F1 in every matched early-fusion comparison, with gains ranging from .014 to .036 macro-F1.Gemma, Qwen, and Mistral improve under RAG for sentence, window, and document prompts, while DeBERTa-v3-base benefits most on average.
  • 7.3 RQ3: Model Family, Scale, and Fusion Strategy: DeBERTa-v3-base with document early-RAG is strongest at .314 macro-F1, above the best zero-shot LLMs at .241.Under this protocol, task supervision matters more than parameter count, and scale is not monotonic across either DeBERTa or LLMs.
  • 7.3 RQ3: Model Family, Scale, and Fusion Strategy: Early fusion is best for both DeBERTa scales when retrieval is held fixed, outperforming the tested alternative fusion strategies.The fusion comparison is encoder-side; late fusion and cross-attention are not evaluated for zero-shot LLM prompting.

8 Analysis

Context and retrieval help different kinds of value decisions: document context supports socially situated interpretations, while moral knowledge aids conceptual boundary decisions. Persistent difficulties include rare or subtle values, although larger instruction-tuned models use retrieval effectively for broad policy-facing categories.

  • Per-value analysis: Document context mainly helps infer HEDONISM, FACE, and TRADITION when sentence-level cues leave social or political motivations implicit.These values are difficult to interpret from isolated sentences naming an event or stance without its surrounding situation.
  • Per-value analysis: Retrieved moral knowledge produces its largest encoder gains for BENEVOLENCE: CARING, STIMULATION, FACE, SECURITY: PERSONAL, and UNIVERSALISM: TOLERANCE.Retrieval helps conceptual boundary decisions when the same sentence plausibly supports multiple value frames.
  • Per-value analysis: FACE benefits from both document context and retrieved knowledge because detection requires identifying the social situation and relevant value definition.This combined benefit reflects complementary contextual and conceptual information.
  • Per-value analysis: The best aggregate score for HUMILITY, SELF-DIRECTION: THOUGHT, and CONFORMITY: INTERPERSONAL remains below .18 F1.These values remain difficult because of rarity and subtle distinctions between ideas, actions, and social harm; model-family changes alter errors without eliminating the difficulty.
  • Prediction-change analysis: DeBERTa changes about 3.5–5.7% of sentence-level label sets across context contrasts, compared with about 5.1–12.2% for zero-shot LLMs.The prediction changes are targeted rather than wholesale interventions.

9 Discussion · 10 Conclusion

The discussion and conclusion establish that added context and moral knowledge help value detection conditionally, depending on model, task, and label ambiguity. They recommend supervised encoders with deliberate context selection and inspectable early-fusion knowledge as a conservative default.

  • 9 Discussion: For supervised encoders, document context and early-fusion moral knowledge are complementary: context recovers political framing, while value descriptions separate neighboring labels.For zero-shot LLMs, retrieved knowledge is more reliable than simply adding longer document prompts.
  • 9 Discussion: DeBERTa-v3-large does not consistently improve over DeBERTa-v3-base, and larger instruction-tuned LLMs do not outperform supervised encoders in zero-shot mode.The results caution against treating scale as a substitute for task design.
  • 9 Discussion: A conservative practical default is to start with a supervised encoder, choose document context carefully, and add moral knowledge when label boundaries are ambiguous.This setup is cheaper to train and run than 70B–123B zero-shot LLMs, easier to reproduce across seeds, and easier to inspect because retrieved chunks are visible.
  • 9 Discussion: LLMs remain useful as complementary systems, particularly for stress-testing label definitions, rather than as automatic replacements for supervised encoders.Their role is complementary within the practical recommendation, not guaranteed by parameter scaling.
  • 10 Conclusion: Simple early-fusion retrieval outperforms the tested late-fusion and cross-attention variants, supporting simpler architectures for encoder-based value detection.The conclusion frames early-fusion moral knowledge as a useful addition under this protocol.
  • 10 Conclusion: Additional information helps only when the model can use it for the relevant label decision, so context and knowledge should be evaluated jointly with model family.Full-document context benefits supervised encoders, while early-fusion moral knowledge is broadly useful under the study protocol.
  • 10 Conclusion: Per-value evaluation is necessary because aggregate macro-F1 can conceal which values are helped, harmed, or unresolved.The conclusion recommends evaluating individual values alongside aggregate performance.

Limitations · Ethical Considerations

The study’s conclusions are bounded by its English political-text benchmark, fixed moral knowledge base, zero-shot LLM design, and non-exhaustive ablations. Ethical use requires treating predictions as inspectable analytical signals for aggregate research, not individual profiling or high-stakes decisions.

  • Limitations: The benchmark covers one broad genre, so findings may not transfer directly to other domains, languages, or communicative settings.Multilingual transfer and language-specific value framing remain open questions.
  • Limitations: The experiments use English task formulations and knowledge-base entries, limiting direct conclusions about multilingual value framing.The paper specifically identifies multilingual transfer and language-specific framing as future questions.
  • Limitations: The manually constructed moral KB improves interpretability but makes results depend on its coverage, wording, chunking, and retrieval configuration.Different retrieval models or automatically generated value explanations could produce different RAG behaviors.
  • Limitations: The fixed top-k retrieval setup is not optimized separately for each model or context condition.This design choice may constrain retrieval behavior across experimental conditions.
  • Limitations: Zero-shot LLM experiments do not establish an upper bound, because few-shot prompting, calibration, instruction tuning, or supervised fine-tuning could change model rankings.Some large-model runs also require quantization or multi-GPU execution in practice.
  • Limitations: The architecture ablations are not exhaustive, and late fusion or cross-attention could require further tuning or alternative retrieval representations.Per-value results are also noisy for rare labels, especially HUMILITY, so broad patterns receive emphasis.
  • Ethical Considerations: Model outputs should be treated as analytical signals requiring human interpretation, not definitive labels about political actors, groups, or communities.Context and retrieved knowledge can both correct and introduce mistakes.
  • Ethical Considerations: The systems should not be used for individual profiling, automated moderation, surveillance, or high-stakes decisions, especially concerning minority or contested political positions.Appropriate use is limited to transparent, auditable aggregate research settings with inspectable retrieval outputs.

A Data and Code Availability … D Reproducibility Details

The paper provides reproducibility materials while respecting benchmark and model licensing constraints. It also specifies the 19-value prediction setup, standardized zero-shot prompting, and benchmark-access assumptions.

  • A Data and Code Availability: The authors release source code, configurations, run scripts, prompt templates, retrieval files, environment and artifact documentation, and analysis scripts.These materials cover training, inference, table construction, and qualitative examples.
  • A Data and Code Availability: Raw benchmark texts are not redistributed, and restricted predictions or qualitative artifacts are replaced with regeneration scripts and configurations.Aggregate results, permitted prediction files, thresholds, and eligible model bundles are also released.
  • B Schwartz 19-Value Taxonomy: The task predicts 19 refined Schwartz values independently as binary multi-label targets, while higher-order regions serve interpretability only.Figure 3 provides an orientation map of the value taxonomy.
  • C Zero-shot LLM Prompt Template: All zero-shot LLM conditions use one prompt structure, with retrieval inserting an optional EXTERNAL KNOWLEDGE block before the input text.Model-specific chat templates wrap the same user prompt without changing its text.
  • D Reproducibility Details: Reproducing the reported experiments requires access to the official benchmark data under its original terms.Table 7 summarizes the main reproducibility settings.
  • D Reproducibility Details: The released resources include a public code repository and, where permitted, a Hugging Face model bundle for the best-performing configuration.Large instruction-tuned LLMs are subject to more limited release provisions.

E Complete Test Results · F Per-Value Results

The complete test results provide aggregated evidence across supervised and zero-shot conditions, while the per-value results support the RQ4 analysis using matched DeBERTa-v3-base comparisons and best observed per-value F1 scores.

  • E Complete Test Results: Table 8 reports the full set of aggregated test results used in the main analysis.
  • E Complete Test Results: DeBERTa results are reported as mean±standard deviation across three fine-tuning seeds.
  • E Complete Test Results: Zero-shot LLM results are based on one completed inference run per condition.
  • F Per-Value Results: Table 9 reports the full per-value evidence used for RQ4.
  • F Per-Value Results: The document and knowledge columns use DeBERTa-v3-base to match the compact RQ4 analysis.
  • F Per-Value Results: The best-F1 column reports the highest mean per-value F1 across all tested model, context, and RAG conditions.

G Qualitative Examples

Table 10 presents representative qualitative examples supporting Section 8, combining supervised DeBERTa context/RAG comparisons, zero-shot LLM RAG comparisons, and failure cases. The examples use paraphrased target sentences and prediction changes rather than verbatim dataset text.

  • G Qualitative Examples: The examples are sampled from qualitative bundles and prediction-change summaries generated by the released analysis scripts.These materials support reproducibility of the qualitative analysis.
  • G Qualitative Examples: Full document contexts can be regenerated from the official data using the released scripts.The table avoids reproducing full contexts while retaining sentence identifiers for reconstruction.
  • G Qualitative Examples: Table 10 covers six qualitative rows: two supervised DeBERTa context/RAG comparisons, two zero-shot LLM RAG comparisons, and two failure-case examples.The table identifies prediction changes as baseline-versus-comparison outputs within the corresponding qualitative bundles.
  • G Qualitative Examples: Target sentences are paraphrased rather than quoted verbatim to comply with the dataset usage agreement.Table 10 provides paraphrased target descriptions and sentence identifiers instead of reproducing full document contexts.
Loading 2605.22641v3…