Enabling Classifiers to Make Judgements Explicitly Aligned with Human Values

1 Introduction

• Human values are non-consensual and culturally variable, so identical content may be judged sexist or non-sexist depending on the values supplied.Figure 1 illustrates this variation in predictions for the same content.Source: §1 · p. 1, Figure 1 · p. 1
• The proposed task separates value definition from model development, enabling predictions based on explicitly provided values rather than one fixed value set.The authors present the approach as supporting more inclusive and explainable value-aligned NLP.Source: §1 · pp. 1–2
• VA-MODELs distill value-based training data from LLMs through prompting, then fine-tune smaller classifiers on that data.The approach is reported as more stable and accurate than directly applying few-shot learning to LLMs.Source: §1 · p. 2
• The authors report that their approach significantly outperforms strong baselines, including LLM few-shot learning and existing text-augmentation methods.They also investigate prompt-generation factors, data quality and size, LLM scale and type, and generalization to unseen value sets.Source: §1 · p. 2

Human Value Alignment

• The paper separates value definition from alignment, arguing that values should be externally defined because they vary across cultures.Prior approaches may intertwine value definition and alignment in an iterative process (Solaiman and Dennison, 2021).Source: §2 · p. 2

Prompt-based Learning

• Prompt-based learning feeds a model examples in a prompt without fine-tuning, and the paper draws on prompt-construction literature (Brown et al., 2020; Chowdhery et al., 2022; Lu et al., 2021; Reynolds and McDonell, 2021; Zhao et al., 2021; Schick and Schütze, 2020).Source: §2 · p. 2

Knowledge Distillation

• Unlike distilling general language-model abilities, the paper transfers value-specific knowledge through generated training data for smaller value-aligned classifiers.This is intended to reduce human-labeling costs and support smaller models specialized for value-aligned judgment.Source: §2 · pp. 2–3

3.1 Task Description

• The task predicts a judgment YV for content C given an explicit human value V, which is defined externally by a user or relevant literature.Here, “value” means qualities, standards of behavior, or beliefs held by individuals or societies, expressed in natural-language phrases or sentences.Source: §3.1 · p. 3
• Unlike conventional classifiers that learn fixed definitions from labeled data, this task requires predictions to change with the supplied values for the same content.Value definition is separated from development of the value-aligned model.Source: §3.1 · p. 3

3.2 Value-aligned Sexism Classification

• The sexism case study predicts sexist, non-sexist, or not applicable (NA), reflecting that sexism definitions evolve and vary across cultures.The authors select sexism because its judgments involve religious, cultural, and personal beliefs and values, and because fine-grained understanding has been emphasized (Parikh et al., 2019).Source: §3.2 · p. 3

4 Methodology

• The two-stage method prompts LLMs with explicit values and instructions to generate training data, then fine-tunes smaller language models on the generated examples.The method is motivated by the lack of existing value-aligned classification training resources and the instability of direct zero-shot LLM use.Source: §4 · p. 3

4.1 Value-Aligned Knowledge Distillation: Prompt-based Data Generation

• Prompt-based generation uses few-shot examples, and the authors select a prompt template by testing five candidates on a smaller sample set.They create a pool of 10 human-labeled samples per value and randomly select and order five for prompts; the prompt construction choices draw on prior work (Lu et al., 2021; Reynolds and McDonell, 2021; Zhao et al., 2021; Yoo et al., 2021).Source: §4.1 · p. 4
• The generation step is model-agnostic, with OPT-175B used in the main experiment and LLM scale and type examined in additional analyses.Source: §4.1 · p. 4
• Generated samples are extracted by pattern matching, deduplicated, stripped of prompt examples, and filtered when shorter than three words.Source: §4.1 · p. 4

4.2 Fine-tuning Smaller Models – Value-Aligned Models

• Smaller transformer classifiers are fine-tuned with value and content formatted as “value [sep] content [sep]” to predict value-aligned judgments.The proposed classifier adds a linear layer over the smaller model’s pooled output; predictions may differ for the same content under different values.Source: §4.2 · p. 4

5 Experiments

• The task labels content as sexist, non-sexist, or NA depending on explicitly provided values.Source: §5 · p. 4

5.1 Dataset

• The dataset uses 19 sexism categories with opposing value and counter-value definitions, and creates 17,720 value-dependent test samples from human-labeled content.Ten items per category were reserved for prompt construction; test samples pair the same content with relevant, counter-, or unrelated values.Source: §5.1 · pp. 4–5

5.2.1 VA-MODELs (Ours)

• VA-ALBERT, VA-ROBERTA, and VA-BART are smaller models fine-tuned on LLM-generated value-aligned data.Generated content is paired with values and counter-values; irrelevant values create NA examples, for 10,722 samples including prompt-construction samples.Source: §5.2.1 · p. 5

5.2.2 Baselines

• Baselines include OPT-175B few-shot prediction and transformer models fine-tuned on human-labeled or semantically augmented data.The fine-tuned baselines use ALBERT, RoBERTa, and BART backbones.Source: §5.2.2 · p. 5

5.3 Experimental setup

• Evaluation reports accuracy, weighted F1-score, precision, and recall; training-data generation uses OPT-175B with top-p 0.7 and temperature 1.Table 1 reports percentage scores averaged over five random seeds and notes statistically significant VA-MODEL performance (t-test p-value < 0.05).Source: §5.3 · pp. 5–6, Table 1 · p. 6

6.1 Main Results

• 15.56∼40.83% W-F1 gain over baselines; VA-ALBERT also surpasses baselines backed by larger models, including Nlp-RoBERTa and HL-RoBERTa.Source: §6.1 · p. 6
• OPT-175B few-shot performs better than random labels and HL-ALBERT, but worse than or comparably to other baselines; Nlpaug raises W-F1 over HL-Models with small accuracy drops.Source: §6.1 · p. 6
• Menstruation-related Discrimination and Pay Gap score above 90%, Internalized Sexism is relatively low, and the nine middle categories are around 80% in Acc. and W-F1.Source: §6.1 · p. 6, Figure 3 · p. 6

6.2 Quality Analysis for Generated Value-Aligned Training Data

• 51.79% overall vocabulary overlap with the test set, but only 0.01% of generated samples exceed 80% overlap with any test sample; category overlaps are generally small and lower for high-performing categories.Source: §6.2 · p. 7
• Human ratings found R&S in 69.44% of Pay gap samples versus 25% of Internalized Sexism samples, consistent with the higher per-category performance for Pay Gap.Source: §6.1 · p. 6, §6.2 · p. 7
• Without human-labeled data, VA-BART and VA-RoBERTa lose 1.65% and 0.63% W-F1 respectively, while VA-ALBERT gains minimally on accuracy and W-F1.Source: §6.2 · p. 7, Table 2 · p. 7

6.3 Generalization Ability on Unseen Values

• On unseen values, all VA-MODELs still outperform baselines, with maximum drops of 17.22% for VA-RoBERTa versus 43.18% for Nlpaug-RoBERTa.Source: §6.3 · pp. 7–8, Table 3 · p. 8

6.4 Ablation Studies

• Larger LLMs continually improve final performance, similarly sized OPT and GPT-Jurassic models differ by only 0.12%, and more than 200 generated samples bring no W-F1 gain except for VA-ALBERT.Source: §6.4 · p. 8, Table 4 · p. 8, Figure 5 · p. 8

7 Conclusion and Future Work

• VA-MODEL generally outperforms strong baselines using LLM-generated value-aligned data to fine-tune smaller classifiers.The approach generates training data from LLMs with prompt-based data generation, then fine-tunes smaller classification models.Source: §7 · p. 8
• Larger LLMs’ generated data helps performance, but too much generated data can reduce it.The paper also reports promising generalization ability for VA-MODEL.Source: §7 · p. 9
• Future work should improve robustness across diverse values, generalization, and generated-data quality through more human curation.Source: §7 · p. 9

Limitations

• The methodology has been tested only in English; applying it elsewhere may depend on LLM capacity and available human value definitions.The authors identify evaluation in other languages as future work.Source: Limitations · p. 9
• Main experimental results use a 175B-parameter model requiring large GPU resources or API access, although 1B and 6B models achieved comparable results.The smaller models may be alternatives for researchers without access to 100B+ models.Source: Limitations · p. 9
• The feasibility study covers sexism alone; the authors call for expansion to racism, toxicity, and other value-aligned classification tasks.Source: Limitations · p. 9

Ethics Statement

• The authors say unconventional and counter-contemporary values in the sexism study describe differing perspectives, not tolerance of sexist behavior or beliefs.Source: Ethics Statement · p. 9
• Value definition should be conducted with society and relevant experts, separately from the model mechanics implemented by engineers.This separation is intended to keep engineers from defining values directly in training data or code.Source: Ethics Statement · p. 9

A.1 Value and Counter Value

• The appendix enumerates 19 sexism categories, each paired with two corresponding values.The full value and counter-value list is provided in Table 6.Source: §A.1 · p. 11

B Prompt Template Choice for Value-aligned Data Generation

• The final generation prompt was selected using OPT-175B experiments; it uses a value and label prompt with five randomly selected examples from ten manually created examples.The prompt asks the model to generate content relevant to the specified value and label.Source: B Prompt Template Choice for… · p. 11
• GPT-Jurassic 6B performed better with prompt template #2, unlike OPT-175B, suggesting prompt-template construction may need model-specific exploration.The authors suggest the difference may relate to distinct training objectives and pre-training resources.Source: B Prompt Template Choice for… · p. 12

C Experimental Details

• Hyperparameters were selected by grid search over learning rates 1e-5 and 5e-5 and batch sizes 32 and 64; runs used early stopping and five random seeds.Training lasted at most 30 epochs with patience 5 on an Nvidia RTX 3090; each epoch took around 2–10 minutes.Source: C Experimental Details · p. 12
• The OPT-175B few-shot baseline used five randomly selected examples per label, generated labels five times at top-p 0.9, and averaged the scores.The prompt is constructed for each test sample from task instructions and examples.Source: C Experimental Details · p. 12

D Vocabulary overlaps of generated training data among sexism categories

• The vocabulary-overlap analysis computes overlap for each sexism category in generated training data.Figure 6 presents these category-level overlaps; the supplied text does not state a result.Source: D Vocabulary overlaps of generated… · p. 12

• Figure 7 presents VA-BART evaluation results for each sexism category on the test set.Source: E Per Category Results · p. 12, Figure 7 · p. 14
• Table 6 lists the values and counter-values for each sexism category.Source: Table 6 · p. 13
• Figure 6 shows vocabulary overlaps among sexism categories, ordered by descending W-F1 from top to bottom and left to right.Source: Figure 6 · p. 14