PaperlayerView exact revision on arXiv

Source-linked AI summary

From Normative Frameworks to Alignment Data: Constructing and Evaluating SFT and Preference Data

Husrev Taha Sencar, Rezart Beka, Danish Naeem, Seda Ozalkan, Majd Hawasly, Ji Lucas, Ala AlFuqaha, Mohamed Abdallah, Recep Senturk

arXiv:2609.35201v1cs.CLcs.AI

TL;DR

Aligning models to a specified normative framework requires translating abstract principles into learnable examples and preference signals. This paper constructs Arabic–English data through expert curation and evaluates it in controlled post-training experiments, finding that SFT improves targeted behavior while preference data adds a smaller, inconclusive benefit.

  • Problem

    Constructing alignment data for a specified normative framework is challenging because appropriate behavior can vary across communities and nuanced frameworks require expertise to address ambiguity and disagreement.

    Source: §1 · pp. 2–3

  • Method

    Seven experts developed Arabic–English SFT demonstrations and preference pairs by probing models, curating responses, and deriving preferences from expert judgments.

    Source: §1 · p. 3, Preference Construction · p. 8

  • Results

    Controlled experiments found that curated SFT improved behavior with respect to the target framework, while preference data added a smaller, statistically inconclusive benefit and standard benchmarks showed no broad capability degradation.

    Source: §7 · p. 23

  • Takeaways & Limitations

    Expert-defined principles can be operationalized as alignment data and evaluated through controlled model training.

    Source: §7 · p. 23

    Results are limited to the studied model family, scale, post-training mixture, and training recipe, and verbosity may have influenced evaluator preferences.

    Source: §6 · p. 22

Abstract

from arXiv · show

Aligning language models with a specified normative framework requires translating abstract principles into concrete examples and preference signals from which models can learn. We present an expert-driven methodology for constructing such alignment data and apply it to a normative framework grounded in Islamic ethical, theological, and jurisprudential traditions. Over approximately one year, seven domain experts systematically probed language models to identify alignment deficiencies, curated desired responses, and constructed preference pairs from model outputs and expert judgments. The resulting Arabic-English datasets contain approximately 2.8K supervised fine-tuning (SFT) examples and 5.4K preference pairs spanning a broad range of normative domains. We evaluate the datasets through controlled post-training experiments comparing a Baseline model with models incorporating the curated SFT data alone and both the SFT and preference data. In blind expert evaluation on 150 separately constructed prompts, the model trained with the curated SFT data was preferred over the Baseline in 51.3% of assessor judgments, compared with 14.4% in the opposite direction (p < .001 at the prompt level). Adding the preference data resulted in a smaller difference, with the model trained with both datasets preferred over the SFT model in 28.0% of judgments versus 20.9% in the opposite direction; this difference was not statistically significant at the prompt level (p = .166). Standard Arabic and English benchmarks show no broad degradation in general-purpose capabilities. These results demonstrate how expert-defined normative principles can be systematically operationalized into alignment data and evaluated through controlled model training.

1 Introduction

Aligning a model to a particular normative framework requires turning abstract principles into learnable examples and preferences, especially where communities disagree about appropriate behavior. This paper introduces NADA and evaluates its SFT and preference data through controlled training experiments and expert assessment.

1 Introduction

  • The normative alignment challenge is translating abstract objectives into concrete training examples and preference signals, particularly when appropriate behavior varies across communities.Domain-specific frameworks can require expertise to address ambiguity and disagreement, motivating systematic methods for dataset construction and validation.Source: §1 · pp. 2–3
  • NADA provides Arabic–English SFT demonstrations and preference pairs for a framework grounded in Islamic ethical, theological, and jurisprudential traditions.The datasets span social, political, legal, economic, medical, technological, cultural, and theological topics; they operationalize a defined framework rather than represent Muslim populations’ opinion distribution.Source: §1 · p. 3
  • Seven domain experts curated approximately 2.8K SFT pairs and 5.4K preference pairs over about one year by probing models and authoring, revising, or validating responses.The data capture expert judgments and model outputs, while the accompanying evaluation set contains 150 prompts.Source: §1 · p. 3
  • Controlled experiments compare a Baseline with models trained on SFT alone or SFT plus preference data, using blind expert judgments on unseen prompts and standard benchmarks.The design isolates the contributions of the SFT and preference datasets while assessing capability retention.Source: §1 · pp. 3–4

2 Alignment Data in the Model Training Lifecycle

Alignment occurs after pre-training has already shaped model behavior, and can also be influenced indirectly through data curation. This paper focuses on post-training, where demonstrations and preference judgments encode a specified normative framework.

2 Alignment Data in the Model Training Lifecycle

  • Pre-training gives models broad linguistic capabilities and world knowledge while shaping biases, assumptions, and behavioral tendencies, so alignment does not begin from a blank slate.The composition of pre-training data influences the resulting model’s behavior.Source: §2 · p. 4
  • Pre-training data selection, filtering, or reweighting can influence alignment, but massive corpora and unavailable foundation-model data constrain these interventions.Continued pre-training on curated corpora is another opportunity to adapt models to particular domains or languages.Source: §2 · p. 4
  • Post-training shapes behavior through SFT demonstrations and preference-based methods, which use alternative responses to optimize alignment objectives.When preferences come from heterogeneous populations, reducing potentially conflicting judgments to one learning signal may obscure differences in values and perspectives.Source: §2 · pp. 4–5
  • This study focuses on post-training datasets that operationalize an Islamic normative framework through supervised demonstrations and preference judgments.The paper situates these methods among other alignment interventions that can constrain or modify behavior at deployment time.Source: §2 · p. 5

3 Methodology

The methodology defines an Islamic normative framework and uses expert teams to discover deficient model responses, curate preferred answers, and derive preference pairs. Review and quality-control steps support the resulting bilingual data.

Defining the normative framework.

  • The framework guides responses to be accurate, ethically responsible, contextually grounded, and precise about legitimate disagreement without requiring a single prescription for every question.It also directs experts to distinguish descriptive claims from normative judgments, communicate clearly, and surface or reframe assumptions in prompts.Source: Defining the normative framework · pp. 5–6

Expert Curation Team.

  • Seven experts with relevant scholarly expertise worked in three independent groups, led by senior scholars who shaped guidelines and reviewed the data over approximately one year.The teams were organized to encourage perspective diversity and independent validation.Source: Expert Curation Team · p. 6

Prompt discovery.

  • Teams independently probed model responses and selected prompts when at least one response was unacceptable, using expert knowledge and targeted questions about normative distinctions.The interaction logs supported both SFT response authoring and later preference-pair construction; prompts were not intended to represent typical user queries.Source: Prompt discovery · p. 7

Response authoring.

  • Experts authored responses from scratch, refined model outputs, or adapted external sources according to the deficiencies and factual needs of each prompt.They aimed for explanatory, respectful answers with detail calibrated to question complexity, while balancing completeness and accessibility.Source: Response authoring · pp. 7–8

Preference Construction.

  • Preference pairs came from discovery-stage judgments by pairing preferred responses with rejected model outputs, or by comparing acceptable and rejected model responses directly.A prompt could yield multiple pairs, so preference data arose from the discovery workflow without a separate preference-annotation stage.Source: Preference Construction · p. 8

Quality Control.

  • Quality control combined team and cross-team review, auxiliary response-quality ratings, and exclusion of cases lacking sufficient expert agreement.The automated ratings assessed general qualities rather than normative alignment; translated samples were also reviewed for meaning preservation, and a final LLM editing pass was restricted to surface-level changes.Source: Quality Control · pp. 8–9

4 Dataset statistics

The SFT data cover 2,546 unique prompts across nine normative topics, with a large theological concentration and team-specific topical patterns. The preference collection contains 5,386 pairs, predominantly grounded in expert-authored accepted responses.

Dataset topic distribution

  • 1,072 of 2,546 unique SFT prompts (42.1%) concern Theological and Religious Issues, the largest of nine meta-topics.Political and Social Issues contribute 409 prompts (16.1%); the remaining categories each account for 10.4% or less.Source: §4 · p. 10, Table 3 · p. 11

Prompt distribution by curation team

  • The three curation teams show distinct topical emphases with substantial semantic overlap in the prompt embedding space.The 2,546 prompts are projected using multilingual-e5-large; the figure colors prompts by originating team and highlights each team against the full distribution.Source: §4 · pp. 10–12, Figure 1 · p. 11
  • Team membership is associated with local semantic clustering: 79.6% of prompts’ ten nearest neighbors share their team, against 39.4% expected under random mixing.The low silhouette score of 0.04 indicates that the teams do not form sharply separated semantic clusters.Source: §4 · pp. 11–12

Preference dataset

  • The preference dataset contains 5,386 pairs from 2,243 unique prompts, with 91.0% of pairs using an expert-authored response as the accepted answer.There are 2,357 unique accepted responses and 5,329 unique rejected responses; 95.8% of prompts have exactly one accepted response, while rejected responses are more varied.Source: §4 · p. 12, Table 4 · p. 12

5 Assessment

The assessment compared post-trained models on held-out prompts through blind expert judgments and standard Arabic and English benchmarks. Curated SFT data improved expert preference over the Baseline, while adding preference data produced a directionally positive but statistically inconclusive difference over SFT alone.

5.1 Evaluation Set

  • The 150 evaluation prompts were separately authored and unused in training; they occupied training-data regions while remaining mostly new questions on covered topics.No prompt appeared verbatim in training; six (4%) were judged close rephrasings, with a maximum estimated two-point win-rate effect if excluded.Source: §5.1 · p. 13, Figure 2 · p. 14

5.2 Trained Models

  • The three-model comparison isolated curated SFT data and the incremental preference data within a full post-training pipeline; experts assessed anonymized responses in randomized order.The curated datasets added approximately 2.8K instruction-response pairs and 5.4K preference pairs to much larger existing training corpora.Source: §5.2 · pp. 14–15, Comparisons · p. 15, Evaluation protocol · p. 15

Inter-annotator agreement.

Effect of curated SFT data (Baseline vs. SFT-Only).

Effect of full curated pipeline (Baseline vs. SFT+DPO).

Response length.

  • 340 (SFT-Only) and 335 (SFT+DPO) mean words versus 152 (Baseline); response length correlated with wins, but may partly reflect required contextualization.The authors note that verbosity independent of alignment quality cannot be ruled out; the two curated models were nearly identical in length.Source: Response length · p. 19, Table 8 · p. 20

5.5 Observed Alignment Failure Modes

  • Experts observed recurring shortcomings: missing context or distinctions, mishandled scholarly disagreement, normative evasion, and inappropriate framing; these were not claimed across all responses.The framing issue included applying an Islamic frame when the question was general or adopting assumptions inconsistent with the target framework.Source: §5.5 · pp. 20–21

5.6 Benchmark Evaluation

  • Across Arabic and English benchmarks, curated models remained broadly comparable to Baseline, with task-dependent gains and losses; PalmX Culture improved modestly while PalmX Islamic remained comparable.These knowledge benchmarks provide complementary rather than direct evidence of adherence to the target normative framework.Source: §5.6 · p. 21

6 Limitations

The study’s limitations concern the scope and construction of its normative framework and evaluation, as well as translation and experimental conditions. Its results also reflect variation in expert judgments and possible response-length effects.

6 Limitations

  • The framework reflects a particular expert-developed perspective, not the distribution of beliefs or preferences across Muslim populations.The curation team comprised seven domain experts, and different expert compositions could yield different judgments on some questions.Source: §6 · pp. 21–22
  • Dataset-construction prompts were selected to expose alignment deficiencies, not to represent naturally occurring user queries.The 150 evaluation prompts were constructed separately and were not selected through model-failure probing, but the evaluation was limited in size and conducted by three senior framework developers.Source: §6 · p. 22
  • LLM-based translation of the Arabic–English parallel data may introduce subtle meaning changes despite attention to canonical references and religious sources.Source: §6 · p. 22
  • The experimental results are limited to the model family, scale, post-training mixture, and training recipe studied.Source: §6 · p. 22
  • Low prompt-level inter-annotator agreement and longer curated-model responses constrain interpretation of evaluator preferences.Response length was associated with preference, so the evaluation cannot fully separate contextualization and explanation from a possible independent effect of verbosity.Source: §6 · p. 22

7 Conclusion

The paper presents an expert-driven method for turning a specified normative framework into Arabic–English alignment data. Controlled experiments find substantial improvement with SFT data, while the incremental benefit of preference data is smaller and statistically inconclusive.

7 Conclusion

  • The expert-driven methodology produced Arabic–English datasets of approximately 2.8K SFT examples and 5.4K preference pairs.The data were constructed through expert prompt discovery, response curation, and preference judgments.Source: §7 · p. 23
  • Controlled post-training experiments show substantial improvement from curated SFT data, while preference data adds a smaller, statistically inconclusive benefit.Source: §7 · p. 23
  • The paper argues that operationalizing abstract principles requires specifying their application to concrete questions, legitimate disagreement, and preferred responses.The authors present their methodology as a potential foundation for systematically operationalizing and evaluating other specified normative frameworks.Source: §7 · p. 23

Appendix A Prompt Discovery and Evaluation Platform

The curation platform supported expert prompt discovery and response comparison, while keeping prompts, judgments, metadata, and curated responses in a linked workflow. Annotators compared randomized model outputs, rated them, and could provide curated responses.

Appendix A Prompt Discovery and Evaluation Platform

  • The custom platform supported prompt submission, response comparison, preference recording, metadata management, user management, model deployment, and response generation.Source: Appendix A · p. 23
  • Prompts, responses, preference annotations, and metadata were linked in one data-management workflow, while curated responses were authored separately and linked back to prompt records.Source: Appendix A · pp. 23–24
  • Annotators viewed multiple model responses side by side in randomized order, assigned one of three preference labels, and could author or edit a curated response.The platform logged these interactions for constructing SFT and preference datasets; Figure A1 depicts the annotation interface.Source: Appendix A · p. 24, Figure A1 · p. 24

Appendix B Detailed Preference Pair Statistics

The appendix details how accepted and rejected responses were distributed across preference-dataset prompts. Most prompts followed the intended annotation design, while the 4+ category reflects questions revisited with different model sets.

Appendix B Detailed Preference Pair Statistics

  • Most prompts followed the intended design of one accepted and two to three rejected responses.Source: Appendix B · p. 24
  • Prompts in the 4+ category represent questions revisited under a different set of available models.The passage refers readers to Section 3 for discussion of these cases.Source: Appendix B · p. 24
  • Table B1 breaks down the number of accepted and rejected responses per prompt in the preference dataset.Source: Table B1 · p. 25

Appendix C Training Infrastructure and Parameters

Post-training ran on two compute nodes, each with eight NVIDIA H100 GPUs; Table C2 summarizes the hyperparameters for each post-training stage.

Appendix C Training Infrastructure and Parameters

Appendix D Pairwise Response Evaluation Guidelines

The guidelines use blind, pairwise review of model responses to user questions, with judgments guided by prioritized evaluation criteria and explicit rules for ties, failures, and critical errors.

Appendix D Pairwise Response Evaluation Guidelines

  • Assessors reviewed every question-response pair across three comparison files, with model identities hidden and responses labeled A and B.Source: Appendix D · pp. 25–26
  • Each comparison file contained 150 assessments, for 450 assessments across all three files.Source: Appendix D · pp. 25–26
  • The prioritized criteria were factual correctness, framework consistency, prompt faithfulness, completeness, reasoning quality, and overall usefulness; acceptable alternative interpretations were not penalized.Source: Appendix D · pp. 26–27
  • Judgments were A better, B better, tie, or both fail, with ties for comparably high-quality responses and both-fail reserved for substantive deficiencies.Source: Appendix D · p. 27
  • Fabricated citations, quotations, references, studies, or statistics were designated critical errors that generally warranted a less favorable judgment.Source: Appendix D · p. 27
  • Comments were reserved for exceptional issues not captured by the labels and were not to replace the assigned judgment.Source: Appendix D · p. 27

Appendix E Detailed Pairwise Evaluation Results

The appendix reports per-assessor results for three pairwise model comparisons, with the SFT-Only advantage over Baseline significant for all team leads and a less uniform comparison between SFT-Only and SFT+DPO.

E.1 Baseline vs. SFT-Only

  • The SFT-Only advantage over Baseline was significant for all three team leads, with the strongest effect on Team 1 prompts across assessors.Source: §E.1 · p. 28, Table E3 · p. 28

E.2 Baseline vs. SFT+DPO

  • For Baseline versus SFT+DPO, the pattern mirrored the SFT-Only comparison, while Team 1 assigned notably more both-fail judgments, particularly on Team 3 prompts.Source: §E.2 · p. 28, Table E4 · p. 29

E.3 SFT-Only vs. SFT+DPO

  • In the direct SFT-Only versus SFT+DPO comparison, only Team 2 found a significant overall advantage for SFT+DPO, while Team 1 observed virtually no difference.Source: §E.3 · p. 29

Appendix F Assessor Bias Analysis

Outcome distributions differed between assessors’ own prompts and others’ prompts for all three team leads, but the analysis does not identify whether prompt-set characteristics, assessor judgment, or their interaction accounts for these differences.

Appendix F Assessor Bias Analysis

  • p < .001 for Teams 1 and 3 and p = .002 for Team 2 indicate own-versus-others prompt outcome distributions differed significantly for all three team leads.Source: Appendix F · p. 29
  • Team 1’s curated-model win rate was 50.0% on own prompts versus 20.7% on others’ prompts, while Teams 2 and 3 had higher both-fail rates on own prompts.Team 2’s curated-model wins were 45.0% on own prompts versus 61.9% on others’; Team 3’s both-fail rates were 22.0% versus 4.0%, concentrated in SFT-Only versus SFT+DPO (χ2(3) = 31.52, p < .001).Source: Appendix F · pp. 29–30
  • The analysis cannot distinguish whether the authorship-associated outcome differences reflect prompt-set characteristics, assessor judgment, or interactions between them.Source: Appendix F · p. 30
Loading 2609.35201v1…