PaperlayerView exact revision on arXiv

Source-linked AI summary

ContextAdapt: Evaluating Contextual Adaptation and Value Alignment in LLMs

Olivia Macmillan-Scott, Mirco Musolesi

arXiv:2609.38260v1cs.CLcs.AI

TL;DR

Whether an LLM applies a value appropriately can depend on professional context, yet models must also remain consistent when context does not change the governing norm. CONTEXTADAPT evaluates these capabilities across professional domains using scenarios grounded in documented standards. Models show high mean appropriateness, but professional grounding varies and some stake changes produce large behavioural shifts despite unchanged obligations.

  • Problem

    Abstract values may imply different appropriate behaviours across professional settings, and it remains important to test whether models adapt when professional norms change while staying consistent when they do not.

    Source: §1 · p. 1, §5 · p. 8

  • Method

    CONTEXTADAPT derives value operationalisations from professional sources and evaluates 12 LLMs on actions and stated justifications across scenarios testing default rules, exceptions, and contextual manipulations.

    Source: §1 · p. 2, §3 · p. 3, Stated justification · p. 5

  • Results

    Models are highly appropriate on average, but professional grounding varies markedly, and stake changes can produce large behavioural shifts despite unchanged professional obligations.

    Source: §5 · pp. 7–8, §7 · p. 9

  • Takeaways & Limitations

    Value-alignment evaluations should test both adaptation when context changes the applicable norm and stability when it does not.

    Source: §5 · p. 9

    The benchmark draws on a limited, predominantly UK, US, and international source set, has incomplete scenario coverage, and does not represent context-independent moral ground truth.

    Source: §6 · p. 9

Abstract

from arXiv · show

Values such as honesty, autonomy, and confidentiality are often regarded as general principles underpinning AI alignment. However, what it means to act in accordance with these values can depend on the context in which a decision is made. In this paper, we ask whether large language models (LLMs) appropriately adapt the application of a value across professional settings, while remaining consistent when contextual changes do not alter the relevant professional norm. To study this, we introduce ContextAdapt, an evaluation framework covering honesty, autonomy, and confidentiality across medicine, law, finance, and national security. Drawing on primary-source professional and regulatory documents, we construct a value x domain framework and use this to develop scenarios testing both default professional rules and recognised exceptions. We evaluate 12 LLMs on both the actions they recommend and the justifications they provide. In our main experiment, models achieve 95.6% mean appropriateness, although the use of the correct domain-specific justification varies substantially across models, from 25.6% to 76.9%. In a separate factorial experiment, explicitly naming the professional domain and changing the role of the model have limited effect on behaviour. Varying stakes, however, reveals severe but localised failures: in some cases, models alter their responses even though the underlying professional obligation remains unchanged. In particular, perceived severity appears to act as a cue for disclosure across both honesty and confidentiality scenarios. These results show that evaluating value alignment requires us to consider not only whether models follow abstract principles, but whether they apply them appropriately across different contexts.

1 INTRODUCTION

Abstract values do not always prescribe the same behaviour across professional settings: norms can change with roles, relationships, institutions, and circumstances. CONTEXTADAPT tests whether models adapt a value when the governing norm changes and remain consistent when it does not.

1 INTRODUCTION

  • CONTEXTADAPT evaluates honesty, autonomy, and confidentiality across medicine, law, finance, and national security, testing actions and their stated justifications in two experiments.The main dataset tests professionally grounded applications under fixed conditions; the secondary dataset manipulates stakes, agent role, and whether the domain is named.Source: §1 · p. 2
  • 95.6% mean appropriateness on the main dataset contrasts with correct domain-specific justification rates of 25.6%–76.9% across models.In the secondary experiment, domain labels and agent role have limited effect, while stakes produce severe but localised failures despite unchanged professional rules.Source: §1 · p. 2
  • Value alignment evaluation must test whether models apply values appropriately across contexts, not only whether they follow abstract principles.The results distinguish appropriate behaviour, professional normative grounding, and sensitivity to contextual changes.Source: §1 · pp. 2–3

2 RELATED WORK

Prior value-alignment work studies how models express or prioritize values, while work on situated norms examines how appropriate behaviour depends on social context. This paper focuses on documented professional norms and whether a fixed value is operationalised appropriately across professional settings.

2 RELATED WORK

  • ValueCompass studies how human–LLM value alignment and value priorities vary across application contexts; this paper instead holds the nominal value fixed and tests its professional operationalisation.Other benchmarks examine everyday moral judgement, broader value orientations, or pluralistic alignment.Source: §2 · p. 3
  • Context-sensitive norm research represents norms relative to social roles and settings, while this work uses formal professional standards as its reference.This permits testing whether changes in model behaviour correspond to changes in the applicable professional norm.Source: §2 · p. 3
  • Research on moral exceptions and value conflicts examines exception recognition or prioritisation among competing values; this paper tests contextual application of one value.Related approaches include MoralExceptQA, case-based alignment, Value Kaleidoscope, and ConflictScope.Source: §2 · p. 3

3 METHODS

CONTEXTADAPT uses professional standards to define value operationalisations and construct scenarios testing default rules and documented exceptions. It evaluates model actions and stated justifications across a main dataset and a secondary set of contextual manipulations.

3 METHODS

  • The four-stage framework derives professional value operationalisations, constructs scenarios, evaluates 12 LLMs in two experiments, and assesses recommended actions and stated justifications.The justification captures the normative framework expressed in a response, not necessarily the model’s internal reasoning.Source: §3 · pp. 3–4, Stated justification · p. 5

3.1 PROFESSIONAL NORM CORPUS

  • The reference corpus contains 24 primary-source documents across four domains and records each norm’s default application, target, and documented override conditions.The corpus is not treated as context-independent moral ground truth; appropriateness means correspondence to the professional operationalisation represented in the sources.Source: §3.1 · p. 4

3.2 SCENARIO CONSTRUCTION

  • Domain-agnostic templates generate matched scenarios while preserving relevant relationships, information asymmetries, authority conflicts, or decisions where possible.Base scenarios test default rules; exception scenarios test documented departures when conditions are met, unmet, or near the boundary.Source: §3.2 · p. 4

3.3 EXPERIMENTAL DATASETS

  • The main dataset tests professional applications under fixed contextual conditions; the secondary dataset varies stakes, agent role, and domain-label explicitness.The secondary dataset has 144 scenarios, with low- and high-stakes variants kept on the same side of the relevant professional boundary.Source: §3.3 · pp. 4–5

3.4 MODELS AND RESPONSE GENERATION

  • The evaluation includes 12 LLMs from several model families, with 10 independent inferences sampled per model and scenario.Command A+ was included only in the main dataset because monthly API limits prevented its secondary-experiment run.Source: §3.4 · p. 5

3.5 RESPONSE EVALUATION

  • Responses are assessed for professional appropriateness and the normative framework stated to justify the action.Appropriateness labels map to 1, 0.5, and 0; stated justifications are classified as domain-specific, domain-neutral, generic, uncertain, or absent.Source: §3.5 · p. 5, Stated justification · p. 5

3.5.1 LLM JUDGE PANEL

  • Three LLM judges independently apply a shared coding manual, with median-label consensus for appropriateness and majority vote for justification categories.Judges are blinded to the generating model’s identity.Source: §3.5.1 · p. 5

3.6 ANALYSIS

  • The analysis examines domain–value appropriateness, justification types, exception under- and over-triggering, and effects of domain labels, role, and stakes.For national-security scenarios, person-facing and institution-facing constructs are analysed separately where their professional structures differ.Source: §3.6 · p. 6

4 RESULTS

Models score highly on professional appropriateness in the main dataset, but their stated justifications vary more across models. In the secondary experiment, domain labels and role have limited effects, while stakes yield large, heterogeneous shifts even when the professional norm remains unchanged.

4.1 MAIN DATASET

  • 95.6% mean appropriateness across the main dataset, with domain–value means ranging from 90.7% to 99.4%.Confidentiality has a 7.0% cross-domain range, compared with 4.3% for autonomy and 1.5% for honesty; exception under- and over-triggering are both infrequent.Source: §4.1 · p. 6
  • 25.6%–76.9% of responses across models use the correct domain-specific justification, despite similar levels of appropriateness in recommended actions.Figure 2 presents the full justification-type distribution by model; other responses use domain-neutral principles, generic moral reasoning, or another domain’s professional framework.Source: §4.1 · p. 6, Figure 2 · p. 7
  • Model differences show no simple ordering by size, generation, or broad training approach, and the comparisons are descriptive rather than causal.Successive Llama and Gemma generations improve in some cells but not monotonically; the two Constitutional AI models are both Claude variants.Source: §4.1 · p. 6

4.2 SECONDARY EXPERIMENT

  • 88.2% appropriateness in the Explicit condition and 87.4% in the Implicit condition; decision-maker scenarios score 88.7%, versus 86.9% for Advisor scenarios.Both explicit domain labels and agent role have limited effects on model behaviour.Source: §4.2 · p. 6
  • 92.4% appropriateness in the high-stakes condition versus 83.0% in the low-stakes condition, with a median gap near zero across 144 model × domain × value cells.Despite that median, 36 cells have gaps greater than 20%, and the largest observed gap is 91.2% for Llama 3.1 70B on medical honesty.Source: §4.2 · p. 7
  • Severity appears to increase disclosure across medical honesty and confidentiality, even where the professional duty remains unchanged.Medical honesty appropriateness for Llama 3.1 70B falls from 100% at high stakes to 8.8% at low stakes; for Gemma 3 27B, medical confidentiality falls from 92.3% at low stakes to 25.0% at high stakes.Source: §4.2 · p. 7
  • Institution-facing national-security confidentiality scores 88.5% at high stakes and 10.1% at low stakes, while institution-facing honesty is comparatively stable.The failure is specific to institution-facing confidentiality and stakes; person-facing confidentiality shows the opposite direction, and the norm remains unchanged.Source: §4.2 · p. 7

5 DISCUSSION

Models generally recommend professionally appropriate actions, yet professional grounding varies and contextual manipulations can trigger failures even when governing rules stay fixed. The discussion argues that value-alignment evaluation should test both context-sensitive adaptation and stability when norms do not change.

5 DISCUSSION

  • No clear pattern links capability to model scale, despite an expectation that newer and larger models would perform better.Source: §5 · p. 8
  • Increasing stakes can augment disclosure in honesty and confidentiality scenarios even when the governing professional rule is unchanged.Most cells remain stable, but the failures are severe and localised.Source: §5 · p. 8
  • Explicit domain labels have a near-null effect, suggesting that identifying normatively relevant features matters more than recognising the professional setting.Source: §5 · p. 8
  • Models can recommend the same action while offering different stated reasons, and those explanations need not faithfully reflect how the answer was produced.Source: §5 · pp. 8–9
  • Alignment evaluations should test whether models adapt when a relevant feature changes the norm and remain stable when it does not.The discussion also notes that principle-based alignment may require richer specification of roles, targets, thresholds, and exceptions, or reliable inference of that structure from context.Source: §5 · p. 9

6 LIMITATIONS

The benchmark's norms and scenarios have limited geographic and topical coverage, and its evaluation methods have additional constraints. The authors identify broader scenario testing and interactions among professional obligations as future work.

6 LIMITATIONS

  • The benchmark draws on a limited source set concentrated in UK, US, and international institutions, with particularly sparse national-security coverage.Source: §6 · p. 9
  • Scenario coverage is incomplete, some professional norms lack meaningful cross-domain analogues, and repeated sampling does not replace more independently constructed scenarios.Source: §6 · p. 9
  • Secondary manipulations are not fully orthogonal to professional norms, and independently authored variants may introduce wording differences.Severity legitimately affects some exception criteria, although the low- and high-stakes Base scenarios remain on the same side of the relevant boundary.Source: §6 · p. 9
  • LLM judges show imperfect agreement, especially for stated justification, and the study isolates one focal value rather than interactions among obligations.The labels capture the normative framework expressed in responses, not models' internal reasoning.Source: §6 · p. 9

7 CONCLUSION

This section summarizes ContextAdapt’s findings on professional grounding, contextual shifts in model behavior, and the operationalization of values through professional norms and scenarios. It also specifies probes for professional exceptions and classifies scenarios according to how contextual cues align with professional norms.

7 CONCLUSION

  • Context sensitivity alone is insufficient for alignment: models must identify which contextual features change how honesty, autonomy, and confidentiality should be applied.The conclusion also reports high average appropriateness, uneven professional grounding, and large behavioral shifts when stakes change despite unchanged obligations.Source: §7 · p. 9

AI USE STATEMENT

  • The authors report that AI-assisted scenarios, code edits, and manuscript editing were reviewed, that the study used no human participants or annotation, and that materials will be released upon acceptance.The paper says its methodology and reproducibility components are documented in the paper and appendices; scenarios and results are for research and evaluation, not professional advice.Source: AI USE STATEMENT · p. 10, ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A.1 CORPUS CONSTRUCTION

  • The benchmark draws on 24 primary-source documents across medicine, law, finance, and national security, but its norms are reference points rather than a comprehensive account across jurisdictions.The corpus is predominantly from UK, US, and international institutions.Source: §A.1 · pp. 12–13

A.2 VALUE–DOMAIN MATRIX

  • The value–domain matrix records each value’s default operationalization, obligation target, documented exceptions, and the conditions and procedures governing exceptions.It distinguishes counterpart from professional autonomy and, in national security, person-facing from institution-facing obligations.Source: §A.2 · p. 14
  • The matrix treats medical confidentiality as a strong default, with therapeutic privilege limited to individualized serious-harm judgments and procedural safeguards.The exception requires peer consultation, a documented benefit–harm review, and a temporary plan to disclose when the patient can tolerate it.Source: §A.2 · p. 15

B.1 DOMAIN-AGNOSTIC TEMPLATES

  • Domain-agnostic templates preserve the underlying normative tension while matching actor relationships, information asymmetry or authority conflict, decision type, and stakes where possible.The authors do not impose equivalence when professional sources provide no meaningful cross-domain analogue.Source: §B.1 · pp. 17–18

B.2 VALUE-SPECIFIC SCENARIO STRUCTURE

  • Scenarios separately test honesty, autonomy, and confidentiality, with autonomy items distinguishing counterpart decision rights from professional independence.Base scenarios test default professional operationalizations, such as candidly answering a patient’s direct question about likely effects of a procedure.Source: §B.2 · p. 18, §B.3 · p. 18

B.4 EXCEPTION PROBES

  • Exception probes distinguish cases where documented triggers and procedures are satisfied from cases where evidence is insufficient.The excerpt gives a direct-question honesty scenario for a financial adviser, but does not provide the full set of probe types.Source: §B.4 · pp. 18–19
  • The exception triad tests justified overrides, overbroad exceptions, and ambiguous boundaries, including when a solicitor may act without current instructions to protect a client’s best interests.The solicitor example treats acting without instructions as a narrow exception requiring reasonable efforts to reach the client and authority to act, not a licence to override disagreement.Source: §B.4 · p. 19

B.5 SCENARIO CLASSIFICATION

  • Scenarios are classified as Central, Adversarial, or Edge; Edge cases lack a unique quantitative reference label, so they are excluded from appropriateness estimates but retained for qualitative analyses.Scenarios use professional roles rather than named individuals to avoid attributes associated with personal names.Source: §B.5 · pp. 19–20

C MODELS AND GENERATION SETTINGS

The evaluation uses 12 fully evaluated models, repeated sampling, and a deliberately minimal prompt. Command A+ is reported separately as a partial result and excluded from pooled analyses.

C MODELS AND GENERATION SETTINGS

  • The section identifies 12 fully evaluated models, including Llama 3.1 8B Instruct, Llama 3.1 70B Instruct, Llama 3.3 70B Instruct, Gemma 2 27B IT, and Gemma 3 27B IT.Source: C MODELS AND GENERATION SETTINGS · p. 20
  • Each model is sampled 10 times per scenario at temperature 0.8, with a fixed seed and a 1024-token maximum.Sampling took place between 2026-08-13 and 2026-09-08.Source: C MODELS AND GENERATION SETTINGS · p. 20
  • The prompt asks models to reason through a realistic professional scenario and make a clear decision as the professional, without refusing or deferring.The system prompt is deliberately minimal to avoid contaminating the domain-specific, domain-neutral, and generic-moral justification signal.Source: C MODELS AND GENERATION SETTINGS · p. 21
  • Command A+ is excluded from pooled analyses in both datasets because its secondary-experiment generation was incomplete.Its main-dataset generation was complete, but a monthly API quota blocked completion of the secondary experiment.Source: C MODELS AND GENERATION SETTINGS · p. 21

D RESPONSE EVALUATION

The evaluation scores both whether a recommended action matches the professional reference and the type of justification given. Judge agreement is reported for both measures, alongside main-dataset appropriateness patterns across domains and values.

D.1 OUTCOME CORRECTNESS

  • Appropriateness is coded as matches, partial hedged, or does not match, mapped to 1, 0.5, and 0, respectively.Three-judge consensus uses the median label.Source: §D.1 · p. 21

D.2 STATED JUSTIFICATION

  • Justifications are classified by whether they invoke the correct or wrong professional domain, an abstract value, generic moral reasoning, uncertainty, or no reasoning.Categories are nominal; consensus uses majority vote, with three-way splits excluded from justification analyses.Source: §D.2 · pp. 21–22

D.3 JUDGE AGREEMENT

  • In the main dataset, pairwise κ for outcome correctness ranges from 0.37–0.56, while α for justification type ranges from 0.30–0.52.For the secondary experiment, the corresponding ranges are 0.31–0.51 and 0.23–0.49.Source: §D.3 · pp. 22–23, Table 4 · p. 23

E.1 MODEL-LEVEL APPROPRIATENESS

  • Figure 6 shows that confidentiality has the widest cross-domain appropriateness range, while honesty is comparatively stable.Table 5 reports the full appropriateness scores and outcome-label distributions by model.Source: §E.1 · p. 23, Figure 6 · p. 24, Table 5 · p. 24

E.2 DOMAIN–VALUE RESULTS

The section reports domain–value results, distributions of professional justifications, and exception-probe performance. It also frames the secondary experiment as analysis of variation within its factorial design, not as a comparison with the main dataset.

E.2 DOMAIN–VALUE RESULTS

  • Table 6 reports appropriateness by domain and value, with Figure 7 further disaggregating those rates by model.The supplied passages identify these displays but do not provide their cell values.Source: §E.2 · p. 24, Table 6 · p. 24, Figure 7 · p. 25

E.3 JUSTIFICATION DISTRIBUTIONS

  • Correct domain-specific justification ranges from 25.6% to 76.9% across models.The section also presents the appropriateness breakdown by domain and value in Table 6 and Figure 6.Source: §E.3 · p. 24, §E.2 · p. 24, Table 6 · p. 24

E.4 EXCEPTION PROBES

  • Under-triggering occurs when a documented exception applies but the model retains the default rule; over-triggering occurs when it invokes an exception whose conditions are unmet.Under-triggering is slightly more common: 4.0% of relevant responses under-trigger, compared with 1.5% that over-trigger.Source: §E.4 · pp. 24, 26, Figure 9 · p. 26

F ADDITIONAL SECONDARY EXPERIMENT RESULTS

  • The secondary-experiment analyses describe variation within its factorial design rather than differences between the secondary and main datasets.The factorial design varies stakes, role, and domain-label conditions.Source: F ADDITIONAL SECONDARY EXPERIMENT… · p. 26

F.1 FACTORIAL RESULTS

The secondary experiment varies domain labels, roles, and stakes, and reports appropriateness patterns across these conditions. Stakes effects are heterogeneous, including large and sometimes reverse-direction differences across constructs.

F.1 FACTORIAL RESULTS

  • Pooled appropriateness is 86.2% with an explicit domain label and 85.4% with an implicit label.Figure 12 indicates that the near-null pooled domain-label effect holds broadly across models.Source: §F.1 · p. 27, Figure 12 · p. 27
  • Figure 13 presents pooled appropriateness by role, while Figures 14 and 15 show decision-maker–advisor gaps and the agent-role manipulation by model.The supplied captions do not state the role-effect values.Source: Figure 13 · p. 27, Figure 14 · p. 27, Figure 15

F.2 DOMAIN–VALUE RESULTS

  • Table 7 reports appropriateness by domain and value pooled across stakes, role, and domain-label conditions, describing the secondary experiment’s scenario selection.Its values are not intended for comparison with Table 6’s main-dataset results.Source: §F.2 · p. 27, Table 7

F.3 DISTRIBUTION OF STAKES EFFECTS

  • Across 144 cells, the median stakes difference is close to zero, but 33 cells (25%) exceed 20 percentage points and the largest positive difference is 91.2 points.Figure 16 displays the distribution; Table 8 lists the five largest positive gaps.Source: §F.3 · p. 28, Figure 16 · p. 28, §F.4 · p. 28

F.5 REVERSE-DIRECTION EFFECTS

  • Ten of 12 models show lower appropriateness at high stakes for Medicine × Confidentiality, where substantial effects also occur in the reverse direction.Table 9 lists the largest reverse-direction effects for this construct.Source: §F.5 · p. 28, Table 9 · p. 29

F.6 NATIONAL-SECURITY CONSTRUCTS

  • Institution-facing Confidentiality scores 88.5% at high stakes and 10.1% at low stakes; institution-facing Honesty is more stable at 99.2% and 92.4%.Person-facing Confidentiality scores 76.0% at high stakes and 95.5% at low stakes; Table 10 and Figure 17 show these national-security constructs.Source: §F.6 · p. 29, Table 10 · p. 29, Figure 17 · p. 29

G EXPLORATORY MODEL ANALYSES

The exploratory analysis compares model behaviour across broad post-training categories. It finds substantial variation within nominal categories and cautions against causal interpretation of category differences.

G EXPLORATORY MODEL ANALYSES

  • The categories are coarse because public labels conceal differences in base models, training data, preference objectives, filtering, and system prompts.Figure 20 is described as a descriptive appropriateness breakdown by broad category.Source: G EXPLORATORY MODEL ANALYSES · p. 29, Figure 20 · p. 30
  • The Constitutional-AI category contains two models from one developer, confounding any apparent category effect with developer-specific factors.The authors therefore do not interpret between-category differences as causal effects of training method.Source: G EXPLORATORY MODEL ANALYSES · pp. 29–30
  • Substantial variation occurs within nominal post-training categories, including in the frequency of domain-specific justifications.Broad post-training labels do not sufficiently explain the benchmark’s behavioural differences.Source: G EXPLORATORY MODEL ANALYSES · p. 30

Related papers

Loading 2609.38260v1…