PaperlayerView exact revision on arXiv

Source-linked AI summary

When Context Misleads: In-context Learning with Jurisdiction in Large Language Models

Pei-lin Li, Qingle Liu, Junyang Feng, Siyu Li, Sunqi Fan, Xin-Sheng Chen, Shuojin Yang

arXiv:2609.27603v2cs.CLcs.AI

TL;DR

Existing ICL methods emphasize learning from demonstrations but give less attention to whether demonstrated rules should govern an answer. The paper introduces FAKECONTEXT-BENCH and J-ICL, which trains models on valid and misleading-rule episodes. Evaluations report that scaling alone is insufficient and that existing ICL fine-tuning can reduce reality-grounded accuracy by up to 14.95 percentage points.

  • Problem

    Existing ICL studies typically assume demonstrations reliably guide the query, leaving limited attention to whether their rules are factually grounded and applicable.

    Source: §1 · pp. 1–2

  • Method

    J-ICL jointly trains on valid-rule episodes and episodes with rules that conflict with reality or the operating system, teaching models when to follow or reject context.

    Source: §1 · p. 2

  • Results

    Existing ICL fine-tuning methods can reduce reality-grounded accuracy by up to 14.95 percentage points, while large-scale pretraining alone is insufficient for reliable context-authority discrimination.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Contextual adaptivity alone may increase vulnerability to misleading context, so models need to assess whether demonstrated rules apply rather than follow them unconditionally.

    Source: §1 · p. 2

    The study uses controlled, predominantly synthetic examples, and its results do not provide a general guarantee of model truthfulness or safety.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

In-Context Learning (ICL) has become a cornerstone of modern LLM deployment. However, existing ICL post-training methods have a critical blind spot: they excel at extracting patterns from demonstrations while often neglecting context authority, the ability to determine whether contextual information should govern the final answer. To benchmark this capability, we introduce FakeContextBench, which contains pseudoscientific claims across seven domains. Our evaluation of commercial and open-source models shows that large-scale pre-training alone is insufficient for reliable context-authority discrimination. Moreover, prevalent ICL fine-tuning methods can increase susceptibility to misleading context, reducing reality accuracy by up to 14.95 percentage points relative to the base model. To address this trade-off, we propose Jurisdiction In-Context Learning (J-ICL), a post-training framework that incorporates context validation into the training objective. Across four model backbones, J-ICL improves ICLEval by an average of 5.84 percentage points and reality accuracy by 9.20 points over the corresponding base models. It also raises the Reality Rate by an average of 18.09 points relative to MetaICL and Symbol Tuning. These results demonstrate that ICL capability and resistance to deceptive context can be improved together. The benchmark is available at https://github.com/peilin717/FakeContext-Bench.

1 INTRODUCTION

In-context learning lets models infer task mappings from demonstrations, but existing work largely assumes those demonstrations are reliable. This paper introduces a benchmark and J-ICL to assess whether contextual rules apply and train models to reject misleading rules while retaining valid adaptation.

1 INTRODUCTION

  • In-context learning lets models adapt to tasks from a few prompt examples without updating their parameters.Prior enhancement methods include meta-training across tasks, replacing meaningful labels with arbitrary symbols, and improving demonstration selection or prompt formatting.Source: §1 · p. 1
  • Existing ICL research largely assumes demonstrations provide reliable guidance, leaving models’ ability to assess whether contextual rules are grounded in reality relatively underexamined.Models that improve at following demonstrations may become more likely to follow misleading rules and answer incorrectly.Source: §1 · p. 1
  • FAKECONTEXT-BENCH tests whether models follow a coherent but unauthorized rule despite possessing the correct knowledge, across seven domains.Each instance includes reality-grounded and context-induced answers, distinguishing preserved facts, misleading-rule following, and mixed responses.Source: §1 · p. 2
  • Scaling and large-scale pretraining do not reliably enable context-authority discrimination, while existing ICL fine-tuning can reduce reality-grounded accuracy by up to 14.95 percentage points.These results expose a trade-off: improved contextual adaptivity may coincide with increased vulnerability to misleading context.Source: §1 · p. 2
  • J-ICL jointly trains on valid J+ episodes and misleading-rule J− episodes to teach when contextual rules should be followed or rejected.Both episode types use the same few-shot format and apply language-modeling loss only to the query answer, without authority labels or auxiliary objectives.Source: §1 · p. 2

2 RELATED WORK

Prior ICL methods improve learning from demonstrations but generally assume demonstrations and queries share the same governing rule. This paper distinguishes authority judgment from ordinary context-following, false-premise, and misinformation tasks.

2 RELATED WORK

  • MetaICL and Symbol Tuning strengthen learning from demonstrations, while other methods target demonstration selection, ordering, calibration, or formatting.These approaches generally assume that demonstrations and queries share the same governing rule.Source: §2 · p. 3
  • FAKECONTEXT-BENCH tests coherent, learnable alternative rules in demonstrations, rather than corrupted mappings or isolated label noise.The task is to decide whether an induced rule applies to the query, not merely to recover the mapping.Source: §2 · p. 3
  • FAKECONTEXT-BENCH separates authority judgment from benchmarks of misconceptions, false premises, sycophancy, and multi-turn misinformation.It records reality-grounded and context-induced answers to distinguish rule-application errors from general failures or factual ignorance.Source: §2 · p. 3

3 FAKECONTEXT-BENCH

FAKECONTEXT-BENCH tests whether models can distinguish reality-grounded rules from coherent but unauthorized rules induced by demonstrations. Evaluations find that scaling does not reliably resolve this failure mode, while activation analyses identify a partially concentrated late-layer pathway associated with misleading demonstrations favoring contextual answers.

3.1 TASK FORMULATION

  • FAKECONTEXT-BENCH records reality-grounded and context-induced answers, classifying responses as REALITY, CONTEXT, or OTHER.Its Reality Rate compares REALITY with CONTEXT responses, excluding unrelated failures from that comparison.Source: §3.1 · pp. 3–4

3.2 DESIGN PRINCIPLES / 3.3 DOMAINS AND CONSTRUCTION

  • The benchmark uses executable misleading rules with consistent demonstrations, pre-established distinct answers, and natural-language applicability cues rather than explicit false-rule warnings.It contains 700 theory families and 3,500 instances across seven domains, with five varied instances per family.Source: §3.2 · p. 4, §3.3 · p. 4

3.4 EVALUATION OF EXISTING MODELS

  • From Qwen3-0.6B to Qwen3-14B, ICLEval accuracy rises 27.40 points, from 44.85% to 72.25%, while Reality Rate falls 3.46 points, from 27.58% to 24.12%.The scaling trend shows that stronger conventional ICL does not naturally yield greater resistance to misleading context.Source: §3.4 · p. 4, Figure 2 · p. 5
  • Removing demonstrations increases Reality Rate by an average of 29.83 points, compared with 7.53 points from removing only the statement and 43.82 points from removing both.The input ablation identifies demonstrations as the primary source of misleading-context effects.Source: §3.4 · p. 4, Figure 2 · p. 5
  • Except for GPT-5.6 Sol at 55.08% Reality Rate, evaluated frontier models score below 50%, ranging from 13.56% to 44.78%.Gemini 3.5 Flash scores 13.56%, indicating severe susceptibility despite its overall capabilities.Source: §3.4 · pp. 4–5, Table 1 · p. 5

3.5 MECHANISTIC EVIDENCE OF DEMONSTRATION-INDUCED CONTEXT BIAS

  • Figure 2c plots standardized Full–No-demo activation change against direct contribution to the Reality-minus-Context output direction; lower-right neurons favor Context.Top candidates cluster in layers 33–35, while activation replacement increases the Reality-minus-Context margin and matched random controls remain near zero.Source: §3.5 · pp. 5–6, Table 2 · p. 5

4 J-ICL

J-ICL jointly trains rule induction and jurisdiction discrimination, distinguishing applicable demonstrations from misleading contextual rules while retaining the authoritative target. Its episodic sampling and query-only objective train this distinction without explicit authority labels.

4 J-ICL

  • J-ICL jointly optimizes rule induction and jurisdiction discrimination without auxiliary classifiers or explicit supervision over authority.Source: §4 · p. 6

4.1 PROBLEM FORMULATION AND EPISODIC CONSTRUCTION

  • The setup contrasts authoritative rule R with contextual rule P when their query outputs differ, and uses yR as target whether demonstrations apply (J+) or not (J−).Source: §4.1 · p. 6
  • Each task cluster yields matched J+/J− episodes, varying whether the contextual rule applies and, for J−, its realization; jurisdiction is conveyed in natural language only.Source: §4.1 · p. 6

4.2 TRAINING AND OPTIMIZATION

  • Only final query-answer tokens receive causal language-modeling loss; jurisdiction statements and demonstrations are masked, with no explicit authority labels or auxiliary classification loss.Source: §4.2 · p. 6
  • Three jointly weighted streams preserve ordinary J+ learning, contrast J+ against J−, and vary negative realizations; paired losses are averaged before one optimizer update.Source: §4.2 · p. 7

5 EXPERIMENT

Across four model settings, J-ICL improves standard in-context learning while generally increasing resistance to misleading demonstrations. Ablations and robustness tests indicate that both jurisdictional episode types matter and that gains persist across tested statement forms and authority cues.

5.1 MAIN RESULT

  • J-ICL achieves the highest ICLEval score in every setting, with Reality Rate gains over Base of 6.44, 12.75, and 22.83 points on Qwen3-8B, Qwen3-14B, and Llama3.1-8B, respectively.On Qwen3-4B, ICLEval rises 5.93 points and Reality accuracy is highest, while Reality Rate remains approximately unchanged from Base.Source: §5.1 · p. 7, Table 3 · p. 8

5.2 ABLATION STUDY

  • Reality Rate falls 16.70 points when all J− data are removed; removing J−/J− pairs lowers it 7.73 points, while matched J+/J− pairs contribute to both ICLEval and Reality Rate.Independent-sampling SFT trails Full by 4.09 ICLEval points and 1.29 Reality Rate points, suggesting pairing benefits beyond the marginal data mixture.Source: §5.2 · p. 8, Table 4 · p. 9

5.3 ROBUSTNESS TO STATEMENT-FORM AND SEMANTIC-CUE SHORTCUTS

  • Across three held-out paraphrase conditions, J-ICL’s Reality Rate gains range from 3.21 to 7.20 points (mean 5.12) and persist under Low-overlap wording.The misleading rule, demonstrations, query, and candidate answers remain fixed while only the scope statement is rewritten.Source: §5.3 · p. 9, Table 5 · p. 9
  • When authority cues conflict but the rule explicitly applies, J-ICL shows near-zero aggregate shifts (∆C = +0.03, ∆R = +0.06 pp) and 96.77% paired-prediction invariance.The diagnostics support robustness under tested conditions, but do not establish fully abstract or compositional authority reasoning.Source: §5.3 · p. 9, Table 6 · p. 9

6 CONCLUSION

The conclusion frames context-authority discrimination as a missing dimension of ICL and describes a trade-off between contextual adaptation and reality-grounded answering. J-ICL addresses this through natural-language jurisdiction cues and a query-only causal language-modeling objective.

6 CONCLUSION

  • The paper identifies context-authority discrimination as a missing ICL dimension and reports that conventional ICL post-training can increase willingness to follow misleading demonstrations.J-ICL uses natural-language jurisdiction cues and a query-only causal language-modeling objective, without authority labels, auxiliary classifiers, or intermediate-reasoning supervision.Source: §6 · pp. 9–10

AI USE STATEMENT

The authors report using generative AI tools for manuscript presentation, data augmentation and synthesis, translation, and experimental scripts; authors reviewed and verified AI-assisted data and code.

AI USE STATEMENT

  • Researchers conducted the experimental design and theoretical analysis independently and take responsibility for the final content.Generative AI tools assisted with manuscript presentation, data augmentation and synthesis, translation, and experimental scripts; the authors reviewed and verified AI-assisted data and code.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The study evaluates resistance to misleading contextual rules using controlled, predominantly synthetic examples and cautions against treating its findings as a general guarantee of truthfulness or safety.

ETHICS STATEMENT

  • The evaluation uses controlled, predominantly synthetic examples of misleading contextual rules.The false or hypothetical rules are for robustness evaluation, not factual claims.Source: ETHICS STATEMENT · p. 10
  • The benchmark could inform misleading-prompt construction, though its intended use is to improve in-context learning reliability.No private or sensitive personal data are used.Source: ETHICS STATEMENT · p. 10
  • The results do not provide a general guarantee of model truthfulness or safety.Publicly sourced materials were paraphrased and de-identified; human involvement was limited to scorer-validation annotation.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The authors state that they will release the artifacts needed to reproduce and audit the work, and illustrate FakeContextBench with a case separating rule induction from rule applicability.

REPRODUCIBILITY STATEMENT

  • The authors will release the benchmark, evaluation code and protocols, training data and scripts, J-ICL adapters, and evaluated models’ raw responses.They say these materials will support reproduction and auditing of training, evaluation, and reported results.Source: REPRODUCIBILITY STATEMENT · p. 10

A FAKECONTEXT-BENCH CASE STUDY

  • The case presents three mutually consistent demonstrations of a coherent but incorrect mechanics rule, while asking for a real object's actual average net force rather than simply applying that demonstrated mapping.The query specifies M = 4 kg, with velocity changing from 12 m/s to 18 m/s in 3 s.Source: A FAKECONTEXT-BENCH CASE STUDY · p. 13
  • In the case study, 24 indicates recovered-but-inapplicable mapping, whereas 8 indicates rejection of the contextual rule in favor of the reality-grounded rule.The registered outcomes for the case are summarized in Table 7.Source: A FAKECONTEXT-BENCH CASE STUDY · p. 13, Table 7 · p. 14

B.1 RULE-BASED EVALUATION

The appendices specify deterministic scoring and validate its mixed-response resolution, audit FakeContextBench’s construction, and document the ICLEval evaluation protocol.

B.1 RULE-BASED EVALUATION

  • The scorer assigns each response to REALITY, CONTEXT, or OTHER, resolving competing answer occurrences by the response’s concluding region and final valid occurrence.It uses conclusion-cue sentences when present, otherwise the final three sentences; it does not use a MIXED category or model-based judge.Source: §B.1 · p. 14
  • Numerical answers are matched as complete tokens using instance-specific absolute tolerance and zero relative tolerance; textual and symbolic answers are normalized before matching.The scorer records matched spans, decision region, and decision path for instance-level auditing.Source: §B.1 · p. 14

B.2 HUMAN VALIDATION OF MIXED-RESPONSE RESOLUTION

  • Human validators judge which registered answer the response ultimately presents as its conclusion, with MIXED reserved for no unique commitment.The validation sample contains responses in which both registered answers occur before scorer disambiguation.Source: §B.2 · p. 15
  • 90.0% agreement and κ=0.807: the deterministic scorer agrees with adjudicated human judgments on 100 sampled mixed responses.For the 97 definitive REALITY/CONTEXT cases, agreement is 92.8% and κ=0.856.Source: Table 8 · p. 16, §B.2 · p. 16

C FAKECONTEXT-BENCH CONSTRUCTION AND AUDIT

  • 700 theory families span seven domains, with 100 families each; 200 draw on pseudoscientific theories and 500 imitate observed reasoning patterns and factual errors.Families define reality-grounded and conflicting contextual rules and provide the semantic-diversity unit.Source: C FAKECONTEXT-BENCH CONSTRUCTION AND… · p. 16
  • Each family expands to five instances by varying parameters, entities, surface realizations, demonstrations, or query conditions while preserving its underlying conflict.The expansion is task-dependent rather than only numeric substitution in a fixed prompt.Source: C FAKECONTEXT-BENCH CONSTRUCTION AND… · p. 16
  • 3,500 records contain five instances per family, with distinct nonempty reality and context answers; the audit reports 3,115 numerical or symbolic, 135 exact symbolic, and 250 short textual items.Structural checks do not replace separate family-level review of semantic correctness.Source: C FAKECONTEXT-BENCH CONSTRUCTION AND… · p. 16

D ICLEVAL EVALUATION PROTOCOL

  • ICLEval evaluation uses 2,040 instances across 17 tasks and reports micro accuracy, with prompts, generation, and scoring following the official protocol.Generation uses greedy decoding; each instance contributes equally to the main score.Source: D ICLEVAL EVALUATION PROTOCOL · p. 17

E CONTEXT-INDUCTION NEURON ANALYSIS

The neuron analysis contrasts paired prompts with and without demonstrations, measures activation changes near the model output, and tests selected neurons on held-out families.

E CONTEXT-INDUCTION NEURON ANALYSIS

  • The paired Full and No-demo prompts preserve the statement and query while removing demonstrations, isolating their addition with possible formatting or sequence-length effects.Five families are excluded for lacking a well-defined first divergent answer token, leaving 695; 486 families are used for discovery and 209 for held-out intervention.Source: E CONTEXT-INDUCTION NEURON ANALYSIS · pp. 17–18
  • For each case, the analysis measures final-position activations after the shared answer prefix and before the first divergent Reality or Context token.Candidate ranking is restricted to layers 24–35 of Qwen3’s 36-layer gated MLP.Source: E CONTEXT-INDUCTION NEURON ANALYSIS · p. 18
  • Candidate neurons are ranked using standardized selectivity, sign consistency, and paired direct attribution, with a threshold requirement on Sctx.The score is an operational ranking criterion, not a multiple-comparison-corrected significance test.Source: E CONTEXT-INDUCTION NEURON ANALYSIS · p. 18
  • Held-out intervention replaces selected neurons’ final-position activations with paired No-demo activations and measures the change in first-divergent-token margin.Positive ΔMi indicates movement toward the Reality answer; comparisons use five random control sets matched approximately by |Sctx|.Source: E CONTEXT-INDUCTION NEURON ANALYSIS · p. 18

F CONSTRUCTION OF JURISDICTION-CONTRASTIVE TRAINING DATA

J-ICL training data pair applicable and inapplicable rules while preserving the correct answer, then vary misleading rules and their presentation across task families. The accompanying experiments examine objective components, hyperparameter trade-offs, and whether performance depends on familiar jurisdictional wording.

F CONSTRUCTION OF JURISDICTION-CONTRASTIVE TRAINING DATA

  • J−/J− pairs vary misleading rules, demonstration views, or jurisdictional wording while keeping the query and authoritative target fixed.J+/J− pairs contrast applicable and inapplicable rules; the misleading rule induces a counter-target that is not used as the supervised query target.Source: F CONSTRUCTION OF… · p. 19, G TRAINING-DATA COMPOSITION · pp. 21–22

G TRAINING-DATA COMPOSITION

  • The corpus combines standard J+ episodes with matched J+/J− and J−/J− pairs across symbolic, numerical, linguistic, semantic, and factual task families.It uses family-balanced sampling and quality controls that preserve shared queries and correct targets while requiring each J− counter-target to differ from the correct answer.Source: G TRAINING-DATA COMPOSITION · pp. 21–22

H COMPONENT-ABLATION DETAILS

  • The component ablations compare independently trained objectives under matched 2,400-update budgets, while an independent-sampling control removes pair correspondence but retains episode exposure.The Full condition accumulates weighted query-only losses from three training streams at each update.Source: H COMPONENT-ABLATION DETAILS · p. 23

I HYPERPARAMETER SENSITIVITY

  • 40% J− preserves conventional ICL best among evaluated mixtures, while the sweeps show trade-offs between ICLEval and context-authority discrimination.At 60% J−, R/(R + C) rises 7.28 points versus Full while ICLEval falls 10.25 points; at 8 × 10−6 peak learning rate, R/(R + C) rises 11.32 points while ICLEval falls 7.85 points.Source: Table 15 · p. 23, I HYPERPARAMETER SENSITIVITY · p. 24

J STATEMENT-FORM SHORTCUT EVALUATION

  • The paraphrase audit finds no complete-string overlap for 33 templates in training instructions, but shared scope concepts mean the paraphrases are not semantically novel.All 3,500 benchmark cases appear in each condition, with only the general scope statement rewritten.Source: J STATEMENT-FORM SHORTCUT EVALUATION · p. 24
Loading 2609.27603v2…