PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs Learn Better In-Context from Rules than from Examples

Xiang Fu, Seungmin Cho, Yukyung Lee, Najoung Kim

arXiv:2609.03213v1cs.CL

TL;DR

The paper asks how reliably LLMs learn novel complex tasks from rules versus examples, and how learner and task properties shape that comparison. It compares the modes on the same tasks and finds rules more effective overall, with the advantage varying by task and model.

  • Problem

    The performance consequences of combining rules and examples, and the learner and task properties that shape their relative efficacy on novel complex tasks, remain to be systematically compared.

    Source: §1 · pp. 1–2

  • Method

    The authors compare rules, examples, and their combination on five novel tasks spanning games, arithmetic, and linguistic inferences, using the same underlying tasks.

    Source: §6 · p. 9

  • Results

    Rules are more effective overall than examples, added examples do not yield significant gains, instruction tuning strengthens rule-based learning without degrading example learning, and the rule advantage varies with task properties.

    Source: §4 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The rule advantage is larger for tasks involving algebraic abstractions and computations, and smaller for tasks requiring distributional sensitivity or recruiting parametric knowledge.

    Source: §6 · p. 9

    The controlled prompt comparisons may limit generalizability to real-world applications, and the authors cannot fully verify that game variations were absent from model pretraining data.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) exhibit in-context learning capabilities, where they can learn new tasks from prompt contexts without weight updates. We compare the learning efficacies of two prominent modes of in-context learning: (1) learning from descriptions of rules (instruction following); and (2) learning from examples of input-output demonstrations (few-shot prompting). Through five learning tasks that cover diverse domains (games, arithmetic, linguistic inferences), we compare two modes of learning (rules vs. examples) specifying the same underlying task. We furthermore explore model and task properties that modulate the learning efficacies. We find that models generally learn more reliably from rules than from examples alone, and additional examples on top of rules or simply scaling up the number of examples do not lead to consistent and significant gains. Instruction tuning amplifies the benefit of rule-based learning while keeping example-based learning capacities intact. Surprisingly, we find no privileged effect of example-based learning in base models, and rules still lead to gains in algebraic task domains. Overall, the comparative efficacy of rules over examples is larger when the task recruits algebraic abstractions and computations, and smaller when the task requires distributional sensitivity and/or recruits parametric knowledge.

1 Introduction

The paper systematically compares rule-based and example-based learning for the same novel tasks, asking how learning mode, model properties, and task properties shape in-context learning. Across domains and model types, rules outperform minimum-coverage examples overall, while the size of that advantage varies by task.

1 Introduction

  • Different mechanisms for rules and examples do not by themselves establish that combining them improves performance.Prior work found partly overlapping mechanisms, motivating a systematic comparison of learning efficacy.Source: §1 · p. 1
  • The study compares rules, examples, and their combination on the same test items while holding the underlying task fixed.The five novel tasks span games, arithmetic, and linguistic inferences.Source: §1 · p. 1
  • The central questions concern whether rules or examples teach complex, novel tasks more reliably and how learner and task properties affect that comparison.The learner question specifically emphasizes instruction tuning, while the task question asks which task properties favor either learning mode.Source: §1 · p. 2
  • Rules outperform minimum-coverage examples overall, while examples added to rules do not produce consistent, significant gains.The advantage is larger for tasks involving algebraic abstractions and computations, and smaller for tasks requiring distributional sensitivity or recruiting parametric knowledge.Source: §1 · p. 2

2 Method

The method compares rules, demonstrations, and their combination on shared test items across five tasks and three difficulty levels. Tasks span games, arithmetic, and linguistic inferences, while example sets use minimum coverage and are sampled and ordered across test items.

2 Method

  • Five task domains are tested with base and instruction-tuned models from several open-weight families, alongside GPT-5.4 as a reference.The open-weight checkpoints span Gemma 3, Qwen 2.5, Qwen 3, and OLMo 3/3.1, with sizes from 7B to 32B parameters.Source: §2 · p. 2, §2.1 · p. 2

2.2 Learning Conditions

  • The three prompt conditions provide rules only, input-output demonstrations only, or both, and all evaluate the same test items.Rule prompts specify the answer format; examples-only prompts provide no explicit task description or formatting instruction.Source: §2.2 · pp. 2–3, §2 · p. 2

2.3 Task Difficulty Conditions

  • Easy, Medium, and Hard levels vary the number of variables, their possible values, and the inference steps needed to compute outputs.The design tests whether rule descriptions yield more robust learning as the space of possible input-output mappings grows.Source: §2.3 · p. 3

2.4.1 Set Game

  • Set Game asks the model to select three cards whose attributes satisfy specified rules, a task designed to recruit algebraic abstractions over variables.Task difficulty varies with the number of attributes and values and with rule complexity.Source: §2.4.1 · pp. 3–4

2.4.2 Tapatan

  • Tapatan presents a board state and asks for a game outcome, with difficulty varying by board size, piece limits, and winning-row length.The task requires recognizing win conditions from coordinate adjacency, which the authors expect to provide strong distributional cues.Source: §2.4.2 · p. 4

2.4.3 Operator Function

  • Operator Function tests applying a newly defined operator composed of familiar primitive arithmetic operators to an expression.The task requires algebraic reasoning and sequential operations; difficulty varies with the number of primitive operators and arguments.Source: §2.4.3 · p. 4

2.4 Task Design

  • Noun Class Agreement tests grammaticality in an artificial language, while Lexical Category Inference tests whether four-word sequences satisfy semantic category membership.The latter uses conjunctive or disjunctive category definitions to examine distributional cues and parametric knowledge; the former varies in noun classes and agreement phenomena.Source: §2.4.4 · p. 4, §2.4.5 · pp. 4–5

2.5 Example Selection

  • Demonstration sets use minimum coverage of output labels, task primitives, and representative cases, with sets sampled and randomly ordered for each test item.The authors do not further optimize demonstration selection; Appendix C examines larger sets.Source: §2.5 · p. 5

3 Results

Rule-based learning significantly outperformed minimum-coverage examples overall, but the size of the advantage varied across tasks. Adding examples or scaling their number did not consistently improve on rules, while instruction tuning and model size affected learning outcomes.

3.1 Effect of Learning Conditions

  • Rule-based learning significantly outperformed example-based learning overall.Open-weight instruction-tuned models showed patterns similar to GPT-5.4, which was excluded from the linear mixed-model analyses.Source: §3.1 · p. 5
  • Rules helped most on Set Game and Operator Function, while benefits were smaller or not significant on Tapatan and the linguistic tasks.Lexical Category Inference showed no clear rule-based benefit; under examples, conjunctive definitions outperformed disjunctive ones, a pattern not consistent under rules or combined learning.Source: §3.1 · p. 6
  • Adding examples to rules provided no statistically significant benefit, and the combined condition closely tracked rule-based learning across tasks.The per-task analysis likewise found no significant gains from the additional examples.Source: §3.1 · p. 6
  • Task difficulty caused monotonic performance declines, but the nearly parallel slopes showed no special robustness advantage for rules over examples.Figure 5 presents the slope of decline as task difficulty increases.Source: §3.1 · p. 6, Figure 5 · p. 6
  • Scaling example counts produced varied trends, with few task-model combinations showing monotonic gains over rules.Gains were often flat, diminishing, or negative; when examples helped, smaller base models tended to benefit more, typically on easier task versions.Source: §3.1 · p. 6

3.2 Effect of Model Properties

  • Instruction tuning significantly improved rule-based and combined learning, without significantly affecting example-based learning.The authors suggest the rule-based gains exceed improved output-format following alone, given that combined learning added little over rules in base models.Source: §3.2 · p. 6
  • Increasing model size produced significant gains for both base and instruction-tuned models.The analysis grouped models into smaller (7B–12B) and larger (14B–32B) sizes.Source: §3.2 · p. 7

4 Discussion

Rules generally support more reliable learning than examples, with little consistent gain from adding demonstrations. The advantage varies with model training and task properties, motivating more controlled follow-up studies.

RQ1

  • Additional examples on top of clear rules do not yield statistically significant gains in general.The authors suggest that their revision of rule prompts for completeness and clarity may help explain the result.Source: §4 · p. 7

RQ2

  • Instruction tuning improves rule-based learning without significantly changing example-based learning, while combined learning also improves.The authors argue that rule-based gains are not merely improved output-format following, since combined learning has little benefit over rules.Source: §4 · p. 7
  • Base models show no general learning advantage from examples, and Operator Function and Set Game significantly benefit from rules.The authors suggest rules surface task, label-space, and output-format information that examples leave to inference; rule-following capacity appears even without instruction tuning, which amplifies it.Source: §4 · p. 7

RQ3

  • Rule superiority is greater for games and arithmetic than for linguistic category-inference tasks.The authors interpret this pattern as stronger rule benefits for algebraic reasoning than for category membership inference.Source: §4 · pp. 7–8
  • Within category-inference tasks, semantic categories benefit less from rules than arbitrary formal categories, and the rule advantage is clearer for disjunctive than conjunctive definitions.Similarity-based inference is expected to be more effective for conjunctive definitions.Source: §4 · p. 8

Future work

  • Future work could test task properties more systematically and examine whether function-vector findings generalize to more complex tasks.The authors propose controlled variations of a single task and mechanistic analyses of tasks like theirs.Source: §4 · p. 8

5 Related Work

Related work examines how in-context learning uses instructions and demonstrations, whether they recruit shared task representations, and how rule and exemplar learning compare across LLMs and humans. The authors situate their contribution as a more detailed comparison of rules and examples and their relation to model and task properties.

Different modes of in-context learning

  • Example-based learning uses few-shot prompt demonstrations, while rule-based learning provides natural-language instructions; example performance is sensitive to selection, retrieval, ordering, prompt design, and example count.Prior work made example-based learning prominent (Brown et al., 2020) and studied instruction-based learning (Sanh et al., 2022).Source: §5 · p. 8
  • Liu et al. (2024) compared rules, examples, and instruction inference for the same tasks, but their rule prompts included examples; this work focuses more directly on rules versus examples and model/task properties.Both studies conclude that correct rules outperform examples.Source: §5 · p. 8

Task representations

  • Davidson et al. (2026) found partly overlapping but distinct mechanisms for rules and examples, while this paper finds no consistent additive performance benefit from combining them.The authors note that distinct mechanisms do not necessarily imply additive performance gains; Yang et al. (2026) identified lexical task representations shared across rule and example prompts.Source: §5 · p. 8

Characterizations of rule following and induction in LLMs

  • Prior work finds that example-derived abstractions can support reasoning, while LLMs can struggle with explicit constraints, compositional rules, and rules conflicting with pretrained regularities.The cited work also connects in-context concept learning to simplicity and task structure (Zhu et al., 2023; Wang et al., 2024a).Source: §5 · pp. 8–9

In-context vs. in-weights learning

  • Prior work distinguishes in-context from in-weights learning, with Cook et al. (2026) finding in-weights instructions less reliable for procedural tasks.Chan et al. (2022) argued that generalization from in-weights and in-context information shows different trends.Source: §5 · p. 9

Rule- and example-based learning in humans

  • Human research has studied rule- and exemplar-based learning, but few studies directly compare efficacy when both specify the same task.The authors identify education, language-learning, and individual-difference research as related areas and suggest further exploration of connections with LLM studies.Source: §5 · p. 9

6 Conclusion

The paper compares rule- and example-based in-context learning through controlled experiments on novel, complex tasks across domains. Across three open-weight model families, rules are more effective overall, and the relative advantage varies with model and task properties.

6 Conclusion

  • Rules are the more effective medium of in-context learning, with no statistically significant gains from adding examples on top of rules.The comparison uses open-weight LLMs from three model families and novel, complex tasks spanning a wide range of domains.Source: §6 · p. 9
  • The rule advantage is greater in instruction-tuned models, while base models show no privileged efficacy for example-based learning.The authors report that base models follow a similar overall trend to instruction-tuned models.Source: §6 · p. 9
  • The rule-example gap is larger for tasks recruiting algebraic abstractions and computations, and smaller for tasks requiring distributional sensitivity or parametric knowledge.Source: §6 · p. 9

Limitations

The controlled prompt comparison supports systematic analysis but may limit generalizability to practical applications. The authors caution that their results do not show rules are always preferable in real-world settings.

Limitations

  • The rule advantage should not be generalized to practical settings where instructions may be ambiguous, incomplete, noisy, or informal.The authors' prompts were revised for clarity without performance-based prompt search, and they say the findings apply when a faithful rule description can be supplied.Source: Limitations · p. 9
  • Pretraining exposure to the original games could influence results for Set Game and Tapatan, and the authors did not audit training data to verify novelty.Source: Limitations · p. 9
  • The experiments use controlled synthetic tasks, so whether the rule advantage persists with less idealized rules and examples remains open.The authors identify this as a direction for future work to establish practical applicability.Source: Limitations · p. 9

A.1 Task Difficulty

The appendix describes task difficulty through increasing representational and inference demands, and documents the model, decoding, and evaluation setup.

A.2 Modeling Details

  • The evaluation includes base and instruction-tuned checkpoints from Gemma, OLMo, and Qwen families, with GPT-5.4 as a separate reference point.GPT-5.4 is excluded from statistical analyses, which use open-weight checkpoints only.Source: §A.2 · p. 12, Table 2 · p. 14
  • The open-weight checkpoints follow their respective license terms, while GPT-5.4 is accessed through the API and its weights are not redistributed.The Qwen3 instruction-tuned checkpoints have thinking mode disabled; base and instruction-tuned prompts use different formatting.Source: §A.2 · pp. 12, 14, 15
  • All experiments are inference-only, with an estimated local budget of approximately 600 GPU-hours; GPT-5.4 API use is excluded from that total.The estimate covers the open-weight main evaluations, OLMo example-scaling experiments, and reruns.Source: §A.2 · pp. 12, 14

A.1 Task Difficulty

  • Easy-to-hard levels vary the primitives models must represent and the inference steps needed across five tasks.The table reports these difficulty manipulations by task and level.Source: §A.1 · p. 12, Table 1 · p. 13

A.2 Modeling Details

  • Models use a shared decoding policy with temperature 0.0, while generation caps vary by model and task and remain fixed across learning conditions.Base models use shorter caps; instruction-tuned models receive larger caps when needed to avoid truncation.Source: §A.2 · pp. 14–15

B Dataset Details

  • Demonstrations are sampled and ordered separately for each test item, while test items and demonstration configurations are matched across checkpoints and example-based conditions.Fixed seeds make these randomized assignments reproducible.Source: B Dataset Details · p. 15

B.1 Dataset Construction

The authors construct task-specific datasets for five domains, then select demonstration sets to cover labels, primitives, and representative cases.

B.1 Dataset Construction

  • Set Game boards contain nine distinct cards and retain boards with at least one valid set, with downsampling to reduce redundancy among easy cases.Each board is paired with a valid set of three cards.Source: §B.1 · p. 15, Figure 8 · p. 15
  • Tapatan examples pair a final board state, without its move sequence, with a win-or-continue outcome, and the splits balance the three labels.Difficulty changes board size, piece limits, and winning-line length.Source: §B.1 · p. 15, Figure 9 · p. 15
  • Operator Function examples pair symbolic expressions with outputs, varying argument and operation complexity; demonstrations include all argument-order permutations.The easy, medium, and hard settings yield 3!, 4!, and 5! demonstration variants.Source: §B.1 · p. 16, Figure 10 · p. 16
  • Noun Class Agreement uses matched grammatical and minimally corrupted sentences, with difficulty varying noun classes, agreement sites, and violation types.Hard examples can include verb-prefix and verb-suffix mismatches.Source: §B.1 · p. 16, Figure 11 · p. 16
  • Lexical Category Inference uses four definitions and a four-word sequence; harder levels use conjunctive or disjunctive multi-condition categories and balanced Yes/No labels.Negative cases cover order, membership, or logical-relation violations.Source: §B.1 · p. 16, Figure 12 · p. 16

B.2 Label and category distributions

  • Main-experiment demonstrations use minimum-coverage sets that cover labels when applicable, all primitives, and representative cases sufficient to infer the task.Dataset statistics are summarized in Table 3, and Table 4 gives demonstration coverage conditions.Source: §B.3 · p. 16, §B.2 · p. 16, Table 3 · p. 17

C Example Scaling Experiments

Example scaling does not yield a consistent advantage over rules: its effects vary by task and model, and can include performance degradation.

C Example Scaling Experiments

  • Example scaling is highly task- and model-dependent, often producing no added gains or performance degradation rather than reliably surpassing rules.The figure plots rules-minus-examples accuracy difference against example count; red regions favor rules and green regions favor examples.Source: C Example Scaling Experiments · p. 17
  • When scaling helps, gains most often occur at Easy difficulty and benefit smaller base models.The leftmost plotted example count matches the main experiment's minimum-coverage set.Source: C Example Scaling Experiments · p. 17
  • In some instruction-tuned Set Game runs, extra examples triggered repetitive analysis and premature truncation, causing severe performance drops.The authors report that a longer truncation window is unlikely to fix this degradation.Source: C Example Scaling Experiments · p. 17

D Lexical Category Inference: Effect of Conjunctive (AND) vs. Disjunctive (OR) Category Definitions

Lexical Category Inference shows an AND advantage for examples but not a stable one for rules or combined prompting; the appendix also presents subgroup analyses and prompt materials.

D Lexical Category Inference: Effect of Conjunctive (AND) vs. Disjunctive (OR) Category Definitions

E Additional Statistical Analyses

F Full Prompts

  • The appendix provides rules-mode prompts for the tasks and describes how examples are placed after rules in combined prompts.Example-based and combined prompt structures are shown in Table 8.Source: F Full Prompts · p. 22, Table 8 · p. 35

Related papers

Loading 2609.03213v1…