Which Differences Matter? Controlled Tests of Rule Transfer, Joint Reliability, and Information Use in Decision Models

Motivation

• Equal accuracy can conceal models that ignore rule and state changes or respond only to states, and valid option selection can coexist with answer-symbol binding errors.The introduction frames the central question as which input changes guide rule-based decisions.Source: §1 · p. 1

Evaluation approach

• Executable arithmetic rules, semantic rule separation, and crossed rule–state tests make transfer and joint success separately measurable.Equivalent rewrites hold behavior fixed while changing presentation; marginal-preserving reassociation calibrates all-four correctness, and altered table values or mappings define alternative answer targets.Source: §1 · pp. 1–2

Results

• Adaptation gains rise with nearest-source answer agreement; fully conflicting quartets show low pooled joint success and opposite gain signs across source splits.Equivalent wording increases the adaptation advantage on binary decisions but reduces it on the other tasks; joint-score departures depend on the associations retained by the calibration reference.Source: §1 · pp. 1–2

Information use

• Paired answer transitions under value and mapping interventions identify directed use of external values beyond aggregate accuracy gains.The decision service changes answers more across equivalent wordings than across identical-text repeats and follows targets implied by altered table values and mappings.Source: §1 · pp. 1–2

Behavioral evaluation and explicit rules

• Behavioral evaluations and rule-following benchmarks motivate testing which inputs guide decisions, while this work combines finite-domain semantic separation with crossed rule and state changes.The cited landscape includes perturbation and contrast-set tests, inference benchmarks, logical systems, and rule-selection evaluations.Source: §2 · p. 2

Generalization, wording, and answer interfaces

• Generalization and answer-interface studies motivate paired tests of unfamiliar rule behavior across equivalent presentations with fixed model weights and intended answers.Related work spans compositional generalization, requested concept order, prompt format, answer-symbol binding, and adaptation.Source: §2 · pp. 2–3

Information use and internal mechanisms

• Prior information-use work examines context and representations, while paired value and mapping interventions test whether supplied information directs answers.Source-selected donor replacements separately test whether recovered rule-value information influences the final selected answer.Source: §2 · p. 3

3.1 States, Rules, and Decisions

• A decision instance combines a state, explicit rule, task family, and ordered options; an oracle supplies the semantic answer and option index for model prediction.The task families include binary decisions, ordinal ratings, minimum routing, and maximum selection, with record identifiers resolving ties.Source: §3.1 · p. 3

3.2 Semantic Separation and State Sampling

• Rules are linked when their signatures match over the finite record domain, and connected components conservatively keep linked rules in one pool.Replication excludes historically exposed components, allocates disjoint source and development pools, and reserves components for evaluation; previously used complete states are excluded across roles.Source: §3.2 · p. 3

3.3 Natural Accuracy and Joint Correctness

• Diagnostic quartets cross two rules with two states, requiring the oracle answer to change along either axis; a deterministic predictor invariant to either factor cannot solve the full quartet.Item accuracy and QAcc are compared on identical diagnostic questions, while natural accuracy retains its original state-sampling distribution; the same item score can mask different rule and state responses.Source: §3.3 · pp. 3–4

4.1 Experimental Setup

• The study uses controlled comparisons of Base, LoRA, frozen readouts, and Jev to test transfer, joint correctness, and directed use of external information.The replication crosses two rule splits with three state seeds; later wording and conflict analyses reuse the reserved target pool.Source: §4 · p. 4, §4.1 · p. 4

4.2 RQ1: How Do Source Conflict and Wording Shape Transfer?

• At full conflict, LoRA’s accuracy fell from 20.47% to 14.37% in split 1 but rose from 22.41% to 31.29% in split 2; QAcc was 0.00% and 2.83%.Pooling concealed the opposite gain signs: the item gain was 1.39 pp and QAcc was 1.42%.Source: §4.2 · p. 5, Table 2 · p. 5
• Equivalent rewrites alter the LoRA–Base advantage in opposite task-level directions, while Jev disagrees more across wordings than across identical-text repeats.For LoRA, the functional rewrite reduces its advantage; the verbal rewrite changes little overall as binary gains offset declines elsewhere. Jev’s across-wording excess persists after interrupted quartets are excluded.Source: §4.2 · p. 6, Figure 2 · p. 6, Figure 3 · p. 6

4.3 RQ2: What Does Calibrated Joint Correctness Show?

• Pooled LoRA and Jev QAcc fall within marginal-preserving reference ranges, while maximum-selection scores exceed them; preserving ordered rule pairs brings its reference means closer to observed QAcc.Departures depend on the associations retained by the reference; falling within a range does not change observed all-four success.Source: §4.3 · pp. 6–7, Table 3 · p. 7

4.4 RQ3: How Do Answers Respond to External Values?

• Complete value tables improve Jev consistently across replication cells, and Jev usually switches to the alternative target implied by altered table values or mappings.The original oracle stays fixed during the value and mapping interventions.Source: §4.4 · p. 7, Table 4 · p. 7, Figure 4 · p. 8

• Base, selected readouts, and LoRA all fail the independent-audit qualification gate in six replication cells, leaving source competence unmatched.The gate requires 90% natural accuracy and 75% QAcc; independent audit matching succeeds in none of the cells.Source: §5 · p. 8
• Source-selected internal donor replacements yield no stable rule-specific improvement over controls.The appendix reports probe results, intervention norms, and explicit paired positive controls.Source: §5 · p. 8
• Validate internal interventions with restoration and positive controls before interpreting recovered rule information as decision-relevant.Source: §5 · p. 8

• Adaptation gains increase with nearest-source answer agreement, while fully conflicting quartets show low pooled QAcc and opposite gain signs across source splits.Source: §6 · p. 8
• The conclusion recommends distinguishing source-consistent success from correct decisions under conflict, specifying calibration associations, and reporting attained source competence alongside target scores.Joint reliability remains a design objective even when individual decision accuracy improves under unfamiliar rules or supplied information.Source: §6 · p. 8

Limitations

• The findings concern tested rule families and evaluation conditions, not broad natural-language policy coverage.The core grammar uses fixed-depth arithmetic rules on three bounded integer fields; Boolean and finite-state extensions test additional formats.Source: Limitations · p. 9
• LoRA–readout comparisons estimate selected training-procedure outcomes because independent source-audit competence remains unmatched.The larger-model comparison also changes system identity, so its effects cannot be attributed to parameter count alone.Source: Limitations · p. 9
• QAcc measures joint success on constructed rule–state quartets, while calibration departures depend on the associations retained by each reference.Small or constant-prediction strata restrict available reassociations; multi-record second states use swapped winning-record fields, so the state-vector reference does not establish reliability on independently sampled state pairs.Source: Limitations · p. 9
• Wording comparisons cover the original rendering and two equivalent templates, while Jev repeats estimate variability only within the observed service window.Table interventions establish directed use of supplied information in answers, not an internal decision mechanism.Source: Limitations · p. 9

Ethics Statement

• The study uses synthetic states and oracle labels, with no human participants or personal records; remote services receive only synthetic inputs.The authors caution that benchmark performance does not establish suitability for consequential decisions.Source: Ethics Statement · p. 9

A.1 Executable Rules and Task Semantics

• The rule grammar yields 125 syntactic candidates, of which 104 pass eligibility checks and form 77 behavioral-equivalence components over the finite integer domain.The components use numeric, binary, ordinal, and weak-order signatures, including ties and field permutations; the audit does not establish symbolic equivalence over real inputs.Source: §A.1 · p. 11

A.2 Rule Pools and Complete-State Isolation

• Natural sampling draws complete states uniformly from the finite domain, and source, validation, development, and replication roles use disjoint complete-state keys.Questions for different rules within a natural role intentionally share complete states; the pool construction also excludes historically exposed components.Source: §A.2 · p. 11

A.3 Semantic Reconstruction and Similarity Audit

• Exhaustive signatures across the finite domain and six field permutations show no cross-pool equivalence, while non-equivalent rules may still agree frequently.Nearest-source oracle agreement is analyzed post hoc under original and best-permuted fields.Source: §A.3 · p. 11

A.4 Exact Eligibility and Pool Construction

• Eligibility rejects rules with simpler-expression matches, unused fields, or constant binary answers, and connected components conservatively group rules linked by any matching signature.Pool selection uses root quotas and seeded matching; the remaining 26 components form the held-out reserve.Source: §A.4 · p. 12

A.5 Counterbalanced Diagnostic Construction

• Diagnostic quartets are sampled to require checkerboard answers across both rules and states, with multi-record states formed by swapping winning-record fields.Candidate-order reversals are separate four-item groups, and diagnostic scores are reported separately from natural accuracy.Source: §A.5 · p. 12

A.6 Evaluated Quartet and Local Decision Prompt

• The evaluated binary example uses a crossed rule-and-state quartet, and the local prompt asks whether the supplied formula is strictly greater than zero.Its illustration was selected by dataset order before model predictions were examined.Source: §A.6 · pp. 12–13

A.7 Routing Example

• The routing example contrasts two models with equal correct counts but different answer positions across two rules and two states.It uses minimum-scoring record selection with the specified pair of rules and states.Source: §A.7 · p. 13

A.10 Macro Metrics and Paired Comparisons

• Macro metrics average natural item scores across rule, task, and option-count strata, while quartet metrics separately measure mean correctness, all-four success, and correct-count distributions.Paired rule- and state-switch outcomes track both correctness and whether semantic predictions change.Source: §A.10 · pp. 13–14

B.1 Input Rendering and Candidate Scoring; B.4 Precision and Interface Qualification

• The compared external systems retain their native interfaces, so these results describe deployed configurations rather than a shared architecture or training recipe.The rank-4 versus controlled-Base contrast also combines readout differences with different candidate-logit projection precision.Source: §B.1 · p. 14, §B.4 · p. 15

B.2 LoRA Adaptation; B.3 Frozen Residual Readouts

• Replication crosses two rule splits with three state seeds, while LoRA and frozen readouts use distinct fitting procedures and differ in parameter count and computation.The readout selection chooses MLP128 in every cell, but split 1/state 202 uses a different feature set from the other cells.Source: §B.3 · p. 15

C.1 Initial System Comparison; C.2 Replication by Rule Split and State Seed; C.4 Correct-Count Distributions and Required Response Changes

• Higher item accuracy coexists with weak all-four correctness, and LoRA and Jev change relative ranking across rule splits.Quartet analysis also distinguishes partial success and failures to change answers when rule or state changes require it.Source: §C.1 · p. 16, §C.2 · p. 16, §C.4 · p. 16

D.2 Replicated Auxiliary Effects

• Complete value tables produce highly variable paired gains: across cells, LoRA item-accuracy differences range from −0.26 to 7.03 pp and Jev differences from 30.47 to 38.02 pp.LoRA QAcc differences range from −3.13 to 9.38 pp, while Jev QAcc differences range from 50.00 to 59.38 pp; the results cover two rule allocations.Source: §D.2 · p. 18, Table 12 · p. 19

D.3 Answer-Binding Positive Controls

• The binding positive control finds that receiving the correct semantic answer does not guarantee that a system will bind it to the displayed option.Its source-validation rates and original new-rule D6 rates use different inputs, so their difference cannot isolate instruction wording.Source: §D.3 · p. 19

E Source-Competence and Optimization Audits

• Selected richer readouts fail the source-competence gate, with source-audit natural accuracy of 41.80–52.08% and audit QAcc of 0–7.29%.The gate requires at least 90% natural accuracy and 75% all-four accuracy on both source-training and independent source-audit questions.Source: E Source-Competence and Optimization… · pp. 19–20

F.1 Frozen Models, Blocks, and Coverage

• LoRA’s reserved-rule advantage over selected readouts is evaluated on fixed models and paired rule blocks, with complete states and feasible quartets retained.The reserved evaluation contains 4,160 natural questions and 254 feasible diagnostic quartets across 13 blocks.Source: §F.1 · p. 20

F.2 Finite-Benchmark Decision Rule

• 14.59 percentage points is the mean reserved-rule natural-accuracy advantage for LoRA over the frozen readout, with all 13 block differences exceeding five points.The block differences range from 9.51 to 19.53 points.Source: §F.2 · p. 21
• The predeclared criterion requires a mean advantage of at least 5 pp and differences above 5 pp in at least ten of 13 blocks.The criterion was fixed before reserved-rule questions or predictions were generated.Source: §F.2 · p. 21
• The pass applies to this reserved finite benchmark, not to a population-level significance test or independent source-retraining confirmation.Blocks share trained models and a finite grammar.Source: §F.2 · p. 21

G.1 Probe Data, Targets, and Fitting

• Probe scores predict oracle values or legal answers from hidden features, and compare real-label performance with shuffled-label controls.The paper cautions that recoverable information alone does not establish causal use in answering.Source: §G.1 · p. 21

G.2 Donor–Receiver Pairs and Source Selection

• Donor–receiver pairs combine different rules and states, and the counterfactual target applies the donor rule to the receiver state.The source-selected patch layer and mode are chosen on source validation and then held fixed for audit and development evaluation.Source: §G.2 · pp. 21, 23

G.1 Probe Data, Targets, and Fitting

• Probe gains over shuffled targets are stronger on source audits than development inputs at many tested LoRA locations.The figure reports real-minus-shuffled-target accuracy for value and answer targets.Source: Figure 10 · p. 22

G.3 Intervention Controls and Numerical Validation

• Source-selected patches produce little change in counterfactual target success relative to the native baseline.The patch choices retain native prediction, restoration, donor, direction, and erasure controls for comparison.Source: §G.3 · p. 23
• Applied BF16 relative norms are 4.84%, 3.39%, 2.19%, and 4.07% for Base/split 1, Base/split 2, LoRA/split 1, and LoRA/split 2, respectively.Prediction changes are 1.56%, 0.00%, 0.00%, and 0.00% in the same order.Source: §G.3 · p. 23

H Joint-Metric Calibration and Conditional Uncertainty

• Reassociation preserves prediction marginals conditional on task, option count, quartet slot, and true label, while breaking associations within original quartets.The reference uses 1,000 reassociations and is descriptive, not an independent-row sampling model or causal test.Source: H Joint-Metric Calibration and… · p. 23
• Both LoRA and Jev exceed the maximum-selection reference range, while other task-level observations lie within their ranges.The pooled result therefore does not imply an absence of joint dependence in every task.Source: H Joint-Metric Calibration and… · p. 24
• Small, singleton, and constant-prediction strata limit how much the reference can change.The reported task-specific ranges are descriptive quantiles rather than confidence intervals or multiplicity-adjusted significance assessments.Source: H Joint-Metric Calibration and… · p. 24

I Common Source Audit and Bounded Competence Matching

• The common audit uses 640 source-natural questions and 320 diagnostic items, forming 80 quartets per cell shared across methods and independent of training and model selection.The competence gate requires 90% natural accuracy and 75% QAcc separately on original training inputs and held-out source-rule audit questions.Source: I Common Source Audit and Bounded… · p. 25
• None of Base, the selected readout, or LoRA meets the competence gate in any of six cells; LoRA reaches 75.43% natural accuracy / 35.94% QAcc on the common source audit.Base reaches 36.31% / 0.00%, and the selected readout reaches 45.70% / 1.39%.Source: I Common Source Audit and Bounded… · p. 25
• Source-validation matching succeeds in 2/6 cells, but independent source-audit matching succeeds in 0/6 under the prespecified 3-point tolerance.The search checks steps 0, 10, 20, 40, 80, and 160; Figure 12 shows the selected checkpoints and audit gaps.Source: I Common Source Audit and Bounded… · p. 25, Figure 12 · p. 26

J.1 Fixed Inputs and Semantic Verification

• Formula-row mismatch reassigns computed-score rows, while record-column mismatch applies a common column permutation; both preserve the table’s value multiset and serialized character count.The column permutation is inapplicable to single-record tasks; 190 of the remaining 192 questions conflict with the original oracle, while formula-row mismatch creates 220 conflicts.Source: §J.1 · p. 26
• The target-value counterfactual changes every question’s table target to the next option in a predefined cycle, while retaining the original task gold answer.Value-range, string-length, and sign properties are matched in 292, 165, and 162 questions, respectively; following the altered table target does not count as correct on the original task.Source: §J.1 · p. 26

J.2 Paired Answer Transitions and Denominators

• Paired-transition rates use T/N across all conflicting questions and T/B among conflicts initially answered correctly with the correct table; strata with B = 0 are undefined.The main-text estimate is pooled micro; the supplement’s macro estimate pools splits within task/K, then averages equally over K and tasks.Source: §J.2 · p. 27

K Execution and Reproducibility

• The reproduction package identifies protocols, generators, executable oracles, model identities, checkpoints, and predictions by SHA256, but excludes model weights and private credentials.BF16 training replay is not bitwise identical; reloaded readouts retain semantic predictions with a maximum checked probability difference of 8.35 × 10−7.Source: K Execution and Reproducibility · p. 27

L.1 Data Roles and Fixed Evaluation; L.2 Precision and Padding Controls

• Batch-sensitive choices qualify wording comparisons, but batch-stable contributions preserve all eight taskwise directions and equivalent rewrites still change answers under FP32 inference with fixed padding.Changing the baseline to the same batch size shifts both pooled gap changes by −0.28 pp; the paired precision-and-padding test finds wording sensitivity beyond the observed BF16 padding effect.Source: §L.1 · p. 27, §L.2 · p. 28

L.3 Nearest-Source Similarity and Conflict Subsets

• Nearest-source rules are selected by oracle-answer agreement over uniform finite-domain inputs, and conflict subsets retain questions where target and reference answers differ.The analysis distinguishes uniform-domain agreement from agreement on accepted natural states; the latter can change through exclusions and finite sampling.Source: §L.3 · pp. 28, 30

L.4 Agreement-Stratified Transfer and Complete-Quartet Conflict Loads

• Natural gains rise with nearest-source agreement, while fully conflicting quartets show low pooled joint success, little pooled gain, and opposite gain signs across source splits.Intermediate conflict loads do not show a monotonic gain trend.Source: §L.4 · p. 30, Table 20 · p. 32, Table 22 · p. 33

L.5 Adjacent Randomized Jev Repetitions

• Across adjacent randomized service calls, across-wording disagreement exceeds same-text disagreement, and the excess remains positive in all five slot-lag strata.The paired comparison averages within-item differences and uses whole-quartet resampling; a sensitivity analysis excludes five quartets with retries or long request gaps.Source: §L.5 · p. 31

L.6 Larger System and Boolean Policies

• Qwen3.5-9B does not uniformly outperform arithmetic-adapted LoRA: it performs better on Boolean policies but has weak exception-handling QAcc, limited transition accuracy, and zero transition-task quartet success.The arithmetic LoRA was reused without policy training, and the reported finite-set results are descriptive.Source: §L.6 · pp. 31–32, Table 24 · p. 34

L.7 Actual Wording and Policy Inputs; L.10 Native Jev Requests and Table Interventions

• The illustrated arithmetic wordings preserve the same state, options, and oracle answer while changing only the question field; the native-request example specifies correct-table, row, column, and target-value interventions.In the routing example, the column intervention changes the target from R1 to R0, whereas the target-value intervention creates a tie resolved in favor of R0.Source: §L.7 · p. 33

L.9 Answer-Transition Weighting Sensitivity

• Answer-transition rates count applicable target conflicts using both all-conflict and originally-correct denominators, with micro, task/K, and equal-cell weighting schemes defined separately.Zero-denominator strata are excluded from each rate, and coverage is recorded.Source: §L.9 · p. 34

L.8 Support of the Reassociation Reference

• The reassociation reference reports finite-cohort departures in joint correctness, with its draw-based difference range explicitly descriptive rather than a confidence interval or range of raw QAcc.The reference holds strata and task-level structure fixed; singleton and constant-prediction strata cannot contribute variation.Source: Table 25 · p. 35

L.10 Native Jev Requests and Table Interventions

• Maximum-selection departures vary across cells and option counts, while the paired-transition analysis distinguishes table changes from changes in the implied answer.The example retains original gold answer R1 in both conflict cases, identifying the conflicts used in every denominator in that analysis.Source: Table 26 · p. 35, §L.10 · p. 35

L.11 Additional Rule Split and Interface Extensions

• Correct-to-new-target transitions under finite-state symbol-mapping swaps are 9/88 (10.23%) for Base, 10/82 (12.20%) for LoRA, and 23/140 (16.43%) for Qwen3.5-9B.All systems have zero QAcc, and the paper cautions that this alone does not establish an additional structural deficit; the tested arithmetic adaptation does not confer broad finite-state rule competence.Source: §L.11 · pp. 35–37
• The fixed native chat template lowers Qwen3.5-9B natural accuracy under all three wordings relative to raw prompting on paired inputs.The comparison holds model weights and question identities fixed and directly scores candidate letters without generated reasoning or additional prompt or checkpoint selection.Source: Figure 14 · p. 36, §L.11 · p. 36
• On the added disjoint split, Base and LoRA score 38.80%/0.00% and 58.46%/19.79% natural accuracy/QAcc; gains are 5.99 pp on conflicts versus 27.69 pp on agreeing answers.Conflict questions carry 36.98% of the full-set weight; the added split is a development replication because its components appeared in reserved evaluation.Source: §L.11 · p. 36

M Comparison of Evaluation Designs

• Quartets make rule and state identities explicit, separating item accuracy from all-four correctness; value-mapping interventions distinguish following supplied values from retaining the original answer.Local perturbation tests measure behavioral sensitivity, whereas compositional benchmarks test generalization across combinations; the designs can complement one another when controlled factors and score denominators are reported transparently.Source: M Comparison of Evaluation Designs · p. 37