Can a Cacheable Decision Model Follow Rules?

1 Introduction

• Certo scores candidate actions and returns a probability for fixed-option decisions, including choice, ordinal scoring, and yes/no tasks.Source: §1 · p. 2
• Reading the state, rules, and candidate together lets the joint scorer apply rules, but candidate representations cannot be reused across states and menu cost grows.Independent encoding reuses each candidate vector across states, but separates state from candidate and may remove the interaction needed for rule application.Source: §1 · p. 2
• The paper asks how much rule-sensitivity survives conversion to cacheable independent encoding, whether training can restore it, and whether it transfers to unseen real rules.Source: §1 · p. 2
• Four experiments find that conversion loses rule-sensitivity, targeted counterfactual supervision restores synthetic-task performance, and transfer to real rules is not clearly established.A shortlist rescue fails, and a cross-domain real-prose pilot does not help.Source: §1 · p. 2

2 Setting and architectures

• Apair concatenates each candidate with the state and rules, while A10 reads the state once and uses a native K-way head capped at ten candidates.Source: §2 · p. 3
• B separately encodes state and candidate into normalized single vectors; C uses per-token vectors and late-interaction MaxSim, and both cache candidate representations.Source: §2 · p. 3
• Rules reside in the state, leaving candidate embeddings reusable even for rule-sensitive decisions.Source: §2 · p. 3
• The protocol matches training examples and optimizer steps, uses disjoint rule-family and document-source splits, and bootstraps confidence intervals by dependence unit.Reported latencies use single-query p50 on one GPU with warm candidate-vector caches and indicate a trend, not an optimized deployment.Source: §2 · p. 3
• Recall@1 is 1.00 for the joint scorer across menu sizes, versus 0.24 for B and 0.08 for C on the 17-candidate rule-sensitive slice.Source: Figure 2 · p. 4

3 Experiment 1: making scoring cacheable loses rule-sensitivity

• The comparison trains all four scorers from the same reference model at a matched budget and measures recall@1 on routing and rule-sensitive selection across menu sizes.Source: §3 · p. 4
• 1.00 rule-sensitive recall@1 for the joint scorer contrasts with 0.24–0.54 for the single-vector encoder; the multi-vector encoder nears chance, while cacheable scoring stays approximately flat.The measured latency gap is 2.9× at K20 (135 vs. 46 ms) and 5.3× at K77 (242 vs. 46 ms).Source: §3 · p. 4, Figure 3 · p. 5
• At k=5 and k=10, rule-sensitive candidate-inclusion recall is 0.72 and 0.80; it reaches 1.00 only at k=17, the full 17-candidate menu.The target was at least 0.98, so none of the tested reduced shortlists both saved compute and met the target.Source: §3 · p. 5
• On routing, candidate-inclusion recall is 0.70, 0.82, and 0.93 at k=5, 10, and 20, while hybrid top-1 accuracy of 0.57–0.64 does not exceed the joint reranker’s 0.65.Source: §3 · p. 5

Experiment 2: counterfactual supervision recovers it on synthetic rules

• Training the single-vector encoder on counterfactual pairs keeps candidates cacheable while flipping the correct action when one rule condition changes.The semantically tempting prohibited action is included as a hard negative, and positives are labelled by an executable rule.Source: §4 · p. 5
• The held-out evaluation covers paraphrased wordings, counterfactual pairs, operator composition, and rule types absent from training.Results are reported as recall@1 against the pre-training encoder and a control continued on unrelated data.Source: §4 · p. 5, Table 2 · p. 6
• 98.7% of counterfactual pairs were answered correctly on both sides, versus 30% for the baseline; performance also extended to paraphrases, operator composition, and some untrained rule types.The result reproduced under a second seed; routing showed no statistically significant change, but the interval still admits a roughly 8-point drop.Source: §4 · p. 6, Figure 4 · p. 6
• The synthetic suite does not isolate whether predictions depend on the stated rule, because its counterfactual pairs change a condition in the state instead.The proposed decisive control would hold facts and candidates fixed while changing only the rule; the suite is also synthetic and templated.Source: §4 · p. 7

Experiment 3: real rules, and a truncation confound

• Right-truncation can remove the joint scorer’s appended candidate, and only 29 of 67 hard-tier items fit both scorers’ trained windows.Rescoring frozen models within the trained window separates skill from this input-validity confound.Source: §5 · p. 7
• 0.861 vs. 0.500 recall@1 on the short tier favors the joint scorer significantly; on the hard tier, its 0.655 vs. 0.483 edge is directional but not significant.The hard-tier comparison uses 29 fitting items; the 38 truncated items are set aside, so 0.655 is not a measured gain across all 67.Source: §5 · pp. 7–8
• Synthetic training showed no convincing added benefit over the pre-trained encoder on real rules, so transfer of the synthetic recovery is not established.The reported treatment-versus-B differences were short d = 0.000 and hard d = +0.069.Source: §5 · p. 8

Experiment 4: a controlled real-prose pilot

• The controlled pilot matched example and step budgets, but the treatment both removed synthetic examples and added privacy-policy decisions.Because these changes occurred together, the observed reduction cannot be attributed separately to either change or their interaction.Source: §6 · p. 8
• −0.093 on the fitting slice and −0.162 on contract-qa were the contract accuracy differences after replacing part of the synthetic mixture with privacy-policy examples.Both differences had confidence intervals fully below zero; the pilot stopped under its pre-registered rule requiring a meaningful gain over continuation.Source: §6 · p. 8
• The pilot supports a narrow conclusion for one privacy-to-contract source pairing in binary format, which does not exercise the joint-versus-cacheable choice gap.It used one seed and did not proceed to a second seed.Source: §6 · p. 8

7 Discussion

• Held-out synthetic generalization does not establish dependence on supplied rules or transfer to unseen-source real rules; the joint scorer wins on short real-rule items.The hard-tier joint-scorer edge is uncertain, and candidate recomputation is its cost.Source: §7 · pp. 8–9
• The pilot treatment fell below both control and untrained starts on contract tests, while synthetic-rule and routing retention was approximately unchanged.The figure reports treatment-minus-control bootstrap intervals fully left of zero for both contract tests; no equivalence test was performed for the approximately unchanged tasks.Source: Figure 6 · p. 9
• At K77, the cacheable encoder’s routing accuracy is 0.422 vs. 0.652 for the joint scorer, making the practical choice application-dependent.The cacheable encoder offers a serving advantage, but applications must decide how much accuracy to trade for latency.Source: §7 · p. 9
• The hard-tier null resulted from right-truncating an appended candidate; an input audit and within-window rescore fixed the confound without more training.Reporting fitting and truncated inputs separately can guard against this evaluation-window confound.Source: §7 · p. 10

8 Future work

• A rule-only counterfactual control should change only the stated rule, separating genuine rule application from a learned template-specific procedure.Deleted-rule and shuffled-rule baselines are proposed as part of this test.Source: §8 · p. 10
• A third pilot arm with reduced synthetic exposure but no privacy supervision would separate the effects of removing synthetic examples from adding privacy prose.The existing treatment changes both factors at once.Source: §8 · p. 10
• A real multi-candidate Choice corpus would test the joint-versus-cacheable gap in a setting where it is clearest.Current real-rule experiments lean on binary and short-clause tasks.Source: §8 · p. 10
• A prefix-reusing joint latency baseline and longer-window evaluation with state-side truncation would better scope caching gains and remove the context-length confound.The current latency comparison uses a joint scorer without prefix caching.Source: §8 · p. 10
• Source-matched training and testing, plus a joint scorer trained on the same new corpus, would distinguish cross-domain transfer limits from architecture effects.The pilot trained on privacy policies and tested on contracts.Source: §8 · p. 10
• A length, mask, or aggregation ablation could test whether long causal state swamps late interaction, the proposed but unisolated cause of C’s MaxSim collapse.A rule-aware multi-vector encoder may also recover more than the single-vector encoder.Source: §8 · p. 10
• Recall@1 on the untrained precedence rule type tops out at 0.73 on both seeds; ordered-rule supervision is a proposed next step.Source: §8 · p. 10
• Calibration under transfer remains unmeasured, leaving open whether a rule-aware cacheable encoder stays calibrated on unseen-source rules.Source: §8 · p. 10

• Real-rule slices contain tens of items, have wide intervals, and are mostly binary or short-clause; the choice gap rests on JevBench slices.Source: §9 · p. 11
• Held-out synthetic performance does not establish dependence on the supplied rule; the rule-only control has not been run.Source: §9 · p. 11
• The real-prose pilot uses one source pairing and seed, so another pairing or a choice-format corpus could differ.Source: §9 · p. 11
• C’s collapse mechanism is proposed rather than isolated, and architecture-specific attribution would require a joint model trained on the same new corpus.Source: §9 · p. 11
• All results use one backbone, Qwen3-4B, and one adapter recipe.Source: §9 · p. 11

• The four experiments ran on RTX PRO 4500 Blackwell Server Edition GPUs, with four GPUs for training and one for latency measurements.Source: §10 · p. 11
• Experiment 1 trained four scorer arms for 1,000 steps from the same reference model at effective batch 64; Experiments 2 and 4 continued the dual arm for 600 and 500 steps.All experiments used LoRA adapters, AdamW at learning rate 1 × 10−4, cosine scheduling, 100 warmup steps, bf16, and gradient checkpointing.Source: §10 · p. 11
• The authors released cacheable-arm checkpoints, scripts for all four experiments, evaluation suites, the locked real-prose test set, and prediction files for recomputing intervals without retraining.Source: §10 · p. 11
• Real-rule and pilot evaluations report inputs fitting the trained window separately from those that do not, and treatment-control differences use bootstraps grouped by dependence unit.Source: §10 · p. 11