MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models

• Existing evaluations often measure which source models follow without establishing whether that source is correct.Source: §1 · p. 1
• MemToC holds tool selection fixed while intervening on the observed return, isolating post-tool-call arbitration.Source: §1 · p. 1
• 542 quality-controlled factual questions combine executable calls, verified answers, model-specific closed-book answers, and controlled correct or incorrect returns.Source: §1 · pp. 1–2
• The benchmark evaluates five open-weight 7–9B models across four source-correctness cases, plus tool-error and no-tool controls.Source: §1 · p. 2
• SFT and DPO are compared with chain-level cross-fitting so evaluation questions remain held out from training.Source: §1 · p. 2

• TMC defines conflicts through disagreement between memory-only and tool-conditioned responses, without knowing which source is correct.Source: §2 · p. 2
• Context-DPO and KnowPO optimize reliance on contextual information or adaptive source selection, rather than conditioning selection on competing-source correctness.Source: §2 · p. 2

Benchmark construction

• 3,912 ToolHop sub-questions are deduplicated and filtered to produce 542 quality-controlled factual questions.Source: §3.2 · p. 3, Figure 2 · p. 3
• Near distractors match the verified answer in type, domain, and plausibility, while far distractors preserve type but differ more in period or country.Source: §3.2 · p. 3

Model-specific evaluation

• For each model and formulation, a closed-book answer m is elicited before presenting the same question with a controlled return r and collecting response y.Source: §3.3 · p. 4
• Comparing m and r with verified answer g assigns four source-correctness cases whose targets distinguish retention, correct-tool following, and rejection of both wrong sources.Source: Figure 2 · p. 3, §3.3 · p. 4
• Two annotators independently preferred explicit disagreement or uncertainty in conflict scenarios, achieving Cohen’s κ=0.785.Source: §3.4 · p. 4

Evaluation design

• Five open-weight models are evaluated, including four instruction-tuned models and a base/instruction-tuned Llama comparison.Source: §4 · p. 4, §4.1 · p. 4
• The primary closed-book prompt requires a best guess, while an abstention-permitting sensitivity analysis leaves some cases nearly empty for refusal-prone models.Source: §4.2 · p. 4

Scoring and metrics

• Responses are extracted from FINAL-marked spans and matched against verified answers, closed-book answers, and tool returns using normalization and alias tolerance.Source: §4.2 · p. 4
• Correct-answer retention, correct-tool following, conflict-resolved accuracy, incorrect-tool following, and tool-error abstention separate desirable and undesirable behaviors.Source: §4.2 · p. 5

Mitigation methods

• The success criterion requires improved correct-answer retention without a detected reduction in correct-tool following.Source: §4.3 · p. 5
• SFT and DPO use LoRA adapters and cross-fitting that keeps complete ToolHop chains and related-question groups within folds.Source: §4.3 · p. 5

Baseline arbitration

• Across four instruction-tuned models, correct-answer retention is 6.5%–17.1%, correct-tool following is 86.0%–93.1%, and incorrect-tool following is 78.4%–86.0%.Models generally favor the tool return regardless of its correctness.Source: §5.1 · p. 6
• Tool-error abstention ranges from 73.9% for Mistral-7B-Instruct-v0.3 to 99.5% for gemma-2-9b-it, without predicting behavior on factual conflicts.Reliable abstention after explicit errors does not ensure resistance to plausible incorrect returns.Source: §5.1 · p. 6

Fine-tuning results

• SFT meets the success criterion for Llama-3.1-8B-Instruct and gemma-2-9b-it, but Mistral gains retention by broadly reducing tool use.Llama retention increases by 14.3 points; gemma retention increases by 17.0 points and correct-tool following by 5.6 points.Source: §5.2 · p. 6
• DPO meets the criterion for Llama and Gemma, while Mistral increases retention by 6.0 points but reduces correct-tool following by 4.5 points.SFT and DPO therefore succeed on the same two models, with outcomes varying more consistently by model than objective.Source: §5.2 · p. 6

Robustness and controls

• Across three formulations, within-model retention varies by 7–15 points and no cross-model ordering remains stable.Qwen2.5-7B-Instruct ranks highest on Reference and lowest on Paraphrase A.Source: §5.3 · p. 6
• Prompting, presentation format, distractor distance, and tool-choice protocol all change measured deference or reliability, limiting interpretation of any single setup.Replacing the executed-tool frame reduces incorrect-tool following by 11 points for Llama-Instruct and 35 for Qwen; schema-constrained decoding repairs Llama tool choice from 0.328 to 0.992.Source: §5.4 · p. 7

Additional analyses

• None of 120 annotated incorrect-tool responses explicitly acknowledges the conflict, while transfer beyond MemToC is positive but partial.Transfer gains depend on model and presentation frame, and adapters also reduce abstention on unanswerable questions.Source: §5.4 · p. 7
• A single 70B probe reaches 21% retention and 96% correct-tool following but 85% incorrect-tool following and 44% tool-error abstention.The result is descriptive because scale is confounded with model generation and quantization and uses one seed.Source: §5.4 · p. 8

Scope and construction

• The evaluation uses one ToolHop-derived benchmark core, primarily open-weight 7–9B models, synthetic typed distractors, and no proprietary model.Most machine-authored distractors came from a single generator, and a second quality-controlled realization remains future work.Source: §6 · p. 8

Evaluation coverage

• Additional training seeds cover only the two models that satisfied the criterion, so seed-specific failures cannot be ruled out for the other two.Training-free conflict-aware decoding was also not evaluated despite being named in the preregistration.Source: §6 · p. 8
• Transfer is evaluated on one external multiple-choice dataset using small verified pools, so null results within a frame may reflect limited power.All transfer estimates are restricted to human-verified examples.Source: §6 · p. 8

Measurement limits

• Forced elicitation makes measured retention rates upper bounds, and neither protocol directly accesses model beliefs.Tool-error abstention is also measured with a single structured service-unavailable payload whose rate depends on the displayed error.Source: §6 · p. 8
• A high tool-error abstention rate is not intrinsically desirable because error payloads change fallback and abstention differently across memory-correct and memory-wrong questions.For one Llama model, a permission error raises correct fallback from 0.09 to 0.35 but lowers abstention from 0.90 to 0.54.Source: §6 · p. 8

• Tool returns strongly dominate elicited closed-book answers, and 0 of 120 annotated conflicts acknowledge the disagreement.The 95% upper bound for conflict acknowledgment is 3%.Source: §7 · p. 8
• Cross-model rankings change across prompt formulations and presentation frames, so tool-deference rates are not stable model properties.Source: §7 · p. 8
• SFT and DPO satisfy the asymmetric success criterion on the same two of four instruction-tuned backbones, with transfer beyond MemToC remaining partial.Source: §7 · p. 8
• 19 of 20 tested method–model combinations reduce abstention after tool errors or on unanswerable inputs.Evaluating only resistance to incorrect returns can therefore reward broad distrust rather than better arbitration.Source: §7 · p. 8

• The benchmark, construction and evaluation code, scoring layer, and per-claim number map are released under Apache-2.0.A verification script recomputes 180 reported values; the separate three-payload tool-error comparison is not part of the release.Source: §8 · pp. 8–9
• The appendix lists every evaluation prompt with its experiment and protocol position, including closed-book, tool-return, intervention, and retrieval-passage templates.Source: A Prompt Templates, by Experiment · pp. 10–11
• A verbatim benchmark episode shows a verified answer, an executable actor lookup, an incorrect tool return, and the model’s closed-book answer.Source: A Prompt Templates, by Experiment · p. 10
• Three prompting interventions alter only the inserted strategy sentence, while question text and episode content remain fixed across formulations.The strategies are warning, source-priority, and abstain-and-flag.Source: A Prompt Templates, by Experiment · pp. 10–11
• The retrieval-passage comparison changes how the same external value is presented while keeping the payload, question, and required answer format fixed.Source: A Prompt Templates, by Experiment · p. 11

Experimental design

• 542 quality-controlled factual questions support executable calls, verified answers, model-specific closed-book answers, and controlled correct and incorrect returns.Source: Figure 3 · p. 11

Evaluation criteria

• The asymmetric success criterion requires improved correct-answer retention without a detected reduction in correct-tool following.Source: B Experimental Setup and Reproducibility · p. 12

Fine-tuning

• SFT and DPO are compared with chain-level cross-fitting, ensuring questions from shared ToolHop chains do not cross training and evaluation folds.The two folds contain 287 and 288 questions from 154 and 155 chains, respectively.Source: B Experimental Setup and Reproducibility · p. 12

Robustness

• Three prompt formulations keep question text, verified answer, tool call, and tool return byte-identical while varying instruction wording.Source: B Experimental Setup and Reproducibility · pp. 12–13

Reproducibility boundaries

• Major benchmark revisions changed incorrect-tool following, retention, and fine-tuning results enough that affected findings are reported as construction-scoped.One revision increased incorrect-tool following to 0.71–0.83, while group cross-fitting reduced a Llama-3.1-8B-Instruct DPO retention gain from +0.24 to +0.058.Source: B Experimental Setup and Reproducibility · p. 13

• Steering is direction-asymmetric: knocking the model off the tool is easier than re-inducing its parametric answer.Source: C Steering the Representation with… · p. 13
• Table 2 maps reported numbers to released files, but absolute fine-tuned Table 1 levels and three previous-version measurement layers cannot be recomputed from the release alone.Source: Table 2 · p. 14
• The memory-ward edit reduces kept-memory behavior by −0.53 while increasing tool-following by only +0.25.Source: C Steering the Representation with… · p. 14
• Generation-preserving edits reach only +0.10 on Gemma and +0.125 on Llama, both below the +0.15 steering bar.Source: C Steering the Representation with… · p. 14
• Stronger Gemma edits cross the nominal bar but annihilate tool-following, collapse outputs into schema echoes, and produce confabulated provenance.Source: C Steering the Representation with… · p. 14
• Only 5 of 80 top mutual-information features are entity- or fact-content features, leaving little content-specific control for the memory-ward edit.Source: C Steering the Representation with… · p. 15

Scorer and protocol validation

• The primary required-best-guess protocol treats retention as an upper bound because allowing abstention leaves some source-correctness cases nearly empty.A separate released-benchmark annotation round found zero acknowledged incorrect-tool responses among 120 labeled responses, bounding rather than estimating acknowledgment prevalence.Source: D Benchmark Construction, Quality… · p. 16

Benchmark construction

• Computational questions are removed so retained answers can meaningfully conflict with parametric knowledge.A rule-based filter rejects answers produced by operations over earlier answers rather than recalled facts.Source: D Benchmark Construction, Quality… · p. 16

Leakage control

• Question-level grouping prevents shared underlying facts from crossing training and evaluation folds.This leakage control applies beyond exact deduplication to near-identical formulations and related questions.Source: D Benchmark Construction, Quality… · p. 16

Distractor design

• Near and far distractors preserve answer type while differing in plausibility or temporal and geographic distance.Near distractors are indistinguishable in type and plausibility; far distractors differ more strongly in period or country.Source: D Benchmark Construction, Quality… · p. 17

Benchmark construction

• 542 quality-controlled factual questions form the released benchmark after review, repair, exclusion, and distractor validation.The construction pool is reduced from 575 to 542 questions after adjudication.Source: D Benchmark Construction, Quality… · p. 18

Human review

• 575 →542 questions remain after blinded review, with repairs changing only incorrect substituted values and exclusions removing unsuitable records.Question wording and verified answers remain frozen byte-for-byte during repair.Source: D Benchmark Construction, Quality… · p. 18

Formulation stability

• Three instruction-wording variants produce overlapping retention ranges and changing model rankings, so cross-model retention orderings are not stable.The question text and episode content remain fixed across Reference, Paraphrase A, and Paraphrase B.Source: E Additional Stability Analyses · p. 20, Figure 4

Prompting trade-offs

• Eleven of twelve pooled prompting comparisons significantly improve retention, but each model pays a model-specific cost.Costs include reduced correct-tool following for some models and reduced tool-error abstention for others.Source: E Additional Stability Analyses · p. 20
• Abstain-and-flag collapses correct-tool following on every model, while gemma preserves tool-error abstention under warning prompting from 100.0 to 89.7.The intervention patterns differ by model and strategy rather than yielding a uniform safety improvement.Source: E Additional Stability Analyses · p. 20

Protocol ablation

• Removing the protocol sentence leaves arbitration retention statistically unchanged but reduces correct-tool following by 0.093.The ablation also reduces no-conflict accuracy by 0.056, while tool-error abstention is read as null.Source: E Additional Stability Analyses · pp. 20–21

Criterion stability

• The asymmetric criterion is formulation-sensitive: only gemma’s SFT arm meets it under every formulation, while some pooled findings fail or reverse by wording.Mistral’s warning arm is −0.016 under Reference versus +0.065 pooled.Source: E Additional Stability Analyses · p. 21

Transfer and scope

• Observable-gate methods face a structural ceiling because 40–42% of conflict cells lack elicitable memory for comparison.The composition analysis identifies these cells as invisible to observables-based policies.Source: E Additional Stability Analyses · pp. 21–22

Transfer results

• Transfer gains are positive but partial, attenuated, and dependent on whether the counterfactual answer appears as a value or passage.The value and passage frames present the same counterfactual answer in different tool-return formats.Source: §F.1 · p. 22, Table 6 · p. 24

Scope and validation

• The verified transfer pools contain 279 value-frame and 185 passage-frame episodes, while the 687-row extension is unverified and excluded from the readings.Human gates validate the matcher and scorer, not the unverified extension’s labels.Source: §F.1 · pp. 22–23

Transfer results

• The anchor model’s DPO arm transfers in the value frame, while its SFT arm transfers in the passage frame; gemma’s DPO transfers in both.Thus, the frame carrying transfer depends on the loss and model.Source: §F.1 · p. 23

Abstention

• 19 of 20 tested method-model combinations reduce abstention after tool errors or on unanswerable inputs.The abstention leak appears on all three transferred arms and is the most consistent measured side effect.Source: §F.1 · p. 23

Scope and validation

• The external test uses one multiple-choice science-domain set, so it tests format transfer but not free-form transfer.Per-frame nulls are power-bounded, with minimal detectable effects of 0.11 to 0.16.Source: §F.1 · p. 23

Scale probe

• The 70B probe shows higher retention and correct-tool following but worse tool-error abstention, making scale a trade rather than a clean improvement.Retention is 0.21 pooled, correct-tool following is 0.96 pooled, and tool-error abstention is 0.49.Source: §F.2 · pp. 23–24

Protocol dependence

• Protocol failures differ by model: Llama fails under auto choice, Mistral under guided calls, and Qwen2.5-1.5B under the two-stage prompt.The traces identify mechanically distinct failures rather than one shared bug.Source: G Tool-Choice Protocols in an Executed… · p. 24

Failure mechanisms

• 458 Llama type errors arise under auto choice, while Mistral has 302 of 302 guided calls rejected for invalid JSON parsing.Other mechanisms include two-stage decision-parse failure and silent abstention from calling the tool.Source: G Tool-Choice Protocols in an Executed… · p. 24

Repairs

• Schema-constrained decoding raises the Llama execution result from 0.328 to 0.992, whereas preference fine-tuning leaves 444 type errors and changes it from 0.328 to 0.360.The controlled repair is the schema-constrained decoder; the tool-finetuned checkpoint is only an uncontrolled comparison.Source: G Tool-Choice Protocols in an Executed… · p. 24

Metric interpretation

• Mistral’s 0.885 answer agreement contrasts with tool execution in only 0.385 of episodes, while Qwen2.5-1.5B scores 0.576 despite executing none.These cases are read as non-calling or silent abstention rather than successful tool use.Source: G Tool-Choice Protocols in an Executed… · pp. 24–25

Serving caveats

• Mistral’s schema-guided results are confounded by a serving defect and must not be interpreted as a model property.A Qwen2.5-7B unusable cell likewise reflects constrained-decoding behavior in the serving stack.Source: Table 7 · p. 24, G Tool-Choice Protocols in an Executed… · p. 25

Reproducibility

• The Llama failure reproduces under different seeds and is deterministic, but decoder attribution rests on cross-arm and cross-formulation evidence.The free auto-choice result changes from 0.984 to 0.312, a delta of −0.672.Source: G Tool-Choice Protocols in an Executed… · p. 25

Measure

• The probe ranks the model’s own elicited closed-book answer at the final input position using a Jacobian lens.It examines arbitration episodes where the closed-book answer is correct and the tool return is a typed distractor.Source: H Whether the Model’s Own Answer Is… · p. 25

Findings

• On both Llama seeds, the correct answer remains rank 1 whether the model keeps it or follows the wrong tool, versus a control median of 421.Gemma shows rank 1 when retaining memory and rank 2 when following the tool.Source: H Whether the Model’s Own Answer Is… · pp. 25–26
• Qwen shows no comparable availability pattern: its median rank is around 2.6 × 10^4, with no episode placing the answer in the top 20.Subtoken scoring leaves Qwen medians at 25,548 and 27,378, still with nothing in the top 20.Source: H Whether the Model’s Own Answer Is… · p. 26

Interpretation

• The probe suggests that, in two of four families, following the wrong tool is not a failure to represent the model’s own correct answer.This is consistent with behavioral interventions changing choices without teaching new information, but remains a probe-based suggestion.Source: H Whether the Model’s Own Answer Is… · p. 26

Caveats

• The rank measures the forced-elicitation memory string rather than the gold answer generally, and the lens projection does not identify a mechanism.The effect is absent on Qwen and partial on gemma, so it is not universal across the tested families.Source: H Whether the Model’s Own Answer Is… · p. 26