MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models

Arseniy Varlamov; Rishat Zinnatullin; Elisei Rykov; Alexander Panchenko; Ilseyar Alimova

Source-linked AI summary

MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models

Arseniy Varlamov, Rishat Zinnatullin, Elisei Rykov, Alexander Panchenko, Ilseyar Alimova

arXiv:2608.26295v1Aug 26, 2026cs.CLcs.AIcs.MAcs.SE

TL;DR

Tool-augmented models must arbitrate between memory and tool outputs, but existing evaluations often lack source-correctness control. MemToC constructs executable, correctness-controlled episodes and tests baseline behavior and fine-tuning, finding that arbitration improves only selectively and must be evaluated jointly with correct tool use and abstention.

Problem
Existing evaluations measure source preference without establishing source correctness, limiting assessment of whether deference is appropriate.
Method
MemToC combines 542 quality-controlled factual questions, verified answers, model-specific closed-book answers, and controlled tool returns, then compares prompting with cross-fitted SFT and DPO.
Results
Correctness-conditioned arbitration improves selectively: SFT and DPO meet the asymmetric criterion on the same two of four instruction-tuned backbones, while 19 of 20 method–model combinations reduce abstention.
Takeaways & Limitations
Fine-tuning can improve arbitration, but gains depend on the model and should be assessed alongside correct tool use, abstention, and formulation robustness.
Takeaways & Limitations
Results come from one ToolHop-based benchmark core and primarily open-weight 7–9B models, with no proprietary model evaluated.

Abstract

from arXiv · show

Tool-augmented LLMs must arbitrate between two fallible sources when a tool return conflicts with their parametric memory, yet existing evaluations measure source preference without establishing source correctness. We introduce MemToC, a controlled benchmark for post-tool-return arbitration with executable tools. MemToC comprises 6,504 evaluation episodes constructed from 542 quality-controlled factual questions, independently elicited model-specific closed-book answers, and controlled tool returns of known correctness. These components instantiate four source-correctness cases; tool-error and no-tool conditions are separate controls. Across five open-weight 7-9B models, tool returns strongly dominate elicited closed-book answers. The four instruction-tuned models retain a verified-correct answer against an incorrect tool in only 6.5-17.1% of eligible cases, follow a correct tool in 86.0-93.1%, and repeat the tool return in 78.4-86.0% of cases where both sources are wrong. No cross-model ordering remains stable across three instruction-wording variants with the question and episode content held fixed. We compare prompting with SFT and DPO using chain-level cross-fitting over ToolHop, so questions sharing an underlying fact never straddle training and evaluation. We apply an asymmetric success criterion: correct-answer retention must improve without a detected reduction in correct-tool following. SFT and DPO meet this criterion on the same two of four instruction-tuned backbones. Improvements rarely come cleanly: 19 of 20 tested method-model combinations reduce abstention after tool errors or on unanswerable inputs. Transfer beyond MemToC is positive but partial and depends on the model and presentation frame. Correctness-conditioned arbitration can be improved through fine-tuning, but gains must be evaluated jointly with correct tool use, abstention, and robustness to formulation.

1 Introduction

MemToC addresses the need to arbitrate between fallible tool outputs and model memory when source correctness is known. It constructs a controlled benchmark and evaluates models, prompting, and fine-tuning under distinct correctness conditions.

Existing evaluations often measure which source models follow without establishing whether that source is correct.
MemToC holds tool selection fixed while intervening on the observed return, isolating post-tool-call arbitration.
542 quality-controlled factual questions combine executable calls, verified answers, model-specific closed-book answers, and controlled correct or incorrect returns.
The benchmark evaluates five open-weight 7–9B models across four source-correctness cases, plus tool-error and no-tool controls.
SFT and DPO are compared with chain-level cross-fitting so evaluation questions remain held out from training.

2 Related Work

Prior work identifies tool–memory conflicts and studies source reliance, but generally does not control executable tool-return correctness during arbitration.

TMC defines conflicts through disagreement between memory-only and tool-conditioned responses, without knowing which source is correct.
Context-DPO and KnowPO optimize reliance on contextual information or adaptive source selection, rather than conditioning selection on competing-source correctness.

3 The MemToC Benchmark

MemToC constructs a correctness-controlled benchmark from ToolHop and evaluates model-specific arbitration by comparing closed-book answers with controlled tool returns against verified answers.

Benchmark construction: 3,912 ToolHop sub-questions are deduplicated and filtered to produce 542 quality-controlled factual questions.
Benchmark construction: Near distractors match the verified answer in type, domain, and plausibility, while far distractors preserve type but differ more in period or country.
Model-specific evaluation: For each model and formulation, a closed-book answer m is elicited before presenting the same question with a controlled return r and collecting response y.
Model-specific evaluation: Comparing m and r with verified answer g assigns four source-correctness cases whose targets distinguish retention, correct-tool following, and rejection of both wrong sources.
Model-specific evaluation: Two annotators independently preferred explicit disagreement or uncertainty in conflict scenarios, achieving Cohen’s κ=0.785.

4 Experimental Setup

The experiments evaluate arbitration across models, prompt formulations, controls, and mitigation methods using fixed artifacts, explicit scoring, and held-out chain-level comparisons.

Evaluation design: Five open-weight models are evaluated, including four instruction-tuned models and a base/instruction-tuned Llama comparison.
Evaluation design: The primary closed-book prompt requires a best guess, while an abstention-permitting sensitivity analysis leaves some cases nearly empty for refusal-prone models.
Scoring and metrics: Responses are extracted from FINAL-marked spans and matched against verified answers, closed-book answers, and tool returns using normalization and alias tolerance.
Scoring and metrics: Correct-answer retention, correct-tool following, conflict-resolved accuracy, incorrect-tool following, and tool-error abstention separate desirable and undesirable behaviors.
Mitigation methods: The success criterion requires improved correct-answer retention without a detected reduction in correct-tool following.
Mitigation methods: SFT and DPO use LoRA adapters and cross-fitting that keeps complete ToolHop chains and related-question groups within folds.

5 Results

MemToC shows that tool returns strongly dominate model memory, while prompting and fine-tuning improve retention only with substantial trade-offs and formulation sensitivity.

Baseline arbitration: Across four instruction-tuned models, correct-answer retention is 6.5%–17.1%, correct-tool following is 86.0%–93.1%, and incorrect-tool following is 78.4%–86.0%.Models generally favor the tool return regardless of its correctness.
Baseline arbitration: Tool-error abstention ranges from 73.9% for Mistral-7B-Instruct-v0.3 to 99.5% for gemma-2-9b-it, without predicting behavior on factual conflicts.Reliable abstention after explicit errors does not ensure resistance to plausible incorrect returns.
Fine-tuning results: SFT meets the success criterion for Llama-3.1-8B-Instruct and gemma-2-9b-it, but Mistral gains retention by broadly reducing tool use.Llama retention increases by 14.3 points; gemma retention increases by 17.0 points and correct-tool following by 5.6 points.
Fine-tuning results: DPO meets the criterion for Llama and Gemma, while Mistral increases retention by 6.0 points but reduces correct-tool following by 4.5 points.SFT and DPO therefore succeed on the same two models, with outcomes varying more consistently by model than objective.
Robustness and controls: Across three formulations, within-model retention varies by 7–15 points and no cross-model ordering remains stable.Qwen2.5-7B-Instruct ranks highest on Reference and lowest on Paraphrase A.
Robustness and controls: Prompting, presentation format, distractor distance, and tool-choice protocol all change measured deference or reliability, limiting interpretation of any single setup.Replacing the executed-tool frame reduces incorrect-tool following by 11 points for Llama-Instruct and 35 for Qwen; schema-constrained decoding repairs Llama tool choice from 0.328 to 0.992.
Additional analyses: None of 120 annotated incorrect-tool responses explicitly acknowledges the conflict, while transfer beyond MemToC is positive but partial.Transfer gains depend on model and presentation frame, and adapters also reduce abstention on unanswerable questions.
Additional analyses: A single 70B probe reaches 21% retention and 96% correct-tool following but 85% incorrect-tool following and 44% tool-error abstention.The result is descriptive because scale is confounded with model generation and quantization and uses one seed.

6 Limitations

The study is bounded by its benchmark construction, model coverage, training-seed coverage, and narrow error-handling and transfer evaluations.

Scope and construction: The evaluation uses one ToolHop-derived benchmark core, primarily open-weight 7–9B models, synthetic typed distractors, and no proprietary model.Most machine-authored distractors came from a single generator, and a second quality-controlled realization remains future work.
Evaluation coverage: Additional training seeds cover only the two models that satisfied the criterion, so seed-specific failures cannot be ruled out for the other two.Training-free conflict-aware decoding was also not evaluated despite being named in the preregistration.
Evaluation coverage: Transfer is evaluated on one external multiple-choice dataset using small verified pools, so null results within a frame may reflect limited power.All transfer estimates are restricted to human-verified examples.
Measurement limits: Forced elicitation makes measured retention rates upper bounds, and neither protocol directly accesses model beliefs.Tool-error abstention is also measured with a single structured service-unavailable payload whose rate depends on the displayed error.
Measurement limits: A high tool-error abstention rate is not intrinsically desirable because error payloads change fallback and abstention differently across memory-correct and memory-wrong questions.For one Llama model, a permission error raises correct fallback from 0.09 to 0.35 but lowers abstention from 0.90 to 0.54.

7 Conclusion

MemToC shows that tool returns often dominate model memory, while arbitration quality and model rankings vary with formulation. Fine-tuning can improve arbitration, but gains are model-dependent and may trade off against abstention and failure handling.

Tool returns strongly dominate elicited closed-book answers, and 0 of 120 annotated conflicts acknowledge the disagreement.The 95% upper bound for conflict acknowledgment is 3%.
Cross-model rankings change across prompt formulations and presentation frames, so tool-deference rates are not stable model properties.
SFT and DPO satisfy the asymmetric success criterion on the same two of four instruction-tuned backbones, with transfer beyond MemToC remaining partial.
19 of 20 tested method–model combinations reduce abstention after tool errors or on unanswerable inputs.Evaluating only resistance to incorrect returns can therefore reward broad distrust rather than better arbitration.

8 Data and Code Availability

The paper provides released benchmark, code, prompts, and verification materials, while documenting the exact evaluation templates and important release boundaries.

The benchmark, construction and evaluation code, scoring layer, and per-claim number map are released under Apache-2.0.A verification script recomputes 180 reported values; the separate three-payload tool-error comparison is not part of the release.
The appendix lists every evaluation prompt with its experiment and protocol position, including closed-book, tool-return, intervention, and retrieval-passage templates.
A verbatim benchmark episode shows a verified answer, an executable actor lookup, an incorrect tool return, and the model’s closed-book answer.
Three prompting interventions alter only the inserted strategy sentence, while question text and episode content remain fixed across formulations.The strategies are warning, source-priority, and abstain-and-flag.
The retrieval-passage comparison changes how the same external value is presented while keeping the payload, question, and required answer format fixed.

B Experimental Setup and Reproducibility

The experimental design emphasizes controlled source correctness, held-out chain-level evaluation, matched prompt variants, validated scoring, and documented construction revisions.

Experimental design: 542 quality-controlled factual questions support executable calls, verified answers, model-specific closed-book answers, and controlled correct and incorrect returns.
Evaluation criteria: The asymmetric success criterion requires improved correct-answer retention without a detected reduction in correct-tool following.
Fine-tuning: SFT and DPO are compared with chain-level cross-fitting, ensuring questions from shared ToolHop chains do not cross training and evaluation folds.The two folds contain 287 and 288 questions from 154 and 155 chains, respectively.
Robustness: Three prompt formulations keep question text, verified answer, tool call, and tool return byte-identical while varying instruction wording.
Reproducibility boundaries: Major benchmark revisions changed incorrect-tool following, retention, and fine-tuning results enough that affected findings are reported as construction-scoped.One revision increased incorrect-tool following to 0.71–0.83, while group cross-fitting reduced a Llama-3.1-8B-Instruct DPO retention gain from +0.24 to +0.058.

C Steering the Representation with Sparse Autoencoders

Sparse-autoencoder steering probes do not restore reliable arbitration: edits can disrupt tool following or corrupt evidence representations without consistently recovering memory-based answers.

Steering is direction-asymmetric: knocking the model off the tool is easier than re-inducing its parametric answer.
Table 2 maps reported numbers to released files, but absolute fine-tuned Table 1 levels and three previous-version measurement layers cannot be recomputed from the release alone.
The memory-ward edit reduces kept-memory behavior by −0.53 while increasing tool-following by only +0.25.
Generation-preserving edits reach only +0.10 on Gemma and +0.125 on Llama, both below the +0.15 steering bar.
Stronger Gemma edits cross the nominal bar but annihilate tool-following, collapse outputs into schema echoes, and produce confabulated provenance.
Only 5 of 80 top mutual-information features are entity- or fact-content features, leaving little content-specific control for the memory-ward edit.

D Benchmark Construction, Quality Control, and Scorer Validation

MemToC’s construction combines controlled source correctness with layered human quality control and scorer validation. These procedures remove computational, redundant, non-self-contained, and invalid items while documenting important measurement boundaries.

Scorer and protocol validation: The primary required-best-guess protocol treats retention as an upper bound because allowing abstention leaves some source-correctness cases nearly empty.A separate released-benchmark annotation round found zero acknowledged incorrect-tool responses among 120 labeled responses, bounding rather than estimating acknowledgment prevalence.
Benchmark construction: Computational questions are removed so retained answers can meaningfully conflict with parametric knowledge.A rule-based filter rejects answers produced by operations over earlier answers rather than recalled facts.
Leakage control: Question-level grouping prevents shared underlying facts from crossing training and evaluation folds.This leakage control applies beyond exact deduplication to near-identical formulations and related questions.
Distractor design: Near and far distractors preserve answer type while differing in plausibility or temporal and geographic distance.Near distractors are indistinguishable in type and plausibility; far distractors differ more strongly in period or country.
Benchmark construction: 542 quality-controlled factual questions form the released benchmark after review, repair, exclusion, and distractor validation.The construction pool is reduced from 575 to 542 questions after adjudication.
Human review: 575 →542 questions remain after blinded review, with repairs changing only incorrect substituted values and exclusions removing unsuitable records.Question wording and verified answers remain frozen byte-for-byte during repair.

E Additional Stability Analyses

Additional analyses show that arbitration outcomes and intervention benefits are sensitive to wording, protocol presentation, and model-specific trade-offs. The authors therefore emphasize pooled, formulation-scoped conclusions and joint evaluation of retention, tool following, and abstention.

Formulation stability: Three instruction-wording variants produce overlapping retention ranges and changing model rankings, so cross-model retention orderings are not stable.The question text and episode content remain fixed across Reference, Paraphrase A, and Paraphrase B.
Prompting trade-offs: Eleven of twelve pooled prompting comparisons significantly improve retention, but each model pays a model-specific cost.Costs include reduced correct-tool following for some models and reduced tool-error abstention for others.
Prompting trade-offs: Abstain-and-flag collapses correct-tool following on every model, while gemma preserves tool-error abstention under warning prompting from 100.0 to 89.7.The intervention patterns differ by model and strategy rather than yielding a uniform safety improvement.
Protocol ablation: Removing the protocol sentence leaves arbitration retention statistically unchanged but reduces correct-tool following by 0.093.The ablation also reduces no-conflict accuracy by 0.056, while tool-error abstention is read as null.
Criterion stability: The asymmetric criterion is formulation-sensitive: only gemma’s SFT arm meets it under every formulation, while some pooled findings fail or reverse by wording.Mistral’s warning arm is −0.016 under Reference versus +0.065 pooled.
Transfer and scope: Observable-gate methods face a structural ceiling because 40–42% of conflict cells lack elicitable memory for comparison.The composition analysis identifies these cells as invisible to observables-based policies.

F.1 Out-of-distribution transfer

The out-of-distribution test finds that fine-tuning gains transfer beyond MemToC, but only partially and with effects that depend on presentation frame and model. The evidence is restricted to verified multiple-choice science pools, with abstention losses recurring across transferred arms.

Transfer results: Transfer gains are positive but partial, attenuated, and dependent on whether the counterfactual answer appears as a value or passage.The value and passage frames present the same counterfactual answer in different tool-return formats.
Scope and validation: The verified transfer pools contain 279 value-frame and 185 passage-frame episodes, while the 687-row extension is unverified and excluded from the readings.Human gates validate the matcher and scorer, not the unverified extension’s labels.
Transfer results: The anchor model’s DPO arm transfers in the value frame, while its SFT arm transfers in the passage frame; gemma’s DPO transfers in both.Thus, the frame carrying transfer depends on the loss and model.
Abstention: 19 of 20 tested method-model combinations reduce abstention after tool errors or on unanswerable inputs.The abstention leak appears on all three transferred arms and is the most consistent measured side effect.
Scope and validation: The external test uses one multiple-choice science-domain set, so it tests format transfer but not free-form transfer.Per-frame nulls are power-bounded, with minimal detectable effects of 0.11 to 0.16.
Scale probe: The 70B probe shows higher retention and correct-tool following but worse tool-error abstention, making scale a trade rather than a clean improvement.Retention is 0.21 pooled, correct-tool following is 0.96 pooled, and tool-error abstention is 0.49.

G Tool-Choice Protocols in an Executed Agent Loop

Executed agent-loop behavior depends strongly on the model and tool-choice protocol, with failures arising from distinct parsing, coercion, and abstention mechanisms. Some apparent answer agreement therefore reflects not executing the tool, and serving defects confound certain guided arms.

Protocol dependence: Protocol failures differ by model: Llama fails under auto choice, Mistral under guided calls, and Qwen2.5-1.5B under the two-stage prompt.The traces identify mechanically distinct failures rather than one shared bug.
Failure mechanisms: 458 Llama type errors arise under auto choice, while Mistral has 302 of 302 guided calls rejected for invalid JSON parsing.Other mechanisms include two-stage decision-parse failure and silent abstention from calling the tool.
Repairs: Schema-constrained decoding raises the Llama execution result from 0.328 to 0.992, whereas preference fine-tuning leaves 444 type errors and changes it from 0.328 to 0.360.The controlled repair is the schema-constrained decoder; the tool-finetuned checkpoint is only an uncontrolled comparison.
Metric interpretation: Mistral’s 0.885 answer agreement contrasts with tool execution in only 0.385 of episodes, while Qwen2.5-1.5B scores 0.576 despite executing none.These cases are read as non-calling or silent abstention rather than successful tool use.
Serving caveats: Mistral’s schema-guided results are confounded by a serving defect and must not be interpreted as a model property.A Qwen2.5-7B unusable cell likewise reflects constrained-decoding behavior in the serving stack.
Reproducibility: The Llama failure reproduces under different seeds and is deterministic, but decoder attribution rests on cross-arm and cross-formulation evidence.The free auto-choice result changes from 0.984 to 0.312, a delta of −0.672.

H Whether the Model’s Own Answer Is Still Available

A dev-stage Jacobian-lens probe asks whether a model’s correct elicited answer remains represented when it follows an incorrect tool. For two model families, the answer often remains highly ranked, suggesting a choice failure rather than simple unavailability, but the probe is narrow and non-mechanistic.

Measure: The probe ranks the model’s own elicited closed-book answer at the final input position using a Jacobian lens.It examines arbitration episodes where the closed-book answer is correct and the tool return is a typed distractor.
Findings: On both Llama seeds, the correct answer remains rank 1 whether the model keeps it or follows the wrong tool, versus a control median of 421.Gemma shows rank 1 when retaining memory and rank 2 when following the tool.
Findings: Qwen shows no comparable availability pattern: its median rank is around 2.6 × 10^4, with no episode placing the answer in the top 20.Subtoken scoring leaves Qwen medians at 25,548 and 27,378, still with nothing in the top 20.
Interpretation: The probe suggests that, in two of four families, following the wrong tool is not a failure to represent the model’s own correct answer.This is consistent with behavioral interventions changing choices without teaching new information, but remains a probe-based suggestion.
Caveats: The rank measures the forced-elicitation memory string rather than the gold answer generally, and the lens projection does not identify a mechanism.The effect is absent on Qwen and partial on gemma, so it is not universal across the tested families.