PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory Is Not Always Needed: Characterizing Conditional Memory in Scientific Reasoning

Zhen Bi, Xueshu Chen, Yan Wang, Zhizhi Peng, Haosen Hong, Zhen Wang, Zhixuan Chu, Bingyu Zhu, Jungang Lou

arXiv:2608.23982v1cs.AIcs.CLcs.LG

TL;DR

Scientific reasoning requires models to retrieve specialized knowledge and integrate it faithfully, yet conditional memory can either repair missing associations or disrupt sufficient reasoning. This paper characterizes when and where memory helps, then routes it using pre-generation input proxies. Across biological and chemical benchmarks spanning two backbone families and six task types, selective routing more consistently preserves benefits and suppresses memory-induced regressions than static or random routing.

  • Problem

    Evidence is limited on whether conditional memory should be activated uniformly across heterogeneous scientific reasoning tasks, where it may help some inputs and disrupt others.

  • Method

    The paper combines behavioral knowledge-boundary analysis with controlled layer-stage interventions and a router that selects memory activation, location, and contribution strength from pre-generation input proxies.

  • Results

    Across biological and chemical reasoning benchmarks, two backbone families, and six task types, routing more consistently preserves beneficial memory effects while suppressing regressions than static and activation-rate-matched random routing.

  • Takeaways & Limitations

    Selective memory allocation is supported as a design principle for reliable scientific reasoning models.

  • Takeaways & Limitations

    Behavioral boundary codes require labels or repeated inference offline and cannot be used directly at inference time.

Abstract

from arXiv · show

Scientific reasoning requires language models to retrieve specialized knowledge and incorporate it reliably into multi-step computation. Conditional memory provides an explicit lookup pathway that complements dense neural representations, but its usefulness is inherently input- and computation-dependent: retrieved information may repair missing scientific associations, yet it may also introduce distracting shortcuts or interfere with reasoning that the base model can already perform correctly. In this work, we systematically investigate when, where, and to what extent conditional memory should participate in scientific reasoning. We characterize the scientific knowledge boundary and controlled interventions on memory-enabled knowledge-circuit nodes. Based on these analyses, we propose a Knowledge Boundary-Aware Router that uses task-specific input proxies available before generation to determine whether memory is activated, which layer-stage nodes receive memory signals, and how strongly these signals contribute. Experiments on biological and chemical reasoning benchmarks, covering two backbone families and six task types, show that memory effects vary substantially across inputs, tasks, and injection locations. Compared with static and activation-rate-matched random routing, our approach more consistently preserves beneficial memory contributions while suppressing memory-induced regressions, establishing selective memory allocation as an important principle for reliable scientific reasoning.

1 Introduction

Scientific reasoning models must retrieve specialized knowledge and integrate it through multi-step inference, but conditional memory can help some inputs while disrupting others. The paper therefore studies selective activation based on knowledge boundaries and computation.

  • Scientific reasoning requires specialized knowledge retrieval and faithful integration throughout multi-step inference.
  • Conditional memory adds a complementary lookup path for recurring scientific entities, motifs, structures, and rules.
  • Memory effects are non-uniform: it can repair missing associations, provide little benefit, or introduce misleading shortcuts across heterogeneous tasks.
  • The study analyzes memory utility through behavioral knowledge-boundary analysis and controlled layer-stage interventions.
  • The Knowledge Boundary-Aware Router determines whether, where, and how strongly memory participates using task-specific input proxies.
  • Across biological and chemical domains, two backbone families, and six task types, routing preserves beneficial memory contributions while avoiding regressions caused by static configurations.

2 Related Work

Related work covers memory architectures, scientific reasoning benchmarks, and methods for diagnosing reasoning failures beyond final-answer correctness. These strands motivate characterizing the base model’s scientific reasoning boundary.

  • Memory-augmented language models supplement dense parameters with selectively accessed stores, including sparse key-value layers, suffix n-gram lookup, memory experts, and datastores.
  • Scientific reasoning benchmarks span broad STEM knowledge, specialized problem solving, and theorem application across multiple scientific domains.
  • Final-answer correctness alone cannot distinguish knowledge gaps from faulty derivations, motivating additional process and stability measures.
  • The paper combines final-answer failure, reasoning-step failure, and repeated-inference instability to characterize the base model’s scientific reasoning boundary.

3 Conditional Memory at Scientific Knowledge Boundaries

The paper characterizes scientific knowledge boundaries behaviorally and studies where conditional memory enters the reasoning computation. It uses these analyses to distinguish repairable failures from harmful intervention.

  • The behavioral view identifies final-answer failure, reasoning-step failure, and prediction instability at the memory-free model’s knowledge boundary.
  • The knowledge-circuit view treats memory as retrieved context injected into hidden states at a selected layer and inference stage.
  • For each example, task-level answers, process-level steps, and repeated-inference consistency define a behavioral boundary code relative to the memory-free backbone.
  • The boundary indicators are nonexclusive, allowing inputs with correct answers but faulty derivations or instability despite one correct sample.
  • The code tests whether memory repairs a failure regime or merely changes aggregate performance.
  • Boundary codes are computed offline using labels or repeated inference, while inference-time routing uses task-specific input proxies available before generation.

a. Conditional Memory at Scientific Knowledge Boundaries

The framework separates behavioral boundary characterization from layer-stage memory intervention and compares memory-free and memory-enabled reasoning outcomes. It distinguishes positive, neutral, and negative memory effects.

  • Behavioral Boundary Characterization: Behavioral Boundary Characterization identifies the memory-free LLM’s boundary code before memory intervention.
  • Layer-Stage Memory Intervention: Layer-Stage Memory Intervention specifies where memory signals enter the model’s reasoning computation.
  • The framework compares memory-free and memory-enabled LLM behavior using positive, neutral, and negative outcome categories.

b. Knowledge Boundary-Aware Routing for Conditional Memory

The paper models conditional memory as a computation whose usefulness depends on where and when retrieved information is injected. This motivates routing memory across inputs and layer–stage knowledge-circuit nodes.

  • Routing proposal: The proposed router uses complementary behavioral and knowledge-circuit perspectives to decide whether, where, and how strongly memory participates.Its external component routes memory access, while its internal component routes layer–stage contributions.
  • Knowledge-circuit view: Each layer–stage pair is treated as a knowledge-circuit node where memory is queried, retrieved, projected, gated, and added to the hidden state.Prefill and decoding at the same layer are distinct intervention units.
  • Knowledge-circuit view: Controlled attenuation separates memory availability from usefulness by testing how changing a node’s contribution affects otherwise fixed backbone computation.The analysis can distinguish helpful retrieved associations from harmful or ineffective injection sites.

4 Knowledge Boundary-Aware Routing for Conditional Memory

The routing design converts pre-inference task-specific proxies into an external memory-access decision and configures internal layer–stage memory contributions. Its parameters are calibrated from boundary analyses and interventions without changing backbone or memory parameters.

  • Routing rationale: Behavioral boundary codes identify when the memory-free base model encounters difficulty, while knowledge-circuit nodes identify where memory can alter computation.Behavioral signals may require labels or repeated inference, whereas node conditions are unavailable before a routed forward pass.
  • External data routing: The External Boundary-Aware Data Router uses pre-inference input proxies to make a global memory-access decision.The proxies describe knowledge demand, reasoning structure, and input ambiguity.
  • External data routing: The external router quantizes continuous proxies into task-specific quantile bins and calibrates additive bucket contributions, interactions, and thresholds by hyperparameter search.Calibration uses associations between quantized regions and boundary codes on existing reasoning samples; no additional classifier is trained.
  • Inference: At inference, routing uses input-side proxies and configured parameters without reference answers or alternative model outputs, while leaving backbone and memory parameters unchanged.Parameters are selected separately for each task and backbone family.
  • Internal parameter routing: The Internal Boundary-Aware Parameter Router selects stages and scales individual memory-enabled layer–stage nodes with coefficients in [0, 1].A zero mask or coefficient suppresses the corresponding memory contribution.

5 Experimental Setup

Experiments cover biological and chemical reasoning across six task types, two Qwen backbone families, multiple memory and adaptation baselines, and task-specific evaluation metrics. Knowledge-boundary analysis uses answer failure, reasoning-step failure, and prediction instability.

  • Datasets: BioProBench evaluates procedural reasoning through Protocol Question Answering, Step Ordering, and Error Correction.These tasks assess procedural facts, causal step dependencies, and safety- or validity-critical errors.
  • Datasets: ChemCoTBench evaluates molecular understanding, molecule editing, and molecular optimization through explicit, verifiable chemical operations.The tasks cover structure comprehension, instruction-guided modification, and property-guided design.
  • Models and baselines: Experiments use Qwen2.5-7B and Qwen3-8B, comparing base, domain-adapted LoRA, memory-only, and Memory+LoRA variants.Routing parameters are task- and backbone-specific and obtained through hyperparameter search.
  • Analysis: Knowledge boundaries are characterized with answer failure, reasoning-step failure, and prediction instability relative to the memory-off base model.These signals define cohorts for boundary and ablation studies.
  • Evaluation: Task-specific evaluators report accuracy, calibration, ordering, optimization, error, and molecular-similarity metrics, with Brier score and MAE interpreted as lower-is-better.The exact metric set varies across BioProBench and ChemCoTBench tasks.

6 Main Results

Across biological and chemical reasoning, memory effects are conditional rather than uniformly beneficial: the router more consistently improves task-specific outcomes while avoiding regressions associated with fixed memory configurations.

  • BioProBench: 0.63 and 0.61 are the router’s strongest ERR and PQA accuracies within the Qwen3 family.Memory+LoRA instead reaches 0.39 PQA accuracy and 0.50 Brier, while the router retains LoRA-level calibration.
  • BioProBench: In Qwen2.5 biological reasoning, the router achieves the highest ERR accuracy, recall, and F1 and the highest PQA accuracy, while matching LoRA ordering scores.Memory alone remains best on macro precision and Brier score, illustrating metric-specific trade-offs.
  • ChemCoTBench: In Qwen2.5 chemical reasoning, the router leads editing accuracy at 0.51, mean property improvement at 0.16, success rate at 0.36, and MAE at 0.50.Memory alone has the highest extraction rate, while non-routed baselines retain the best molecular-understanding accuracy and TMS.
  • ChemCoTBench: In Qwen3 chemical reasoning, the router leads editing, all optimization metrics, and understanding accuracy, but memory-only retains the lowest MAE and LoRA the highest TMS.No fixed memory configuration dominates the full metric suite.

7 Ablation Study

Ablations show that memory’s effects depend on boundary regime, input selection, injection location, and task, motivating both external and internal routing rather than uniform activation.

  • 7.1 Boundary-Dependent Memory Effects: Memory effects reverse across boundary cohorts: Bio prediction-instability cohorts gain 2.09 and 1.82 points, while corresponding Chem cohorts decline 4.45 and 3.81 points.Chem reasoning-failure-positive examples instead improve by 4.71 points, demonstrating both repair and interference regimes.
  • 7 Ablation Study: Engram attains a higher parsing success rate than MeKi on BioProBench ERR across three repeated runs.The comparison motivates examining differences between their retrieval and residual-injection paths.
  • 7.3 External Routing versus Activation-Rate-Matched Random Routing: The empirical feature-based Data Router outperforms activation-rate-matched random routing in every displayed Bio ERR and ChemCoTBench setting.The comparison also includes Always OFF and Always ON configurations, with means and standard deviations over repeated runs.
  • 7.4 Internal Routing of Knowledge-Circuit Nodes: Internal routing improves MolEdit accuracy from 0.495 to 0.513 and MolUnd accuracy from 0.562 to 0.573, while reducing MolUnd MAE from 0.536 to 0.501.The same-cohort offline replay validates the routing mechanism rather than independent generalization, and effects remain metric-dependent.
  • 7.5 Task-Dependent Memory Placement: Late injection at layers 27–28 yields the strongest ERR accuracy at 0.562, middle injection at layers 15–16 yields the highest PQA accuracy at 0.528, and memory-free remains best for ORD exact match at 0.206.PQA scores are conditioned on successful parsing and should be read alongside the middle-layer configuration’s lower parsing coverage.

8 Conclusion

The conclusion frames conditional memory as useful but non-uniform: its value depends on input boundaries and knowledge-circuit placement. The proposed router therefore allocates memory selectively across inputs and internal nodes.

  • 8 Conclusion: Memory utility depends on whether the base model reaches a knowledge boundary and whether retrieved signals enter usable knowledge-circuit nodes.The study characterizes this dependence through behavioral indicators and controlled layer-stage interventions.
  • 8 Conclusion: Across two benchmarks, two backbone families, and six task types, memory effects vary across inputs, tasks, metrics, and injection locations.Feature-based routing beats activation-rate-matched random routing, while targeted internal routing attenuates harmful contributions.
  • 8 Conclusion: Conditional memory should be treated as a selective computational resource rather than a uniformly beneficial augmentation.The router uses pre-inference input proxies to control global access and internal node contributions without reference answers or alternative outputs.

Additional Definition

The paper defines boundary-aware memory gains alongside task-specific evaluation metrics for biological and chemical reasoning. It also specifies balanced error-correction scoring, molecular validity handling, and repeated-run reporting procedures.

  • Scientific Conditional Memory Benefit: SCMB compares aggregate, boundary-set, and non-boundary-set gains to measure whether memory helps more near the scientific knowledge boundary.The definition distinguishes gains on boundary cases from gains on cases the base model already solves.
  • Experimental reporting: Reported task-performance results average three stochastic generation-and-evaluation runs with matching checkpoints, splits, prompts, and routing configurations.Tables report arithmetic means and standard deviations, while CPU preprocessing and metric computation are excluded from latency and GPU-memory comparisons.
  • BioProBench: BioProBench evaluates ERR with accuracy and macro precision, recall, and F1; ORD with exact match and Kendall’s τ; and PQA with accuracy and Brier score.Higher values are preferred except for Brier score, while official evaluators provide the reported task metrics.
  • BioProBench: ORD exact match tests complete protocol-order recovery, whereas Kendall’s τ provides partial credit for pairwise ordering agreement.Kendall’s τ pools concordant and discordant step pairs across instances.
  • BioProBench: ERR’s balanced extension gives equal class weight to error detection and correct-step recognition, while reporting parser failures separately.Macro precision, recall, and F1 average the corresponding False and True class scores.
  • ChemCoTBench: ChemCoTBench separates molecular understanding, editing, and optimization using error, accuracy, similarity, improvement, success, validity, and extraction metrics.For optimization, mean improvement and success rate distinguish improvement magnitude and frequency, while validity and extraction rate measure output usability.
Loading 2608.23982v1…