PaperlayerView exact revision on arXiv

Source-linked AI summary

$A^3$-Bench: Benchmarking Memory-Driven Scientific Reasoning via Anchor and Attractor Activation

Jian Zhang, Yu He, Zhiyuan Wang, Zhangqi Wang, Kai He, Fangzhi Xu, Qika Lin, Jun Liu

arXiv:2601.09274v1cs.AI

TL;DR

Existing scientific reasoning benchmarks do not directly evaluate the memory activation mechanisms involved in multi-step reasoning. A^3-Bench addresses this gap with a dual-scale benchmark, SAPM annotations, and AAUI, and experiments show improved accuracy with controllable token consumption while exposing differences in model memory use.

  • Problem

    Existing benchmarks mainly evaluate final answers or process consistency, leaving memory retrieval and activation insufficiently assessed in scientific reasoning.

  • Method

    A^3-Bench annotates 2,198 cross-domain problems with anchors and attractors through SAPM and evaluates their activation using a dual-scale framework and AAUI.

  • Results

    Memory activation improves accuracy with controllable token consumption and reveals substantial differences in how models utilize memory across experiments.

  • Takeaways & Limitations

    A^3-Bench provides a memory-centric, interpretable, cognitively aligned evaluation paradigm for scientific reasoning and model development.

Abstract

from arXiv · show

Scientific reasoning relies not only on logical inference but also on activating prior knowledge and experiential structures. Memory can efficiently reuse knowledge and enhance reasoning consistency and stability. However, existing benchmarks mainly evaluate final answers or step-by-step coherence, overlooking the \textit{memory-driven} mechanisms that underlie human reasoning, which involves activating anchors and attractors, then integrating them into multi-step inference. To address this gap, we propose $A^3$-Bench~ https://a3-bench.github.io, a benchmark designed to evaluate scientific reasoning through dual-scale memory-driven activation, grounded in Anchor and Attractor Activation. First, we annotate 2,198 science reasoning problems across domains using the SAPM process(subject, anchor & attractor, problem, and memory developing). Second, we introduce a dual-scale memory evaluation framework utilizing anchors and attractors, along with the AAUI(Anchor--Attractor Utilization Index) metric to measure memory activation rates. Finally, through experiments with various base models and paradigms, we validate $A^3$-Bench and analyze how memory activation impacts reasoning performance, providing insights into memory-driven scientific reasoning.

1 Introduction

A^3-Bench addresses the gap between evaluating reasoning outcomes and evaluating the memory activation mechanisms that support scientific reasoning. It introduces a dual-scale benchmark and AAUI metric, then validates their use across models and paradigms.

  • Motivation: Existing scientific reasoning benchmarks emphasize final-answer correctness and process consistency but do not directly evaluate memory activation.They therefore cannot distinguish flawed logical inference from inadequate retrieval or activation of necessary memory.
  • Motivation: Human-inspired evaluation should represent reusable knowledge units, experience-based templates, and context-dependent activation across multiple reasoning steps.The paper motivates this design from hierarchical memory and selective activation by contextual cues.
  • Benchmark design: A^3-Bench annotates 2,198 scientific reasoning problems across domains with anchors, attractors, and question-level anchor–attractor mappings using SAPM.Anchors model foundational knowledge units, while attractors model experience-based templates.
  • Benchmark design: The benchmark introduces a dual-scale memory evaluation framework and AAUI to quantify memory activation rates.AAUI leverages human-like context-dependent episodic recall.
  • Evaluation: Experiments across base models and paradigms validate evaluation of anchor–attractor activation and utilization, while memory enhancement improves accuracy with controllable token costs.The authors present this as support for cognitively aligned evaluation and model development.

2 Preliminaries

The preliminaries define memory-driven reasoning around anchors, attractors, activation, and attractor basins. Memory activation is modeled as movement toward a state that best explains the input, after which reasoning settles into a stable inference outcome.

  • Anchor and Attractor: An anchor constrains the initial state and focuses reasoning on relevant knowledge, while an attractor guides reasoning along a specific path.Together, activated anchors and attractors form the state space of an Attractor Basin.
  • Memory Activation: Memory activation minimizes a formula involving the input query, posterior representation, and prior knowledge distribution.The internal state then evolves through gradient descent.
  • Memory Activation: Gradient descent drives the internal state toward the attractor that best explains the input, corresponding to activation of memory structures.This supplies the mechanism connecting optimization with memory activation.
  • Memory-Augmented Reasoning: Memory-augmented reasoning maps an input query to an initial internal state, identifies candidate attractors, and evolves reasoning through the resulting state dynamics.The final output is expressed as a stable inference outcome after settling within the activated attractor basin.

3 The A3-Bench Dataset

A^3-Bench constructs a cross-domain scientific reasoning dataset through hierarchical subject standards, expert-developed anchors and attractors, problem refinement, and memory mapping. Its pipeline combines model-assisted examination and refinement with repeated assessment and expert annotation.

  • SAPM: SAPM comprises subject benchmarking, anchor-and-attractor development, problem reconstruction, and memory mapping.The process links questions to relevant anchor–attractor sets and is illustrated through the dataset’s examples and guidance.
  • Subject Benchmarking: The benchmark covers math, physics, and chemistry using integrated classification systems that yield 8 math, 5 physics, and 5 chemistry subdomains.The discipline standards are based on AMS, international physics standards, and IUPAC.
  • Anchor and Attractor Development: For each subdomain, three experts label anchors and attractors; anchors include concepts, principles, and formulas, while attractors include schemas and exemplars.Anchors set initial conditions and attractors constrain reasoning to predefined frameworks.
  • Problem Reconstructing: Questions are examined by GPT-5, Deepseek-V3.2, and Qwen-30B, retaining any question answered incorrectly by at least one model for refinement.The models provide varied responses across parameters, capabilities, and openness.
  • Problem Reconstructing: Three models cross-evaluate reasoning, experts revise retained questions with multi-step reasoning, and the revised problems receive standard answers.Assessment then uses 30 answers per question from three models solving each question 10 times to determine difficulty.
  • Memory Mapping: Memory mapping combines model subdomain voting with expert selection of relevant anchors and attractors, producing the final associated problem set.Each question may receive up to 6 anchors and 4 attractors.

4 Experiments

The experiments evaluate memory-driven scientific reasoning across memory paradigms, base models, and metrics. The framework instantiates memory activation with HybridRAG and measures both answer accuracy and Anchor–Attractor utilization.

  • Benchmarking Method: HybridRAG implements memory activation through a dense vector store, a knowledge graph, and a Context Fabric Composer.The Vector Needle retrieves semantically similar nodes, while the Graph Needle recovers logical links before composing the final context.
  • Memory Paradigms: A3-Bench evaluates three paradigms: No memory, Full memory, and Gold memory with human-labeled activation subsets.
  • Base Models: The experiments use ten LLMs spanning different scales, architectures, and access types.
  • Evaluation Metrics: Accuracy is measured by matching each model’s final answer against the ground truth.
  • Evaluation Metrics: AAUI measures how well models activate expert-annotated Anchors and Attractors during reasoning.It combines anchor and attractor recall with an interaction term rewarding simultaneous activation.

5 Analysis

Memory activation improves scientific reasoning across models and subjects, with especially large benefits on difficult and competition-level problems. AAUI tracks reasoning fidelity, while activation also reduces inference time and selected error types.

  • 5.1 Main Results: 34.71% to 48.19% (+13.48): Annotated Activation raises average accuracy over Vanilla across ten LLMs.Model gains range from GPT-5-Mini +3.37 and Gemini-2.5-Flash +4.74 to GLM-4-32B +22.75 and Qwen3-4B +21.16.
  • 5.1 Main Results: Hard-problem gains are substantial, including Physics improvements of +25.00 for Grok-4-Fast and +15.56 for GLM-4-32B.The analysis attributes many hard-problem failures to missing or misselected Attractor solution templates.
  • 5.1 Main Results: AAUI generally aligns with accuracy: Grok-4-Fast reaches AAUI=0.66 with Avg.=56.69%, versus GPT-5-Mini at AAUI=0.09 and Avg.=18.74%.The metric is presented as capturing compatible co-activation of Anchors and Attractors and their conversion into correct reasoning.
  • 5.2 Generalized Analysis: 11.12 points over Vanilla and 6.35 points over CoT: Anchor–Attractor Activation improves average OlympiadBench scores across all ten models.DeepSeek-V3.2 reaches 33.60% with activation, improving +12.35 versus Vanilla and +6.63 versus CoT.
  • 5.2 Generalized Analysis: 7.63% versus 1.69% (∼4.5×): activation raises Qwen3-4B’s Physics Competition score on OlympiadBench.Competition-level subsets show the most pronounced improvements, while activation supplies starting principles and supporting Anchors for viable solution paths.
  • 5.4 Inference-Time Analysis: +13.5% performance with 2.1 seconds lower average inference time occurs under Annotated Anchors & Attractors.Nearly all models show accuracy improvements alongside reduced inference times.
  • 5.5 Error-Type Distribution: Reasoning and Knowledge errors decrease substantially in the +Both and +Annotated paradigms, while Calculation, Formatting, and Comprehension errors change little.The error distributions are compared across five experimental modes.

6 Related Works

Related work covers external and long-horizon memory methods for LLMs and benchmarks for scientific reasoning. The reviewed benchmarks span multimodal, expert-level, visual, and discovery tasks but do not measure memory utilization during scientific reasoning.

  • Memory: LLM memory methods include externally retrievable stores such as RAG and virtual-memory systems such as MemGPT.These approaches target knowledge updates, retrieval, long-horizon interaction, and context limits.
  • Scientific Reasoning Benchmarks: Scientific reasoning benchmarks include OlympiadBench, EMMA, Humanity’s Last Exam, MathVista, and ScienceBoard.They cover areas including bilingual multimodal problems, multimodal reasoning, expert-level questions, visual mathematics, and scientific discovery.
  • Scientific Reasoning Benchmarks: Existing scientific reasoning benchmarks do not measure memory utilization during scientific reasoning.

7 Conclusion

A3-Bench formalizes memory-driven scientific reasoning through hierarchical anchors and attractors, a free-energy-based activation model, and a structured annotation process. Its dataset and framework support interpretable evaluation across scientific domains.

  • Dataset and framework: A3-Bench annotates 2,198 scientific reasoning problems across math, physics, and chemistry with structured anchor and attractor signals.The dataset uses the SAPM process and maps each question to an anchor–attractor set.
  • Activation model: The framework defines memory activation as minimizing free energy that balances question fit, anchor-based prior knowledge, and entropy.The resulting dynamics connect an external query with internal anchor-based structures through Attractor Dynamics and Free Energy Minimization.
  • Activation model: Reasoning follows gradient flow toward stable attractors, with a non-positive free-energy derivative ensuring dissipative convergence to a fixed point.The formulation treats the subject domain as a manifold and anchors as local minima or attraction basins.
  • Annotation pipeline: The annotation pipeline defines subdomains, identifies anchors and attractors, refines multi-hop questions, and pairs each problem with a complete reference solution.Experts use model outputs to diagnose errors, compare discrepancies, enforce coupled knowledge points, and verify discipline assignments.
  • Dataset and framework: Anchors provide foundational reasoning primitives, while attractors provide reusable schemas and episodic exemplars that operationalize them into solution pathways.The annotation process also records relations between the two memory types to keep their roles coherent and non-redundant.
  • Annotation pipeline: A3-Bench organizes the resulting memory units into traceable libraries and a hierarchical taxonomy spanning three scientific domains.The libraries use JSON entries with identifiers, definitions, and explicit relations, while the taxonomy distinguishes domain-specific subdomains.

D Different Memory Paradigms

The memory-paradigm experiments compare anchor-only, attractor-only, and dual activation. They show complementary contributions, domain- and model-dependent sensitivity, and a limitation in integrating both memory types.

  • Single-memory activation: For 7/10 models, attractor-only activation yields higher overall accuracy than anchor-only activation.Examples include Grok-4-Fast, GLM-4-32B, Llama-3.1-70B, and Qwen3-30B.
  • Dual activation: Dual annotated activation outperforms attractor-only activation, indicating that procedural templates require conceptual grounding for the best performance.Grok-4-Fast rises from 57.60% to 65.10%, while Qwen3-4B rises from 38.90% to 58.92%.
  • Domain sensitivity: Memory-type sensitivity varies by subject: Chemistry shows smaller anchor–attractor differences, whereas Math and Physics more often favor attractor-only activation.The reported pattern is consistent with Chemistry requiring precise property definitions and Math and Physics relying more on theorem selection and template-based derivations.
  • Model sensitivity: Model robustness to partial memory is heterogeneous, with large performance drops for Qwen3 models but only modest changes for GPT-OSS-120B.The authors identify integrating both Anchors and Attractors as a key limitation exposed by the dataset.

E Other Analysis

The paper adds statistical significance testing and noisy-memory robustness analysis to examine the role of memory augmentation in scientific reasoning.

  • Additional analyses: Additional experiments include a task-appropriate McNemar’s test for statistical significance.
  • Additional analyses: These analyses are designed to quantify the necessity of memory augmentation for scientific reasoning.
  • Additional analyses: The analysis also evaluates robustness under noisy memory interference.

E.1 Significance Test

Significance tests show that external memory improves scientific reasoning, combined activation is strongest, and irrelevant memory progressively harms performance.

  • Significance testing: p < 0.001: Single-memory activation significantly outperforms the question-only baseline, showing that parametric knowledge alone is insufficient for these problems.
  • Significance testing: p < 0.001: Combined Anchor–Attractor activation significantly outperforms either memory component alone.The result supports joint activation of declarative and procedural memory components.
  • Significance testing: p = 0.120: Anchor-only and attractor-only activation do not differ significantly, supporting complementary rather than dominant roles.
  • Noisy-memory robustness: Accuracy falls from 65.1% with fully annotated memory to 32.5% under full noise replacement for Grok-4-Fast.The decline becomes pronounced once noise exceeds 40–60%, as irrelevant memory disrupts anchor–attractor alignment.

F Case Study

The case study shows that reasoning succeeds when activated declarative knowledge and procedural attractors coherently align, but fails when retrieval introduces distracting or irrelevant memories. Correct topic identification alone is insufficient for reliable derivation.

  • Successful activation: Successful activation retrieves the Pitman–Yor definition, a closed-form expectation formula, a Gamma-function reasoning schema, and an episodic exemplar for computing E[K_n].The Memory Twin-Needle Activator combines conceptual grounding with procedural guidance.
  • Failure activation: Failure activation retrieves the correct subject definition but adds a distracting closed-form expression and an irrelevant Dirichlet-process attractor.These noisy activations shift reasoning away from the intended recurrence-based approach.
  • Failure activation: 2.5 is the incorrect value produced after GLM-4-32B follows an incompatible procedure and fails to derive a valid result.The case indicates that identifying the correct topic alone does not ensure reliable reasoning.
  • Case-study implication: Reliable reasoning requires coherent alignment between declarative Anchors and procedural Attractors.The paired cases connect memory-activation quality with the resulting reasoning outcome.
  • Successful activation: HybridRAG composes relevant anchors and attractors with the question and produces the correct answer, 187/64.The successful Grok-4-Fast case demonstrates aligned memory activation on TheoremQA.
Loading 2601.09274v1…