PaperlayerView exact revision on arXiv

Source-linked AI summary

ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models

Chonghan Qin, Xiachong Feng, Weitao Ma, Xiaocheng Feng, Lingpeng Kong

arXiv:2604.08064v2cs.AI

TL;DR

Existing LLM memory benchmarks emphasize explicit factual recall, leaving implicit behavioral adaptation insufficiently assessed. ImplicitMemBench evaluates three forms of implicit memory across 17 models and finds no model exceeds 66% overall, exposing limitations beyond parameter scaling.

  • Problem

    Existing LLM memory benchmarks emphasize conscious factual retrieval rather than implicit memory expressed as automated behavior after experience.

  • Method

    ImplicitMemBench evaluates procedural memory, priming, and classical conditioning through a unified Learning/Priming–Interfere–Test protocol with first-attempt scoring.

  • Results

    No model exceeds 66% overall, with severe behavioral asymmetries and bottlenecks across the three implicit-memory paradigms.

  • Takeaways & Limitations

    The findings expose architectural limitations in consolidating experience into automated behavior that require innovations beyond parameter scaling.

  • Takeaways & Limitations

    The benchmark omits perceptual learning, habit formation, motor skill acquisition, and emotional conditioning.

Abstract

from arXiv · show

Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval. This gap is critical: effective assistants must automatically apply learned procedures or avoid failed actions without explicit reminders. We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three cognitively grounded constructs drawn from standard cognitive-science accounts of non-declarative memory: Procedural Memory (one-shot skill acquisition after interference), Priming (theme-driven bias via paired experimental/control instances), and Classical Conditioning (Conditioned Stimulus--Unconditioned Stimulus (CS--US) associations shaping first decisions). Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring. Evaluation of 17 models reveals severe limitations: no model exceeds 66% overall, with top performers DeepSeek-R1 (65.3%), Qwen3-32B (64.1%), and GPT-5 (63.0%) far below human baselines. Analysis uncovers dramatic asymmetries (inhibition 17.6% vs. preference 75.0%) and universal bottlenecks requiring architectural innovations beyond parameter scaling. ImplicitMemBench reframes evaluation from "what agents recall" to "what they automatically enact".

1 Introduction

ImplicitMemBench addresses the gap between explicit fact retrieval and implicit memory, evaluating whether LLM agents automatically adapt behavior through procedural memory, priming, and classical conditioning. Its 300-item, first-attempt protocol reveals substantial performance ceilings, paradigm asymmetries, and limitations of explicit memory augmentation.

  • Motivation: Existing memory benchmarks mainly test consciously prompted factual retrieval, leaving unconscious behavioral adaptation under-evaluated.Prior systems use query-response protocols that actively trigger recall, whereas implicit memory concerns experience becoming automated behavior.
  • Benchmark design: ImplicitMemBench operationalizes three non-declarative constructs: procedural skill acquisition after interference, theme-driven priming, and CS–US conditioning shaping first decisions.The constructs target internalized routines, context-driven adaptation without explicit instruction, and associations influencing initial choices.
  • Benchmark design: 300 items use a unified Learning/Priming–Interfere–Test protocol with first-attempt scoring to isolate automatized behavior from explicit recall.The framework maps cognitive constructs to text-based agent evaluation through functional isomorphism and combines scenario-driven testing with first-attempt outcomes.
  • Findings: 17 evaluated models show no model exceeding two-thirds overall accuracy, with top performers remaining far below human baselines.The results indicate a severe ceiling effect across proprietary and open-source systems.
  • Findings: Performance varies sharply across paradigms: procedural memory is most tractable, while classical conditioning is a substantial bottleneck and priming remains narrowly moderate.The benchmark exposes dramatic behavioral asymmetries rather than uniformly weak performance across implicit-memory mechanisms.
  • Implications: Memory-augmented agents do not consistently improve on IMPLICITMEMBENCH, suggesting implicit memory requires more than explicit storage and retrieval.The authors identify universal bottlenecks that may require architectural innovations beyond parameter scaling.

2 Related Work

Prior LLM memory benchmarks primarily assess explicit memory through active retrieval, despite increasingly diverse tasks and scales. ImplicitMemBench addresses this gap by evaluating procedural learning, priming, and classical conditioning as unconscious behavioral adaptation, distinct from explicit storage and recall.

  • Explicit Memory Benchmarks: Existing benchmarks predominantly evaluate explicit memory through active retrieval triggers across dialogue continuity, long-context reasoning, emotion support, factual and reflective memory, state tracking, and information integration.The cited benchmarks span contexts from roughly 9k tokens to 1.5M tokens and include 300 conversations, 160 dialogues, and multi-platform long-horizon environments.
  • Research Gap: Prior work covers increasingly diverse tasks and scales but leaves implicit memory unaddressed.Table 1 summarizes this distinction between explicit-memory evaluation and the missing implicit-memory setting.
  • ImplicitMemBench: ImplicitMemBench is presented as the first systematic evaluation of implicit memory through procedural learning, priming, and classical conditioning.It uses efficient ~500-token protocols to assess unconscious behavioral adaptations.
  • External Memory Systems: External memory modules and retrieval-based long-term memory systems primarily target explicit storage and recall rather than implicit memory.The paper states that implicit memory is not reducible to explicit retrieval alone.

3 IMPLICITMEMBENCH

ImplicitMemBench evaluates procedural memory, priming, and classical conditioning as complementary forms of implicit adaptation in LLM agents. Its unified Learning-Interference-Test designs test whether models automatically apply learned rules, transfer themes, or avoid conditioned stimuli without explicit reminders.

  • Benchmark Scope: Three paradigms operationalize distinct implicit adaptations: procedural memory acquires behavioral patterns, priming transfers thematic context, and classical conditioning forms stimulus-response associations.The benchmark focuses on these paradigms under a taxonomy of non-declarative memory and excludes non-associative learning.
  • Procedural Memory: Procedural evaluation provides a rule and 1-3 examples, 15 misleading turns, and novel probes scored by first-attempt success.The design isolates proceduralization by testing rule execution despite extensive interference and format variations.
  • Priming: Priming uses matched evocative-versus-neutral exposure, cognitive-buffer interference, and creative generation to measure thematic transfer without reminders.Experimental-control differences isolate causal priming effects, while LLM evaluation detects thematic alignment beyond surface keywords.
  • Classical Conditioning: Classical conditioning pairs conditioned stimuli with unconditioned stimuli, inserts unrelated tasks, and scores the first action after stimulus reintroduction.Tasks target automatic avoidance or adaptation in API selection, filetype choices, and directory restrictions.
  • Dataset Construction: 300 items span 18 task families and three paradigms, with 100 items per paradigm and quality controls applied to an initial pool exceeding 1,000 candidates.The dataset is balanced across paradigms and follows the unified learning-interference-test protocol.

4 Experiments

Experiments evaluate 17 language models under standardized, zero-shot conditions across procedural memory, priming, and classical conditioning, with first-attempt scoring and a 100% human baseline. Results show a ceiling below 66% overall, strong cross-paradigm asymmetries, and persistent bottlenecks in inhibition, conditioning, and context-dependent adaptation.

  • Experimental Protocol: 17 language models were evaluated under identical zero-shot conditions, with deterministic generation for procedural memory and conditioning and test-only stochasticity for priming.All models used standardized prompting, controlled generation parameters, and responses capped at 4096 tokens.
  • Human Baseline: 100% accuracy was achieved by all five human participants across the 300-item benchmark, with 100% interannotator agreement.Participants completed the same Learning/Priming–Interfere–Test protocol and rubric used for model evaluation.
  • Metrics: First-Try Accuracy counts only correct initial responses for procedural memory and conditioning, while Priming Influence Score quantifies thematic influence.Self-corrections and revisions are ignored to measure reflexive first-attempt behavior rather than iterative refinement.
  • Performance Results: 66% was the overall ceiling, with DeepSeek-R1 at 65.30%, Qwen3-32B at 64.13%, and GPT-5 at 63.00%, all below the 100% human baseline.Only three systems exceeded 63% overall, while median performance was 55%.
  • Cross-Paradigm Analysis: 35 points separated Claude-4.1-opus’s 76.67% procedural score from its 41.67% classical-conditioning score, demonstrating capability dissociation across paradigms.Procedural memory was most tractable, while classical conditioning was substantially weaker and priming clustered at 42–52%.
  • Behavioral Bottlenecks: 17.6% inhibition versus 75.0% preference-based adaptation produced a 57.4-point gap, while universal bottlenecks included jargon avoidance at 4% ± 5% and API distrust at 21% ± 17%.Other challenging categories were context-dependent behavior at 28% ± 23%, API aversion at 45% ± 29%, and emotion-driven strategy shift at 55% ± 18%.

5 Conclusion

The conclusion presents IMPLICITMEMBENCH as a systematic benchmark for implicit memory in LLMs, using procedural memory, priming, and classical conditioning. Across 17 models, results expose severe behavioral asymmetries and architectural bottlenecks that parameter scaling alone does not resolve.

  • 5 Conclusion: IMPLICITMEMBENCH systematically evaluates implicit memory through procedural memory, priming, and classical conditioning.The benchmark is presented as the first systematic assessment of these three cognitively grounded constructs in LLMs.
  • 5 Conclusion: 66% overall is the highest score any of 17 evaluated models achieves.The evaluation reveals fundamental limitations across the model set.
  • 5 Conclusion: 17.6% inhibition versus 75.0% preference reveals a severe behavioral asymmetry.The conclusion identifies inhibition and preference as sharply unequal behavioral outcomes.
  • 5 Conclusion: Universal bottlenecks persist across all architectures, exposing limitations that require innovations beyond parameter scaling.The benchmark establishes reproducible protocols for assessing implicit memory while highlighting architectural limitations.

Limitations · A Detailed Models List

The benchmark covers three core implicit-learning paradigms but omits several complementary phenomena. The study evaluates 14 proprietary and open-source language models across developers, families, architectures, training paradigms, and scales.

  • Limitations: IMPLICITMEMBENCH evaluates procedural memory, classical conditioning, and priming as core implicit-learning mechanisms.These paradigms are drawn from cognitive science.
  • Limitations: The benchmark does not include perceptual learning, habit formation, motor skill acquisition, or emotional conditioning.These phenomena represent additional aspects of implicit memory.
  • Limitations: Future work could expand the benchmark to cover these complementary implicit-memory phenomena.The proposed expansion addresses the evaluation’s current scope.
  • A Detailed Models List: 14 models were evaluated across proprietary and open-source systems.The proprietary systems include OpenAI’s GPT series, Anthropic’s Claude, and Google’s Gemini; open-source alternatives include Qwen, LLaMA, DeepSeek, and GLM.
  • A Detailed Models List: The evaluated models span OpenAI’s GPT series, Anthropic’s Claude, Google’s Gemini, Qwen, LLaMA, DeepSeek, and GLM.These models are organized by developer and model family in Table 8.
  • A Detailed Models List: The selection supports comparisons across architectural choices, training paradigms, and scale configurations.Table 8 provides the complete list of evaluated models.

B Additional Evaluation Validations · B.1 Priming Influence Score

This section notes that Table 9 presents the exact scoring protocol for the Priming Influence Score evaluation.

  • B.1 Priming Influence Score: Table 9 presents the exact scoring protocol for the Priming Influence Score evaluation.

B.2 Judge Robustness · B.3 Context-Length Sensitivity

Independent judging produced highly stable system rankings, while increasing interference length caused a sharp performance decline that plateaued after approximately 500 tokens. These findings support both judge robustness and the use of an approximately 500-token context budget.

  • B.2 Judge Robustness: Top 11 and bottom 2 system positions were identical across GPT-4o-mini and Gemini-2.5-Flash judges.All 17 systems were re-evaluated with Gemini-2.5-Flash as an independent second judge.
  • B.2 Judge Robustness: Middle-tier ranking changes across the two judges were minor.The reported stability concerns the system ranking across the two judge models.
  • B.2 Judge Robustness: All 17 systems were evaluated under both GPT-4o-mini and Gemini-2.5-Flash judging.Gemini-2.5-Flash served as an independent second judge because LLM-as-Judge evaluation may introduce model-family bias.
  • B.3 Context-Length Sensitivity: Performance then plateaus beyond approximately 500 interference tokens.The passage reports a sharp decline before the plateau but does not provide the corresponding performance values.
  • B.3 Context-Length Sensitivity: Approximately 500 tokens were selected as an efficient context budget.The plateau supports the claim that this budget is already sufficient to move beyond short-term retention effects.

B.4 Human Baseline Details

The human baseline used five computer science Ph.D. students who completed all 300 benchmark items under the model-matched protocol. Independent scoring by two additional Ph.D. students yielded 100% interannotator agreement, and all participants achieved 100% accuracy.

  • Five computer science Ph.D. students completed the full 300-item benchmark under the same Learning/Priming–Interfere–Test protocol as the evaluated models.
  • 100% interannotator agreement was obtained when two additional computer science Ph.D. students independently scored participants using the model-experiment rubric.
  • 100% accuracy was achieved by all five human participants.

C Additional Analysis of Memory Frameworks

Mem0 and MIRIX expose memory interfaces that typically depend on external or manual storage decisions, unlike the automatic unconscious adaptation targeted by IMPLICITMEMBENCH. An oracle-style Mem0 evaluation shows that even perfect storage of critical information yields inconsistent paradigm-specific gains, so implicit memory is not reducible to retrieving stored content.

  • Framework distinction: Mem0 and MIRIX are agent frameworks with memory interfaces, not automatic memory modules, and often require external logic or manual storage decisions.Their operational mode therefore differs fundamentally from the benchmark’s target of automatic, unconscious adaptation.
  • Oracle evaluation: Oracle-style Mem0 manually stores each task’s critical rule, core priming content, or exact CS–US pairing, assuming perfect extraction and storage.This setup is substantially stronger than the benchmark’s intended setting.
  • Oracle evaluation: Oracle memory improves overall performance, but gains remain highly inconsistent across Procedural Memory, Priming, and Classical Conditioning.For DeepSeek-R1, improvement concentrates mainly in Priming; for Qwen2.5-7B-Instruct, larger gains appear in Procedural Memory and Classical Conditioning.

D Memory Augmented Agents · E Discussion

External memory produces inconsistent benefits on implicit-memory tasks, while ImplicitMemBench advocates experience-based training and evaluation of behavioral internalization rather than retrieval alone.

  • D Memory Augmented Agents: MemAgent yields a modest overall gain (+3.9), MemGPT a smaller gain (+2.3), and MEM1 decreases overall performance (-2.1).These comparisons use MEM1 with Qwen2.5-7B, MemAgent with Qwen2.5-14B, and MemGPT with Yi-34B-200K.
  • D Memory Augmented Agents: MemAgent improves Procedural Memory (51.00 →60.00), whereas MEM1 and MemGPT reduce it (50.67 →27.00 and 50.00 →44.00).The pattern suggests explicit retrieval can interfere with immediate rule execution in procedural tasks.
  • D Memory Augmented Agents: Classical Conditioning improves for all three memory-augmented agents, but MemAgent and MemGPT remain low at 22.00 and 25.00.The passage indicates that recording past failures is insufficient to produce robust conditioning.
  • E Discussion: ImplicitMemBench’s compact interaction protocols provide a blueprint for training data in which models learn from experience rather than explicit instructions alone.This implication extends the benchmark beyond metric reporting toward future agent design.
  • E Discussion: ImplicitMemBench shifts evaluation from retrieval to internalization by testing whether exposure becomes behavior.This contrasts with LongMemEval, which mainly tests whether models can recover information from long contexts.
  • E Discussion: The benchmark asks whether models internalize experience, rather than merely recovering information from long contexts.This reframing treats learned behavior as the evaluation target instead of explicit information retrieval.

F LLM Prompts · G Illustrative Examples

The appendix specifies the prompt architecture and validation rules used to generate, curate, and evaluate ImplicitMemBench instances across memory paradigms. It also reports that procedural-memory accuracy reaches an effective interference threshold near ~500 tokens, after which performance plateaus.

  • F LLM Prompts: The appendix organizes prompts for data generation, curation, and evaluation, with controls for task difficulty, cross-paradigm consistency, and LLM-as-Judge criteria.These prompts constitute the complete prompt set used throughout the ImplicitMemBench pipeline.
  • F.1 Data Generation: 58.4% accuracy at ~200 tokens falls to 50.1% at ~500 tokens, then plateaus at 49.8% and 49.5% for ~1000 and ~2000 tokens.The appendix labels ~200 tokens insufficient, ~500 tokens the effective threshold, and longer contexts a plateau.
  • F.1.1 Data Generation Prompt for Procedural Memory: Procedural-memory prompts require a distinct, contextually relevant interference phase and a test probe applying the learned core skill in a new context or format.The prompt specifies exactly 15 user-assistant dialogue turns, or 30 messages, for interference.
  • F.1.1 Data Generation Prompt for Procedural Memory: Procedural-memory outputs must be raw JSON with explicit learning, interference, test-probe, and expected-pattern fields whose dialogue remains logically consistent with the rule description.Validation also requires direct dialogue-turn objects and exact interference-message counts.
  • F.1.2 Data Generation Prompt for Priming: Priming generation creates matched experimental and control instances with one shared probe, while reusing the experimental interference messages verbatim in the control instance.Experimental priming is evocative and imagery-dense, whereas control priming is neutral, technical, and low-imagery.
  • F.1.2 Data Generation Prompt for Priming: Priming validation enforces exactly 2 interference messages, shared probe objects, group labels, forbidden-lexicon compliance, and required-content checks.The output schema requires parallel experimental_instance and control_instance structures with meta_checks.
  • F.1.3 Data Generation Prompt for Classical Conditioning: The conditioning examples operationalize avoidance as selecting an alternative approach or issuing a warning after a cue-associated failure, without system-context hints.Interference consists of 2-3 unrelated Q&A turns, and the probe naturally triggers the learned association.

H Further Analysis · Illustrative Example: Procedural Memory: Reversed Parameter Protocol · Illustrative Example: Classical Conditioning: Conditioned Protocol Preference

The further analysis profiles model-specific strengths and weaknesses across implicit-memory task families, revealing tradeoffs among procedural learning, classical conditioning, and priming. Illustrative examples show reversed-parameter procedural learning, theme-based creative-naming priming, and conditioned protocol preference.

  • Illustrative Example: Procedural Memory: Reversed Parameter Protocol: The procedural-memory example teaches the reversed protocol copy_file('destination_path', 'source_path') before unrelated interference.The example tests whether the learned parameter order is applied after an interference phase.
  • Illustrative Example: Procedural Memory: Reversed Parameter Protocol: A priming example contrasts creative naming after a Volcanic Eruption theme with a Dewey Decimal Classification control condition.The paired conditions illustrate experimental-versus-control testing of theme-driven creative behavior.
  • Illustrative Example: Procedural Memory: Reversed Parameter Protocol: The experimental priming condition begins with an evocative volcanic-eruption paragraph describing pressure, ash, molten rivers, destruction, and regrowth.The passage represents the learning phase for the volcanic-eruption source theme.
  • Illustrative Example: Classical Conditioning: Conditioned Protocol Preference: The classical-conditioning example repeatedly pairs insecure http URL generation with warnings before testing protocol choice.The sequence includes repeated http requests that trigger insecure-protocol warnings, followed by an https request that succeeds.
  • Illustrative Example: Classical Conditioning: Conditioned Protocol Preference: The conditioned protocol-preference task illustrates how prior warning-associated interactions can shape a subsequent first protocol decision.The example contrasts warning-triggering http behavior with successful https generation.
  • H Further Analysis: Table 14 profiles each evaluated model’s reliable behavioral strengths and recurring weaknesses across implicit-memory task families.The analysis emphasizes model-specific capability profiles and tradeoffs among procedural learning, classical conditioning, and priming.
Loading 2604.08064v2…