PaperlayerView exact revision on arXiv

Source-linked AI summary

When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization

Jiayu Hou, Lei Wang

arXiv:2608.29327v1cs.CL

TL;DR

Domain specialization must improve in-domain performance without exposing harmful adaptation effects to every out-of-domain input. Engram Adapter repurposes conditional memory for frozen LLMs using local n-gram matching, occupancy tracking, and gated residuals. It improves target-domain accuracy while preserving general capabilities, with mechanistic analyses showing that false activations still produce very small residual and output drift.

  • Problem

    Existing PEFT methods are typically always on, applying learned perturbations to every input and risking degraded out-of-domain performance during domain specialization.

  • Method

    Engram Adapter repurposes DeepSeek Engram as a post-hoc adapter for frozen LLMs, combining local-pattern matching, occupancy tracking, and learned gating.

  • Results

    Engram Adapter improves target-domain performance while preserving 99.4%--100.1% of average OOD performance; OOD residuals remain below 0.08% of hidden-state norm on average.

  • Takeaways & Limitations

    Conditional activation is a promising route toward modular, retention-preserving domain specialization over frozen backbones.

  • Takeaways & Limitations

    The experiments cover two model scales and three evaluation domains, while larger models and more challenging corpora remain untested; the adapter also trails LoRA by 2–4 percentage points across experiments.

Abstract

from arXiv · show

Large language models deployed in specialized domains must improve in-domain performance without sacrificing general capabilities. Existing parameter-efficient fine-tuning methods are typically always on: their learned perturbations are applied to every input, which can degrade out-of-domain (OOD) performance. We propose Engram Adapter, a framework that repurposes pretraining-time conditional memory as a post-hoc adapter for frozen LLMs. It uses multi-channel matching over local n-gram patterns with explicit occupancy tracking as a lightweight selectivity prior, making residual injection more likely on in-domain inputs while a learned scalar gate suppresses incoherent OOD retrievals. We evaluate on Qwen3-4B and Qwen3-8B with AG-News and MedMCQA as adaptation tasks and OOD benchmarks spanning reasoning, translation, code generation, and legal reasoning. Engram Adapter improves in-domain accuracy while preserving 99.4%--100.1% of average OOD performance; on LegalBench it slightly exceeds the frozen base model on average, whereas comparable always-on baselines degrade sharply. Mechanistic analyses show that although OOD activations are non-zero, gate and projection attenuation reduce residuals to approximately 0.08% of hidden-state norm, yielding small KL drift and negligible accuracy change. These results suggest conditional activation is a promising route toward modular, retention-preserving domain specialization over frozen backbones.

1 Introduction

The paper addresses the retention problem in domain specialization by replacing always-on adaptation with conditional residual injection. Engram Adapter uses local-pattern selectivity, gating, and limited layer insertion to improve target-domain performance while preserving general capabilities.

  • Motivation: Most PEFT methods expose learned perturbations to every input, creating a trade-off between in-domain improvement and out-of-domain retention.The paper frames the central question as when domain knowledge should be invoked, not only how it should be injected.
  • Motivation: Soft gating over hidden representations can be unreliable in low-resource settings because domain boundaries are not cleanly separable.The paper motivates explicit local n-gram patterns as a more stable domain signal.
  • Approach: Engram Adapter repurposes DeepSeek Engram as a post-hoc adapter for frozen language models, using occupancy-tracked multi-hash matching, scalar gating, and projection attenuation.The design injects residuals selectively and suppresses incoherent retrievals under false activation.
  • Approach: The framework turns domain specialization from uniform parameter perturbation into selective knowledge injection driven by local pattern matching.Domain-relevant inputs are more likely to activate the adapter, while weakly matched OOD inputs receive masked or gate-attenuated residuals.
  • Evaluation: Experiments on Qwen3-4B and Qwen3-8B use AG-News and MedMCQA for adaptation and evaluate reasoning, translation, code generation, and legal reasoning for retention.The reported experiments show improved target-domain performance with more stable general-capability preservation.
  • Findings: Mechanistic analyses connect domain selectivity to residual suppression, small output drift, and stable OOD behavior.The paper also notes that direct multi-domain adapter-switching evaluation remains future work.

2 Related Work

Related work mainly improves adaptation efficiency or limits update magnitude, whereas Engram Adapter conditions residual exposure on explicit local-pattern membership.

  • PEFT and retention: PEFT methods reduce training and storage costs by adapting frozen models with small additional or reparameterized parameter sets.Representative approaches include LoRA, adapters, prefix tuning, and (IA)3.
  • PEFT and retention: Typical PEFT perturbations are exposed unconditionally at inference, so these methods address efficient adaptation rather than when adaptation should intervene.The paper distinguishes this always-on behavior from conditional residual exposure.
  • Forgetting-aware adaptation: Forgetting-aware methods mitigate degradation by constraining parameter changes, allocating task-specific capacity, replaying data, or changing update magnitudes and token-level loss weights.These approaches still produce a single adapted model whose updates are exposed to every input.
  • Conditional memory: Engram Adapter repurposes hash lookup over local n-gram patterns to retrieve a domain-specific residual and control whether that residual is exposed.Its occupancy-based membership signal controls a neural residual path rather than functioning as standalone set membership.
  • Conditional memory: Because false activations cannot be eliminated in natural language, the method combines membership filtering with a learned gate and projection attenuation.This design targets residual suppression beyond activation sparsity alone.

3 Engram Adapter

Engram Adapter attaches a small number of conditional residual modules to a frozen backbone, using shared hash memories, occupancy masks, projection, and learned gating before injection.

  • Conditional-Activation Adapter Framework: Four hook-based adapters are inserted into self-attention layers L = {1, 3, 7, 15}, while the frozen backbone receives no parameter updates.Layers outside this set pass hidden states through unchanged.
  • Conditional-Activation Adapter Framework: The adapter computes new = u(ℓ) + α · g(ℓ) · mem(ℓ)(x1:t), with α = 0.2, a learned gate g(ℓ), and Engram memory retrieval mem(ℓ).The residual is exposed only through domain-matching memory and gating.
  • Memory retrieval: Shared hash-indexed memory tables retrieve local suffix and previous-context n-gram representations for every injected layer.The tables are shared across layers, giving them a common domain-pattern inventory and amortizing memory cost.
  • Occupancy masking: Occupancy-tracked joint masking zeroes all retrieved embeddings in an order group when any of its k hashes maps to an unfilled bucket.This coordination supplies the selectivity prior used by the retention mechanism.
  • Projection and gating: Filtered memory chunks are concatenated, projected into the hidden dimension, and scaled by the learned scalar gate before residual injection.Ablations report that linear gate terms account for nearly all observed OOD suppression, while the dot-product term contributes negligibly.
  • Implementation: The Qwen3-4B configuration has approximately 154M trainable adapter parameters, larger than LoRA r=64 at approximately 132M, while injecting adapters into only 4 of 36 layers.Many memory rows remain inactive under sparse occupancy.

4 Experiments

The experiments compare Engram Adapter with LoRA across two Qwen3 scales, two adaptation domains, and diverse OOD capability benchmarks, measuring both specialization and retention.

  • Evaluation design: The evaluation measures domain adaptation effectiveness and preservation of general capabilities across two model scales, two adaptation tasks, and 12 legal reasoning tasks.This design explicitly tests both specialization and retention preservation.
  • Evaluation design: Qwen3-4B uses 1,000 AG-News or MedMCQA examples and evaluates ARC-Challenge, the opposite domain, FLORES, and MBPP.Qwen3-8B uses 10,000 AG-News examples and evaluates MedMCQA, ARC-Challenge, and LegalBench.
  • Experimental setup: Engram Adapter and LoRA baselines are run with three seeds on Qwen3-4B and Qwen3-8B.The adapter uses layers L = {1, 3, 7, 15}, suffix orders (2, 3), previous-context order (3), and k = 4.
  • Experiments: Tables 1 and 2 report complete results for AG-News adaptation on Qwen3-4B and the corresponding medical-domain experiment.The medical experiment is presented as a generality check.

5 Mechanistic Evidence for Retention

Retention is supported by a measured perturbation pathway: OOD inputs can activate the adapter, but gating and projection attenuation keep residuals, output drift, and accuracy changes small.

  • Activation: OOD activation reaches roughly 26–28% of token positions, so retention cannot be explained by exact OOD transparency.The analysis examines the two highest-activation cross-domain pairs and reports non-zero activation across retained n-gram channels.
  • Activation: The occupancy-tracked mask provides relative selectivity, while false activations remain possible because natural-language hash inputs are correlated.Empirical false positives can exceed the idealized ρk reference because shared token-ID inputs and non-uniform n-gram distributions correlate collisions.
  • Residual: ≈0.08% mean residual norm and ≈1.1–1.2% 95th-percentile activated-token perturbation show strong suppression despite 26–28% activation.The residual is defined as ∆u = α · g · mem, and the measured ratio is ∥∆u∥2/∥u∥2.
  • Output: Mean output KL remains 0.04–0.09, while prediction changes affect only 2–8% of examples.For AG → MedMCQA, accuracy changes from 53.65% to 53.60%; for Med → AG-News, only 2.16% of predictions change.
  • Behavior: Net accuracy changes range from −0.05 to +1.43 percentage points, consistent with stable OOD behavior despite non-negligible activation.The reported pathway links activation, residual suppression, limited output drift, and negligible downstream accuracy change.

6 Scaling and Baseline Ablations on Qwen3-8B

Qwen3-8B experiments compare Engram Adapter with LoRA learning-rate controls and TALR, showing a stronger adaptation–retention balance for Engram Adapter than for the always-on baselines.

  • Experimental setup: Qwen3-8B evaluation adds LoRA learning rates of 10−5 and 10−6, TALR, and 12 LegalBench tasks to test scaling and retention controls.The experiment uses AG-News adaptation and evaluates MedMCQA, ARC-Challenge, and LegalBench for OOD retention.
  • Engram Adapter: 86.47% AG-News accuracy accompanies 56.83% MedMCQA, 89.52% ARC-Challenge, and 72.53% LegalBench accuracy for Engram Adapter.These are the reported Qwen3-8B AG-News results for the adapted model.
  • Baseline comparison: 90.29% in-domain accuracy from LoRA r=64 comes with 44.67% LegalBench accuracy, illustrating a sharper retention trade-off.The Figure 3 comparison places LoRA variants at higher in-domain accuracy in some settings but lower OOD retention.
  • Baseline comparison: 96.0% average OOD retention for LoRA lr=10−5 remains below Engram Adapter despite near-matched in-domain accuracy of 86.96% versus 86.47%.The baseline also drops to 61.08% on LegalBench.
  • Baseline comparison: 79.57% versus base 79.49% AG-News accuracy shows TALR preserves OOD scores while providing only marginal in-domain improvement.The more conservative LoRA lr=10−6 remains below the base model on MedMCQA, indicating complementary baseline failure modes.

7 Conclusion

Engram Adapter repurposes conditional memory for input-conditioned specialization over frozen backbones, improving target-domain performance while preserving general capabilities. The paper identifies direct multi-domain isolation and composition as future work.

  • Contribution: Engram Adapter repurposes DeepSeek’s Engram conditional memory as a post-hoc adapter for frozen large language models.Its design combines hash-indexed local-pattern matching, occupancy tracking, and a learned scalar gate.
  • Findings: The adapter improves target-domain performance while preserving general capabilities across reasoning, translation, code generation, and legal reasoning benchmarks.Mechanistic analyses attribute stable OOD behavior to suppressed residuals, KL drift, prediction changes, and downstream accuracy.
  • Future work: Direct multi-domain isolation and composition experiments remain future work.The stated scope boundary concerns evaluating multiple specialized domains independently and in combination.

Limitations

The main limitations concern evaluation scope, inference cost, parameter overhead, and weaker in-domain performance than LoRA. The method is also not presented for high-stakes deployment without further validation.

  • Evaluation scope: Validation covers only Qwen3-4B and Qwen3-8B across news, medicine, and law, leaving larger models and harder corpora untested.The authors specifically identify 27B-scale models and more challenging domain corpora as future validation targets.
  • In-domain performance: Engram Adapter trails LoRA by 2–4 percentage points in in-domain performance across experiments, motivating larger datasets, alternative n-gram orders, or tuned α values.The authors leave open whether these changes can close the gap without sacrificing retention.
  • Efficiency: Inference overhead reaches approximately 22% per token on 128-token MBPP generation and 1.45%–39.9% for OOD prefill.Overhead depends on input length, batching, Python-level hashing, and memory access; the implementation uses unoptimized Python lookup.
  • Intended use: The method is not presented as deployable for high-stakes legal or medical decision making without additional validation.The stated use is limited to research training and evaluation with publicly available artifacts and their original licensing terms.
  • Parameter overhead: The Qwen3-4B adapter has approximately 154M trainable parameters, roughly 1.2× LoRA r=64’s approximately 132M.The shared memory tables account for the main parameter cost, while four injected layers add approximately 16M parameters.

C.2 Zero-Training Transparency

Before training, explicit occupancy masks make the adapter exactly transparent, so the adapted model matches the frozen base model. After training, however, hash collisions and shared patterns can activate the adapter on OOD inputs, making preservation an empirical question.

  • Zero-training transparency: At initialization, zero occupancy flags mask every retrieved chunk, producing zero residuals and exact functional equivalence to the frozen model.All hidden states and output logits match the base model, with F(x) = 0 for every input.
  • Post-training scope: After training, OOD inputs can trigger false activations through hash collisions or shared local patterns, so initialization-time transparency is not a post-training guarantee.Post-training preservation depends on mask selectivity, learned gate attenuation, and projection-level residual suppression.
  • Occupancy tracking: Explicit occupancy tracking separates membership from embedding values, avoiding false negatives when observed-pattern embeddings are numerically close to zero.This is presented as an advantage over zero-initialized embeddings used as the activation test.
  • Idealized analysis: The independent-hashing false-positive analysis is an idealized reference rather than a distribution-free bound because natural-language hash inputs are correlated and non-uniform.Empirical OOD activation can exceed the idealized ρ^k rate.
  • OOD suppression: The learned gate and incoherent projections attenuate false-positive residuals after occupancy-mask activation.The gate reduces activation values, while projected memory signals tend to cancel across retrieved chunks.

D.1 Analysis

Engram Adapter improves target-domain accuracy while maintaining strong OOD retention across model scales, adaptation tasks, and capability benchmarks. Its retention is generally competitive with or better than PEFT baselines, while some always-on methods suffer severe forgetting.

  • Retention: 99.38%±0.32% and 100.07%±0.24% are the Engram Adapter’s average OOD retention in the AG-News and MedMCQA 4B experiments.Retention spans reasoning, classification, translation, and code-generation benchmarks.
  • Baseline comparison: PiSSA drops AG-News to 0.00% and ARC-Challenge to 27.30% after MedMCQA training, despite reaching 55.63% in-domain accuracy.After AG-News training, PiSSA reduces MedMCQA to 32.23% and ARC-Challenge to 19.88%.
  • Baseline comparison: 99.38%±0.32% is the Engram Adapter’s AG-News average retention, tying LoRA r=32 with lower variance, while 100.07%±0.24% leads on MedMCQA.The comparison notes that LoRA r=32’s AG-News average is partly driven by positive transfer.
  • Domain adaptation: 87.93%±0.64% AG-News accuracy improves 2.79 percentage points over the 4B base, while MedMCQA reaches 55.79%±0.74%, 2.14 points above base.On 8B AG-News, accuracy is 86.47% versus 79.49% for the base model, a 6.98-point improvement.
  • Cross-domain consistency: The adapter’s retention remains consistent across Qwen3-4B and Qwen3-8B, news and medicine adaptation, and reasoning, classification, translation, coding, and legal reasoning.LegalBench is included among the OOD evaluations supporting this cross-domain pattern.

E Full LegalBench Results

Engram Adapter preserves LegalBench performance while comparable always-on baselines suffer substantial task-level and average degradation. Gate diagnostics compare domain signals under matched in-domain acceptance, with Engram n-gram density providing the native selectivity signal.

  • LegalBench results: 72.53% average accuracy across 12 LegalBench tasks slightly exceeds the base model’s 71.91%.Engram Adapter preserves performance on 10 of 12 tasks.
  • LegalBench results: 37.9% relative degradation reduces LoRA r=64 from 71.91% to 44.67% average LegalBench accuracy.LoRA r=64 degrades on 10 of 12 tasks, including function_of_decision and personal_jurisdiction.
  • LegalBench results: 61.08% LegalBench accuracy for LoRA at lr=10−5 trails the base model despite near-matched AG-News accuracy.The AG-News comparison is 86.96% for LoRA versus 86.47% for Engram Adapter.
  • LegalBench results: 68.54% LegalBench accuracy for LoRA at lr=10−6 is second among adapted models, but still below the base model’s 71.91%.The conservative learning rate preserves performance on 9 of 12 tasks while retaining notable task-specific drops.
  • Gate selectivity diagnostic: Engram n-gram density scores the fraction of valid prompt tokens whose n2, n3, or p3 groups activate all four occupied hash entries.The diagnostic threshold accepts approximately 95% of held-out AG-News examples and reports OOD false-accept rates.

F.2 Gate Ablation

Gate ablations show that disabling the gate substantially increases OOD residual perturbations, while removing its explicit dot-product coherence term has little effect. OOD activations can still occur, but attenuation limits their downstream impact.

  • Gate ablation: 1.8× higher mean residual ratios occur when the gate is disabled for both AG→Med and Med→AG OOD pairs.The ablation measures residual perturbation among activated tokens.
  • OOD perturbation: <0.08% of hidden-state norm is the residual perturbation produced by the gate on the two highest-activation cross-domain pairs.Approximately 27% token-level activation still yields minimal KL divergence and negligible accuracy change.
  • Gate ablation: The learned linear gate term provides most OOD residual attenuation, whereas removing the dot-product coherence term has virtually no effect.Mean residual ratios remain nearly unchanged without the dot-product term, with slightly lower P95 residuals.
  • OOD perturbation: A larger residual in the LegalBench-only no-gate result does not necessarily produce large average-accuracy loss on every OOD benchmark.This bounds the interpretation of residual-level attenuation as a mechanism-level result rather than a universal accuracy guarantee.

G Inference Efficiency and Activation Profile

The current implementation has benchmark-dependent inference overhead and higher in-domain than OOD activation rates, supporting selectivity but not a direct latency–activation relationship. Efficiency comparisons remain setting-specific, and several practical limitations are explicit.

  • Inference efficiency: +1.45%–+1.56% forward overhead occurs on MBPP, compared with +39.9% on AG→MedMCQA.The measured profile uses benchmark-dependent sequence lengths and evaluation windows.
  • Inference efficiency: Higher activation does not imply higher latency: MBPP has higher AG-News-adapter activation than AG→MedMCQA but much lower prefill overhead.Input lengths, batching, Python-level lookup overhead, memory access, and measurement variance also affect runtime.
  • Activation profile: 0.356 versus 0.052–0.221 n3 activation distinguishes AG-News in-domain inputs from OOD sets for the AG-News adapter.For the MedMCQA adapter, the corresponding contrast is 0.601 versus 0.096–0.145.
  • Engineering limitations: Unoptimized Python-level hashing and occupancy lookup are identified as sources of overhead, with fused GPU kernels proposed for future reduction.Inference optimization is left to future work.
  • Baseline limitations: Three gate-only LoRA trials produced empty LegalBench outputs, preventing a meaningful formal accuracy comparison.The variant lacked calibrated supervision for unrelated OOD domains and was trained only on target-domain data.
  • Retention trade-offs: 101.9% OOD retention for TALR on standard benchmarks coexists with only marginal in-domain improvement, 79.57% versus a 79.49% base.The passage contrasts training-time loss reweighting with conditional inference-time intervention.
Loading 2608.29327v1…