Source-linked AI summary
From Retrieval to Weights: Parametric Individualization of Small Language Models with Individual Text Corpora
Christoph Wigbels, Ali Abusaleh, Markus T. Jansen, Alexander Mehler, Markus J. Hofmann
TL;DR
The paper asks whether individual knowledge can be modeled beyond population-level language-model knowledge. It incorporates personal search-history corpora through retrieval and per-participant DoRA adapters, finding that adapters write individual text into model weights but improve generic knowledge rather than person-specific answer alignment.
Problem
Modeling knowledge at the individual rather than aggregate level remains a distinct and largely open challenge.
Method
The study combines individual text-corpus retrieval with per-participant DoRA adapters applied to a small language model and evaluates answer alignment using content-based likelihood readouts.
Results
The adapters fit their own participant’s held-out text better than other participants’ texts (dz = 1.27), while log-loss match improves but PMI match accuracy does not, and retrieval adds nothing on top.
Takeaways & Limitations
Individual text corpora can be consolidated into small-model weights, but this consolidation does not reproduce participants’ individual answer behavior on the generalized knowledge test.
Takeaways & Limitations
The analyses use a stratified subsample of 150 participants with trimmed corpus-size extremes, and the predominantly young, female, highly educated sample may limit generalizability.
Abstract
from arXiv · showhide
We approach a cognitive simulation perspective on episodic and semantic memory in multiple-choice question answering by incorporating text from individual text corpora (ITC) into retrieval-augmented generation and DoRA fine-tuning. We web-crawl the search histories of 515 participants who answered 36 multiple-choice knowledge items and analyze a stratified subsample of 150 participants. For each participant, one DoRA adapter consolidates their ITC into a small language model (SLM) whose baseline correctness falls below the participants' lowest quartile. The adapter measurably writes the ITC into the weights: it fits its own participant's held-out text better than other participants' texts (dz =1.27), an individuality effect that increases with ITC size in rank order. On the generalized knowledge test, however, the adapter adds knowledge rather than alignment with the individual: log-loss match improves, whereas match accuracy under a bias-corrected PMI readout does not, and retrieval adds nothing on top. Our results demonstrate that ITCs can be consolidated into the weights of SLMs, an encouraging basis for individualized tutoring agents, and we discuss how to move from there toward a realistic simulation of episodic and semantic memory at the individual level.
1 Introduction
The study targets the open challenge of modeling knowledge at the individual rather than aggregate level by consolidating personal text corpora into model weights and comparing this route with retrieval.
- Motivation: Individual knowledge varies substantially, yet most language-model research recovers or aligns knowledge at the population level.The paper frames individual-level knowledge reproduction as a distinct and largely open challenge.
- Approach: Individual text corpora provide a route from personal experience toward semantic memory by writing participant-specific text into model weights with DoRA adapters.Retrieval keeps the corpus external, whereas DoRA implements consolidation through a low-rank weight update.
- Hypotheses: The study tests whether participant-specific adapters improve answer alignment across retrieval conditions, with match accuracy and log-loss match as key outcomes.The hypotheses also examine whether adapter and retrieval effects substitute for or complement each other.
- Design: A 2×2 design crosses frozen versus adapter-equipped models with answering without retrieval versus ITC-RAG retrieval.All four conditions are evaluated using match accuracy, log-loss match, and CK-accuracy.
2 Related Work
Related work motivates individual text corpora as proxies for acquired knowledge, contrasts retrieval with parametric adaptation, and motivates content-based answer readouts for non-instruction-following models.
- Memory and individualization: Individual text corpora approximate acquired knowledge, while retrieval leaves that corpus external and weight consolidation provides its semantic counterpart.The framing draws on complementary learning systems, distinguishing fast episodic retrieval from slower semantic consolidation.
- Personalized adaptation: Parameter-efficient fine-tuning stores a small adapter per user while sharing the base model, and DoRA decomposes weights into magnitude and direction.The paper positions parameter adaptation as complementary to user-specific retrieval or conditioning.
- Answer readout: Forced-choice first-token readouts can reflect option-identity biases and diverge from generated text, especially as instruction-following ability decreases.These findings motivate scoring the likelihood of option contents instead of option IDs.
3 Methods
The study uses a frozen Qwen3-0.6B base model with per-participant adapters and German search-history corpora to model individualized semantic memory alongside retrieval cues.
- Model: Qwen3-0.6B serves as the frozen base model, while each participant receives a dedicated adapter.The adapters are specified as the mechanism for incorporating participant-specific information into the shared model.
- Participants: Participants needed German fluency, at least one year of Google-account use, and an individual corpus containing at least 2,500 unique word types.Recruitment combined university flyers, SONA, and personal recruitment; participants completed a 60–90 minute survey.
- Memory model: The study treats crystallized knowledge as semantic memory built when fluid intelligence is invested in learning directed toward particular domains.The conceptual framework links person-specific Gc to environmental enrichment and accumulated knowledge.
no RAG IC-RAG
The evaluation contrasts answering with and without retrieval while measuring answer alignment from option-content likelihoods, using personal texts as either retrieval cues or adapted model information.
- Measures: The study evaluates match accuracy and log-loss match as primary measures, with CK-accuracy as a secondary measure.The answer-level analyses use N=138 participants, while text-level analyses use all N=150.
- Conditions: ITC-based retrieval supplies participants’ own texts as episodic cues at test time, while the study compares this route with parametric adaptation.The figure materials frame the central question as whether the model reproduces the individual’s answers.
- Readout: Option-content scores use length-normalized sequence likelihoods renormalized across four options, with PMI correcting preferences for generic option strings.A permutation-balanced control bank confirmed that the readout was content-driven in both base and adapter arms.
- Decomposition: Knowledge gain compares adapter-versus-base shifts on correct options against shifts on wrong options, while error alignment tracks the participant’s chosen wrong option.These decompositions clarify what CK-accuracy cannot distinguish.
4 Results
The adapters write participant-specific information into model weights, but answer-level gains reflect added general knowledge rather than reliable alignment with individual responses. Retrieval adds little beyond the adapter, and individuality is larger for participants with larger corpora in rank order.
- Parametric individualization: 0.107 nats per token better fit to their own participants’ held-out text demonstrates an individual signal beyond generic domain gain.The individuality contrast was significant (dz = 1.27; p < .001).
- Answer-level outcomes: −0.008 match-accuracy change under PMI was null, whereas log-loss match improved by 0.079 nats under PMI and 0.248 nats under LN.The length-normalized match-accuracy gain (+0.050, dz = 0.91) is attributed to generic option-string preferences removed by PMI.
- Answer-level outcomes: Generic knowledge gain carries the log-loss improvement: adapters add probability to correct options but move away from participants’ specific errors.Under PMI, knowledge gain on correctly answered items was +0.039, while chosen-option mass on incorrect items fell by 0.043 and error alignment was −0.017.
- Contamination control: The base model scored .417 versus .292 on public versus unpublished items, while participants scored .672 versus .645; the adapter shrank this gap to .072.The item-set split serves as a contamination control, with added knowledge landing disproportionately on items unavailable during base-model pretraining.
- Retrieval interaction: 0.004 was the primary-PMI adapter-by-retrieval interaction for match accuracy and −0.008 for log-loss match, indicating no retrieval advantage on top of adapters.Both primary-PMI interactions were null; positive values would mean retrieval helps more with the adapter than with the base model.
- ITC size: ρ = .29 (p < .001) showed that individuality tended to be larger for participants with larger corpora, despite no linear association (r = .15, p = .069).Partialing out log10 ITC size left the contrast essentially unchanged: size-adjusted dz = 1.29 versus 1.27 unadjusted.
5 Discussion and Outlook
The adapters consolidated individual text into model weights, but answer-level effects reflected added knowledge rather than reliable person-specific alignment. The small-model setup improved log-loss while remaining below participants’ correctness, and retrieval added little.
- Findings: dz = 1.27: Each participant’s adapter fit its own held-out text better than other participants’ texts, demonstrating person-specific information in the weights.This establishes consolidation for the individual text corpus itself.
- Findings: Log-loss match improved, but match accuracy did not improve in the primary PMI view, indicating that the adapter’s answer-level gains were not reliably person-specific.The correctness split attributed the log-loss gain to generic knowledge rather than alignment with the participant.
- Outlook: The base model’s correctness was .33 versus the participants’ lower quartile of .56, and fine-tuning did not raise simulation to participants’ correctness level.The authors identify the small-model choice as an inherent drawback and discuss larger or tailored base models as potential remedies.
- Outlook: Log-loss values remained above the chance-level baseline of −ln(0.25) = 1.39 nats, so calibration to participants’ own answers remained limited.The authors also caution that these values are not directly comparable with the prior study because the scoring views differ.
- Interpretation: The adapter added probability to correct options while moving away from the participant’s specific errors, dissociating knowledge from individual answer behavior.The paper presents this as a distinction between consolidated knowledge and person-specific alignment.
- Interpretation: ITC-RAG added nearly nothing on top of the semantic route, and retrieval did not interact with the adapter effect.The comparison supports a limited incremental role for retrieval in this setup.
Limitations
The analyses are limited by a trimmed, demographically narrow participant sample, a German-only design, and a generalized test that captures only knowledge shared across participants.
- Sample: The analysis uses a stratified subsample of 150 participants with trimmed ITC-size extremes, limiting conclusions about very small or very large corpora.The underlying sample was also predominantly young, female, and highly educated, which may limit demographic generalizability.
- Language: The study is monolingual: German items and German running text define the training signal, while English content enters only through machine translation.Whether parametric individualization generalizes to other languages remains open.
- Measurement: The generalized knowledge test measures only knowledge participants hold in common, making answer-level individualization results a lower bound.An assessment targeted to each participant’s own knowledge could reveal more individualization.
Ethical considerations and data availability
The study received ethics approval, documented data-protection measures for potentially person-related ITCs, and obtained participant consent for sharing search histories and possible scientific reuse.
- Data protection: Ethics approval and documented technical and organizational safeguards governed the handling of potentially person-related individual text corpora.The paper states that raw URLs were removed and relative rather than precise timestamps were retained.
- Consent: Participants consented to share their Google search histories for potential scientific reuse, and some consented to storage of their names for identifiability testing.The study treats these corpora as data requiring continued protection.
- Data availability: A planned shared task will allow registered teams to participate under strict data-protection conditions.Evidence that identifiability is absent would not establish that it is impossible, so the data will remain protected.
A Knowledge Item Example
The knowledge items use a four-option multiple-choice format, illustrated by a German question about mitosis with an English translation and four answer choices.
- Item format: The 12 BEFKI GC-K core items use a documented four-option format, while 24 extended items remain unpublished to reduce contamination risk.The example is reproduced from the public BEFKI GC-K documentation because no official English translation exists.
- Example: “Wozu dient die Mitose?” is translated as “What is the function of mitosis?”The item demonstrates how the German assessment questions are presented for readers.
- Answer choices: The four answer choices concern metabolic regulation, reproduction, formation of germ cells, and cell proliferation during growth.The item’s listed alternatives show the content-option structure used in the example.
B Text-Level Scoring Protocol
Held-out running text is scored with a standardized tokenization, packing, and causal-shift procedure so per-token NLL remains comparable across participants despite differing corpus lengths.
- Documents are concatenated with EOS delimiters, tokenized without special tokens, and packed into independently scored 2048-token blocks.A final tail block is retained when it contains at least 2 tokens.
- NLL is summed in fp32 across all blocks, while the first token of each block is excluded from prediction under the standard causal shift.
- The protocol makes per-token NLL comparable across participants regardless of corpus length.