Source-linked AI summary
Mostly Reading: What Open-Book Scores Measure in Document-Level LoRA Adapters
Martino M. L. Pulici, Cuong Xuan Chu, Evgeny Kharlamov, Volker Tresp
TL;DR
Open-book scores may reward document-level LoRA adapters for reading supplied text as well as for holding its content, making their value as mergeable or retrievable knowledge units uncertain. The paper trains adapters on Wikipedia articles and compares open-book scoring with document-withheld cloze recall. Random adapters recover much of the specialists’ open-book advantage but little of their cloze advantage, and merging shows a similar gap.
Problem
Open-book scoring with the document supplied cannot distinguish an adapter that holds its content from one that has learned to extract answers from text, complicating assessment of adapters as document-level knowledge units.
Source: §1 · p. 1Method
Results
82 % of specialists’ open-book advantage transfers to a random adapter, versus 2 % of their cloze advantage, while merges at 16 documents or more recover 12 % to 16 % of cloze advantage against 94 % for joint training.
Source: §1 · pp. 1–2Takeaways & Limitations
The authors recommend pairing open-book scoring with a document-withheld probe, averaging comparisons over random adapters, and using consistent precision and decoding settings.
Source: §8 · p. 8The experiments use one base model, one adapter rank and training schedule, 150 encyclopedic articles, and collections no larger than 32 documents, limiting generalization across models, capacities, genres, and larger collections.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Single-document LoRA adapters have been proposed as units of knowledge to be merged and retrieved; yet open-book scoring, with the document in the prompt, can credit a document's own adapter (its specialist) for reading that document as readily as for holding it. Training one adapter for each of 150 Wikipedia articles on Llama-3.1-8B-Instruct, we find that open-book scores mostly measure a reading behaviour every adapter shares: a random adapter trained on another document recovers 82 % of the specialist's open-book advantage over the base model but only 2 % of its advantage on a cloze probe that asks for the same document's facts without providing the source text, a share we cannot distinguish from zero. Content does reach the weights, since the specialists beat the base model in cloze recall by 0.094, and the geometry of their updates follows the documents they read; yet, in a test of 30 queries, adapters chosen by weight-space proximity or text embedding are not distinguishable from random ones on either probe, although that test cannot exclude an advantage of about four fifths of what a query's own adapter gains over a random one. At 16 documents or more, correctly scaled merges recover 87 % to 89 % of the specialists' open-book advantage but only 12 % to 16 % of their cloze advantage, with intervals that run from below zero to at most 32 %, against 94 % for joint training.We also report four silent errors, each of which produced a clean, wrong result.
1 Introduction
Open-book scoring may reward an adapter for extracting answers from supplied text rather than for holding that document’s content. The experiments find that open-book advantages transfer broadly across adapters, while cloze recall and merge results show a much smaller content advantage; retrieval tests and four silent errors qualify the findings.
1 Introduction
- Specialists gain 0.094 over the base model on cloze recall, and update distances follow the domain and text read rather than the questions.When questions and text come from different documents, update geometry follows the text.Source: §1 · p. 1
- 82 % of the specialists’ open-book advantage transfers to a random adapter, but only 2 % of their cloze advantage, a share not distinguishable from zero.At 16 documents or more, merges recover 12 % to 16 % of the cloze advantage, each CI including zero, versus 94 % for joint training.Source: §1 · pp. 1–2
- With 30 queries, weight-space and text-embedding selections are not distinguishable from random on either probe, but the cloze CI still permits an advantage of about four fifths of the own-adapter margin.The bank excludes the query’s own adapter.Source: §1 · pp. 1–2
- At four documents or more, merges retain 0.93 to 1.01 of specialists’ open-book scores, compared with 0.88 to 0.97 for random adapters.The open-book result contrasts with the weaker cloze recovery at 16 documents or more.Source: §1 · p. 2
- Four silent errors produced clean but wrong results, including a training-length default that reversed a comparison and a merge-scaling error that looked like a merging cost.The other errors were a repetition penalty left in evaluation code and a mismatch between evaluation and training precision.Source: §1 · p. 2
2 Experimental setup
The study trains document-specific LoRA adapters on 150 English Wikipedia articles, then evaluates open-book and document-withheld recall using held-out questions and a cloze probe. It also compares adapter composition methods across document collections, with paired bootstrap intervals for uncertainty.
2 Experimental setup
- 150 English Wikipedia articles, 25 in each of six domains, supply eight training QA pairs and four held-out items per article.The base is Llama-3.1-8B-Instruct; rank-8 adapters train for 500 steps on every token of each training string.Source: §2 · p. 2
- Five training conditions keep each article’s QA pairs fixed while varying accompanying text from the natural article to no context or another article’s prose.The other conditions are a style-normalized rewrite and a synthetic template retaining six extracted facts.Source: §2 · p. 2
- Open-book evaluation supplies the document, while closed-book evaluation withholds it; both score held-out answers using SQuAD token-level F1.The cloze probe presents declarative sentences with a blank, without the document, and scores recall on three factual items.Source: §2 · p. 2
- 95 % percentile confidence intervals use 10 000 paired bootstrap resamples, with the resampling unit varying by analysis.Units are documents except for composition collections, retrieval queries, and transfer source–target pairs.Source: §2 · p. 2
- Composition compares merges, joint training, and random outside adapters across within-domain and cross-domain collections of 2 to 32 documents.Retention is the method’s held-out score divided by the document specialist’s score; random scores average five draws.Source: §2 · p. 3
3 Content and geometry
Training-text comparisons show that cloze recall tracks whether an adapter read the scored document’s full content, while open-book and closed-book scores can favor conditions that did not. Adapter updates also reflect document semantics, and context swaps indicate that both the read document and question supervision shape outcomes.
3.1 Training text and encoding
- Cloze gains clear of zero only when an adapter read the scored document’s full content; the natural adapter gains +0.094 over the base model.No-context adapters score 0.132 below the base model on cloze, despite ranking second of five conditions on open-book scoring.Source: §3.1 · p. 3, Table 1 · p. 3
- Open-book and closed-book rankings can favor adapters that read least of the scored document, while cloze recall tracks content exposure.The no-context adapter scores 0.773 open-book, 0.049 behind the natural adapter, and leads both rewrites despite reading no document text.Source: §3.1 · p. 3
3.2 Orthogonality and semantic structure
- Mean update cosine is 0.0066, nearly four times the 0.0018 reference, while layerwise updates remain nearly orthogonal.The comparison indicates A matrices move toward shared input directions even though the updates stay nearly orthogonal for merging.Source: §3.2 · pp. 3–4
- Update distances correlate with document sentence-embedding distances, and Llama-3.1-8B-Instruct reaches held-out ρ = 0.296 using selected final-third layers.Across all layers, six nearest neighbours share an adapter’s domain at rate 0.708 versus chance 0.161.Source: §3.2 · p. 4
3.3 Context swap
- Context swaps show update geometry follows the donor document’s semantics (ρ = +0.217), not the source questions’ semantics (+0.033).Swapped adapters gain +0.056 cloze recall on the document they read but are not distinguishable from the base model on their source; on 99 donor documents they recover 59% of the natural adapter’s open-book advantage, with question supervision supplying the rest.Source: §3.3 · p. 4
4 Behaviour and content
The section compares adapter transfer, random-adapter performance, and retrieval from a bank lacking each query’s specialist. Foreign adapters recover much of the specialist’s open-book advantage, but little cloze advantage, and retrieval comparisons are inconclusive.
4 Behaviour and content
- The section reports adapter results as shares of specialists’ advantage over the base model, distinct from retention ratios that divide scores.The share is (s̄ − s̄_b)/(s̄_s − s̄_b); retentions are reported as decimals.Source: §4 · p. 4
4.1 Transfer to unseen documents
- A foreign adapter from the target’s domain recovers 88% of the specialist’s open-book advantage over 120 source–target pairs.Its 95% CI is [77%, 101%], so the portion that does not transfer cannot be distinguished from zero; no-context adapters recover 78%.Source: §4.1 · p. 4
4.2 Random adapter decomposition
- 82% of specialists’ open-book advantage but 2% of their cloze advantage is recovered by a random adapter over collections of 16 documents or more.The cloze share’s 95% CI is [−19%, +17%], so it is not distinguishable from zero.Source: §4.2 · pp. 4–5
4.3 Retrieval by weight-space proximity
- Weight-space and text-embedding selection are not distinguishable from random on either probe across 30 queries.The primary cloze contrast for weight-space selection is +0.020 (95% CI [−0.038, +0.075]); its open-book contrast is −0.024 (95% CI [−0.061, +0.013]).Source: §4.3 · p. 5, Table 2 · p. 5
- The retrieval test cannot rule out a cloze advantage as large as about four fifths of the query’s own-adapter margin over random.That own-adapter margin is 0.091, and the primary contrast’s CI reaches +0.075.Source: §4.3 · p. 5
5 Composition
Section 5 compares adapter merges with random-adapter and joint-training references on open-book and cloze probes. Its central finding is that strong open-book retention does not imply comparable retention of document facts in cloze recall.
5 Composition
- The composition grid compares merges, random out-of-collection adapters, and joint training on both open-book and cloze probes because open-book scores credit shared reading behaviour.The random reference averages five independent adapter draws per set.Source: §5 · p. 5
5.1 Merges on open book
- At rank 8, merges retain 0.93 to 1.01 of specialist open-book scores across nine N ≥4 cells, compared with 0.88 to 0.97 for random adapters.√ LERP, update merging, and TIES each exceed random in several paired cell contrasts, though retention near one is less informative because random already recovers about four fifths of specialist advantage.Source: §5.1 · pp. 5–6, Table 3 · p. 6
5.2 Scale and cross terms
- 0.176 to 0.348 lower open-book retention resulted when LERP factors were each averaged at 1/N, scaling own updates by 1/N^2; matched-scale cross terms shifted retention by −0.035 to +0.008.The cited comparisons found at most 0.034 retention cost from rank-8 truncation, and tuning the merge coefficient added nothing over 1/N.Source: §5.2 · p. 6
5.3 Joint training
- 0.038 to 0.045 of specialists’ open-book score and 0.172 to 0.183 of their cloze recall separated joint training above the merges for collections of 16 documents or more.No pooled CI lower end fell below +0.023 on open book or +0.114 on cloze; at cross-domain N = 32, joint training was not distinguishable from specialist parity.Source: §5.3 · pp. 6–7
5.4 Merges on cloze
- At 16 documents or more, merges recover 12 % to 16 % of specialists’ cloze advantage over the base model, versus 87 % to 89 % of their open-book advantage.Each cloze-share CI includes zero; pooled over 56 collections, merges exceed random cloze retention by 0.043 to 0.052, about a third of joint training’s 0.150.Source: §5.4 · p. 7
6 Four silent errors
Four implementation and evaluation errors produced clean results that the authors were prepared to report. They affected training length, decoding, merge scaling, and evaluation precision.
A training-length default
- A 1024-token training default truncated answers in 44 % of documents, reversing the open-book comparison between question-only and full-text training relative to a 2048-token window.With the shorter window, question-only adapters scored 0.168 above full-text adapters; with 2048 tokens, they scored 0.049 below them.Source: §6 · p. 7
A decoding setting in our own code
- A cached repetition penalty of 1.1 mixed penalized and unpenalized evaluations, making scores appear irreproducible and placing update merging’s best cell above parity.Under one configuration, all 150 scores repeated exactly.Source: §6 · p. 7
A scaling error in our own merge
- Scaling both LERP factors by 1/N made each adapter’s own update 1/N^2 of its trained magnitude, creating an apparent decline from 0.82 retention at four documents to 0.62 at 32.The corrected √ LERP showed scale accounted for the decline; the error was in the authors’ implementation.Source: §6 · p. 7
A mismatch in evaluation precision
- Evaluating adapters trained on a 4-bit base after merging them into a bf16 base lowered specialists’ cloze gain by 0.045 and changed significance for eight of 22 composition contrasts.Uniform evaluation footing also made the synthetic condition’s cloze gain no longer clear of zero.Source: §6 · p. 7
7 Related work
The related work situates document-level adapters within parametric retrieval, adapter selection, and model merging. The paper distinguishes evidence from open-book scoring from evidence on a probe that withholds the document.
Document adapters and knowledge in weights
- Parametric RAG sums updates from retrieved passage adapters, but re-scored by answer presence it does not consistently beat retrieval in context; the authors’ same-population control supports document-specific gains only on the source-withheld probe.The prior re-implementation and control are discussed alongside the distinction between open-book and cloze evidence.Source: §7 · pp. 7–8
- Back et al. (2026) identify routing as a bottleneck, while Rakotonirina et al. (2026) report degrading cloze results as more documents are retrieved; both connect to this paper’s adapter-selection and merge findings.The paper also relates its reading-behaviour result to Zhou et al. (2023) on instruction tuning and Allen-Zhu and Li (2024) on how training presentation affects extraction.Source: §7 · p. 8
Merging and its scale
- Factor-wise LoRA merging changes both the scale of each adapter’s own update and introduces cross terms, unlike methods acting on the full update.The cross terms are noted by Prabhakar et al. (2025) and isolated by Zheng et al. (2025).Source: §7 · p. 8
Adapter selection
- LoraHub and AdapterFusion presuppose that adapter choice matters; here, cloze supports that premise more than open-book scoring does.Source: §7 · p. 8
8 Conclusion
The conclusion argues that open-book scoring mostly credits a reading behaviour shared across adapters, while cloze better tests whether document content survives substitution or merging. It recommends evaluation controls and notes joint training’s retraining trade-off for known collections.
8 Conclusion
- On Llama-3.1-8B-Instruct, a random adapter recovers about four fifths of specialists’ open-book advantage but only 2 % of their cloze advantage, a share not distinguishable from zero.The conclusion says update geometry records what adapters read, but in the bank it did not select adapters that help on another document’s content on either probe.Source: §8 · p. 8
- The authors recommend pairing open-book scores with source-withheld probes, averaging random-adapter baselines over several draws, and using uniform precision and decoding settings.Source: §8 · p. 8
- For collections of up to 32 documents known in advance, joint training keeps what merges lose but requires retraining when the collection changes.Source: §8 · p. 9
Limitations
The study’s findings are bounded by a single main model, adapter rank, training schedule, and 150 Wikipedia articles; retrieval and composition tests also leave important settings unresolved.
Limitations
- Most experiments use one base model, one adapter rank and schedule, and 150 encyclopedic articles, so the measured shares may change with model size, capacity, or genre.The geometry analysis was also replicated on other models, but the section 4 shares were not.Source: Limitations · p. 9
- Composition was tested only on fixed collections of at most 32 documents, leaving larger-scale joint-training parity and incremental composition versus retraining untested.At these sizes, joint training was not yet distinguishable from parity with specialists.Source: Limitations · p. 9
- The retrieval test has only 30 queries, and its confidence interval still permits a weight-space selection advantage around four fifths of the query’s own-adapter margin over random.The specialist itself was not distinguishable from the base model on cloze in this test.Source: Limitations · p. 9
- The generated items were validated only by string match, so alternate valid answers may go unrecognized; the study did not measure this error.Cloze and open-book probes also score different items, and the base model may already know some Wikipedia facts.Source: Limitations · p. 9
- All adapters shared one instruction format, while the base model’s open-book score used the raw template, so adapter gains may partly reflect format adaptation.The retrieval bank also excludes each query’s own adapter, limiting that result to substitution rather than banks containing every document adapter.Source: Limitations · p. 9
A.1 Training and software
- The study estimates that all experiments, including superseded and preliminary runs, used a few hundred GPU hours, with 20,971,520 trainable parameters per reported Llama-3.1-8B adapter.Most experiments used Llama-3.1-8B-Instruct; the geometry was additionally replicated on Llama-3.2-3B and two Qwen models.Source: §A.1 · pp. 11–12
A.2 Held-out items and the metric
The section distinguishes open-book scoring from factual recall without the source text, then reports how context swapping and update geometry relate to those evaluations. It also describes the generated probes and document variants used in those analyses.
A.2 Held-out items and the metric
- The true/false item behaves differently from factual questions: open-book accuracy is 0.987 for the base model versus 0.933 for natural adapters, so cloze scores factual items alone.Open-book results include all four held-out items; closed-book true/false accuracy is below the base model for every training condition.Source: §A.2 · p. 12
A.3 Cloze harness
- The cloze harness adds the model’s chat template, and its 35-document comparison cannot exclude a change in specialist cloze gain as large as the gain itself.Eight of 900 specialist and base-model generations hit the 64-token limit, all from specialists; the maximum possible recall reduction is 8/450 = 0.018.Source: §A.3 · p. 12
B Prompts; B.1 Instruction template; B.2 Generation of question-answering pairs
- The fixed instruction template was used across documents, while generated QA pairs and cloze variants supplied source-based questions and declarative one-blank probes.Questions and document variants were generated with Claude Sonnet 4.6; generations were validated against source text by string match.Source: §B.1 · p. 12, §B.2 · pp. 12–13
B.3 Document variants
- Document variants included neutral style-normalized rewrites and synthetic templates retaining six extracted facts without adding information.The rewrite prompt instructed preservation of factual content and approximately the original length.Source: §B.3 · p. 14
C.1 Training-text conditions
- Training-text conditions held each document’s QA pairs fixed while varying the accompanying text, including source text, rewrites, no context, and swapped context.Open-book evaluations could therefore present different texts across conditions.Source: §C.1 · p. 15
C.3 Update geometry
- Update geometry compares normalized weight updates and their subspaces, distinguishing whether adapters use shared directions from whether their updates point alike.The overlap reference across layer-module combinations is 0.051, and the measured overlap is 0.155.Source: §C.3 · p. 15
C.2 The context swap on its donor documents
- On the 99 donor documents, swapped adapters reached 84 % of the natural adapter’s open-book score on the text they read, while cloze gain over base was +0.051.The natural adapter’s cloze lead was 0.039, but its confidence interval included zero once the true/false item was also scored.Source: §C.2 · p. 15, Table 4 · p. 16
C.3 Update geometry
- Across adapter pairs, mean update cosine was 0.0066; subspace overlap correlated with document semantic similarity at r = 0.122, accounting for roughly 1.5 % of variance.The cosine mean exceeds the stated 0.0018 reference, while the semantic-similarity association is small despite p < 0.001.Source: §C.3 · p. 16
C.4 Geometry across architectures
The geometry analysis finds domain structure in adapter updates, while the random-adapter baseline varies substantially across individual draws. Selected-layer clustering improves domain-label recovery, but its measured purity is sensitive to selection.
C.4 Geometry across architectures
- Held-out layer selection reduces neighbourhood purity from 0.746 in-sample to 0.616, while the selected layers’ correlation optimism is 0.022.The analysis evaluates layers on held-out adapter splits after selecting them on the other half.Source: §C.4 · p. 16
- Clustering the 150 adapters recovers domain labels at NMI 0.511 across all layers and 0.587 over selected layers, with structured splits in economics and history.Economics separates macroeconomic topics from game theory and welfare; history separates political and military events from social and economic history.Source: §C.4 · p. 17
- The training-window correction changes full-aggregation correlations by at most 0.019 and late-layer correlations by at most 0.021, leaving the geometry result largely unchanged.The same late layers carry the correlations after correction.Source: §C.4 · p. 17
D.1 The random-adapter baseline
- For collections of at least 16 documents, random-adapter open-book shares are 0.79 across domains and 0.87 within one, but individual draws range from 0.57 to 1.19 of specialist score.The random baseline averages five independent out-of-collection adapter draws per collection.Source: §D.1 · p. 17
D.2 Retrieval by weight-space proximity
Weight-space proximity and text embeddings do not reliably select adapters that outperform random choices on the reported retrieval probes. For merging, factor scaling and cross terms matter, while correctly scaled rank-8 merges retain open-book performance better than cloze advantage.
D.2 Retrieval by weight-space proximity
- At top-1, weight-space proximity and text embedding are not distinguishable from random selection on cloze; the reported contrasts include zero.The bank excludes each query’s own adapter; the passages also report no distinguishable selection advantage on either probe.Source: §D.2 · p. 17, Table 7 · p. 20
- Across the factual-item splits, adapter selection gains 0.018 over random on cloze and is best for 0.18 of documents.With only one factual item and the true/false item for selection, it is best on the scoring part for none of the documents.Source: §D.2 · p. 19
E.1 Merge operators and scale
- For N ≥4, update merging retains 0.93 to 0.99 of specialists’ open-book score, while keeping little of their cloze advantage.The wanted update’s share relative to the other updates falls from 1.00 at N = 2 to 0.20 at N = 25.Source: §E.1 · p. 19
- At N = 8, erroneous factor scaling makes each adapter’s own update 1/64 of its trained magnitude rather than 1/8.Factor averaging scales both sums by a further N relative to √LERP; factor merges and update merges also differ in cross terms.Source: §E.1 · p. 19
E.2 Merges at matched rank
- Rank-8 projection loses 0.002 to 0.034 of retention against the full-rank update, despite retaining only 0.38 of squared singular values at N = 32.The retained spectral share is 0.72 at N = 4; six of nine projection-loss confidence intervals exclude zero.Source: §E.2 · p. 20, Table 8 · p. 21
- At matched rank, TIES exceeds update merging in eight of nine cells, with a sign-test p = 0.039.Three paired-contrast confidence intervals exclude zero: two favor TIES and one favors update merging.Source: §E.2 · p. 20, Table 10 · p. 22
E.3 Joint training and the full grid
Joint training’s open-book retention is broadly flat at larger collection sizes, while cloze results show a weaker merge pattern and stronger joint-training results. Pairwise interference is not anticipated by semantic or weight-space distance, and earlier prompt truncation varied substantially by domain.
E.3 Joint training and the full grid
- At N = 16, scaling joint-training budget to 8000 steps does not significantly change retention within or across domains.The matched-minus-scaled contrasts are +0.017 within domains and +0.006 across them, and neither confidence interval excludes zero.Source: §E.3 · p. 22
E.4 Cloze composition grid
- At N ≥16, merge cloze contrasts with random range from −0.037 to +0.063, while joint training leads random by 0.164 to 0.237.Four of five joint-training confidence intervals exclude zero; for merges, only some √LERP and one update-merging intervals do.Source: §E.4 · pp. 22–23, Table 12 · p. 24
E.5 Pairwise interference
- Across 200 TIES pairs, semantic and weight-space distances do not anticipate accuracy changes; within-domain pairs lose more than cross-domain pairs, −0.026 versus −0.005.The distance correlations stay within |r| ≤0.016, and weight-space distance versus change has Spearman ρ = 0.000.Source: §E.5 · pp. 23–24
F.1 Truncation by domain
- Under the 1024-token training default, the share of documents with no training answer token retained ranges from 4 % in law to 80 % in biology.At the 2048-token evaluation window used throughout, no prompt is cut; some earlier 512-token evaluations cut documents to about 330 tokens.Source: §F.1 · p. 24
F.2 The merge scaling error and prior work on scale
A factor-averaging scaling error reduced each adapter’s own contribution, and evaluation footing shifted cloze composition results. Repeated runs matched exactly on a defined set of 4-bit evaluations, while related work differs in probe, routing, and merge setups.
F.2 The merge scaling error and prior work on scale
- At N = 32, the scaling error made each adapter’s own update 1/1024 of its trained magnitude rather than 1/32.The same arithmetic affected pairwise LERP and earlier top-3 retrieval merges, which the authors do not report.Source: §F.2 · p. 24
F.3 Evaluation footing
- On the uniform 4-bit footing, specialist cloze scores move by −0.039, while the base model moves by +0.006.The 0.045 reduction in specialists’ cloze gain over the base model lies almost entirely on the specialists’ side; archived bf16 evaluations also did not repeat exactly.Source: §F.3 · p. 27
F.4 What repeats exactly
- Across five runs and four runner versions, the 4-bit cloze runs agree string for string on all 1393 compared generations.The passage limits this reproducibility standard to these runs and notes that two archived bf16 runs agreed on only 883 of 1050 generations.Source: §F.4 · p. 27
G Further related work
- Back et al. (2026) route among document adapters and merge top-ranked modules, whereas this paper tests adapter-to-adapter selection and open-book scoring.Their TIES performance falls as the number merged grows from one to five, consistent with this paper’s cloze grid despite differences in probe, merge operator, and metric.Source: G Further related work · p. 27
- Tang et al. (2025b) report token F1 rewarding short template-like answers; Rakotonirina et al. (2026) find several merge methods within about one point at five documents.The cited setups differ from this paper’s evaluations and do not establish a shared merge ranking.Source: G Further related work · p. 27