SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking

1 Introduction

• SHELF varies subject, genre, audience, register, geography, and other facets independently, while recording writing model, prompt style, and sampling settings.The controlled design addresses the correlation of genre and subject in natural collections and the influence of cataloguer, time, and language on real records.Source: §1 · p. 2
• SHELF can generate documents after future model training cutoffs, but synthetic benchmark scores do not establish performance on production catalogue data.The released documents postdate evaluated embedding checkpoints; this out-of-sample claim does not cover later decoder models or remove source bias.Source: §1 · p. 2
• The evaluation covers classification, clustering, retrieval, pair classification, and instruction retrieval across six bibliographic facets, using only methods compatible with each task.Compared method families include sparse methods, encoders, and a subject-classification-only decoder comparison.Source: §1 · pp. 1–2
• Subject classification reaches 0.8887, while classification across 133 genre forms reaches 0.2605; subject retrieval reaches 0.7104, but genre-form retrieval reaches 0.1173.Register and audience pairs remain near chance; sparse methods compete with dense encoders in classification but trail them in retrieval.Source: §1 · p. 2
• SHELF’s scores support method comparison and diagnosis, not estimates of production accuracy; external checks find rankings transfer more reliably than absolute scores.The checks compare subject tasks with Project Gutenberg passages and LCSHBench catalogue records.Source: §1 · p. 2
• SHELF turns a labelled taxonomy, writing specifications, and a generation budget into bibliographic benchmark data and evaluation tasks.The release includes 62,899 documents from 25 models, with 20,367 specification-backed documents and an 18,345-document generator-balanced factorial component.Source: §1 · p. 2

2.1 Automated subject cataloguing

• LCSHBench evaluates heading assignment on real catalogue records, while SHELF measures representation separation under controlled conditions; cataloguers share concept-level headings more often than identical sets.The reported concordance is 93.3% for sharing a concept-level heading and 39.4% for assigning identical sets.Source: §2.1 · pp. 2–3

2.2 Embedding benchmarks

• MTEB and related embedding benchmarks cover many tasks but omit bibliographic work, while model comparisons also show task-level strengths and cost can differ despite similar aggregate scores.The cited Embedder’s Dilemma compares 26 embedding models and ten language models; SHELF’s encoder set does not test that frontier.Source: §2.2 · p. 3

2.3 Domain-specific benchmarks

• Domain-specific benchmarks address gaps left by general leaderboards, but benchmark validity also depends on criteria such as reliable annotation, which SHELF does not yet meet.FinMTEB reports that general rank does not reliably predict finance-task rank; CaseHOLD builds legal tasks for an intended gap.Source: §2.3 · p. 3

2.4 Model-generated evaluation data

• Model-written evaluation data can be easier than human-written originals, and source bias may affect SHELF’s all-machine-written corpus rankings by an unknown amount.SHELF’s subject transfer test addresses part of the concern that model-generated benchmarks require direct testing; real-document benchmarks have also been reported to reproduce human-curated rankings.Source: §2.4 · p. 3

2.5 Contamination

• Post-cutoff generation protects against direct overlap but does not remove source bias; SHELF’s unlikely facet combinations instead test whether methods separate attributes that usually co-occur.The cited contamination work reports accuracy drops on a fresh matched test set and rewritten samples that evade n-gram overlap checks.Source: §2.5 · p. 3

2.6 Instruction-conditioned retrieval

• Instruction-retrieval benchmarks report failures to use constraints, while SHELF combines controlled facet separation, compatible sparse and encoder comparisons, and a subject-only decoder comparison with natural-corpus checks.FollowIR reports failures to use detailed instructions; NevIR finds bi-encoder and sparse systems at or below random performance.Source: §2.6 · pp. 3–4

3.1 Corpus composition and splits

• The 62,899-document pooled corpus combines records with different provenance and split protocols; specification-level leakage protection applies only to its later component.The generator-balanced v0_4_core component contains 18,345 later documents; shared specifications are kept together in grouped splits, avoiding near-duplicate versions across splits.Source: §3.1 · p. 4

3.2 Independence by construction

• Independent facet assignment creates uncommon subject–genre combinations; in the generator-balanced slice, writing-model association with subject and genre labels is negligible.Cramér’s V is 0.016 for subject class and 0.027 for genre category; wider v0.4 generation has subject-class association of 0.037.Source: §3.2 · p. 4

3.3 Scale and generation models

• The aggregate corpus is not generator-balanced, while a separate 3,016-passage human-written, human-catalogued Gutenberg slice tests whether SHELF rankings carry over to natural text.The largest writing model supplies 47.7% of the aggregate corpus; Gutenberg passages are kept outside the main corpus to preserve the transfer test distinction.Source: §3.3 · p. 4

3.4 Quality control

• Quality control rejects malformed, off-topic, self-labelled, or duplicate documents, but one model returned reasoning tokens without documents for 14.6% of calls, mostly on short-text requests.The output-token allowance was increased and affected records removed; the revised label check flagged 2.48% of the earlier corpus and less than 0.5% of the newer corpus.Source: §3.4 · p. 5

3.5 Evaluation protocol

• The protocol varies panel size by task and evaluates classification, retrieval, clustering, and pair tasks with macro-F1, nDCG@10, adjusted Rand index, and AUC, respectively.Classification and clustering use 25 configurations; retrieval and instruction retrieval use 25 retrieval-compatible methods, while released pair configurations come from an earlier 21-model sweep.Source: §3.5 · p. 5

4 Benchmark results

• The comparison evaluates lexical methods and encoders from 22M to 596M parameters, reporting macro-F1 for classification and adjusted Rand index for clustering.An adjusted Rand index near zero indicates clusters no better than a random partition.Source: §4 · p. 5

4.1 Evaluated methods

• The encoder panel ranges from early sentence encoders to models trained more directly for embedding and retrieval, but excludes frontier-scale models.Inputs omit model-card-recommended query or document prefixes, which may understate prefix-dependent models and affect their ranking.Source: §4.1 · pp. 5–6

4.2 Classification

• TF-IDF leads register classification, ranks third on subject, and sits mid-table on genre form, while models at 335M parameters and above rank as low as fifteenth on subject.Three of those larger models rank behind a sparse baseline.Source: §4.2 · p. 6

4.3 Clustering

• 0.5152 is the subject-clustering score, the only useful leaderboard among the four tasks; geography, register, and genre-form clustering remain weak.The best scores for geography, register, and genre form are 0.0068, 0.0678, and 0.0921, respectively.Source: §4.3 · p. 6

4.4.1 Retrieval

• 0.1173 is the best genre-form retrieval score, versus 0.7104 for subject retrieval, making genre form difficult in both classification and retrieval.RoBERTa leads category retrieval despite ranking last on subject and genre-form classification; this is treated as task-specific, not evidence of generally stronger representations [21].Source: §4.4.1 · p. 6
• BM25 ranks 23rd to 25th on the three retrieval tasks, although TF-IDF leads register classification and ranks third on subject.Sparse-method strength is task-specific.Source: §4.4.1 · p. 6

4.4.2 Pair classification

• Pair performance is measured by AUC, which orders positive above negative pairs by cosine similarity without choosing a test-set threshold.BM25 pair scores average its score for each document queried against the other.Source: §4.4.2 · p. 6
• Models distinguish subject, topic, and topic-overlap pairs at median AUC 0.69–0.71, genre-form pairs at 0.62, while register and audience remain near chance.Each pair task is balanced, with 2,000 positive and 2,000 negative pairs.Source: §4.4.2 · p. 6
• The reported 0.8474 same-subject pair score uses released pairs from the original corpus component, not the pooled corpus.The two corpora yield different pair sets and scores; the released configurations are used because readers can download them.Source: §4.4.2 · p. 6

4.4.3 Instruction retrieval

• Subject-anchored retrieval instructions are followed to a useful degree, whereas form- or audience-anchored instructions are not.The task does not isolate instruction following from the strength of the underlying label signal.Source: §4.4.3 · p. 6

4.5.1 Differences between tasks

• 0.2605 is the best genre-form score, compared with 0.8887 for subject; mean performance falls from 0.8226 to 0.1698.Every model loses between 0.60 and 0.70 points.Source: §4.5.1 · p. 7
• All 133 forms appear in every split, the rarest test form has 25 documents, and the same documents carry both labels.The task difficulty difference is therefore not an artifact of unseen classes.Source: §4.5.1 · p. 7

4.5.2 Agreement in model rankings

• Across 24 models, subject and genre-form rankings correlate at 0.815, with a bootstrap interval of [0.58, 0.94].A practitioner choosing between these models gets roughly the same ordering from either task.Source: §4.5.2 · p. 7

4.5.3 Interpreting the genre-form task

• Genre form remains useful because scores stay low, not because it produces a new model order.The top twelve subject scores span only 0.046, while every genre-form score remains low; its best score is 0.2605.Source: §4.5.3 · p. 8
• The early nine-model ranking correlation of 0.450 came from a small, cheap-model-skewed group and did not support a claim.No model in that partial run exceeded 109M parameters; the full-model correlation later rose to 0.815.Source: §4.5.3 · p. 8

4.6 Zero-shot decoder comparison

• The decoder results are a separate zero-shot reference, not part of the main 25-configuration panel or its rank analysis, and they lack the embedding panel’s temporal out-of-sample guarantee.The decoder models postdate the original public release and may have encountered it during training.Source: §4.6 · p. 8

4.6.1 Classification results

• All four decoders beat the 0.048 expected macro-F1 of a uniform guess across 21 classes, but this does not show whether they encountered the labels during pretraining.The models predict all 21 classes and receive no labelled SHELF examples.Source: §4.6.1 · p. 8
• 0.8887 for the best encoder versus 0.5860 for the best decoder, but the encoder used 37,795 labelled documents and the zero-shot decoders used none.The comparison is a reference for the tested models and prompts, not a general zero-shot floor.Source: §4.6.1 · p. 8

4.6.2 Prompt and generator effects

• Prompt selection matters: chat templates raised mean macro-F1 two to three times over raw completion, and prompts were selected on validation data.Across tested settings, one model’s macro-F1 changed by more than 0.4, and different prompts worked best for different models.Source: §4.6.2 · p. 8
• Own-family accuracy differences were inconsistent across the four judges, while their rank correlations across twelve generator groups ranged from 0.622 to 0.902.The results describe associations and do not identify their cause.Source: §4.6.2 · p. 8

4.6.3 Measured inference rates

• TF-IDF with a logistic head processed 2,271 documents per second on CPU at 0.8686 macro-F1, nineteen times the best encoder’s measured rate and 0.0201 macro-F1 lower.These measurements use one run on this harness and machine; they are not each model’s best possible throughput.Source: §4.6.3 · p. 9
• EmbeddingGemma-300M gained 0.0872 macro-F1 over MiniLM-L6 at about one sixth its measured processing rate, while parameter count alone did not determine rate.The two rate measurements—20.3 documents per second for Gemma-4-E2B and 11.5 for Qwen3.5-0.8B—also reflect architecture, implementation, and batching.Source: §4.6.3 · p. 9
• At frozen input limits, Qwen3.5-0.8B processed 9.6 documents per second at 2,048 tokens versus 11.5 at 512, and Qwen3.5-2B processed 7.2 whole documents per second versus 11.2 at 512.The 512-token rates do not describe the exact configurations used for the table’s macro-F1 values.Source: §4.6.3 · p. 9

5.1.1 Measuring label terms

• SHELF documents named their own subjects about three times as often as matched-length real documents, which is consistent with higher within-corpus scores but does not quantify how much it causes.The real-document rate was about 6%.Source: §5.1.1 · p. 9

5.1.2 Removing the label terms

• Removing label terms lowered macro-F1 by about 3.4 points on SHELF and 1.2 on Gutenberg, while sham token removal moved neither score materially.Retrieval showed the same pattern on SHELF: −0.0396 masked versus +0.0008 sham.Source: §5.1.2 · p. 10
• Label-term availability raised scores but barely changed model order: masked-versus-unmasked rank correlations were 0.987 on SHELF, 0.988 on Gutenberg, and 0.983 on LCSHBench.The cross-corpus rankings also remained similar after masking.Source: §5.1.2 · p. 10

5.1.3 Changes between generations

• Genre-form repetition fell by 81% between generations, while topic repetition fell less and remained common; the difference tracks whether labels reach the writing model as descriptions or exact strings.The generation comparison cannot assign the change to one pipeline change.Source: §5.1.3 · p. 10, §5.1.4 · p. 10

5.1.4 Topic leakage and mitigation

• Instructions to develop a subject without topic-label words reduced leakage from 82.2% to 4.4% for one model and 61.5% to 23.1% for another.Plain instructions reduced judged topic coverage by 11.7 points; a trustworthy gloss reduced it by 6.5 points, but its interval included zero, so the released generation kept the original prompt.Source: §5.1.4 · pp. 10–11

5.2 Weak and failed tasks

• Subject clustering reached a median adjusted Rand index of 0.3439 and a best score of 0.5152, while genre peaked at 0.0921, register at 0.0678, and geographic clustering at 0.0068.Geographic clustering does not support model comparison; the rewrite was rejected after exact label copying rose from 29.3% to 35.5% and pilot TF-IDF macro-F1 rose from 0.639 to 0.724.Source: §5.2 · p. 11
• The leakage-reducing topic instruction also lowered blind-judged coverage, and the evidence does not support a free improvement in both measures.The gloss-based variant’s coverage-loss interval included zero, but the released generation retained the original prompt.Source: §5.2 · p. 11

6.1 Transfer of absolute scores

• The diagonal measure shows within-corpus classification is much easier for SHELF (0.8796) than for either natural corpus, Gutenberg (0.5101) or LCSHBench (0.5559).Source: §6.1 · p. 12
• Scores fell in every transfer direction: a SHELF-trained probe scored 0.3193 on Gutenberg, while a Gutenberg-trained probe scored 0.2823 on SHELF.Equalizing corpus size and truncating documents to 200 words did not remove the within-corpus difficulty gap.Source: §6.1 · p. 12
• SHELF-trained probes outscored Gutenberg-trained probes on LCSHBench, 0.4078 to 0.2135, and on Gutenberg, 0.3193 to 0.2800, but scores still do not predict production accuracy.SHELF and LCSHBench share Library of Congress vocabulary; reducing SHELF to 3,016 documents lowered its Gutenberg score to 0.2341.Source: §6.1 · p. 12

6.2 Transfer of model rankings

• Subject-model rankings on SHELF were similar to those on both natural corpora, though agreement was lower than the 0.958 correlation between the natural corpora.The intervals overlap and do not establish a difference between the two SHELF-to-natural-corpus correlations.Source: §6.2 · p. 12
• All four subject-task formulations passed the preregistered ranking-agreement rule against both natural corpora, and the clustering row passed its stability check across five seeds.Median pairwise rank correlations were 0.991 on SHELF, 0.932 on Gutenberg, and 0.962 on LCSHBench.Source: §6.2 · p. 12
• All eight capable-only comparisons still passed, but the formulations reuse labels, documents, and representations, so they are not four independent validations.The weakest cell was subject classification against LCSHBench at 0.655, with interval [0.22, 0.93].Source: §6.2 · p. 12

7 Scope and limitations

• SHELF supports comparisons on its released tasks and diagnosis across controlled bibliographic facets, but its scores do not show whether errors matter to cataloguers.No cataloguer-annotated sample is available to provide a human agreement rate or expert ceiling.Source: §7 · p. 12, §7.1 · p. 12

7.1 Reference points and external validity

• The zero-shot decoder comparison covers only subject classification with four models and no labelled decoder examples, unlike the encoders fitted on 37,795 examples.The external comparison also covers subject labels only; it does not validate genre, audience, register, geography, or instruction tasks, and uses one lexical probe.Source: §7.1 · pp. 12–13

7.2 Corpus and split provenance

• The pooled corpus combines original records with document-level splits and later records grouped by specification ID, so the grouped-split guarantee applies only to the later component.The authors cannot test whether versions of one original brief appear on both sides of a split.Source: §7.2 · p. 13
• The pooled corpus is not balanced across writing models: its largest model supplies 47.7% of documents, so model comparisons require the balanced factorial component.The released pair tasks come from the original corpus component.Source: §7.2 · p. 13

7.3 Models, tasks, and implementation

• The encoder panel spans 22M to 596M parameters, omits the largest embedding systems, and a larger panel may change model order.The unprompted input format may favor models needing less prompting; several tasks, including geographic clustering and register and audience pairs, remain at or near chance.Source: §7.3 · p. 13
• Timing results come from one run with different hardware and input-length conditions, so they are observed rates for this harness, not hardware-neutral latency, cost, or energy measures.The measurements have no repeated-trial intervals.Source: §7.3 · p. 13

• Methods classify broad subjects well but struggle with fine genre forms, register, audience, and several clustering tasks; sparse methods remain strong classifiers, while dense encoders lead retrieval.Source: §8 · p. 14
• Controlled variation across subject, genre, audience, register, and writing model aids diagnosis, but synthetic scores are not equivalent to production accuracy.Label terms make SHELF somewhat easier, some tasks remain near chance, and the corpus lacks human catalogue judgments as a reference.Source: §8 · p. 14
• Subject-model rankings are more stable across SHELF, Project Gutenberg, and LCSHBench than absolute scores, so SHELF cannot predict accuracy on a library catalogue.Source: §8 · p. 14
• SHELF can generate new documents after a future model’s training cutoff while preserving labels and task design.Source: §8 · p. 14

Data availability

• The dataset is available on Hugging Face, with evaluation code and stored per-task results on GitHub.Source: Data availability · p. 14

AI assistance

• Large language models assisted with portions of the work, and the author takes responsibility for all content, including errors or omissions.Source: AI assistance · p. 14

Limitations

• The stated interpretation limits include the absence of a human reference and a frontier-scale encoder.Source: Limitations · p. 14