Source-linked AI summary

(Whose defaults?) Is artificial intelligence reorienting archaeological methods?

Lorenzo Cardarelli, Roberto Ragno

arXiv:2609.11198v1cs.CYcs.AIcs.CLcs.HC

TL;DR

Whether LLMs are narrowing archaeological methods remains understudied, in a discipline whose interpretative goals differ from those of software engineering. This paper models method composition in archaeological publications and finds that σβ exceeds σγ by a factor of 2.35, while literature-wide method diversity has not yet declined.

  • Problem

    Whether LLMs narrow archaeological methods remains understudied, in a discipline whose interpretative goals differ from those of software engineering.

    Source: §1 · p. 2
  • Method

    The study models computational-method counts compositionally within archaeological sub-disciplines using a Bayesian hierarchical Dirichlet-Multinomial regression.

    Source: §3.3 · p. 8
  • Results

    σβ exceeds σγ by a factor of 2.35, and the study found no reduction yet in literature-wide method diversity.

    Source: §6 · p. 18, S7.6 Posterior summary · p. 58
  • Takeaways & Limitations

    The key issue is integrating LLMs into existing scientific workflows, as code delegation reconfigures epistemic practice rather than merely substituting tools.

    Source: §6 · p. 18

    The corpus’s 546 qualifying journals reflect the ASJC 1204 filter and retrieval-date Source List, which is revised periodically.

    Source: S3.4 Regenerating the source list · p. 26

Abstract

from arXiv · show

Generative AI and the practice of "vibe coding" are changing how archaeologists carry out computational research, but their effects on the discipline's range of methods is still understudied. In this paper, we evaluate whether large language models (LLMs) are narrowing the variety of methods archaeologists use. We first analysed approximately 119,000 archaeology abstracts from Scopus, covering publications from 2010 to 2025. Using a locally run LLM, we identified the computational methods reported in each abstract and organised them into 25 broad categories (L2) and 241 finer clusters (L3). A Bayesian Dirichlet-multinomial model of method composition within sub-disciplines found a small but credible shift in method use after 2023. However, this shift was smaller than the variation already present across the full study period. No individual technique showed a significant change, and overall methodological diversity increased rather than declined. We then ran a controlled experiment to see whether LLMs recommend a narrower set of methods than archaeologists have used in practice. Two different open-weight models were asked to suggest methods for 28 archaeological research problems, with prompts providing three levels of methodological guidance: novice, intermediate, and expert. Recommendation diversity was much lower than in the published literature, particularly without methodological guidance. The models also tended to favour methods that were widely used before 2023, and their recommendations more closely resembled the post-2023 literature. Taken together, these results are consistent with LLMs pushing methodological choice towards convergence, although our study cannot establish a causal effect. They raise a broader question: how can archaeology retain methodological diversity as LLMs become more involved in research?

1 Introduction · 2 Aims · 3 Materials and Methods

The paper examines whether LLM defaults narrow archaeological computational methods, using corpus trends and model recommendations while developing a reflective framework for engaging with these tools. It describes the taxonomy, analyses of published methods and recommendations, and tests of factors associated with recommendations.

1 Introduction

  • Because code mediates how archaeological evidence is filtered, classified, and interpreted, delegating its authorship to an LLM has epistemological implications.The authors support using these tools in programming workflows under firm conditions, while examining their risks rather than rejecting them.Source: §1 · pp. 2–3

1.1 Background and Theoretical Framework

  • Vibe coding can bypass researchers’ formalisation of tacit knowledge, while LLM defaults may favour common methods and risk homogenising archaeological practice.The authors also identify risks to researchers’ ability to evaluate code and to equitable access to the tools.Source: §1.1 · p. 4

2 Aims

  • The study tests the mean-collapse hypothesis through Scopus literature trends and local LLM recommendations, then reflects on the epistemological implications of vibe coding.Its framework aims neither to reject these tools nor to accept them uncritically.Source: §2 · p. 5

3.1 Dataset and Bibliometric Stream

  • The Scopus corpus covers 2010–2025 and comprises 119,327 cleaned journal articles; its pre-2023 records provide a baseline for comparison with 2023–2025.The dataset was retrieved programmatically from journals assigned Scopus’s Archaeology classification; licensing restrictions prevent public release of the raw metadata.Source: §3.1 · p. 5

3.2 Method Classification and Taxonomy Creation

  • A local Qwen3.5-9B pipeline extracted explicitly stated methods, identifying 8,404 papers with at least one and organizing methods into 25 L2 categories and 241 retained L3 clusters.The extraction could enumerate multiple heterogeneous methods per abstract, unlike single-dominant-topic approaches.Source: §3.2 · pp. 5–7
  • The taxonomy pipeline produced 241 L3 method categories, embedding and clustering terms before validation and conservative L3-to-L2 assignment review.Qwen models labelled clusters and assigned them to 25 L2 categories; validation removed one non-method cluster, affecting six of 8,404 articles, and reassignment required unanimous agreement across three review calls.Source: §3.2 · p. 8

3.3 Analysis of the Bibliometric Stream

  • A hierarchical Dirichlet-Multinomial regression modelled technique shares within sub-disciplines, separating baseline shares and pre-existing trends from additional post-2023 shifts.The compositional model tests whether techniques gained or lost share relative to others, rather than treating technique counts as independent observations; hierarchical pooling regularises estimates toward sub-discipline means.Source: §3.3 · pp. 8, 10

3.4 LLM Experiment Stream

  • The controlled experiment elicited method recommendations for 28 neutral archaeological questions under novice, intermediate, and expert prompting, using two local LLMs.Across 252 question–descriptor iterations, all three profiles received the same archaeological content; outputs were mapped to the L3 taxonomy, with unanimous mappings distinguished from majority-vote cases flagged for review.Source: §3.4 · pp. 10–11

3.4.1 Concentration Analysis

  • Recommendation concentration was compared with pre- and post-2023 literature using the inverse Simpson index and Bayesian posterior distributions over method shares.The analysis used a uniform prior and 4,000 posterior samples per condition; a footnote reports that bootstrapping widened credible intervals by up to 2.9 times in the most clustered condition, without closing the reported LLM–literature gap.Source: §3.4.1 · p. 11, §3.4.2 · p. 11

3.4.2 What Drives LLM Method Recommendations?

  • A Bayesian negative binomial regression tested whether pre-2023 method prevalence and post-2023 share gain predicted LLM recommendation counts across models and profiles.The authors fitted eight model/profile combinations, including novice, intermediate, expert, and pooled profiles, to assess whether prevalence-related recommendations varied with methodological guidance and across models.Source: §3.4.2 · p. 12

4 Results · 5 Discussion

The literature became more methodologically diverse after 2023, while LLM recommendations covered a narrower range of methods and tracked pre-2023 prevalence. The discussion argues that diversification and homogenization may coexist, with their effects shaped by researchers’ expertise, access, and use of AI within the research workflow.

4 Results

  • The effective number of literature methods rose from 87.6 to 111.2 after 2023, although the aggregate shift was modest against a baseline variation 2.35 times greater.None of the 241 individual methods showed credible change at the 90% threshold.Source: §4 · p. 12, Figure 3 · p. 13
  • LLMs recommended 31.6 effective methods (Qwen3) and 28.8 (Gemma), versus 87.6 before and 111.2 after 2023 in the literature (Table 3).Generic novice prompts yielded around 21 methods for each model, while expert prompts yielded 60.7 for Qwen3 and 46.8 for Gemma; neither matched the breadth of published articles.Source: §4 · pp. 12, 14, Table 3 · p. 14
  • LLM recommendations tracked methods’ pre-2023 prevalence, most strongly for novice prompts, while post-2023 momentum had no detectable association with recommendations.The pre-2023 association was weakest for expert prompts, where more comprehensive problem formulations left less scope for default methods.Source: Figure 6 · p. 15, §4 · p. 15

5 Discussion

  • The experiments indicate a potential homogenizing mechanism, but published abstracts show no contraction; the analyses capture different stages, and current adopters may be expert users.Lower technical barriers could also enable broader methods, but the study cannot determine whether this explains the post-2023 diversity increase or establish a causal effect of AI-assisted coding.Source: §5 · pp. 15–16
  • As generative AI adoption grows, a “Slow AI” approach faces practical constraints, while proprietary tools raise privacy and governance concerns and local alternatives require substantial resources.The discussion also raises questions about the environmental costs and sustainability of AI infrastructure.Source: §5 · pp. 16–17
  • Novice users are most exposed to model defaults, connecting methodological homogenization with unequal access to the tools and expertise needed to navigate those defaults.The discussion notes that technical barriers are reduced, but cost differences can disadvantage students and researchers with fewer resources.Source: §5 · pp. 16–17
  • Vibe coding may broaden who can build analytical pipelines, but it also risks de-skilling and unaudited analyses when users cannot evaluate or troubleshoot generated code.Whether this expansion is beneficial depends partly on how archaeological training and peer review adapt.Source: §5 · p. 17
  • The authors argue that LLMs should support, not replace, researchers’ inferential workflow: vague prompts can turn hypothesis-driven analysis into open-ended exploration.They frame LLMs as infrastructural tools for executing analytical choices, while researchers retain responsibility for hypotheses, data collection, and modelling decisions.Source: §5 · pp. 17–18

6 Conclusions and Prospects … S1: Computational environment

The study finds no decline in methodological diversity after LLMs’ introduction, but identifies conditions under which methods could converge. It argues that LLM use reshapes analytical responsibility and requires attention to transparency and reproducibility.

6 Conclusions and Prospects

  • The introduction of ChatGPT and other LLMs has not yet reduced method diversity in archaeological literature, though unclear user engagement could enable measurable convergence.The study’s concentration analysis suggests the conditions for such a shift are already in place.Source: §6 · p. 18
  • LLM-assisted code writing reconfigures epistemic practice by blurring the boundary between researchers’ technical skill and theoretical reasoning.The paper frames the key issue as integrating LLMs into scientific workflows and dividing analytical responsibility between researchers and machines.Source: §6 · p. 18
  • Auditing AI-assisted analysis requires documenting prompts, model versions, and interaction histories, while stochasticity still prevents full reproducibility.Structured prompting, including ‘vibe engineering’ or spec-driven development (Piskala, 2026), may improve transparency; AI literacy and locally hosted models are also presented as ways to encourage critical use and control data flows.Source: §6 · p. 18

Declarations

  • The authors report joint contributions to the study, no specific funding, and no competing interests.Source: Declarations · p. 18

Data and Code Availability

  • Raw Scopus metadata and corpus identifiers cannot be redistributed, but released downstream artefacts support reproducing the analyses in §4 without Scopus access.The release includes code, the taxonomy, experiment stimuli, and both models’ recommendations and mappings; prompts and a verification script are also provided.Source: Data and Code Availability · pp. 18–19

Supplementary Materials

  • The supplementary materials cover the machine-learning pipeline and experiment design, while the Bayesian modelling stream and its sensitivity checks are documented separately in S7–S12.Source: Supplementary Materials · p. 24

S1: Computational environment

  • All machine-learning inferences ran locally on consumer hardware without commercial APIs, avoiding hosted-model version drift and supporting auditability and re-execution.The Bayesian stream was fitted separately in R 4.6 via cmdstanr.Source: S1: Computational environment · p. 24

S2: Model cards and decoding settings

The supplement consolidates decoding settings for the three models used across the pipeline, combining settings from §§3.2 and 3.4 with parameters not reported there.

Model cards and decoding settings

  • The settings table covers three models across pipeline stages and consolidates settings reported in the main text with additional parameters.The passage identifies §§3.2 and 3.4 as the main-text sources; the table caption specifies its scope as models and decoding settings by pipeline stage.Source: S2: Model cards and decoding settings · p. 24, Table S2.1 · p. 24
  • The experiment-generation entry lists Qwen3.5-9B-Instruct with Q8_0 quantization, a 4,096 context length, temperature 0.3, seed 42, and a 30-token maximum.The entry names build_taxonomy_supervised.py as its source.Source: S2: Model cards and decoding settings · p. 25
  • The adversarial-review entries name Qwen3.5-9B-Instruct and Gemma 4 E4B-it.The passages identify adversarial review as §S4.8 but do not provide further settings in the supplied text.Source: S2: Model cards and decoding settings · p. 25

S3: Scopus retrieval protocol

The supplement specifies how Scopus sources were selected and queried, which metadata were requested, how records were reduced to the analysis corpus, and how the source list can be regenerated.

S3.1 Query construction

  • The ASJC 1204 filter identified 546 Scopus Source IDs, queried through the Search API; sources exceeding its result ceiling were split by publication year from 2010 to 2025.Requests used view=COMPLETE and a page size of 25, were throttled and retried, and progress was checkpointed so retrieval could resume.Source: S3.1 Query construction · p. 25

S3.2 Metadata fields retrieved

  • The API request covered 26 metadata fields, with some fields retained in the raw response but not carried into the flat CSV export.Author affiliations, for example, were retrieved as a nested structure and flattened into separate columns.Source: S3.2 Metadata fields retrieved · p. 25, Table S3.2 · p. 25

S3.3 From raw records to the analysis corpus

S3.4 Regenerating the source list

  • The 546 Scopus Source IDs are not redistributed because of licensing restrictions, but the corpus can be regenerated from Elsevier’s public Scopus Source List using the ASJC 1204 filter.The corpus was retrieved on 29 April 2026, when 546 journals satisfied the filter. Because the Source List is revised periodically, later regeneration may produce a slightly different set.Source: S3.4 Regenerating the source list · p. 26

S4: Method extraction and taxonomy construction

The appendix documents the executed pipeline for extracting named methods from abstracts and building a taxonomy through normalization, clustering, validation, labeling, assignment, and review.

S4.1 Extraction of methods from abstracts

  • One LLM call per each of 119,327 cleaned abstracts extracted named computational methods, preserving the abstract’s specificity and excluding software or languages.The prompt requests all techniques using common names and an output of technique names only, or “None” when no method is present.Source: S4.1 Extraction of methods from… · p. 27

S4.2 Phase 1: Text normalisation

  • Normalization standardized spelling, hyphens, capitalization, and plurals; an over-aggressive plural rule deformed “Principal Component Analysis” but did not alter grouping.The deformed canonical term is an internal key, not a display label.Source: S4.2 Phase 1: Text normalisation · pp. 27–28

S4.3 Phase 2: Fuzzy deduplication

  • Fuzzy deduplication grouped normalized terms at a token_sort_ratio threshold of 88/100, collapsing word-order variants and retaining each class’s most frequent variant as canonical.Source: S4.3 Phase 2: Fuzzy deduplication · p. 28

S4.4 Phase 3: Semantic embedding and L3 clustering

S4.5 Phase 4: Automatic L3 labelling

S4.6 Phase 4.5: Garbage validation (not described in the main text)

  • Among 242 L3 clusters, exactly one was flagged and removed; it touched 145 article–method pairs, but removal cost only 6 articles, reducing the analyzed count from 8,404 to 8,398.The 241 L3 clusters reported throughout the main text are the 242 produced minus the one removed.Source: S4.6 Phase 4.5: Garbage validation… · p. 29

S4.7 Phase 5: L3-to-L2 assignment

  • Each L3 cluster was assigned one of 25 L2 categories according to methodological paradigm rather than archaeological application; all 242 clusters received valid assignments.Source: S4.7 Phase 5: L3-to-L2 assignment · p. 30

S4.8 Phase 6: Adversarial review

S5: The full taxonomy

The full taxonomy organizes archaeological computational and analytical methods into broad L2 categories and finer L3 clusters, distinguishing researcher-defined groupings from data-driven clusters. It spans statistical, chronological, spatial and geophysical, imaging, laboratory, machine-learning, and other computational approaches.

S5: The full taxonomy

  • L1 groups the taxonomy for readers, L2 comprises 25 predefined categories, and L3 comprises 241 automatically labelled clusters discovered from the corpus.Only L2 definitions and representative methods were supplied verbatim in the assignment and review prompts; L1 has no computational role.Source: S5: The full taxonomy · p. 31

S5.1 The 25 L2 categories

  • The 25 L2 categories span statistical, spatial, imaging, computational, and specialist analytical methods, with definitions and representative methods for each.Examples include Bayesian inference, spatial statistics, remote sensing, machine learning, chronological modelling, and archaeometry.Source: S5: The full taxonomy · p. 31, S5.1 The 25 L2 categories · pp. 32–35

S5.2 The 241 L3 clusters

  • The 241 L3 clusters are ordered by mentions; distinct absorbed terms and five frequent member terms are shown to help assess cluster coherence.Counts exclude the garbage cluster; mentions count article-method pairs, whereas article counts count distinct articles, including an article once per L2 category.Source: S5.1 The 25 L2 categories · p. 31, S5.2 The 241 L3 clusters · p. 36, Table S5.2 · p. 36
  • Spatial and geospatial methods include GIS analysis, viewshed analysis, digital terrain modeling, and remote sensing.The listed clusters also include spatial suitability analysis and SAR interferometry.Source: S5.2 The 241 L3 clusters · pp. 38, 40, 41
  • Statistical methods span hypothesis tests, regression, clustering, dimensionality reduction, and multivariate analysis.Examples include chi-square and t-tests, linear and logistic regression, hierarchical clustering, and multidimensional scaling.Source: S5.2 The 241 L3 clusters · pp. 38–40
  • The taxonomy also lists machine-learning and computational approaches, including random forests, convolutional networks, transformers, NLP, and graph analysis.Related clusters include semantic segmentation, object detection, transfer learning, and generative adversarial networks.Source: S5.2 The 241 L3 clusters · pp. 38, 39, 42
  • Laboratory and material-characterization methods include mass spectrometry, chromatography, spectroscopy, tomography, and stable-isotope analysis.The examples range from GC-MS and Raman spectroscopy to X-ray computed tomography and isotope-ratio analysis.Source: S5.2 The 241 L3 clusters · pp. 38–41
  • The L3 clusters include chronological methods such as wiggle matching, ancient DNA analysis, radiocarbon calibration, age-depth modeling, and luminescence dating.Source: S5.2 The 241 L3 clusters · pp. 43–45
  • Statistical and modeling clusters span resampling, optimization, maximum entropy and likelihood, fuzzy clustering, Gaussian processes, and survival analysis.Source: S5.2 The 241 L3 clusters · pp. 43, 44, 46
  • Digital and imaging clusters include image similarity and recognition, denoising, parametric design, virtual reality, and attention mechanisms.Source: S5.2 The 241 L3 clusters · pp. 43, 45
  • Spatial and geophysical clusters cover resistivity and GPR prospecting, remote-sensing indices, point-cloud processing, GPS/GNSS, and interpolation.Source: S5.2 The 241 L3 clusters · pp. 44, 46, 47
  • Laboratory and specialist analytical clusters include thermal analysis, spectroscopy, infrared thermography, dental microwear, pollen quantification, and radiometric dating.Examples include nuclear magnetic resonance spectroscopy, polymerase chain reaction variants, and micromorphological residue analysis.Source: S5.2 The 241 L3 clusters · pp. 48, 50, 51, 52
  • Geospatial and imaging clusters range from orthorectification and histogram enhancement to triangulation, landscape characterization, and handcrafted texture descriptors.The listed examples also include cost-surface modeling, local-relief modeling, and Voronoi and Delaunay methods.Source: S5.2 The 241 L3 clusters · pp. 48, 49, 51, 52
  • Statistical clusters include non-parametric rank correlation, Bayesian multilevel modeling, time-series analysis, probability methods, and outlier detection.The examples also include Poisson point-process modeling, clustering validation metrics, and linear mixed-effect models.Source: S5.2 The 241 L3 clusters · pp. 49, 51, 52
  • Computational and modeling clusters include self-organizing neural networks, word embeddings, encoder-decoder architectures, association-rule mining, and glaciological modeling.The taxonomy also lists ensemble learning and variogram modeling.Source: S5.2 The 241 L3 clusters · pp. 50, 52, 53

S6: LLM experiment: design and quality control

The experiment compared method recommendations across three researcher profiles using 28 lexically neutral archaeological questions, with identical content within each comparison. It also specified procedures for mapping, parsing, and checking model outputs.

S6.1 The 28 research questions

S6.2 The 9 vague methodological descriptors

S6.3 Sampling design

  • Across 252 question–descriptor–category triplets, all three profiles received the same archaeological content, yielding 756 generation calls per model plus mapping calls.Each iteration paired one research question, one L2 category, and one vague descriptor; the profiles differed only in methodological framing.Source: S6.3 Sampling design · p. 54

S6.4 The three profile prompts

  • The three prompts shared system framing and output instructions, varying only the researcher’s stated prior knowledge.The novice had no specific computational background; the intermediate profile had a rough methodological idea, while the expert was given an L2 category.Source: S6.4 The three profile prompts · pp. 54–55

S6.5 The L4→L3 mapping prompt

  • L4 methods were assigned L3 labels by majority vote across three independent runs, with consistency requiring unanimous agreement.Responses were matched through a priority cascade; unresolved responses were flagged inconsistent. Generation outputs were parsed into method–justification pairs, and unparseable responses were retained as UNPARSED rows. Table S6.6a reports mapping consistency by model, but no values are supplied here.Source: S6.5 The L4→L3 mapping prompt · p. 55, Table S6.6a · p. 56

UNPARSED

The recommendations largely used methods already represented in the L3 taxonomy, with coverage counts depending on whether the garbage cluster is included. Only one mapping attempt fell outside the taxonomy, due to a malformed code rendering.

UNPARSED

  • Supplementary Table S6.6b is scoped to L4 method items by researcher profile.Source: Table S6.6b · p. 56

S6.7 Coverage of the taxonomy by the recommendations

  • Qwen’s 2,904 distinct L4 strings mapped to 194 L3 labels, while Gemma’s 1,746 mapped to 167.The figures count distinct strings after normalising case and whitespace.Source: S6.7 Coverage of the taxonomy by the… · p. 56
  • Against the 241 analysable labels, coverage was 193 labels for Qwen and 166 for Gemma, excluding the garbage cluster.The reported figures of 194 and 167 use the 242-label taxonomy denominator, which includes the garbage cluster L3-241.Source: S6.7 Coverage of the taxonomy by the… · p. 56
  • The coverage check used all mapping attempts, rather than only the majority-consistent subset used in the preceding analyses.Out-of-taxonomy output could be detected without cross-run agreement.Source: S6.7 Coverage of the taxonomy by the… · p. 56
  • Exactly one mapping attempt fell outside the taxonomy: Gemma returned L3-27, a malformed rendering of the taxonomy code L3-027.The main-text footnote attributes the issue to a malformed L2 label, but this passage identifies it as an L3 label.Source: S6.7 Coverage of the taxonomy by the… · p. 56

S7: The Dirichlet-Multinomial model

The model finds a credible but modest post-2023 shift: σγ is above zero, yet σβ is 2.35 times larger and no individual L3 method shows a credible change. The supplement documents the model’s assumptions and implementation, including convergence limitations that affect efficiency rather than biasing exploration.

S7.1 Model specification

  • The model estimates baseline log-shares, linear trends, and post-2023 level shifts relative to the pre-existing trend, with σγ as its primary estimand.With only three post-2023 years, it cannot distinguish an abrupt change from gradual acceleration; inference therefore targets the across-technique scale rather than individual γ estimates.Source: S7.1 Model specification · p. 56

S7.2 Priors and implementation details

  • Asymmetric hyperpriors make a large σγ harder to obtain, while individual γ estimates remain uncertain, motivating inference on the scale and directional patterns.The posterior for ϕ concentrates near 1,100, far above earlier fixed values of 10 and 50, which attributed structural variation to noise and were superseded.Source: S7.2 Priors and implementation details · p. 57
  • The implementation uses a weak sum-to-zero constraint rather than a reference technique and pins unused padded parameters near zero to speed sampling without affecting reported parameters.The rectangular count array uses Kmax = 20; unused slots do not enter the likelihood but would otherwise still be explored by the sampler.Source: S7.2 Priors and implementation details · pp. 57–58

S7.4 Sampler configuration

S7.5 Convergence

  • Tree-depth saturation truncates trajectories and reduces efficiency but does not bias exploration; σγ has the model’s lowest bulk ESS, 640.The passage attributes saturation to adapt_delta = 0.99 on a 1,500-parameter posterior and notes that the sampling run was long.Source: S7.5 Convergence · p. 58

S7.6 Posterior summary

  • σβ exceeds σγ by 2.35, while σγ is credibly above zero and none of the 241 L3 methods has a 90% credible interval excluding zero.The posterior mean of ϕ is near 1,100, placing the likelihood close to multinomial; the aggregate signal does not localise to any named method, and the 20 methods furthest from zero all cross zero.Source: Table S7.4 · p. 58, S7.6 Posterior summary · p. 58, Figure S7.1 · p. 59

S8: Bayesian workflow checks

Bayesian workflow checks found that the priors span the observed diversity range and the model reproduces all fitted sub-disciplines. Fake-data recovery supports identification of σγ, though it is weak for a single sub-discipline; cross-validation was not performed.

S8.1 Prior predictive check

  • The prior predictive 5th–95th percentile range for inverse Simpson index was [2.76, 11.57], against an observed range of [1, 12.76].The priors cover the observed range without concentrating on it.Source: S8.1 Prior predictive check · p. 59

S8.2 Posterior predictive check

  • All 25 of 25 sub-disciplines fell inside the 0.05–0.95 posterior predictive band.For each group-year cell with data, 200 replacement-count draws were used to recompute the inverse Simpson index.Source: S8.2 Posterior predictive check · p. 59

S8.3 Fake-data simulation

  • The fake-data recovery posterior contracted toward the true σγ value relative to the Exponential(4) prior, supporting identification rather than prior-driven inference.The posterior remained wide for a single sub-discipline with three post-2023 years; in the full model, aggregation across 25 sub-disciplines sharpens the estimate. Cross-validation was not performed because only one model class was fitted.Source: S8.3 Fake-data simulation · p. 60, Table S8.1 · p. 60

S9: The concentration analysis

The analysis found that LLM method recommendations were less diverse than the literature, while recommendation diversity increased with methodological guidance. A conjugate Dirichlet–multinomial analysis estimated each condition’s effective number of methods using the inverse Simpson index.

S9.1 Statistical method

  • The analysis used a uniform Dirichlet(1) prior and 4,000 posterior draws to estimate the inverse Simpson effective number of methods.Conjugacy provided an exact posterior without MCMC; adding one pseudo-count to every L3 category inflated diversity most for concentrated conditions, so the LLM–literature gap may be understated.Source: S9.1 Statistical method · p. 60

S9.2 Results

  • Gemma’s effective-method counts were 28.8 overall, 20.5 novice, 29.5 intermediate, and 46.8 expert.The respective 90% CIs were [27.6, 30.0], [19.0, 21.9], [27.3, 31.7], and [43.4, 50.2].Source: Table S9.1 · p. 60, S9.2 Results · p. 61
  • Both models’ recommendations sat below both literature profiles with no interval overlap, while literature diversity increased after 2023 and LLM diversity rose strictly from novice to expert.The expert profile reached roughly half to two thirds of the pre-2023 literature’s breadth without matching it; consistency across two models is suggestive but not conclusive.Source: Figure S9.1 · p. 61, S9.2 Results · p. 61

S10: What drives LLM recommendations

The regression found that recommendations tracked pre-2023 method prevalence, especially with less methodological guidance. The post-2023 movement predictor was inconclusive because its underlying technique-level estimates were noisy.

S10.1 Model

  • The negative-binomial model jointly entered pre-2023 prevalence and post-2023 movement to separate training-distribution reproduction from tracking recent disciplinary change.It modeled overdispersed recommendation counts, with a few techniques recommended often and most rarely.Source: S10.1 Model · pp. 61–62

S10.2 Results

  • Across all eight fits, b_pre was credibly positive (posterior probability > 0 = 1.00); its novice-to-expert decline was steeper for Qwen3 (1.039 to 0.419) than Gemma (0.675 to 0.511).The association was strongest under novice prompts and weakest under expert prompts, consistent with less guidance producing recommendations that more closely track the pre-2023 canon.Source: Table S10.2 · p. 62, S10.2 Results · p. 62
  • The b_γ coefficient was indistinguishable from zero across fits, but this null is uninformative because all 241 primary-model technique estimates were individually non-credible.Sensitivity C addresses the question without relying on individual γ estimates.Source: S10.2 Results · p. 62

S11: Robustness and sensitivity

Robustness checks support the main findings across alternate breakpoints, method filters, taxonomy mappings, distributional comparisons, and a direct diversity analysis. They also qualify the distributional evidence: similarity to post-2023 literature does not establish causation.

S11.1 Breakpoint at 2022

  • Under the 2022 breakpoint, qualitative conclusions were unchanged: σβ remained stable, no technique reached credibility, and per-technique γ estimates correlated at 0.77.The smaller σγ was expected because including 2022 in the post-treatment period dilutes the contrast; the result supports a shift located at 2023 rather than 2022.Source: S11.1 Breakpoint at 2022 · pp. 62–63, Table S11.1 · p. 63

S11.2 Sensitivity A: count threshold

  • Restricting the analysis to techniques with at least 50 papers in 2023–2025 left 33 of 242, yet the overall coefficient remained −0.138 with P(β > 0) = 0.438.The null result remained slightly negative, indicating it was not an artefact of rare methods.Source: S11.2 Sensitivity A: count threshold · p. 63

S11.3 Sensitivity B: taxonomy remapping

  • Remapping the earlier experiment’s taxonomy raised coverage from 53 to 205 of 242 v3 methods (85%), while the overall β remained −0.339 with P(β > 0) = 0.368 and all profiles negative.This check applies only to the earlier experiment run; current experiment data were classified directly against v3.Source: S11.3 Sensitivity B: taxonomy remapping · p. 63

S11.4 Sensitivity C: distributional test

  • LLM recommendation distributions were credibly closer to post-2023 literature, with the strongest alignment under the weakest guidance.The profile gradient matches the mean-collapse prediction and recovers a signal the regression did not detect; the test establishes similarity, not causation, and cannot separate model influence from training-data trends.Source: S11.4 Sensitivity C: distributional test · p. 63, Table S11.2 · p. 63

S11.5 Sensitivity D: direct diversity trajectory

  • All 25 sub-discipline γ intervals straddled zero, while σβ exceeded σγ by an order of magnitude, indicating no measurable sub-discipline-level homogenisation.The direct diversity trajectory analysis used inverse Simpson index posterior draws from the primary model, propagating upstream uncertainty into a second-level model.Source: S11.5 Sensitivity D: direct diversity… · pp. 63–64, Table S11.3 · p. 64

S12: Reproducibility

The reproducibility materials document how the corpus enters the model and what can be rerun from deposited data. The early pipeline requires Scopus access, but analyses from §3.3 onward can be reproduced from the deposited count array.

S12.1 Corpus flow into the model

  • The corpus falls to 8,398 articles after removal of the garbage cluster, followed by one further modelling-stream filter before the array passed to Stan.The verification script re-derived every figure from the deposited artefacts.Source: S12.1 Corpus flow into the model · p. 64
  • 1,043 articles are removed by year filtering—403 from 2026 and 640 before 2010—leaving 14,138 observations: 8,442 pre-2023 and 5,696 post-2023.2026 is excluded as a partial year because retrieval occurred in April 2026 and Scopus indexing lags publication.Source: S12.1 Corpus flow into the model · p. 64
  • The taxonomy contains 242 L3 clusters, but 241 are carried into analysis; LLM recommendation coverage uses 242 as its denominator.Both models mapped some items into the discarded cluster.Source: S12.1 Corpus flow into the model · p. 64

S12.2 Software environment

  • The modelling stream's software is documented in Table S12.2, while the machine-learning stream's Python environment is given in S1.The table caption identifies the modelling stream as its scope.Source: Table S12.2 · p. 65

S12.3 Scripts and deposited artefacts

  • Scopus-dependent scripts 00 and 01 and their outputs are deposited; readers can start from stan_data.rds to reproduce every analysis reported in §4.Deposits include the taxonomy, stimulus set, model recommendations and justifications, mapping votes, Stan models, posterior summaries, and analysis scripts.Source: S12.3 Scripts and deposited artefacts · p. 65

List of supplementary tables

The supplementary tables document the study’s data, methods, modeling, and sensitivity analyses, providing a cross-reference to information in the manuscript.

  • Tables cover the computational environment, model settings, Scopus metadata and corpus construction, embedding and clustering settings, and the L2 and L3 method taxonomies.The inventory includes the 25 L2 categories with definitions and corpus counts and the 241 L3 clusters ordered by mentions.Source: List of supplementary tables · p. 65
  • Tables list the 28 research questions, vague methodological descriptors, L4-to-L3 mapping consistency, and L4 method items by researcher profile.Source: List of supplementary tables · p. 65
  • Tables document Dirichlet-multinomial and negative-binomial model priors, sampler settings, convergence diagnostics, global-parameter summaries, and fake-data recovery of σγ.Source: List of supplementary tables · p. 65
  • Results tables report effective L3 method counts by condition, model-profile posterior summaries, breakpoint sensitivity, recommendation similarity to pre- and post-2023 literature, and diversity trajectories.The inventory also includes the corpus flow into the Dirichlet-multinomial model and software for the modeling stream.Source: List of supplementary tables · pp. 65–66

List of supplementary figures

The supplementary figures present posterior summaries of method shifts and methodological diversity.

  • Supplementary figures show posterior γ estimates with 90% credible intervals for the 20 L3 methods with largest |γ|, and effective-number-of-L3-method distributions by condition.Source: List of supplementary figures · p. 66
Loading 2609.11198v1…