LLM knowledge for improved Out-of-Distribution Tabular Prediction - A Cross-Validated Regularisation Approach

1 Introduction

• External knowledge can encode plausible feature effects that source data may overemphasise or fail to distinguish as transferable under domain shift.Ridge instead shrinks coefficients toward zero, a generic belief that effects are small unless the data indicate otherwise.Source: §1 · p. 2
• Expert elicitation is slow, hard to scale, and does not guarantee a useful prior; LLMs offer a repeatable, low-cost alternative, but their numerical outputs can contain sign, scale, or uncertainty errors.LLMs are not presented as substitutes for human experts.Source: §1 · p. 2
• LLM-elicited coefficient priors improve OOD prediction on four of five real domain splits, with placebo tests attributing the lift to feature-specific content rather than prior scale.The evaluation uses a cross-provider, cross-family selection pipeline driven only by source-domain data.Source: §1 · p. 3
• For classifiers, fixed decision thresholds can substantially overstate gains when domain shifts alter class balance; reliable benefits lie in probabilistic accuracy and proper scoring rules.The reported contribution contrasts probabilistic calibration with threshold-dependent metrics.Source: §1 · p. 3
• On three of five datasets, priors improve in-domain accuracy as much as OOD accuracy; only where in-domain gains disappear does the prior specifically improve transfer.That dataset has enough training data for the in-domain regularisation gain to disappear.Source: §1 · p. 3
• Source-domain cross-validation margins are smaller than fold-level noise, so selection is better at avoiding poor priors than identifying the best one.Depending on the LLM variant and prior family, a prior can improve or degrade OOD performance relative to the zero-centred baseline.Source: §1 · p. 3

2 Related work

• Riegler et al. found that LLMs often identify feature-effect direction but calibrate prior width poorly, motivating this paper’s OOD focus and estimator design.They identified OOD prediction as a future use case.Source: §2 · p. 3
• A separate line asks LLMs for feature-specific penalty scores rather than coefficient values, as in LLM-Lasso [25].The broader comparison also notes that an ML ensemble outperforms LLM prediction alone, with only a small gain from hybrid fusion [28].Source: §2 · pp. 3–4
• Other approaches use LLMs as in-context regressors or row embeddings, with reported robustness under conditional shift inconsistent unless labelled target examples are available.Requeima et al. [26] study in-context probabilistic regression; Zeng et al. [27] study embeddings of serialised rows.Source: §2 · p. 3
• AutoElicit [23] aggregates priors across rephrased prompts for in-distribution prediction, whereas this work uses repeated coefficient samples and source-domain cross-validation to set prior strength.Moment-matching AutoElicit’s mixture yields a diagonal Gaussian prior comparable to the Independent-Prior.Source: §2 · pp. 3–4
• LoID evaluates token-prediction-derived priors under OOD covariate shift, while this work uses sampled coefficients, full-covariance uncertainty, source-domain strength selection, and disjoint domain splits.LoID’s target includes the training region; this study tests on a separate domain.Source: §2 · p. 4

3.1 Notation and problem setting

• Source and target risks can differ under domain shift, and source-only data or cross-validation cannot identify which source relationships transfer to deployment.Real-covariate splits may combine covariate, label, and concept shift; the study does not assume the conditional outcome mechanism is preserved.Source: §3.1 · p. 5
• Global knowledge comprises feature–response claims not derived from either sample and expected to hold across populations.Examples include domain theory, mechanistic understanding, prior studies in other populations, and practitioner judgement.Source: §3.1 · p. 5

3.2 Penalised regression as maximum a posteriori estimation

• Ridge regression is maximum a posteriori estimation under a zero-centred Gaussian prior, with λ determining prior precision.A large λ concentrates the prior at zero; λ approaching zero recovers the unpenalised estimator.Source: §3.2 · pp. 5–6
• Because ridge centres every coefficient at zero, it reduces variance but does not encode which feature effects may transfer between populations.A nonzero target can move coefficients away from the origin; it reduces target-risk bias only if it is closer to the target optimum than the unshrunk estimate.Source: §3.2 · p. 6

3.3 Priors that encode external knowledge

• A general Gaussian prior yields a penalised estimate whose target µ and structure P encode beliefs, while λ scales the penalty; ridge is the zero-target, identity-structure special case.The intercept is not penalised because it reflects response base rate and feature centring rather than beliefs about feature–response relationships.Source: §3.3 · p. 6
• The structure P can encode equal-strength target beliefs, feature-specific confidence through variances, or dependence among coefficients through a full covariance.Greater structure requires more elicited quantities and can be harder to estimate reliably from finite elicitation samples.Source: §3.3 · p. 6
• With nonzero µ, increasing λ drives estimates toward the prior target rather than zero, so a misspecified target can move estimates away from the target optimum.The balance between trusting data and reducing variance is favourable only to the extent that µ is informative.Source: §3.3 · p. 7
• The method blends baseline and fully informed predictions on the response scale, averaging probabilities for classification.The blend recovers the baseline at γ = 0 and the fully informed model at γ = 1.Source: §3.3 · p. 7

3.4 Selecting prior hyperparameters using LLMs

• Elicitation supplies prior shape while source-domain cross-validation chooses λ, separating relative feature beliefs from absolute prior strength.The procedure can drive λ toward zero to switch off an uninformative or misleading prior; this section’s framework is independent of whether coefficient vectors come from experts or LLMs.Source: §3.4 · p. 7

3.5 Pipeline overview

• The OOD partition is reserved for evaluation: preprocessing, prior strength, variant, and prior family are determined using source-domain data only.The data, elicitation, and modelling branches meet at model fitting and evaluation.Source: §3.5 · pp. 7–8, Figure 1 · p. 8
• Model families vary in shrinkage target and structure, from zero-centred ridge through elicited targets and uncertainty to full covariance; Mixed blends predictions, while unregularised fitting is the λ = 0 reference.The number of elicited quantities grows from none, through p and 2p, to p+p(p+1)/2.Source: §3.5 · p. 8, Table 1 · p. 9

3.6 Datasets and domain splits

• The five reported datasets span two regression and three classification tasks, with training sizes from 242 to about 27,000 and shifts defined by real covariates.The trade-off is one plausible deployment shift per dataset rather than a controllable family of shifts.Source: §3.6 · p. 9
• Removing Boston Housing’s ethically problematic feature reduced its best resplit average from +11.5% to +7.9% and eliminated the Prior-Mean advantage.The elicitation prompts still described the twelve-feature problem, creating an omitted-variable mismatch in the fitted eleven-feature specification.Source: §3.6 · p. 9
• A small dataset with 124 training rows was excluded because its flat validation curve selected λ ≈0 for every family, leaving no meaningful prior selection.The authors identify a validation surface with enough signal to distinguish λ values as a practical requirement.Source: §3.6 · p. 10

3.7 Eliciting priors from LLMs

• Prompts combine source-domain context, feature descriptions, author-written hints, and source-training feature-scale summaries; they do not disclose the OOD target population.Directional prior content is partly supplied by the authors, while magnitudes, per-feature variances, and covariance structure come from the model.Source: §3.7 · pp. 10–11
• The elicitation samples are archived because dated API snapshot identifiers were not recorded, so re-querying a model family cannot reproduce the stored draws.The paper treats the 60 stored bundles, rather than model names, as the reproducible artefact.Source: §3.7 · p. 11, Table 3 · p. 11

3.8 Prior quantities and model fitting

• Repeated elicited coefficient vectors provide the prior mean, feature-specific variances, and covariance; the correlated precision is formed by inverting the covariance with diagonal jitter.Regression solutions are closed-form, whereas logistic fits use L-BFGS; correlated-prior logistic fits show small optimiser-path sensitivity.Source: §3.8 · pp. 11–12

3.9 Tuning, selection and evaluation

• Five-fold cross-validation on the source training pool selects λ and the best variant per family, using MSE for regression and AUC for classification.All variants use identical folds, and the OOD set is excluded from tuning and selection.Source: §3.9 · p. 12
• Single-deployment uncertainty uses paired bootstrap resamples of the OOD test set, capturing target-row variability but not training, tuning, or elicitation variability.For classification, resampling is stratified by class.Source: §3.9 · p. 12

3.10 Follow-up analyses

• Classification thresholds are selected separately for each model by maximising source-domain cross-validation F1, then each model is evaluated on OOD data at its own threshold.The OOD-maximising threshold is recorded only as a diagnostic upper bound.Source: §3.10 · p. 12
• Placebo priors permute feature assignments or flip coefficient signs while preserving scale and correlation structure, then receive the same cross-validation retuning as real priors.Adult is excluded because cross-validation drives λ to the grid floor and prior families collapse onto the baseline.Source: §3.10 · p. 13
• The reported intervals pool OOD bootstrap scores across repeated source splits, with λ retuned and models refitted within each split; the deployed variant remains fixed in the separate resplit analysis.Prediction intervals describe a future pipeline run, while confidence intervals for mean lift average the source-split component over S repetitions; elicitation variability is not covered.Source: §3.10 · p. 13

4.1 OOD prediction on regression tasks

• Prediction intervals cover zero almost everywhere although confidence intervals often exclude it; Boston’s 151 OOD observations leave target-sample uncertainty at roughly half its confidence-interval variance.Repeated source splits reduce source-split uncertainty but do not add target observations or reduce uncertainty from the finite OOD sample.Source: §4.1 · pp. 14–15
• Five of six prior-informed regression results have positive means and confidence intervals excluding zero; Boston Correlated-Prior (Opus 4.5) reduces OOD MSE 7.9%, while Cancer Prior-Mean averages +2.4%.Only Boston Housing’s Prior-Mean does not exclude zero.Source: §4.1 · pp. 14–15, Table 4 · p. 15
• Boston Housing, Heart Disease, and Diabetes gain at least as much in-domain as out-of-domain, whereas Cancer gains +0.1% in-domain and +3.9% out-of-domain.The priors therefore act as better regularisation than shift correction on the first three datasets; Cancer is the exception with a shift-specific benefit.Source: §4.1 · p. 15

4.2 OOD prediction on classification tasks

• Heart Disease log loss improves by more than 10% for every family (Prior-Mean: 0.4709 → 0.4327), while its AUC intervals include zero; Diabetes improves on both metrics.On Adult (ntrain ≈27,000), cross-validation drives λ to the grid floor and the prior families collapse onto the baseline. Threshold-dependent metrics are omitted because label prevalence shifts can make fixed-threshold comparisons misleading.Source: §4.2 · p. 16, Table 5 · p. 17

4.3 Placebo priors: the priors’ content, not their scale

• Across 24 (dataset, prior family, placebo) comparisons, real priors beat placebo distributions in 20 at p < 0.05; permutation and sign-flipping were each rejected in 10 of 12, implicating feature identity and direction.Permutation attaches elicited numbers to wrong features; sign-flipping preserves magnitudes but discards direction.Source: §4.3 · pp. 18–19

4.4 What the selection step can and cannot do

• Source-domain cross-validation selects mid-range candidates, including Boston Housing at +2.7% versus the best achievable +11.2%; its winner gap is 0.06 standard errors, and selection avoids rather than finds the best candidate.Winner gaps are 0.06 standard errors on Boston Housing, Cancer, and Adult; 0.25 on Heart Disease; and 1.63 on Diabetes. Selected models avoid the worst candidates in these experiments.Source: §4.4 · pp. 19–20
• Candidate OOD lifts span −2.3% to +11.2% on Boston Housing and −2.9% to +6.1% on Cancer; the worst candidate falls below baseline on every dataset.Figure 3 shows the full candidate range alongside within-family and cross-family selections.Source: §4.4 · p. 19, Figure 3 · p. 20

4.5 What the elicited priors agree on

• Across twelve variants, dominant-feature signs agree while disagreement concentrates on smaller, ambiguous features; a single extreme Gemini Flash property-tax draw illustrates that outliers remain consequential.That draw pulls the 50-sample mean to about −148 in standardised space, though tuning and selection contain the damage and the variant is not deployed on Boston Housing.Source: §4.5 · pp. 20, 22, Figure 4 · p. 21

5.1 When LLM priors help

• Four of five datasets improve OOD, but single-deployment prediction intervals cover zero almost everywhere, so an average benefit does not guarantee a benefit in any one deployment.Ten of 24 prior-informed results have confidence intervals excluding zero in the beneficial direction.Source: §5.1 · p. 22
• Cancer alone shows a substantial OOD gain (+3.9%) with negligible in-domain gain (+0.1%), supporting transfer beyond ordinary regularisation in only that dataset.Boston Housing, Heart Disease, and Diabetes improve in-domain at least as much as OOD; their training sets have 242–316 rows for 7–18 features.Source: §5.1 · p. 22
• More datasets with negligible in-domain gains but meaningful domain shift are needed to test whether the transfer-specific benefit generalises.The paper positions the method as most useful when source data cannot identify stable coefficients; the stronger claim of shift correction is supported here by Cancer alone.Source: §5.1 · p. 22

5.2 What selection does, and what it does not

• Candidate priors can either improve or degrade OOD performance, and source-CV margins are too noisy to identify the best; selection mainly avoids poor candidates.The selected model falls in the middle of the candidate range on every dataset; source-only validation need not rank candidates by target performance.Source: §5.2 · pp. 22–23

5.3 Evaluating classifiers under label shift

• Under label shift, fixed-threshold F1 can create an apparent advantage that disappears when the baseline receives the same source-tuned threshold freedom.The paper recommends proper scoring rules and, when threshold-dependent metrics are reported, tuning every model's threshold on source data only.Source: §5.3 · p. 23

5.4 Limitations and future work

• The study's scope is constrained by possible benchmark contamination, one source–target split per dataset, and unresolved uncertainty about whether sign hints contributed useful directions.The matched-baseline comparisons test elicited priors against zero-centred shrinkage, not against other ways of using LLM knowledge; more target samples and distinct splits would strengthen the conclusions.Source: §5.4 · pp. 23–24

5.5 Conclusion

• The conclusion is conditional: prior quality and structure matter, while source-only selection offers safety rather than optimality and fixed-threshold metrics can inflate apparent gains.Proper scoring rules provide a more reliable classification comparison when class distribution shifts.Source: §5.5 · p. 25

Data availability; Code availability; Competing interests

• The paper makes its datasets and analysis artifacts available, reports no competing interests, and notes that stored coefficient samples permit reproduction without further LLM API calls.The county-level cancer compilation is redistributed because its original hosting page no longer resolves.Source: Data availability · p. 25, Code availability · p. 25, Competing interests · p. 26

Supplementary figures

• Figure S1 shows lift distributions across ID resplits with λ retuned each time; the single-split Table 4 values lie in the upper part of those distributions.Lift is MSE reduction for Boston Housing and Cancer and AUC gain for classification datasets.Source: Figure S1 · p. 29
• Figure S2 shows improved Brier score and log loss calibration on Heart Disease and Diabetes while AUC remains essentially unchanged.The figure pairs reliability curves with histograms of predicted probabilities.Source: Figure S2 · p. 29
• Table S1 reports OOD lift across repeated ID resplits and the fraction of resplits where the prior-informed model is worse than baseline.It includes 50 resplits for Boston Housing and Cancer, 30 for Heart Disease and Diabetes, and 10 for Adult.Source: Table S1 · p. 30

Threshold-dependent metrics under label shift

• At threshold 0.5, F1 gains are +0.048 [+0.017, +0.080] for Prior-Mean on Heart Disease and +0.089 [+0.039, +0.143] on Diabetes.All eight results across four prior families and two datasets are significant at this fixed threshold.Source: B Threshold-dependent metrics under… · p. 31
• Higher OOD prevalence makes 0.5 an inappropriate neutral comparison point, because upward-shifted probabilities can appear better on F1 without a better model.Diabetes prevalence rises from 0.212 in the source pool to 0.495 OOD; Heart Disease rises from 0.459 to 0.530.Source: B Threshold-dependent metrics under… · p. 31
• Source-CV thresholds remove the apparent F1 gains; on Diabetes, the recalibrated baseline reaches F1 = 0.7236 versus 0.6472 for Prior-Mean at 0.5.All eight prior F1 gains lose significance when every model, including baselines, receives the same source-selected threshold; proper scoring-rule improvements remain unaffected.Source: B Threshold-dependent metrics under… · p. 31, Figure S3 · p. 31

Sensitivity to the variance floor

• The sensitivity analysis varies ε while retuning λ for prior families whose precision depends on elicited spread, and tests Ledoit–Wolf covariance shrinkage for Correlated-Prior.The deployed setting uses ε = 10^-8.Source: C Sensitivity to the variance floor · p. 32, Table S2 · p. 32
• Boston Housing's Correlated-Prior lift is +11.15% at the deployed setting and +11.10% with Ledoit–Wolf; Cancer remains flat to two decimal places across the range.Lifts decay only when ε exceeds the elicited variances, causing Independent-Prior to approach Prior-Mean.Source: C Sensitivity to the variance floor · p. 32
• Heart Disease's Independent-Prior lift falls from +0.75% to +0.09% as the variance floor changes, making it the most floor-sensitive reported case.The authors flag this result as one to read with the floor in mind.Source: C Sensitivity to the variance floor · p. 32

• The reproduced example uses the Diabetes dataset and non-reasoning sampling mode.Source: D Example elicitation prompt · p. 32
• Prompts combine a fixed system part with a dataset-specific user part; other datasets differ only in the user part.Source: D Example elicitation prompt · p. 32
• Feature-scale summaries use only the source training pool, and the domain context omits the OOD target population.Source: D Example elicitation prompt · p. 32

• The system prompt requests plausible regression priors from the perspective of an endocrinologist and applied statistician experienced in type 2 diabetes screening data.Source: System prompt · p. 32
• It asks the model to reason about both coefficient signs and magnitudes using feature definitions and scale summaries.Source: System prompt · p. 33
• Coefficients must remain in the model matrix’s raw-feature units, without standardization, rescaling, or an intercept.Source: System prompt · p. 33
• The output should be one plausible draw from the belief distribution, retaining uncertainty rather than collapsing distinct plausible settings into an average.Source: System prompt · p. 33
• The draw should be realistic, interpretable, conservative, and domain-plausible, avoiding unjustified extreme magnitudes.Source: System prompt · p. 33

• The task is binary logistic regression for the positive class in the Pima Indians Diabetes dataset, with the outcome defined as diabetes within five years of the visit.Source: User prompt · p. 33
• The source domain comprises younger patients, while the prior should encode physiology associations expected to transfer beyond that age range.Source: User prompt · p. 33
• The cohort is Pima Indian women aged 21 and older; Age defines the OOD split but is excluded from predictors.Source: User prompt · p. 33
• Observed zeros in several clinical features are treated as likely missing values and imputed with source-domain medians; DiabetesPedigreeFunction is a unitless family-history score on roughly a 0–3 scale.Source: User prompt · p. 33
• The instructions call for conservative, feature-scale-aware values, 0/1 interpretation for indicators, and conservative signal allocation when predictors overlap.Source: User prompt · pp. 33–34
• The prompt provides feature meanings and source-matrix summaries, including medians, interquartile ranges, central 95% ranges, and missingness for several predictors.Source: User prompt · pp. 33–34
• Glucose is identified as the strongest single predictor, with positive effects expected for BMI, pregnancies, and pedigree score; other listed effects are smaller or noisier and may be near zero.Source: User prompt · p. 34