PaperlayerView exact revision on arXiv

Source-linked AI summary

BLADE: Distilled LLM Regularization for Calibrated Knowledge Graph Completion

Ibne Farabi Shihab, Rabeya Bosri Tamanna, Abdo El Karaky, Sanjeda Akter, Anuj Sharma

arXiv:2609.29487v1cs.LG

TL;DR

Knowledge graph completion models optimize ranking, but their scores need not be calibrated probabilities for the candidates downstream users encounter. BLADE separates latent truth from graph recording and distills offline language-model judgments into a frozen training regularizer. Across declared candidate pools, it largely preserves ranking while improving probability quality, with calibration claims limited to those pools.

  • Problem

    KGC scores can rank candidate entities without being calibrated probabilities, while calibration depends on the candidate distribution actually encountered.

    Source: §1 · p. 1

  • Method

    BLADE uses variational embeddings to model latent truth and graph recording separately, with offline language-model judgments distilled into a frozen teacher regularizer.

    Source: §1 · p. 1

  • Results

    BLADE largely preserves ranking while improving probability quality on declared pools; adaptive ECE falls 60.1% versus deep ensembles and 78.1% versus temperature-scaled RotatE.

    Source: §7.2 · p. 6, §8 · p. 7

  • Takeaways & Limitations

    BLADE offers a route to better calibrated KGC when the deployment candidate distribution can be specified and audited.

    Source: §8 · p. 7

    Calibration is supported only on constructed binary pools; the tested near-miss shift does not establish calibration for all unobserved triples or arbitrary deployment mixtures.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Knowledge graph completion models optimize ranking, although many downstream applications require calibrated probabilities. We present BLADE, a variational model that separates latent truth from graph recording and distills offline language-model judgments into a frozen teacher regularizer. The LLM is absent during inference. Posterior samples provide predictive probabilities and epistemic uncertainty, while the compact teacher remains available only as an optional triage factor. Across five benchmarks, BLADE remains competitive under a common ranking protocol and reduces adaptive ECE by a macro-average of 60.1% relative to deep ensembles and 78.1% relative to temperature-scaled RotatE. On identical FB15k-237 candidate sets, BLADE also improves ECE, Brier score, and NLL over validation-selected histogram binning and a matched generative ComplEx2 model, with these improvements persisting on a prespecified near-miss pool. Under controlled injected missingness, the full triage score achieves a mean AUC-PR of 0.863, compared with 0.805 for its strongest non-teacher variant. Leakage stress tests show that aligned semantics matter, but they cannot exclude knowledge acquired during LLM pretraining. We therefore claim calibration only for the declared candidate distributions, not for all unobserved triples.

1 Introduction

Knowledge graph completion models rank candidate facts, but their scores are not necessarily probabilities calibrated for the candidates people actually encounter. BLADE separates latent truth from graph recording and uses offline language-model judgments as a training regularizer, with evidence supporting preserved ranking and improved calibration on specified candidate pools.

1 Introduction

  • KGC scores can rank candidate entities effectively without representing probabilities that are calibrated for the candidates users encounter.Curators and reasoning systems need confidence values whose stated probabilities match observed correctness on those candidates.Source: §1 · p. 1
  • Under the open-world assumption, absent triples may be false, ambiguous, or true but unrecorded, and calibration depends on the candidate distribution.The paper treats transfer from random corruptions to plausible near misses as a distribution shift, not evidence of distribution-free calibration.Source: §1 · p. 1
  • BLADE distills offline language-model labels into a frozen plausibility teacher that regularizes variational KGC training, then is absent during inference.The benchmark teacher candidates are constructed from the training graph, so its experimental role is consistently called a regularizer.Source: §1 · pp. 1–2
  • Evidence supports largely preserved ranking, improved calibration on specified exact and near-miss pools, and better recovery of deliberately hidden positives.The paper frames its contribution as separating and jointly evaluating these roles, not as the first probabilistic KGE, variational embedding, distillation method, or calibrated link predictor.Source: §1 · p. 2

2 Related work

Prior work supplies ranking models, calibration methods, uncertainty estimates, and semantic side information, but these approaches target different outputs and evaluation protocols. BLADE uses language-model distillation as a training regularizer rather than an inference-time text scorer.

2 Related work

  • Classical KGE models learn unnormalized compatibility scores, so converting them to probabilities requires a declared negative distribution and fitted calibration protocol.The paper therefore compares matched Platt, isotonic, beta, and histogram-binning methods and evaluates probabilistic models on identical candidate identifiers.Source: §2 · p. 2
  • Distributional KGE, Bayesian graph uncertainty, MC dropout, and deep ensembles motivate uncertainty estimates, while conformal KGE targets set coverage rather than scalar probability calibration.BLADE uses a factorized variational density with embeddings coupled through a relational encoder, not an unrestricted joint posterior.Source: §2 · p. 2
  • Text-enhanced and LLM-assisted systems demonstrate semantic side information, while BLADE distills a larger model into a frozen teacher for training-time regularization.Its teacher is not used as an inference-time text scorer; the cited systems are situated within distinct evaluation protocols.Source: §2 · p. 2

3 Problem setup

The setup distinguishes whether a triple is true from whether it was recorded, and defines separate evaluation outputs for ranking, calibration, and hidden-positive triage. Each output depends on a specified candidate construction.

3 Problem setup

  • The open-world assumption separates latent truth from recording: an unrecorded triple is not necessarily false.The setup defines truth y_i and recording o_i as distinct binary variables for each candidate triple.Source: §3 · p. 2
  • Filtered ranking, binary-pool calibration, and hidden-positive triage are distinct outputs with separate metrics and candidate constructions.Ranking uses MRR and Hits@k; calibration uses adaptive ECE, Brier score, and NLL; triage uses AUC-PR and precision at a fixed review budget.Source: §3 · p. 2
  • The candidate-pool construction is part of the calibration and triage estimand, not an incidental implementation choice.The setup distinguishes a standard pool, a prespecified near-miss shift, and injected missingness.Source: §3 · p. 2

4 Model

BLADE separates offline semantic distillation, probabilistic modeling of truth and recording, variational inference, regularized training, and posterior-based prediction. Its benchmark teacher is data-dependent, and the recording-propensity factor is treated as a proxy rather than a causal or identified account of missingness.

4 Model

  • BLADE separates offline distillation, teacher-regularized KGC training, and LLM-free inference.Offline scores are fixed before training, and inference consumes learned models without issuing LLM queries.Source: §4 · pp. 2–3, Figure 1 · p. 4

4.1 Latent truth and graph recording

  • The model represents latent truth and graph recording separately, but recording propensity is a regularized proxy rather than an identified causal missingness mechanism.Only the product of truth probability and recording propensity is observed; parameters are regularized and selected on injected-missing validation splits.Source: §4.1 · p. 3

4.2 Variational posterior

  • A relational encoder produces parameters for a factorized Gaussian posterior, sharing graph-conditioned representations without unrestricted covariance.The encoder aggregates typed incoming and outgoing messages.Source: §4.2 · p. 3

4.3 Distilled LLM plausibility

  • Offline language-model plausibility scores train a compact frozen teacher that supplies a bounded energy during KGC training.The benchmark labeling stage excludes validation and test triples; the energy aligns the structural predictor with the teacher without requiring new labels at inference.Source: §4.3 · p. 3, Figure 1 · p. 4

4.4 Teacher regularization and the prior regime

  • The benchmark teacher energy is a data-dependent regularizer; a prior interpretation requires all energy-defining quantities to be independent of the analyzed sample.The separate prior regime uses a proper base prior and has a finite normalizer because the energy is nonpositive.Source: §4.4 · p. 4

4.5 Training objective

  • The benchmark objective combines observed-data reconstruction, a KL penalty against a Gaussian base prior, and the expected teacher regularizer.The paper does not describe this benchmark objective as an ELBO under a data-independent LLM prior.Source: §4.5 · p. 4

4.6 Posterior prediction and triage

  • Posterior samples estimate predictive truth probabilities and epistemic uncertainty, while a separate optional score ranks unobserved triples for false-negative triage.Ranking and calibration use the predictive probability without the teacher factor; the triage score’s confidence and teacher factors are heuristic modifiers, not a generally Bayes-optimal posterior.Source: §4.6 · p. 4

5 Theoretical scope

The theory gives a local covariance ordering and an exact false-negative identity under known truth and recording probabilities, while keeping benchmark claims conditional.

5 Theoretical scope

  • The conditional theory neither makes the benchmark-dependent teacher a Bayesian prior nor identifies the true recording process.Source: §5 · p. 5
  • Σλ ⪯ Σ0 for λ ≥ 0 under A ≻ 0 and positive-semidefinite teacher curvature, comparing Laplace covariances at one fixed expansion point.The ordering does not guarantee improved calibration when the teacher is inaccurate.Source: §5 · p. 5
  • With known pi and ρi, an unobserved triple's posterior truth probability is (1−ρi)pi/(1−ρipi).This is the exact identity motivating the first factor of the triage score.Source: §5 · p. 5
  • The PAC–Bayes statement requires Eq. 8 to define a complete hypothesis independently of an i.i.d. analyzed sample, and is not invoked for transductive benchmarks.Source: §5 · p. 5

6 Experimental design

The experiments use shared protocols to separate ranking, candidate-conditional calibration, controlled hidden-positive triage, component effects, and leakage stress tests.

6 Experimental design

  • Direct comparisons share candidate identifiers and metric code, separating controlled reruns from external published numbers.Source: §6 · p. 5

6.1 Datasets

  • The five datasets span general, lexical, web-extracted, and biomedical graphs: FB15k-237, WN18RR, CoDEx-M, NELL-995, and Hetionet.Source: §6.1 · p. 5

6.2 Baselines

  • Common ranking reruns include RotatE, KG2E, a RotatE ensemble, and SimKGC; calibration and generative comparisons use shared binary pools and matched ComplEx2.Post-hoc calibration families and hyperparameters are selected on source-query-clustered validation before test evaluation.Source: §6.2 · p. 5, Table 1 · p. 6

6.3 LLM teacher protocol

  • LLaMA-3-70B-Instruct supplies offline labels from 20,000–60,000 queries per dataset; one compact teacher is fitted per dataset and frozen across five KGC seeds.Queries include observed positives, type-compatible negatives, and random negatives; a strict parser rejects invalid outputs.Source: §6.3 · p. 5

6.4 Leakage and robustness controls

  • Leakage stress tests vary descriptions, entity names, and relation semantics, then permute scores within relations; these controls cannot erase facts encoded during pretraining.Source: §6.4 · pp. 5–6

6.5 Metrics

  • Calibration uses adaptive-bin ECE, Brier score, and NLL on exact and frozen hard pools; two-stage paired bootstrapping preserves candidate dependence within queries.The calibration table reports adaptive ECE/Brier on each graph’s declared binary candidate distribution.Source: §6.5 · p. 6, Table 2 · p. 6

7 Results

Across five seeds, the results assess ranking, calibration, injected-missingness triage, and component effects using paired two-stage bootstrap inference.

7.1 Link prediction

  • BLADE trails SimKGC by 0.003–0.006 MRR on three datasets, is effectively tied on NELL-995, and leads by 0.013 on Hetionet.Paired tests support improvement over RotatE on all five datasets and over SimKGC only on Hetionet; ranking is a preservation criterion, not a state-of-the-art claim.Source: §7.1 · p. 6

7.2 Calibration

  • On identical FB15k-237 candidates, BLADE–RotatE improves over selected histogram binning by 0.005 ECE, 0.010 Brier, and 0.030 NLL on the exact pool.On the hard pool, the advantages grow to 0.016 ECE, 0.020 Brier, and 0.060 NLL; across five declared binary pools, adaptive ECE falls by 60.1% versus deep ensembles and 78.1% versus temperature-scaled RotatE.Source: §7.2 · pp. 6–7, Table 3 · p. 7

7.3 False-negative detection

  • Under controlled injected missingness, the full triage score reaches 0.863 mean AUC-PR, versus 0.805 for the mean-plus-observation variant.These results concern deliberately hidden positives under the declared matching protocol, not real-world KG missingness.Source: §7.3 · p. 7

7.4 Component roles and leakage stress tests

  • Removing the observation model worsens ECE and hidden-positive AUC-PR despite little non-significant MRR change; point embeddings degrade all three metrics.Removing the teacher retains a structural model but loses much of the calibration and triage gain; decoder-control gains largely transfer to ComplEx, without establishing decoder independence.Source: §7.4 · p. 7
  • Aligned semantics matter: anonymizing entity strings causes a larger drop than removing descriptions, while within-relation score permutation moves ECE and triage toward chance.The controls cannot determine whether the useful semantics came from benchmark-adjacent pretraining.Source: §7.4 · p. 7

8 Conclusion

BLADE separates offline semantic regularization from an LLM-free variational predictor, improving probability quality on declared candidate pools while preserving ranking and supporting controlled triage.

8 Conclusion

  • The appropriate conclusion is better calibrated KGC when the deployment candidate distribution can be specified and audited, not universal calibration or ranking dominance.Source: §8 · p. 7

Limitations

BLADE’s reported probabilities are calibrated only on specified binary candidate pools, not all unobserved triples. Its recording-propensity factor is an assumption-dependent proxy, and its mean-field posterior and sampling impose representational and computational limits.

Calibration scope

  • Calibration is supported only on the constructed binary pools; the hard near-miss pool does not certify behavior on arbitrary candidate distributions or all unobserved triples.The reported probabilities also do not establish calibration for future graphs or arbitrary deployment mixtures.Source: Limitations · p. 7

Missingness assumptions

  • The observation factor is non-identifiable: low latent truth and low recording propensity can explain the same observation.Degree and relation features make recording propensity a regularized proxy in injected-missing experiments, not a causal estimate of real graph omissions.Source: Limitations · pp. 7–8
  • The recording proxy assumes omissions are conditionally missing at random given relation and local node-degree features, an assumption systematic, adversarial, extremely sparse, or temporal omissions may violate.Source: Limitations · p. 8

Representation and cost

  • The mean-field posterior cannot represent unrestricted graph-level covariance, teacher controls cannot erase LLM pretraining knowledge or prevent temporal and aliasing errors, and posterior sampling adds cost when latency and ranking alone matter.Source: Limitations · p. 8

Ethical Considerations

The paper documents its data and language-model use, and limits its claims about leakage, calibration, and missingness. Its extended materials distinguish related evaluation objects and detail model, calibration, and audit procedures.

Ethical Considerations

  • The study uses publicly released benchmarks under their original licenses, collects no new data, and reports locally hosted LLaMA-3-70B-Instruct use for teacher scoring and audit adjudication.The language-model section also records prompts, decoding settings, parser outcomes, retries, and rejected responses for inspection.Source: Ethical Considerations · p. 8
  • Teacher pools exclude validation and test triples, but the leakage interventions cannot remove facts encoded during LLM pretraining, so calibration claims remain limited to declared candidate distributions.Source: Ethical Considerations · p. 8
  • The authors recommend human-reviewed triage rather than autonomous high-stakes assertions, note possible inherited teacher bias, and report no demographic-bias audit.They also describe the recording-propensity factor as a non-causal proxy whose assumptions may fail for systematic, adversarial, or temporally driven omissions.Source: Ethical Considerations · p. 8

B Extended related-work positioning

  • The related-work discussion separates ranking, scalar calibration, generative probability, embedding uncertainty, prediction-set coverage, and semantic augmentation because these objects and protocols are not directly interchangeable.It positions BLADE’s offline teacher signal against systems that retain language models in scoring or augmentation, while treating transfer-oriented reasoners as complementary.Source: B Extended related-work positioning · pp. 10–11

C Relational encoder details

  • The relational encoder couples predictive representations, but the variational posterior remains factorized and cannot express unrestricted entity, relation, or graph-level covariance.The appendix describes a low-rank-plus-diagonal posterior as a possible extension.Source: C Relational encoder details · p. 11

D Common-protocol ranking details

F Post-hoc and generative calibration

G LLM teacher configuration

N Hyperparameters and compute

Cross-dataset testing favors BLADE over RotatE in MRR on all five datasets, while leakage controls show semantic alignment effects without ruling out pretraining exposure. The appendices specify selection, compute, and statistical procedures.

N Hyperparameters and compute

  • Hyperparameters are selected on validation MRR subject to an ECE constraint, and posterior evaluation uses 64 samples on four datasets and 96 on Hetionet.Training the variational system takes approximately 2.3× one RotatE model’s runtime; the appendix says standard KGE remains preferable when latency and rank alone are the objectives.Source: N Hyperparameters and compute · pp. 14–15

O Statistical testing

  • Statistical comparisons use paired seed and source-cluster resampling with 95% percentile intervals, two-sided bootstrap p-values, and Holm correction within declared families.Intervals are conditional on the validation-selected pipeline because selection decisions remain fixed inside the bootstrap.Source: O Statistical testing · p. 15
  • Cross-dataset recomputation favors BLADE over RotatE in MRR on all five datasets, but over SimKGC only on Hetionet; full triage beats mean plus observation on all five graphs.Removing the observation model has clear ECE and triage effects but a small, nonsignificant MRR effect; entity anonymization affects ranking and triage, while its smaller ECE change is not significant.Source: O Statistical testing · p. 15

P Additional leakage-control results

  • Leakage controls indicate that descriptions help modestly, entity names contribute more, relation text retains a weaker signal, and randomly aligned scores damage calibration and triage.These interventions cannot rule out facts encoded during LLM pretraining.Source: P Additional leakage-control results · p. 15

Q Conditional data-independent prior result

The conditional PAC–Bayes result requires sample-independent prior construction, while extended experiments test calibration on verified negatives, calibration complementarity, pool difficulty, and teacher-construction stability.

Q Conditional data-independent prior result

  • A genuine prior requires the teacher, candidate set, energy hyperparameters, and proper base prior to be fixed independently of an i.i.d. sample; the benchmark regime violates these conditions.The theorem is conditional on an external-teacher regime, not a benchmark guarantee.Source: Q Conditional data-independent prior… · p. 16

R Reproducibility materials

  • The supplementary package records split manifests, teacher candidates and scores, calibration pools, five-seed configurations, prediction files, decoder controls, and evaluation scripts.Raw LLM responses are included only when the model license permits redistribution.Source: R Reproducibility materials · p. 16

S Failure cases

  • Failure cases include temporally wrong facts, sparse biomedical entities whose variance can hide rare positives, and relation aliases that mismatch curated meanings.The authors retain posterior variance and the observation model so no single triage signal dominates.Source: S Failure cases · p. 16

T Extended Experiments

  • Five extensions examine verified negatives, estimand-matched calibration, difficulty versus prevalence, teacher variability, and multi-LLM adjudication; calibration advantages are smaller on harder pools.Selection is frozen before test labels are read, and each experiment states when claims should be narrowed.Source: T Extended Experiments · p. 17

T.1 Experiment 1: Calibration on released verified CoDEx-M negatives

  • BLADE–RotatE improves over the strongest non-BLADE comparator by >0.006 ECE and >0.010 Brier, without reducing AUC-PR.The paired ECE and Brier intervals exclude zero on verified negatives.Source: §T.1 · p. 18, Table 21 · p. 19

T.2 Experiment 2: Estimand-matched KGE Calibrator comparison

  • BLADE plus KGE Calibrator is complementary, not a replacement: it leads on calibration metrics while the rank-preserving calibrator leaves ranking unchanged.BLADE improves the probabilistic representation, while KGE Calibrator supplies a query-conditional post-hoc map.Source: §T.2 · p. 19

T.3 Experiment 3: Separating negative difficulty from prevalence

  • 0.0135 ECE difficulty effect versus 0.0045 prevalence effect for BLADE–RotatE; the difficulty-by-prevalence interaction was small, with intervals including zero.The fixed-pool, single-dataset effects are interpreted descriptively, not as universal deployment coefficients.Source: §T.3 · p. 19, Table 30 · p. 21

T.4 Experiment 4: Teacher-construction uncertainty

  • 0.001 ECE and 0.003 AUC-PR between-teacher standard deviations fall below predeclared instability bounds, but variance-share intervals remain wide with three constructions.The result supports stability within the declared construction, not invariance to arbitrary teachers.Source: §T.4 · pp. 20–21

T.5 Experiment 5: Evidence-grounded multi-LLM adjudication

The multi-LLM adjudication evaluates BLADE on naturally unobserved candidates using frozen textual evidence and consensus labels. BLADE improves calibration and triage against structural baselines, but those labels are not human-verified truth.

T.5 Experiment 5: Evidence-grounded multi-LLM adjudication

  • Fleiss’ κ = 0.690 with interval [0.678, 0.702], and Panel-HC clears every audit gate; teacher–judge correlations remain below the 0.80 circularity-warning threshold.The audit precedes evaluation on naturally unobserved candidates.Source: §T.5 · p. 22
  • Every judge-removal subset stays within the predeclared instability bounds of 0.010 ECE and 0.030 AUC-PR, but the labels remain an LLM-consensus proxy rather than human ground truth.Reported prevalences are conditional on selected strata and do not estimate prevalence among all unobserved triples.Source: §T.5 · pp. 21, 23, 24
  • On unanimous, high-confidence LLM-consensus candidates, BLADE improves calibration and ranks supported unobserved triples better than structural baselines; full triage is strongest across strata and datasets.The labels are a proxy, not human-verified truth.Source: §T.5 · p. 24

T.6 Shared statistical analysis

  • The shared analysis uses 10,000 paired bootstrap replicates, resamples seeds before candidate clusters, and applies Holm correction within declared metric families.Intervals for LLM-derived labels are conditional on the frozen panel; three judges are not treated as a random sample of LLMs.Source: §T.6 · p. 24

T.7 Compute and budget estimate

  • The three-model panel cost about USD 500–1,500 at run-time provider prices; replication costs should be recomputed against current pricing.API prices and reasoning-token accounting change over time.Source: §T.7 · p. 24
Loading 2609.29487v1…