Coupling Noisy Pairwise Knowledge to the DAG Posterior for Causal Discovery

1 Introduction

• External reports may identify a direct relation, reverse its direction, or describe an influence mediated by another measured variable.Repeated queries can also disagree or abstain, making their influence something to estimate alongside observational data.Source: §1 · p. 1
• HB-NoisyKG models each report as a noisy observation of a DAG-implied direct pair state, with pair-specific reliability and a shared error distribution.The feature-conditioned reliability prior and shared error matrix link report learning to graph inference through pair-state marginals.Source: §1 · pp. 1–2
• The evaluation spans budget-controlled discrete benchmarks and a physical light-tunnel task, plus synthetic source-estimation and continuous-posterior studies.Other studies examine trust assignment, report accuracy and coverage, elicitation, and sparse graphs up to 125 variables.Source: §1 · p. 2
• The report reward depends only on graph marginals, supporting alternating graph, reliability, and confusion updates across graph-only and joint inference.The same observation layer serves BDeu, BGe, and linear and nonlinear joint posteriors.Source: §1 · p. 2
• Continued updating reduces mean reliability estimation error and held-out report log loss compared with one-time estimation.The experiments also demonstrate structural gains over data-only inference on five classical networks and a physical system.Source: §1 · p. 2

Bayesian structure learning and knowledge guidance

• HB-NoisyKG assigns feature-conditioned reliability to repeatedly reported pairs and estimates shared incorrect-state probabilities alongside the graph posterior.Amirkhani et al. instead estimate or marginalize expert-level accuracy when scoring structures; VaMSL [3] incorporates elicited feedback through a graph prior.Source: §2 · p. 2

LLMs, noisy labels, and Dawid–Skene coupling

• HB-NoisyKG retains repeated LLM reports in an observation likelihood with pair-specific reliability and a shared error matrix inferred alongside one DAG.Other approaches use structural priors, query-based orientation seeds [23], consistency amendments [21], or causal order to address direct-versus-mediated ambiguity.Source: §2 · p. 2

3 Method

• The same repeated-report likelihood augments either the collapsed graph posterior or the joint graph-and-parameter posterior.The data likelihood remains setting-specific, while reports observe the graph through its implied direct pair states.Source: §3 · p. 2, §3.1 · p. 3

3.1 Core hierarchical model

• The hierarchical model combines a DAG, optional data-model parameters, and report parameters for the direct state of each variable pair.Repeated reports are modeled as observations of those graph-implied states.Source: §3.1 · p. 3

3.2 Noisy pairwise observation model

• Direct states are none or one of two orientations; reliability is the probability an observed report matches the state, while a shared error matrix distributes mistakes.A feature-conditioned prior pools reliability information across pairs, and a shared matrix pools off-diagonal error counts.Source: §3.2 · p. 3

3.3 Closed-form reward and alternating inference

• Pair-state marginals determine expected matching and incorrect report counts, which update reliability and the shared error matrix before report rewards inform the next graph update.Because the noisy-knowledge layer needs only the graph marginal, it is reusable in graph-only and joint settings.Source: §3.3 · p. 4

3.4 Instantiating the alternating inference procedure

• The procedure uses DiBS with SVGD for continuous joint inference and Top-K local search for discrete benchmarks and graph-only Sachs.Before the first reliability update, a one-time blend of data-only pair-state marginals and report frequencies is weighted by report agreement.Source: §3.4 · p. 5

3.5 Properties of the observation layer

• When report rewards vanish, inference reduces to data-only Bayesian structure learning; reliability concentration assumes a converged true pair state and conditionally i.i.d. reports.With imperfect pair marginals, the finite-report bound separates graph-state error, sampling variation, and prior shrinkage; more reports reduce the latter two terms.Source: §3.5 · p. 5

4.1 Setup

• The study uses five discrete benchmarks, continuous Sachs, a physical light-tunnel task, and controlled synthetic studies, with comparisons spanning data-only, knowledge-guided, and other structure-learning methods.Report prompts and comparison budgets vary by study; the matched discrete design shares observations and a Top-K backend.Source: §4 · p. 6, §4.1 · pp. 6–7

4.2 Primary evidence: discrete Bayesian network recovery

• Mean SHD is 16.06 for HB versus 22.39 for no-KG with 80 restarts; HB improves on all five networks, while the fixed channel has lower mean SHD on four.The matched direct-parent study reports mean SHD of 16.03 for HB versus 24.03 without reports, and 16.25 for the fixed-channel control; the five-seed benchmark suite reports HB mean SHD 14.84 and AUROC .925.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 8

4.3 External reports in a physical system

• At N = 1000 with random IDs and physical descriptions retained, HB lowers mean SHD from 39.00 to 27.30 and raises AUROC from .738 to .764.The reports cover 8.8% of unordered pairs under IDs; their aggregated direct-edge F1 is .395.Source: §4.3 · p. 8, Table 3 · p. 9

4.4 Reliability learning and knowledge use

• Continued alternation reduces mean pair-reliability error from .240 to .152 and held-out report log loss from .874 to .762 versus one-time estimation.At α = .75, full HB has SHD 2.40 versus 3.83 for an intercept-only reliability prior; at α = 0, the values are 13.40 and 7.53, while no-KG gives 11.10.Source: §4.4 · pp. 8–9, Table 4 · p. 9

4.5 Robustness under knowledge degradation

• HB retains lower mean SHD throughout the tested missing/spurious-report grid, but severe direction reversals and mixed errors exceed no-KG.With 75% independent abstention, gains remain 2.6 on Asia and 1.2 on Sachs; complete abstention recovers no-KG exactly.Source: §4.5 · p. 11, Table 8 · p. 11

4.6 Cross-family portability and continuous inference

• On continuous Sachs, graph-only BGe with reports raises mean AUROC from .702 to .823 and lowers E-SHD from 18.13 to 10.38; joint-inference AUROC gains over DiBS are .012 and .031.Across four ER settings, mean E-SHD is 1.39 to 4.70 lower for HB, with graph-level Holm-adjusted p-values from .25 to .50.Source: §4.6 · p. 12, Table 9 · p. 12

4.7 Computational cost

• On sparse binary graphs with 100 and 125 variables, HB lowers SHD by 47.9 and 55.4 at roughly three times no-KG runtime, using .202 and .254 GiB resident memory.The added cost reflects repeated graph updates; the report layer remains usable beyond one hundred variables.Source: §4.7 · p. 12, Table 10 · p. 12

• HB-NoisyKG uses a common observation likelihood for repeated causal reports and combines report evidence with discrete or continuous data in graph-only or joint inference.Report influence is estimated during Bayesian graph inference.Source: §5 · p. 13
• Matched benchmark comparisons and the physical light-tunnel task demonstrate structural gains over data-only inference.The conclusion also reports sparse recovery at 125 variables.Source: §5 · p. 13
• Extending the interface to richer graph classes requires corresponding pair states and a suitable graph posterior.The conclusion also says report coverage and source quality determine available information, and completely incorrect reports remain challenging.Source: §5 · p. 13

• When pairwise rewards are zero, the graph update reduces to the corresponding data-only Bayesian posterior.With every report abstaining, the rewards are zero; the explicit joint family likewise yields data-only Bayesian posteriors.Source: A Proofs and conditional bounds · p. 16
• Adding a state-independent constant to each pair reward leaves graph rankings unchanged.Each unordered pair contributes one latent state under every graph, so the added terms do not distinguish graphs.Source: A Proofs and conditional bounds · p. 16
• Under positive Beta and off-diagonal confusion parameters and finite non-abstain report counts, every centered reward is finite and bounded for each pair.The proof concludes that each pair has a finite bound on its centered reward.Source: A Proofs and conditional bounds · pp. 16–17
• The posterior variance of pair reliability is O(1/Rij), giving a posterior standard-deviation rate of O(1/√Rij).The variance bound is at most 1/[4(aij + bij + Rij + 1)].Source: A Proofs and conditional bounds · p. 17
• The conditional finite-sample bound separates graph-marginal error, report-sampling error, and prior bias, and recovers consistency as ϵij → 0 and Rij → ∞ for fixed priors.The bound is stated for a pair with conditionally i.i.d. reports and a graph marginal satisfying the specified concentration condition.Source: A Proofs and conditional bounds · p. 18
• Correlated reports provide less information; a concentration inequality can use effective sample size under a specified mixing model, while general dependence needs its own assumptions.The Beta update counts matches and mismatches regardless of the distribution among wrong labels; heterogeneous off-diagonal errors affect joint inference through the graph posterior.Source: A Proofs and conditional bounds · p. 18
• Exact block maximization, or accepted improving inexact updates, cannot decrease the variational objective, whose values converge if it is bounded above.This conditional ascent argument does not itself establish parameter convergence or graph recovery, which require additional conditions.Source: A Proofs and conditional bounds · p. 18
• Top-K and SVGD approximate the graph block, and repeated inaccurate reports can support an incorrect graph explanation when observational evidence is weak.Approximate graph updates may fail to improve the objective at every iteration; iteration limits and numerical or cycle monitoring are practical diagnostics.Source: A Proofs and conditional bounds · p. 18

B Implementation details

• Prompts can target immediate parent edges, directed-path existence, or nonzero total effects, which are distinct from direct-edge claims.The benchmark prompt requests causal candidate edges; direct-parent prompts exclude relations supported only by mediators.Source: B Implementation details · pp. 18–19
• Pair features summarize report confidence, coverage, agreement, and abstentions; expected report-match counts update pair reliability under a Beta prior.The implementation also uses non-abstain report counts and pair-state marginals in the expected match count.Source: B Implementation details · p. 19
• ABSTAIN marks missing evidence, while NONE, FWD, and REV contribute to reliability, error-matrix, and reward updates.The shared error matrix has a zero diagonal and updated off-diagonal rows.Source: B Implementation details · p. 19
• A warm start blends data-only pair-state marginals with agreement-weighted report frequencies before the first reliability update.Unreported pairs retain their data-only marginal; empirical-Bayes updates then estimate the reliability hyperparameters.Source: B Implementation details · p. 19
• Graph inference uses Top-K search or DiBS-SVGD, with BDeu scoring for discrete data and BGe scoring for continuous graph-only inference.Study-specific budgets and early-stopping rules are listed separately; graph-only HB and its matched no-KG control use corresponding Top-K searches.Source: B Implementation details · pp. 20–21, Table 12 · p. 20
• In one Sachs joint-particle diagnostic, effective sample size 1 made edge marginals equal the highest-weight graph, with both E-SHD and point SHD equal to 16.The other seven particles had zero numerical mixture weight.Source: B Implementation details · p. 21

C Knowledge graph quality analysis

• Directed-edge report precision ranged from .41 to .73 and recall from .36 to 1.00, yielding both useful edges and incorrect claims.A true positive required both the correct parent-child pair and the correct direction.Source: Table 13 · p. 22, C Knowledge graph quality analysis · p. 22

C.1 Prompt semantics and observation targets

• Target-matched F1 was 1.000 for direct-parent prompts, 1.000 for path-existence prompts, and .945 for total-effect prompts.The total-effect prompt selected five reachable pairs with zero total effect as positives despite cancellation.Source: Table 14 · p. 22, §C.1 · p. 22
• For direct-graph SHD from the same path reports, paired tests found direct-state and path-aware HB differences inconclusive.The comparison paired four source batches with twenty data seeds.Source: §C.1 · p. 22, Table 15 · p. 23

C.2 Variable names and source quality

• SHD reductions over no-KG were 2.38 on Asia, 5.55 on Sachs, 9.60 on Child, 7.96 on Insurance, and 14.50 on Alarm.Removing canonical names lowered inferred reliability on every network; some anonymous or pseudo-scientific conditions reproduced no-KG, while incorrect anonymous Sachs reports increased SHD by .51 (p = .444).Source: §C.2 · p. 23, Table 18 · p. 24

D Source comparison across LLM providers

• Across providers, within-network standard deviations of mean inferred reliability were .049 on Asia, .027 on Sachs, and .003 on Child.These summaries describe the fitted report layer; calibration against known generating accuracy is assessed separately.Source: Table 19 · p. 24, D Source comparison across LLM providers · p. 24

E.1 Pair-specific and pooled reliability

• Pair-specific reliability has a small equal-network SHD advantage, but pooling yields higher mean AUROC and lower Brier score, and no corrected test establishes a pair-specific advantage.In benchmark diagnostics, correct reports receive mean inferred reliability .24 higher than incorrect reports; reliability correlates .87, .85, and .86 with reported confidence, existence agreement, and directional agreement.Source: E Additional discrete diagnostics and… · p. 25, Figure 3 · p. 26, §E.1 · p. 27

E.2 Independently learned common reliability

• Full HB has lower mean SHD than independently learned common reliability on Insurance and Alarm after correction, but at α = 0 its SHD is higher by +9.2 on Asia and +18.6 on Sachs.Common reliability has lower equal-network mean SHD than pair-intercept; the α = 0 generator independently samples incorrect labels, so repeated labels can provide exclusion information.Source: §E.2 · p. 29, Table 28 · p. 30, Table 29 · p. 30

E.3 Alternating inference and report-feature controls

• Full alternation has lower aggregate reliability MAE and held-out report log loss than one-time estimation, though at α = .75 one-time estimation has lower MAE and both log losses round to .777.The intercept-only prior has lower SHD at α = 0, while the feature prior has lower SHD at α = .75; disjoint-R = 5 and reused-R = 10 each use ten reports allocated differently.Source: Table 30 · p. 30, §E.3 · p. 31, Table 31 · p. 31

E.4 Using fully accurate reports

• 20.1933 mean SHD reduction: HB improves on every network with fully accurate positive-edge reports, achieving approximately 93% of the frozen channel’s SHD gain.The frozen channel at w = .99 reduces SHD by a further 1.62, and hard use by 4.2133.Source: §E.4 · p. 31, Table 32 · p. 31

E.5 Accuracy over the full range and the number of reports

• As report count rises, reliability calibration slopes approach one and MAE declines; at α = 0, HB-minus-no-KG SHD gaps shrink from +5.0 to +1.8 on Asia and +10.0 to +1.2 on Sachs.Across the full accuracy sweep, inferred reliability increases strictly with generating accuracy, with fitted slopes below one indicating shrinkage at the tested report count.Source: §E.5 · pp. 31–32, Table 33 · p. 32

E.7 Abstention and selective refusal

• At 75% abstention on Sachs, selective refusal yields −5.8 SHD difference while adverse refusal yields +1.3; at 100%, all runs match no-KG on SHD and E-SHD.The shared-B model also retains mean SHD gains in all four tested heterogeneous and correlated error conditions, including their combination.Source: §E.6 · p. 32, Table 34 · p. 32, §E.7 · p. 33

E.8 Increasing the search budget

• 4.36 and 3.93 SHD reductions: raising the fixed-channel budget helps on Insurance and Alarm; no-KG improves by 4.70 on Child.These are corrected-significant budget effects; the supplied passages do not state the corresponding HB-versus-control results.Source: §E.8 · p. 34, Table 36 · p. 34

F Procedurally generated synthetic ER checks

• Harmonized-Prior rewards path existence using shared confidence, but even its best swept setting falls below the no-KG baseline on the 11 asserted Sachs edges in graph-only inference.A wrong-direction claim can be rewarded by a detouring chain; separate procedural ER studies examine graph-only and joint inference on generated structures to avoid LLM benchmark familiarity.Source: F Procedurally generated synthetic ER… · p. 34
• HB-NoisyKG remains competitive overall and is stronger on the denser ER4-D20 family in two d=20 graph-only checks.The checks use linear-Gaussian SEM data and controlled noisy KGs; Table 38 reports mean ± std over five repeats, but its supplied caption does not provide cell values.Source: F Procedurally generated synthetic ER… · p. 34, Table 38 · p. 35
• The paired ER study compares HB with DiBS-joint across linear-Gaussian and nonlinear MLP SEMs, but computation and base priors differ from no-KG.It uses d ∈{20, 30}, expected degree 2, N = 500, report accuracies α ∈{.60, .75, .90}, and R = 5 rounds; Table 39 reports lower-is-better structural-recovery metrics, with outcomes not supplied here.Source: F Procedurally generated synthetic ER… · p. 35, Table 39 · p. 35

F.1 Graph-level analysis and comparison methods

• NOTEARS and GOLEM achieve lower SHD than the tested HB and DiBS joint-inference configurations on the linear generators.The comparisons cover 75 conditions based on 25 observational datasets and five graph draws.Source: §F.1 · p. 36, Table 42 · p. 36
• Mean learned reliability rises monotonically with generating accuracy, approximately following .71α + .10 at d = 20 and .74α + .08 at d = 30.In this ER study, reported confidence equals α, so both labels and confidence change across conditions.Source: §F.1 · p. 36, Table 43 · p. 36
• As report accuracy rises from .60 to .90, mean HB SHD also rises from 19.36 to 21.40 at d = 20 and from 43.84 to 46.80 at d = 30.The passage cautions that reliability ordering and graph recovery capture different aspects of inference.Source: §F.1 · p. 36
• 4.44–5.08 at d = 20 and .80–2.76 at d = 30 are the nonlinear E-SHD reductions favoring HB at each tested accuracy.Each accuracy-specific comparison averages seeds within graphs before the graph-level sign-flip test.Source: §F.1 · p. 36, Table 44 · p. 37
• 3.57 lower mean E-SHD than DiBS is reported for the linear d = 50 extension, with graph-level sign-flip p = .125.The extension uses five graphs, five data seeds per graph, and three report accuracies.Source: §F.1 · p. 36, Table 45 · p. 37

F.2 Joint inference with a larger computational budget

• The larger budget changes the linear HB-minus-DiBS mean E-SHD difference from +6.0 to −3.4 and increases the nonlinear advantage from 4.2 to 5.0.Within HB, mean E-SHD falls from 26.0 to 15.0 on the linear example and from 18.4 to 14.6 on the nonlinear example; these are single-graph comparisons.Source: §F.2 · p. 37, Table 46 · p. 37

F.3 Accurate reports in joint inference

• With fully accurate reports and the stronger joint budget, HB’s E-SHD differences are +.5 and −.3 for the linear and nonlinear examples, while the frozen channel reduces E-SHD by 3.9 and 1.5.The comparison uses ten paired runs per family.Source: §F.3 · p. 38, Table 47 · p. 38

G Empirical recovery of ρ and B under known G

• With G known, one-hot pair-state marginals reduce the reliability and confusion updates to pair-wise Beta and global Dirichlet closed forms.The simulation uses 105 unordered pairs and R ∈ {3, 5, 10}.Source: G Empirical recovery of ρ and B under… · p. 38
• Reliability MAE and B Frobenius error decrease with report-round budget R under both symmetric and asymmetric error matrices.Figure 8 plots the recovery curves with 95% confidence bands over 50 Monte Carlo seeds per (R, B⋆) setting.Source: G Empirical recovery of ρ and B under… · p. 38, Table 48 · p. 38, Figure 8 · p. 39
• This experiment evaluates reliability and shared-error recovery with the graph held known; for uncertain graphs, the finite-sample bound additionally accounts for pair-state marginal error.The known-graph setup provides an empirical reference for Proposition 2.Source: G Empirical recovery of ρ and B under… · p. 38

H Hyperparameter sensitivity

• SHD ranges from 6.3 to 11.3 on Asia and from 11.0 to 17.3 on Sachs across warm strengths, while fixed-strength variation over βvar and τshape gives spreads of at most .7 and 1.0.The warm parameter scales the initial pair-state blend.Source: Table 49 · p. 39, H Hyperparameter sensitivity · p. 40
• At α = 0, mean reliability falls from .373–.506 after the first E-step to .260–.326 at the final update; at α = 1, it rises from .728–.774 to .835–.872.Reliability responds to report quality across the tested warm strengths.Source: H Hyperparameter sensitivity · p. 40
• At α = 0, HB and the fixed-channel control give SHD 10.0 and 21.3 on Asia, and 19.5 and 19.8 on Sachs; both remain above no-KG SHD of 5.5 and 9.1, respectively.At α ≥ .6, the fixed channel has .2–1.4 lower mean SHD, but none of these differences is significant after correction.Source: H Hyperparameter sensitivity · p. 40, Table 50 · p. 40
• Neither conflict-based selection nor two-fold posterior-predictive selection consistently improves performance over fixed λwarm = .5 in the synthetic report setting.The comparison evaluates differences relative to fixed .5.Source: H Hyperparameter sensitivity · p. 40, Table 51 · p. 41

I.1 Runtime and larger graphs

• HB runtime ranges from 0.34 seconds on Asia to 253.49 seconds on Alarm, compared with no-KG times of 0.14 and 55.10 seconds, respectively.Figure 9 uses logarithmic axes; reported log-log slopes are 4.22 for HB and 3.94 for no-KG.Source: §I.1 · p. 42, Figure 9 · p. 42
• 47.9 and 55.4 are the SHD reductions reported for sparse binary d = 100 and d = 125 experiments, respectively.The paired-mean 95% intervals are [−61.3, −34.5] and [−60.7, −50.1].Source: §I.1 · p. 42

I.2 Stability under weak data and inaccurate reports

• The fifty-update stress test spans exact inference, Top-K, and DiBS-SVGD across specified sample sizes, report accuracies, warm strengths, and report conditions.It covers 5,508 settings; exact and Top-K vary report counts, while DiBS uses five reports.Source: §I.2 · p. 42
• The primary dynamical endpoint is numerical failure or persistent period-two behavior; non-stabilization, boundary concentration, and statistical harm are reported separately.The study defines tolerance attainment as three consecutive joint-state changes of at most 10^-3 within the 50-update budget; boundary flags can also mark a correctly concentrated posterior.Source: §I.2 · p. 42, Table 52 · p. 43
• 1656/3240 Exact and 887/2160 Top-K runs trigger performance-loss flags, with approximately 77% occurring at α ≤1/3.At α = .75, both backends improve mean Brier score, but Exact AUROC decreases.Source: §I.2 · p. 42, Table 53 · p. 43
• Residual damping reduces period-two events from two to one but slows tolerance attainment, making it an optional safeguard.Practical monitoring uses an iteration cap and checks for non-finite values and persistent cycles, with data-only inference available as a fallback.Source: §I.2 · p. 43

J Observation models for other graph classes

• Finite-candidate HB fits use five reports per each of six pairs, alternating reliability and shared-error updates with normalization over the full candidate set.The protocol uses a warm strength of .5, at most 20 updates, and a 10^-6 stopping tolerance; no-KG omits report terms.Source: J Observation models for other graph… · p. 44

J.1 Markov equivalence and CPDAGs

• CPDAG pair states distinguish absent and oriented edges from undirected edges when DAGs in an equivalence class disagree on orientation.The study groups four-node DAGs into 185 classes by skeleton and unshielded colliders, and uses a 4 × 4 report-error matrix.Source: §J.1 · p. 44
• HB raises true equivalence-class mass from .889 to .933, .925, and .967, and reduces pair-state error from .045 to .028, .031, and .015.The values correspond to report accuracies .60, .75, and .90, respectively.Source: §J.1 · p. 44, Table 54 · p. 44

J.2 Hidden confounding and cyclic graphs

• The hidden-confounding and cyclic studies test finite four-node candidate sets with pair states that include bidirected edges or allow directed cycles.The cyclic study's support allows directed cycles but not reciprocal edges; its separately reported DAG control uses BGe scoring and Top-K search.Source: §J.2 · p. 45, Table 55 · p. 45
• With compatible cyclic support, HB attains SHD zero at α = 1, while the DAG-only BGe comparison gives E-SHD 2.02.Any DAG must omit at least one edge of the cycle; the main DAG implementation retains its causally sufficient, acyclic scope.Source: §J.2 · p. 45, Table 55 · p. 45

K.1 Data and report elicitation

• The optical task uses 20 described variables from a physical light tunnel, with random-ID prompts retaining descriptions and reports elicited in batches.Inference uses individual report rounds; the reference graph is used only to evaluate inferred structures.Source: §K.1 · pp. 45–46

K.2 Matched inference and evaluation

• All methods use BGe, the same .01 base edge prior, up to 19 parents per node, and four graph calls with 20 search restarts each.Each fit has one data-only initialization and three updates; report-based methods use a .5 warm blend and report-reward scale one.Source: §K.2 · p. 46
• Full HB uses feature-conditioned pair reliability, while the common-reliability control fits one Beta reliability and fixed-reliability HB sets ρ = .8 while learning B.The fixed-reliability comparison holds the likelihood form fixed while replacing inferred reliability with a fixed value.Source: §K.2 · p. 46
• SHD, E-SHD, AUROC, and Brier evaluate thresholded structure, expected adjacency disagreements, edge ranking, and probability error, respectively.Exploratory intervals resample data blocks and report batches separately while preserving method pairing.Source: §K.2 · p. 46
• HB improves all four metric means over no-KG and independent common reliability in each condition.At N = 1000, HB has lower SHD, E-SHD, and Brier than fixed-reliability HB; fixed-reliability HB performs better at N = 250 and has higher AUROC at both sizes.Source: §K.2 · p. 46, Table 57 · p. 47