PaperlayerView exact revision on arXiv

Source-linked AI summary

CreateScore: Domain-Theory-Informed Bayesian Routing for LLM-Based CV Screening

Rupsa Roy

arXiv:2610.02972v1cs.AIstat.AP

TL;DR

Whole-query cascades cannot reflect criterion-specific uncertainty, motivating CreateScore, an inspectable Bayesian router for criterion-level decisions. Routed escalation reduced token use and raised exact agreement over local-only evaluation, but its uncertainty signal did not identify local-model errors.

  • Problem

    Whole-query routing cannot represent candidates who are clear on some rubric criteria but uncertain on others, motivating criterion-level uncertainty routing.

    Source: §1.1 · p. 1, §1.3 · p. 2

  • Method

    CreateScore uses a hand-specified Bayesian network to produce criterion-level posterior uncertainty, routes low-uncertainty decisions locally, and escalates uncertain ones using a threshold calibrated on training data and frozen for testing.

    Source: §1.2 · p. 2, §2.3 · p. 5

  • Results

    Routed escalation reduced token use by 65.2% versus reference evaluation of every criterion and raised exact agreement from 32.8% to 42.6%, while the uncertainty signal had AUROC 0.473 for predicting local–reference disagreement.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    CreateScore is supported as an auditable cost-reduction mechanism, not yet as a targeted error detector, and is intended for cohort-level triage rather than autonomous hiring.

    Source: §4.1 · p. 6

    The reference is a single 120B model draw, not human adjudication, so agreement measures consistency with a larger model rather than correctness.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

Large language models (LLMs) can support rubric-based screening of CVs, but applying a high-capability model to every candidate and criterion is costly. We present CreateScore, a domain-theory-informed Bayesian network for criterion-level LLM routing. A hand-specified directed acyclic graph with Dirichlet-multinomial conditional probability tables converts CV evidence into posterior uncertainty; low-uncertainty decisions are resolved by a local 8B model and uncertain ones are escalated to a 120B reference model. The graph is causally motivated, but the system performs standard Bayesian conditioning, not causal inference. The escalation threshold is calibrated on a training fold (target: 70% resolved locally) and then frozen. On 200 synthetic Data Science CVs (139 training and 61 test candidates, five criteria), 77.7% of criterion decisions were resolved locally (237 of 305). Relative to a reference condition in which the 120B model adjudicated every criterion, routed escalation reduced token use by 65.2% and raised exact score agreement from 32.8% (8B alone) to 42.6% (95% CI 31.0-55.1%); at n = 61 the gain was not statistically distinguishable. The uncertainty signal did not, however, identify the decisions on which the 8B model erred: disagreement with the reference was 16.2% among escalated and 19.4% among locally resolved decisions (AUROC 0.47, 95% CI 0.39-0.56), no better than random selection. We also document how an earlier evaluation was invalidated when truncated reasoning-model outputs were silently replaced by local labels, and we recommend safeguards for cascade evaluation. CreateScore is supported as an auditable cost-reduction mechanism, not yet as a targeted error detector, and is not an autonomous hiring system.

1 Introduction

The paper motivates criterion-level routing because query-level cascades cannot represent criterion-specific uncertainty, then evaluates an inspectable Bayesian router for reducing reliance on high-capability models.

1.1 Background

  • Existing cascades generally route whole queries and escalate them wholesale, missing candidates whose rubric criteria differ in certainty.Source: §1.1 · p. 1

1.2 Motivation

  • CreateScore uses a hand-specified Bayesian network to derive inspectable posterior uncertainty for independent criterion-level routing, rather than learning a router from preference data.Its domain assumptions are expressed as priors that can be revised or contested without retraining.Source: §1.2 · p. 2
  • Its graph is causally motivated, but the system performs standard Bayesian conditioning, not causal inference or hierarchical parameter learning.Source: §1.2 · p. 2

1.3 Aim

  • The study asks whether posterior uncertainty identifies local-model disagreements and how routed escalation compares with local-only and all-reference evaluation, testing one fixed-window payload policy.Three alternative payload policies are implemented in the harness, but their corrected re-evaluation is left to future work.Source: §1.3 · p. 2
  • The calls-and-token reduction prediction was supported, but the prediction that uncertainty would concentrate local-model disagreements in escalated decisions was not.Source: §1.3 · p. 3
  • CreateScore is framed as cohort-level triage and auditing, not autonomous hiring; deployment conclusions require human-labelled heterogeneous CVs, fairness analyses, shift robustness, and oversight.Source: §1.3 · p. 3

2. Study Design

The study tests criterion-level Bayesian routing in a controlled CV-screening setup, separating the decision to escalate from the context sent with an escalated decision. It specifies a synthetic corpus, a five-criterion rubric, the routing network, and a three-arm evaluation.

2.1 About the Dataset

  • The controlled design uses synthetic rather than representative CVs to isolate routing behavior from document heterogeneity.The 200 generated documents have regular formatting and coherent career histories, so they do not test real-CV artifacts such as multi-column layouts or contradictory self-reporting.Source: §2 · p. 3, §2.1 · p. 3

2.1.1 Inclusion-Exclusion Flowchart

  • All 200 CVs met the five-criterion extraction eligibility rule, and the fixed stratified split produced 139 training and 61 held-out test candidates.Stratification used MH_1, the minimum-hire gate, to reduce the risk of skewed folds; calibration used only the training fold.Source: §2.1.1 · pp. 3–4, Figure 1 · p. 3

2.2 Job Order Payload Design

  • The rubric has five criteria, with MH_1 as a deterministic experience gate and each criterion scored as hit, inferred, or miss.The intermediate inferred category preserves indirect evidence as distinct from explicit support; five criteria also keep conditional-table cells meaningfully supported at n = 139.Source: §2.2 · p. 4, Table 1 · p. 4

2.3 Bayesian DAG Architecture & Pipeline Design

  • A hand-specified Bayesian network uses two latent candidate states and criterion-specific parent sets to estimate posterior outcome uncertainty.Dirichlet-multinomial tables receive training-fold counts added to hand-elicited priors; normalized-entropy confidence sets the routing signal, with a 0.2253 threshold calibrated on training data and frozen for test decisions.Source: §2.3 · pp. 4–5, Figure 2 · p. 4

3.1 Evaluation design

  • The evaluation compares local-only scoring, routed escalation, and reference-model scoring on all five criteria, measuring agreement, routing discrimination, and token use.The routed pipeline applies the escalation-context policy after the routing split; A2–A4 were implemented but appear only in preliminary analyses, and their shared fallback quota is heuristic rather than optimized.Source: §3.1 · p. 5, Figure 3 · p. 5, §2.3 · p. 5

3. Methodology

The methodology fixes the routing mechanism and evaluates one escalation-context policy, using a local model for extraction and a larger reference model for re-adjudication.

3. Methodology

  • The 8B local model extracts criterion decisions, and the 120B reference model re-adjudicates escalated decisions.The network and frozen threshold yield 68 escalations among 305 decisions, identical across context policies.Source: §3 · p. 5
  • The reported evaluation uses the fixed-window slice, policy A1, as its single escalation-context policy.Escalated criteria for a candidate are sent together with a section-based excerpt.Source: §3 · p. 5

4. Analysis & Results

Routed escalation reduced token use and raised exact score agreement relative to local-only evaluation, but the gain was not statistically established. The uncertainty signal did not identify local-model errors better than random selection.

4. Analysis & Results

  • On the 305 test criterion-decisions, 237 (77.7%) were resolved locally and 68 (22.3%) escalated, compared with a 70% local-resolution target calibrated on training data.At candidate level, 44 of 61 required at least one reference call and 17 were resolved entirely locally; a single uncertain criterion triggers escalation.Source: §4 · pp. 5–6

4.1 Results

  • 42.6% exact score agreement with the reference under routing versus 32.8% locally, alongside a 65.2% reduction in token use versus adjudicating every criterion.At n = 61, the agreement gain was not statistically distinguishable.Source: §4.1 · p. 6, Table 3 · p. 6
  • Routed evaluation raised exact agreement to 26 of 61 candidates from 20 of 61 under local-only evaluation, while the paired shift was not significant (p = 0.146).The 95% limits of agreement remained wide, from −10.35 to +11.66 points on the 30-point scale.Source: §4.1 · p. 6, Table 3 · p. 6, §4.2 · p. 7
  • All corrected-evaluation calls returned valid classifications without replacing reference outputs with local labels; reasoning tokens used 84.4% of reference completion tokens.A mean of 387 reasoning tokens per full-rubric call left almost nothing of the 400-token completion limit for the visible answer.Source: §4.1 · p. 6

4.2 Interpretation.

  • AUROC was 0.473 (95% CI 0.392–0.557) for uncertainty predicting disagreement; disagreement was 16.2% among escalated decisions versus 19.4% among locally resolved decisions.The interpretation describes this signal as carrying no detectable information about local-model error; a proposed explanation based on within-stratum predictability is untested.Source: §4.1 · p. 6, Figure 4 · p. 7, §4.2 · p. 7

5. Discussions

CreateScore reduced reference-model use and moved scores toward the reference, but its posterior uncertainty did not identify local-model errors. The discussion limits these findings to a controlled synthetic setting and outlines evaluation safeguards and follow-up tests.

5.1 Inference

  • 77.7% of reference-model calls and 65.2% of reference-model tokens were removed, while escalated answers moved candidate scores toward the reference.The inspectable posterior also makes each routing decision traceable for screening operations that need to explain why a judgment was sent for further review.Source: §5.1 · p. 7
  • The posterior did not direct the expensive resource toward judgments where the local model was least reliable, so high-confidence labels do not establish trustworthy extraction.The authors characterize traceability as distinct from discrimination.Source: §5.1 · p. 7
  • A 400-token completion limit exhausted the reference model's mean 387 reasoning tokens per full-rubric call, silently substituting local labels and invalidating the earlier evaluation.The authors recommend logging finish reasons and failures, treating failed calls as missing, and checking token totals against call counts and payload lengths.Source: §5.1 · p. 7
  • The earlier four-policy payload comparison is unreliable; exploratory score overlap for policies A1, A2, and A4 does not establish that payload design matters little.The fixed-window run recorded 94.5% of corrected input tokens and 91.0% of corrected output tokens; scores coincided for 58 or 59 of 61 candidates.Source: §5.1 · p. 7

5.2 Limitations

  • The evidence comes from 200 synthetic CVs, one simplified five-criterion rubric, and a single reference model; all candidates had extractable evidence, unlike cases with truly absent signals.The authors say deployment would need a distinct not-evaluable state and a policy for it.Source: §5.2 · p. 7
  • The graph was elicited for one job family, and synthetic documents omit noisy upstream extraction; the evaluation also lacks random-routing and alternative-uncertainty comparisons and covers one test fold.The capture analysis is limited to 305 decisions in that fold.Source: §5.2 · pp. 7–8
  • The reference is a single 120B-model draw rather than human adjudication, intervals are wide, and the fixed-threshold savings projection does not match the procedure's 70% resolution-rate retargeting.The metrics also treat a 0–30 score as additive across five unequally weighted criteria, and the frozen threshold assumes train and test distributions match.Source: §5.2 · p. 8

5.3 Future Research Scope

  • Future work proposes within-stratum uncertainty signals, corrected replicated comparisons, structured uncertainty payloads, active updating, richer routing designs, stakes-weighted escalation, and human adjudication.Deployment claims additionally require validation on heterogeneous real CVs with human-adjudicated labels, a second reference model, and subgroup fairness analysis.Source: §5.3 · p. 8

6. Streamlit Application

The companion Streamlit application helps plan runs by scaling measured evaluation rates and costs to a candidate pool. It does not score or route candidates, and its projections inherit the limits of the measurements and assumptions used as inputs.

6. Streamlit Application

  • The planner scales observed escalation rates and per-candidate token costs to estimate escalated criteria and compare reference-token use with an escalate-everything baseline.It accepts a candidate count or uploaded job-order payload and projects a stratified split, test schedule, and wall-clock runtime.Source: §6 · p. 8
  • The application performs no scoring, routing, or posterior computation and does not predict agreement; its adjustable constants come from the corrected evaluation.Its token and timing estimates derive from a single observed run, and escalation rates depend on the corpus and calibrated threshold.Source: §6 · p. 8
  • Figure 5 presents the CreateScore Interactive App interface for planning and study design.The application projects run schedules, escalation counts, timing, and reference-token comparisons.Source: §6 · p. 8, Figure 5 · p. 9

7.2 Statistical construction of the DAG

The DAG construction derives latent variables and posterior confidence from extracted CV signals and training-fold counts, then applies a frozen confidence threshold to route criterion decisions. The accompanying app supports planning with measured rates and costs, while the corrected evaluation documents an earlier harness defect.

7.2 Statistical construction of the DAG

  • SS1 bands extracted experience into junior, mid, or senior, while SS2 deterministically classifies extracted B_2–B_4 evidence as industry-leaning, academically-leaning, or mixed.Neither latent-variable derivation uses a language model or held-out labels.Source: §7.2 · p. 10
  • Table 4 identifies the DAG's parent sets and prior shapes; full pseudocount values are in the accompanying source.The caption does not report the cell values.Source: Table 4 · p. 10
  • Posterior concentration sums hand-elicited Dirichlet pseudocounts and training-fold counts; posterior means are normalized, with confidence defined as one minus normalized entropy.The priors use weakly informative cell totals of roughly two to four and a global strength parameter of 1.0.Source: §7.2 · p. 10
  • The training-fold confidence quantile targets 70% local resolution, yielding a frozen 0.2253 threshold that routes decisions locally at or above the threshold and escalates lower-confidence decisions.The threshold is applied unchanged to the test fold.Source: §7.2 · p. 10
  • A corrected run used a 4,000-token limit, explicit failure recording, and cached call metadata; the earlier 400-token harness silently replaced failed reference outputs with local labels, invalidating its metrics.In a diagnostic replay, seven of the first twelve calls ended in an unparseable length limit; the corrected condition and fixed-window policy were each run once on 61 test candidates.Source: §7.2 · p. 10

7.3 Usage of the Streamlit app

  • The app takes candidate-pool and run settings, then returns a split, batch schedule, time projection, escalation count, tier breakdown, and token comparison.It can infer candidate and criteria counts from an uploaded job-order JSON; launch instructions specify Streamlit and pandas dependencies.Source: §7.3 · p. 10
  • The default timing is about 16 seconds per candidate unthrottled and 34 seconds across the run including rate-limit pauses; both are single observations that vary with conditions.The token projection scales measured reference-model tokens per candidate and is a descriptive planning estimate, not a cost guarantee.Source: §7.3 · pp. 10–11
Loading 2610.02972v1…