Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making

1 INTRODUCTION

• Existing approaches address safety or efficiency separately, while selectively queried feedback and model updates make it difficult to learn alignment and keep safety checks valid.The section identifies a need to maintain safety while learning from human feedback to reduce future intervention.Source: §1 · pp. 1–2
• CARE combines model-aware learning from feedback with model-agnostic risk control, aiming for safe, human-aligned decisions with fewer human queries.Its modules include residual rectification, adaptive calibration, escalation, and active acquisition.Source: §1 · p. 2

1.1 RELATED WORK

• Related work includes AI-assisted supervision, model-assisted inference, and conformal calibration, but the cited methods do not directly meet CARE’s combined goal of adaptive learning and per-time-step risk control.The described calibration gap includes history-dependent selection and online model adaptation.Source: §1.1 · p. 3

1.2 PROBLEM SETUP

• The binary-classification setup assumes an i.i.d. stream and correct human labels, treats AI as potentially inaccurate and uncalibrated, and minimizes end-to-end query cost under a risk tolerance.The end-to-end task includes training, validation, calibration, and deployment, with human queries supplying any required labels.Source: §1.2 · pp. 3–4

2 CAREFUL JUDGE: A FRAMEWORK FOR RISK-CONTROLLED ADAPTIVE AI-ASSISTED DECISION MAKING

• The risk decomposition identifies aleatoric ambiguity, excess risk, epistemic uncertainty, and systematic bias as distinct sources that require different responses.The corresponding responses include escalation, calibration, informative query selection, and rectification.Source: §2 · p. 4
• CARE reuses queried labels to rectify AI bias and epistemic uncertainty, calibrate the rectified model, and escalate decisions that remain risky.The rectified decision is accepted only when confidence exceeds the calibrated threshold and the sample is not selected for active acquisition.Source: §2 · p. 4
• CARE is modular: its rectifier, confidence measure, escalation threshold, and acquisition function can be plugged in, with safety and efficiency as shared objectives.Offline checkpoints refit rectification and rebuild calibration; between checkpoints, the system updates sequentially online.Source: §2 · pp. 4–5

3 RESIDUAL GAUSS: LOCALITY-AWARE BAYESIAN RECTIFICATION

• Residual correlations let feedback on one query improve decisions on related inputs, potentially reducing escalation over time.The section motivates modeling both global systematic bias and local epistemic uncertainty in the human–AI residual.Source: §3 · p. 5
• A Gaussian-process prior models the residual, with its mean capturing global bias and its kernel representing correlations that transfer feedback across inputs.The prior can combine a learned kernel with AI-imputed uncertainty, and its parameters are refit at offline checkpoints.Source: §3 · p. 5
• The posterior predictive probability supplies a rectified prediction and confidence measure; online assumed-density filtering updates the residual posterior after acquired feedback.The online update uses moment matching and a rank-one covariance correction.Source: §3 · p. 5
• When the GP is well-specified and the human–AI residual is small in L2(PX), rectified inference enjoys the variance reduction standard in model-assisted inference.The paper states this result under both the model-specification and residual-size conditions.Source: §3 · p. 5

4 CONFIDENT JACK: CALIBRATING ADAPTIVE ESCALATION AND LEARNING

• Confident Jack reuses labels used for rectifier learning to calibrate trust in the evolving model while accounting for how queries were selected.It is designed to control risk at any fixed time, regardless of the underlying AI model or rectifier.Source: §4 · p. 6
• Its confidence-based nonconformity score penalizes confident mistakes more than uncertain ones and is tailored to the escalation rule.The score uses the rectified model’s confidence and prediction.Source: §4 · p. 6
• Weighted cross-conformal calibration uses leave-one-fold-out models and replay-based weights to account for history-dependent acquisition and model improvement.During online operation, calibration uses offline feedback while scores and weights adapt to the current model; new online feedback joins the calibration pool at the next offline checkpoint.Source: §4 · pp. 6–7
• Theorem 1 provides adaptive risk control at any offline checkpoint or online step for any risk tolerance ϵ in (0, 1).Co-evolving particles can make prediction sets less conservative as models improve, yielding fewer escalations.Source: §4 · p. 7

5 EXACTING GATES: SAFETY ESCALATION THRESHOLD ON CONFIDENCE

• The confidence gate tests whether the model is confident enough to reject the opposite label, using a threshold that adapts across time and input space.The threshold is based on the agreeing particles' nonconformity scores and is inflated to account for instability.Source: §5 · p. 7
• At every offline checkpoint or online step, CARE's ensemble risk is bounded by the user-specified tolerance ϵ.The guarantee applies for any ϵ in (0, 1).Source: §5 · p. 7
• The risk-control proof relaxes weight and score calculations to accommodate approximation errors and computational constraints, both offline and online.Source: §5 · p. 7

6 ACTIVE MAX: ACTIVE ACQUISITION FOR LONG-TERM EFFICIENCY

• Active acquisition is kept independent of safety escalation to avoid replaying a policy that depends on a threshold known only after calibration, which could create nested dependencies and exponential computational complexity.It can also prioritize queries that may reduce future escalation frequency.Source: §6 · p. 7
• Active Max targets efficiency by selecting queries that may reduce future escalation frequency, rather than maximizing inference precision.Source: §6 · pp. 7–8
• Actively escalated queries update the rectifier and calibration particles between offline checkpoints, while safety-threshold-only queries resolve current decisions and are used for learning at the next checkpoint.Source: §6 · p. 8

7 EXPERIMENTS

• The evaluation covers collision detection, hate speech detection, robotic tool-use safety, and workplace hazard detection, comparing CARE with three baselines and three ablations.CARE uses GP rectification with an offline–online learning and calibration schedule.Source: §7 · p. 8, Figure 2 · p. 8
• At a 5% risk tolerance on iSafetyBench, CARE escalates 21.7% of queries to humans versus 57.1% for SE, a 62% reduction.Figure 3a plots total human query rate against empirical mean risk.Source: §7 · p. 8, Figure 3 · p. 9
• On ASIMOV, CARE variants achieve higher early-stage efficiency; Residual Gauss adapts faster than PPI, while Confident Jack addresses PPI+SE's data inefficiency.Figure 3b–c tracks learning over rolling windows of 100 samples.Source: §7 · p. 8, Figure 3 · p. 9
• In nuPlan, CARE achieves the largest model-risk reduction in all six regions, with the strongest corrections in R1 and R2, where the original AI struggles.Figure 4 presents successive snapshots and final regional heatmaps on 2,000 held-out scenarios.Source: §7 · p. 8, Figure 4 · p. 9
• On HateBench, CARE reduces human queries and rectified-model error across eight black-box AI models, with query savings of 49–66% versus SE and 26–48% versus PPI+SE.The models' raw error rates range from 14.5% to 44.7%.Source: §7 · p. 9
• On iSafetyBench, switching from inferred to direct (AI-provided) confidence increases query rates for SE and PPI variants by 28.0 pp and 11.8+ pp, respectively.CARE gains robustness by also rectifying the confidence measure.Source: §7 · pp. 9–10

8 CONCLUSION

• CARE combines model-aware residual learning with model-agnostic adaptive calibration to improve future predictions while maintaining control of final-system risk.Source: §8 · p. 10

• Ensemble risk counts errors over all processed examples, queried cases add no error, query rate includes preparation and online escalation, and rectified risk evaluates pre-intervention predictions.Source: A EXTENDED EXPERIMENTAL SETUPS AND… · p. 13
• Ten randomized trials pair each data stream across methods, risk targets, and preparation budgets; nuPlan and HateBench also use reserved evaluation panels.The reserved panels are not used for training or calibration and are excluded from query complexity.Source: A EXTENDED EXPERIMENTAL SETUPS AND… · p. 13
• Comparators include unqueried AI, static linear or GP rectifiers with calibrated escalation, frozen static CARE, adaptive CARE, and CARE with a linear rectifier.Source: A EXTENDED EXPERIMENTAL SETUPS AND… · p. 13
• Static methods reserve a stream fraction for preparation; adaptive methods checkpoint after queried-label batches from 32 to 2,048, repeating the final interval.Source: A EXTENDED EXPERIMENTAL SETUPS AND… · p. 13
• Experiments align active escalation with safety escalation and use truncated-horizon sampled replays and shorter EP iterations; reported empirical risk approximates marginal risk guarantees.Source: A EXTENDED EXPERIMENTAL SETUPS AND… · p. 13
• The default operating point uses 20% preparation and a 5% risk target.Source: Table 1 · p. 13

A.1 NUPLAN

• At nuPlan’s primary operating point, CARE achieves 4.36% ensemble risk with 26.6% queries, versus 50.3% for SE, 45.3% for PPI+SE, and 45.8% for static CARE.These correspond to query reductions of 47%, 41%, and 42%, respectively.Source: Figure 5 · p. 14, §A.1 · p. 14, Table 2 · p. 15
• On nuPlan, CARE’s final 1,000-arrival window has 25.6% queries and 8.6% model risk, versus 43.8% and 13.1% in the first 1,000; final-window ensemble risk is 3.7%.Across nuPlan’s six regions, final mean CARE ensemble risks range from 3.0% to 4.5%, below the 5% target.Source: §A.1 · pp. 14–15, Figure 6 · p. 15

A.2 HATEBENCH

• With Perspective at HateBench’s 5% target and 10% static preparation, CARE achieves 3.71% ensemble risk with 21.5% queries, versus 54.0% for SE, 31.8% for PPI+SE, and 28.3% for static CARE.Similar trends hold with other models.Source: Figure 7 · p. 16, §A.2 · p. 16, Table 4 · p. 17
• Across HateBench’s eight detectors, CARE uses 15.6–30.2% queries with mean ensemble risks of 3.71–5.31%, while model-error reductions range from 15.5–70.5%.With Perspective, CARE’s final 500-arrival window has 16.8% queries and 9.4% model risk, compared with 41.4% and 14.3% in the first 500.Source: §A.2 · pp. 16–17, Figure 8 · p. 17, Table 5 · p. 18

A.3 ISAFETYBENCH

• At iSafetyBench’s 5% target and 10% static preparation, CARE achieves 3.79% risk with 21.7% queries using direct confidence, and 2.86% risk with 24.8% queries using inferred confidence.Mean CARE risk is below every tested target under both confidence measures.Source: Figure 9 · p. 18, §A.3 · p. 18
• On iSafetyBench, direct-confidence CARE falls from 25.3% queries and 9.3% model risk in the window ending at 300 clips to 6.9% and 4.8% in the final 100 clips.Across four targets, GP CARE’s absolute mean query-rate change between direct and inferred confidence is at most 3.1 points.Source: §A.3 · pp. 18–19, Figure 10 · p. 19, Table 6 · p. 20

A.4 ASIMOV

• At ASIMOV’s 5% target and 20% static preparation with inferred confidence, CARE achieves 4.76% ensemble risk with 50.8% queries, versus 90.5% for SE, 95.3% for PPI+SE, and 60.3% for static CARE.Mean CARE risk is below every tested target under both confidence measures.Source: Figure 11 · p. 20, §A.4 · p. 20
• On ASIMOV, GP-based CARE queries 47.4% after 500 samples versus 65.6% for CARE with PPI, and its average query rate over the stream is 50.8% versus 57.9%.At the 5% target, changing confidence measure changes CARE’s query rate by −1.5 percentage points, with mean ensemble risks of 4.76% and 4.88%.Source: §A.4 · p. 20, Figure 12 · p. 21

B RISK DECOMPOSITION

• When residual uncertainty is heteroskedastic, the training-conditional predictor can have strictly lower Brier risk by adapting to local uncertainty; with homoskedastic independent residuals, the two predictors have equal risk.The decomposition follows from the law of total variance.Source: B RISK DECOMPOSITION · p. 21

C.1 PRIOR REGRESSION

• CARE places a Gaussian-process prior on human–AI residuals with a linear mean and a kernel mixing learned and AI kernels, fitting prior parameters by marginal likelihood and an EM procedure.The EM E-step computes posterior moments and the M-step increases the expected Gaussian log prior.Source: §C.1 · p. 22

C.2 POSTERIOR INFERENCE

• Posterior inference integrates over the residual process to obtain a posterior process, and uses its posterior mean as the rectified prediction.The process formulas are exact for exact posterior moments; implementation approximates the latent posterior using EP or Laplace moments.Source: §C.2 · p. 22

C.3 EP AND ADF APPROXIMATION

• EP updates Gaussian likelihood sites by moment matching, while online ADF processes each newly queried label once and updates the residual process using conditional moments.Each EP update requires a one-dimensional integral and rank-one covariance correction; when the query variance is zero, the process is unchanged.Source: §C.3 · p. 23

C.4 LAPLACE AND MAP APPROXIMATION

• Laplace approximation uses the MAP residual vector and local curvature for a Gaussian approximation, with online Newton updates and predictions integrated over that process.The online update substitutes its local Gaussian tilted mean and variance into the process update.Source: §C.4 · p. 24

D EXTENDED ANALYSIS OF CALIBRATION

• The nonconformity score is zero when the true label agrees with the prediction and equals model confidence when it disagrees, so confident errors receive higher scores.The construction is tailored to the escalation rule and accommodates model-provided confidence measures.Source: D EXTENDED ANALYSIS OF CALIBRATION · p. 24
• Adaptive data streams make left-out datasets nonexchangeable, so conformity scores must be weighted according to the chance each group supplies the test score.The method conditions on arrival-slot information and considers assignments within a permutation subgroup.Source: STEP 2. WEIGHTED CONFORMITY · p. 25
• The method calculates test-slot weights by considering counterfactual streams under a permutation subgroup where the learning method is permutation-invariant.Smaller subgroups reduce likelihood-calculation effort but can concentrate weights and reduce particle diversity.Source: STEP 2. WEIGHTED CONFORMITY · pp. 25–26, STEP 4. COUNTERFACTUAL LIKELIHOOD · p. 27
• Each sample group’s calibration weight is its probability of occupying the test slot, and that weight determines its contribution to the comparison with the random test group.The weight depends on the test label and input, though the notation suppresses those dependencies.Source: STEP 3. RISK CONTROL · p. 26
• After an offline checkpoint, weights can be adapted online without replay only when offline learning is order-invariant and permutations leave the online stream unchanged.The online particles can then be co-evolved to maintain the relevant counterfactual histories.Source: STEP 4. COUNTERFACTUAL LIKELIHOOD · p. 28

E.1 PROOF OF COROLLARY 1

• The ensemble risk is bounded by prediction-set noncoverage, which is at most ϵ under Theorem 1.The proof reduces the accepted-error event to the event that the true label falls outside the prediction set.Source: §E.1 · pp. 28–29
• Under covariate shift, excluding the opposite label need not imply that the predicted label is in the prediction set, but the risk bound still permits safe acceptance.The stated implication for equal weights does not generally hold under covariate shift.Source: §E.1 · p. 29

E.2 APPROXIMATE WEIGHTS AND SCORES

• If the exact weights and scores lie in a high-probability uncertainty set, choosing the adjusted threshold guarantees excess risk of O(ϵ).The passage gives score and total-variation weight approximation bounds as one example.Source: §E.2 · p. 29

F EXTENDED DISCUSSION OF ACTIVE ACQUISITION

• CARE’s modular design permits plugging in any acquisition strategy from active learning or inference.The discussion separately motivates active acquisition as a way to prioritize queries that may reduce future escalation frequency.Source: F EXTENDED DISCUSSION OF ACTIVE… · p. 29
• Active acquisition targets both short-term overlap with safety escalations and long-term reductions in future escalation, approximating a difficult objective with a smoothed confidence-maximizing rule.The full objective is hard to evaluate because adaptation makes the future difficult to estimate, the safety threshold is unknown, and decision indicators are not smooth.Source: F EXTENDED DISCUSSION OF ACTIVE… · p. 30