PaperlayerView exact revision on arXiv

Source-linked AI summary

Guaranteed Adaptive Modality Acquisition: When the Policy Chooses Its Own Calibration Group

Melika Baghi

arXiv:2608.15520v1cs.LG

TL;DR

Adaptive acquisition makes terminal-pattern calibration difficult because the policy itself determines the grouping. RouteCert provides finite-sample certification through terminal-pattern calibration or simultaneous policy–pattern certification, achieving 71.2% answered ECG cases at 7.4% disagreement and 48.8% of full ordinal acquisition cost.

  • Problem

    Adaptive policies make terminal acquisition patterns calibration-dependent, so fixed-group conditional calibration arguments may not provide finite-sample guarantees.

  • Method

    RouteCert certifies adaptive acquisition using terminal-pattern calibration or simultaneous certification of complete policy–pattern pairs.

  • Results

    71.2% of held-out ECG patients were answered at 7.4% disagreement and 48.8% of the prespecified full-acquisition cost, with all three terminals certified.

  • Takeaways & Limitations

    Pattern-wise guarantees can remain valid for adaptive modality acquisition when routing is calibration-blind or policy–pattern pairs are jointly certified.

  • Takeaways & Limitations

    Unrestricted transfer from calibration-independent grouping is invalid, although additional structure or direct simultaneous certification can preserve validity.

Abstract

from arXiv · show

A multimodal system may begin inference holding only some of its inputs and may acquire the rest at a cost. With adaptive acquisition, the policy determines which inputs are ultimately observed, so we state the guarantee conditional on that terminal input pattern. Conditional calibration normally assumes the grouping map is fixed independently of the calibration sample, which policy-induced grouping does not satisfy. We characterize when pattern-conditional guarantees remain valid and give two finite-sample constructions: threshold-free routing with calibration applied at the terminal pattern, and simultaneous certification of complete policy-pattern pairs, which lets calibration data select the deployed policy. A counterexample shows that a guarantee proved for a calibration-independent grouping map need not transfer once the policy makes the terminal group calibration-dependent. We call the resulting method RouteCert. On a clinical electrocardiogram task with a staged, cost-ordered lead protocol, the certified policy answers 71.2% of held-out patients at an observed 7.4% disagreement with the cardiologist's diagnosis at 48.8% of the prespecified ordinal cost of acquiring every stage, and all three acquisition stages carry their own certificate. On masked multimodal benchmarks, certifying pointwise at each terminal pattern holds observed worst-pattern selective risk, measured against the full-information reference decision rather than the true label, at 0.034 where a pooled design reaches 0.145 against a 0.10 cap, at a comparable answered fraction (0.350 vs 0.342); under the budget-matched simultaneous comparison the answered fraction falls to 0.305.

1 Introduction

RouteCert provides finite-sample, pattern-conditional certification for multimodal systems that adaptively acquire costly inputs, addressing the mismatch between policy-induced terminal patterns and fixed-group calibration. It offers threshold-free terminal-pattern calibration and simultaneous policy–pattern certification, including calibration-data policy selection while preserving pattern-specific caps.

  • Motivation: Adaptive acquisition decides per input whether current evidence suffices or whether to pay for additional sources.Applications include powering a second sensor, ordering another assay, or retrieving an unread scan.
  • Problem: The central problem is retaining finite-sample guarantees conditional on terminal acquisition patterns generated by an adaptive policy.Standard split conformal calibration assumes the grouping rule is fixed independently of calibration data, creating an assumption mismatch under policy-induced grouping.
  • Method: RouteCert samples plausible missing-source completions and acquires the source or coalition with the best expected answer-set shrinkage per unit cost until one answer remains.Completions guide acquisition but are never treated as acquired evidence; certification is applied at the pattern the case reaches.
  • Certification framework: Two finite-sample constructions certify adaptive acquisition: calibration at the terminal pattern and simultaneous certification of complete policy–pattern pairs.The simultaneous construction permits deployed-policy selection on the same calibration data while preserving each answered pattern’s cap, at a stated finite-sample cost.

2 Related Work

Related work either addresses missing-modality prediction and dynamic acquisition without answering the two calibration questions, or provides guarantees while fixing the grouping map. RouteCert instead certifies the terminal pattern produced by a selected policy, complementing work that permits calibration-dependent selection with a fixed evaluation group.

  • Closest work: Missing-modality prediction and dynamic modality selection do not address calibration-data policy selection or guarantees conditioned on policy-created groups.These methods return point predictions or select inputs dynamically, but the paper identifies neither as answering its two distinguishing questions.
  • Guarantees with fixed groups: Conformal and selective prediction provide guarantees through fixed grouping maps, including group-conditional, shift-weighted, and risk-controlling extensions.The cited literature also includes abstention methods, but the paper’s central contrast is that their grouping map is fixed.
  • Concurrent work: Concurrent work allows calibration-dependent selection while keeping the evaluation group fixed.Prinster et al. (2026) and Yu and Liu (2026) relax the policy-selection axis rather than the policy-induced-group axis.
  • Concurrent work: BCEA incorporates a fixed visual-evidence acquisition policy into its score and globally recalibrates on post-acquisition outcomes.Its evaluation group remains fixed, making this direction complementary to RouteCert.
  • RouteCert: RouteCert certifies conditional on the terminal pattern produced by the selected policy, treating that pattern as part of the certified object.The paper contrasts this with approaches that hold the evaluation group fixed while moving policy selection.

3 Problem Formulation

The paper formulates adaptive acquisition as a complete policy that chooses what to purchase, when to stop, and whether to answer. It evaluates answers by terminal acquisition pattern, using either a full-information reference decision or the true label, while distinguishing calibration-blind from calibration-aware routing.

  • Problem setup: Each case has M sources, an observed pattern S, missing complement S̄, observed blocks x_S, per-source costs c_m, and budget B.A group A of missing sources has aggregate cost c(A) defined as the sum of its source costs.
  • Problem setup: An acquisition policy repeatedly purchases affordable source groups, updates the observed pattern, stops at S_fin(d, x), and then answers or declines.Thus, the policy is a complete decision rule covering acquisition, stopping, and answering.
  • Evaluation targets: The default target is the full-information model decision y_full, while ground-truth calibration settings instead target the true label.The reference target measures whether a partial-information answer remains stable relative to the decision made with every source present.
  • Evaluation targets: For terminal pattern S, answer rate a_S(d) is the answered fraction and selective risk R_S(d) is the error probability among answered cases.These quantities evaluate performance conditionally on the cases reaching each pattern.
  • Calibration dependence: Calibration-blind policies route using only inputs and models, whereas calibration-aware policies use calibration-derived quantities that can change which cases reach each terminal pattern.This distinction matters because a calibration-estimated cutoff can also determine when acquisition stops.

4 Methodology

RouteCert separates adaptive acquisition from certification: an input- and model-driven loop selects the terminal pattern without consulting calibration, then Phase 2 applies a pattern-specific guarantee. It supports risk-based threshold routing and coverage-based singleton-set routing, with acquisition gains estimated by hypothetical completions and fresh residual sets.

  • Two-phase design: RouteCert has an acquisition loop followed by a single guarantee, with only Phase 2 carrying the formal certification.The residual set is used for routing during Phase 1, not for the guarantee itself.
  • Residual label set: Two disjointly trained models generate K plausible completions for missing blocks and average their predictions into a residual label set.The completion model supplies plausible missing-source values, while the reference model produces the predictions used by the residual set.
  • Certification variants: RouteCert-Risk answers when the emitted class probability clears a validated pattern-specific cutoff, whereas RouteCert-Cov answers only when its conformal prediction set is a singleton.RouteCert-Risk is the deployed default and is used for all headline selective-risk results.
  • Adaptive acquisition: The acquisition policy repeatedly buys the coalition with the greatest expected residual-set reduction per unit cost until ambiguity collapses, no affordable group helps, or the budget is exhausted.Candidate gains are estimated from J hypothetical group versions, with fresh residual sets recomputed from K completions before the true block is revealed.
  • Adaptive acquisition: At M=5, restricting each round to singles and, only when needed, pairs yields 1.3 ms per case rather than 2M actions.This scoring rule is an implementation choice, not part of the guarantee.
  • Terminal-pattern calibration: Calibration is consulted only after stopping, at the terminal pattern, using calibration cases passed through the same loop; acquired blocks are then fixed to their observed values.The loop depends only on inputs, models, and its seeded randomness, while completions marginalize only sources that remain missing.

5 Validity of a Post-Acquisition Guarantee

RouteCert’s post-acquisition guarantees are valid when routing is calibration-independent, enabling pattern-conditional coverage and selective-risk certification. Calibration-dependent policy selection instead requires simultaneous certification of complete policy–pattern pairs, because fixed-map guarantees can fail for policy-induced patterns.

  • Failure under calibration dependence: A calibration-dependent routing map can produce pattern-conditional coverage of 0, so fixed-map guarantees cannot transfer to groups defined by the calibration sample.In the counterexample, cases routed into S are excluded from their own answer sets because routing and answering reuse the same calibration quantile.
  • Pattern-specific selective risk: RouteCert-Risk uses split calibration and one exact Clopper–Pearson test, deploying a pattern cutoff only when its answered-case error bound is at most α.Patterns with insufficient data abstain rather than guess, and this rule underlies the reported selective-risk results.
  • Pattern-conditional validity: Under policy-consistent calibration, terminal-pattern answer sets achieve conditional coverage at level 1 − α for every pattern with positive probability.The guarantee averages over calibration and test draws, with the target defaulting to the full-information reference decision or, under direct calibration, the true label.
  • Validity object: RouteCert’s validity contribution is identifying that calibration methods must certify the policy-induced conditioning group, rather than claiming novelty for Mondrian conditioning, exact binomial validation, or family-wise testing individually.The distinction is between certifying a group fixed independently of grading data and certifying the group realized by the selected policy.
  • Simultaneous policy certification: Simultaneous certification indexes hypotheses by complete policy–pattern pairs, allowing calibration data to select the deployed policy while retaining the α cap for every answered pattern.The policy family must be fixed independently of certification data, and uncertified patterns decline; continuing acquisition creates a different complete policy that must also be certified.

6 Experiments

Experiments evaluate policy-created calibration groups, adaptive acquisition, and conditional risk rather than state-of-the-art performance. RouteCert certifies adaptive policies across terminal patterns, with clinical and multimodal results showing controlled risk but costs from simultaneous certification and thin calibration pools.

  • True-label risk control: 46.7% of MHEALTH-5 cases were answered at 1.0% true-label error, while CMU-MOSEI and IEMOCAP abstained when no cutoff passed the test.Ground-truth calibration after each policy reached its final state provides the strongest guarantee, and every MHEALTH-5 arm remained under the 0.10 cap.
  • Coalition acquisition: +3 to +12 points over the best single source were guaranteed where the needs-a-pair fraction was substantial, even after calibration costs for an extra terminal pattern.The coalition advantage was predicted before deployment from the fraction of cases that some pair resolves but no single source does.
  • PTB-XL clinical acquisition: 71.2% of PTB-XL patients were answered at 7.4% disagreement and 48.8% of prespecified full-acquisition cost, with all three terminals certified.The policy answered 1355/1904 patients at mean ordinal cost 2.442 of 5.0; disagreement was measured against cardiologist-assigned myocardial-infarction labels.
  • Simultaneous certification: 0.350 → 0.305 answered fraction for CMU-MOSEI and 0.203 → 0.188 for IEMOCAP under joint certification, while MHEALTH fell from 0.362 → 0.014.The simultaneous comparison uses a family-wise 90% guarantee and pays for multiple-testing and cutoff-selection corrections, with the thinnest pools suffering most.
  • Adaptive acquisition validation: Every policy’s realized and worst-pattern risk stayed at or below target under actual adaptive acquisition, and coalitions guaranteed more cases than matched single-source implementations.Six policies ran cold-start at budget two, with RouteCert-Risk applied at each reached pattern; an open-loop commit control matched the coalition while a replanning planner guaranteed less.

7 Conclusion and Limitations

RouteCert certifies adaptive modality acquisition when terminal grouping is policy-determined, using calibration-blind routing or jointly certified policy–pattern pairs. Its pattern-wise guarantees trade conditional resolution against calibration size, while evidence remains limited to replayable masking and retrospective single-site ECG simulation.

  • Conclusion: RouteCert certifies policy-determined terminal groups through calibration-blind routing or joint certification of fixed policy–pattern pairs.Both constructions preserve pattern-wise guarantees when the policy determines the terminal group.
  • Conclusion: Pattern-wise guarantees trade conditional resolution against calibration size.The conclusion identifies this tradeoff as a cost of the certification approach.
  • Limitations: Evidence is limited to replayable masking and a retrospective, single-site, class-asymmetric ECG simulation.Naturally missing data and externally validated clinical deployment remain unaddressed.

Appendix

The appendix documents the benchmark suite, assumptions, proofs, algorithms, experiments, and reproducibility configuration, with appendix artifacts using an “S” prefix. It also specifies the computing environment and run counts supporting reported results.

  • Appendix contents: The appendix covers benchmarks and provenance, complete proofs, the deployable acquisition-scoring algorithm, secondary and control experiments, and reproducibility configuration.Appendix sections, tables, figures, and equations use an “S” prefix distinct from the main text.
  • Computing infrastructure: All experiments ran on a Slurm cluster using one node with two Intel Xeon Gold 6226 processors, 4 CPU cores, 32 GB RAM, and one NVIDIA V100.The PTB-XL study used PyTorch 25.10 with Python 3.12; other experiments used Python 3.10 with PyTorch 2.1.0.
  • Computing infrastructure: Verification of the shipped artifacts needs no GPU and under 2 GB of RAM.
  • Run counts: Main-paper Table 1 and per-pattern risk-control tables use one trained model per benchmark evaluated over 50 calibration/evaluation resplits.
  • Run counts: End-to-end acquisition tables and coalition comparisons use five independently retrained seeds, each averaged over 50 resplits.

S1 Full benchmark suite

The benchmark suite comprises four real datasets spanning controlled multimodal masking and a staged clinical electrocardiogram protocol. MHEALTH is evaluated in two configurations for distinct result types.

  • Datasets: Four real datasets form the benchmark suite: CMU-MOSEI, IEMOCAP, MHEALTH, and PTB-XL.CMU-MOSEI, IEMOCAP, and MHEALTH are controlled-masking multimodal benchmarks, while PTB-XL is the staged clinical electrocardiogram protocol.
  • Dataset configurations: MHEALTH uses three sensor packs for risk-control results and five per-sensor streams for coalition results.The five-stream configuration is designated MHEALTH-5.
  • Clinical benchmark: PTB-XL supplies the staged clinical electrocardiogram benchmark used in section S7.It is identified as a headline benchmark in the suite.

S2 Completion-model robustness and the true-label screen

RouteCert’s conformal guarantees remain distribution-free in the completion model, while the residual-set screen is only a routing heuristic. A separate audited true-label layer converts reference-relative guarantees into conditional correctness guarantees and can abstain when the reference is too weak.

  • Completion-model robustness: Conformal coverage and selective-risk validity are distribution-free in qϕ because calibration uses the same completion model, regardless of its quality.Completion quality affects only the heuristic semantic interpretation of L as answers genuinely in play.
  • Completion-model robustness: Mean |L| rises 1.96→3.46 on CMU-MOSEI from K=5 to 50, so the residual-set size is reported separately and never supports the guarantee.The screen counts distinct argmax classes, discards probability mass, and can be inflated by rare draws; it only routes acquisition.
  • True-label screen: An independent labeled audit with a one-sided Clopper–Pearson bound yields a true-label guarantee of ro + ¯εref with probability at least 1 −δ −δ′.The audit point estimate concentrates at rate O(1/√m), and the resulting slack vanishes with audit size.
  • True-label screen: εref ≈0.05 on MHEALTH-5 and εref ≈0.44 on CMU-MOSEI show that reference-relative risk implies near-true-label control only when the reference is strong.On MHEALTH-5, the α-guaranteed reference risk implies a true-label cap near α + 0.05; on CMU-MOSEI, the guarantee reports decision stability rather than correctness.
  • True-label screen: RouteCert answers exactly on patterns that meet practitioner-specified correctness γ, while weak-reference patterns abstain rather than emit an unguaranteed answer.Every pattern clears γ ≈α + 0.05 when ¯εref ≈0.05 on MHEALTH-5, whereas only a loose γ is attainable when ¯εref ≈0.44 on CMU-MOSEI; direct true-label calibration is an alternative.

S3 Assumptions and proofs

RouteCert’s guarantees rely on threshold-free terminal-pattern routing and augmented exchangeability, enabling finite-sample calibration conditional on each acquired-input pattern. Reusing calibration thresholds for routing can invalidate coverage, while pattern-specific calibration yields selective-risk control under stated answer-rate conditions.

  • Calibration procedure: Calibration fits the completion model, reference model, and policy on training data, routes calibration points without calibrated quantities, and calibrates separately within each terminal pattern.RouteCert-Cov uses a conformal threshold τS, whereas RouteCert-Risk uses a validated confidence cutoff λS; these quantities are never interchanged.
  • Exchangeability and coverage: Threshold-free routing makes each terminal pattern a fixed measurable function of the augmented input, auxiliary randomness, costs, and budget, independent of calibration labels and thresholds.Because fitted components and the policy are disjoint from calibration, conditional exchangeability applies within routed patterns, including random pattern counts.
  • Selective-risk corollary: Calibrating at α′ = αamin keeps answered-case risk at most α when every pattern answers with probability at least amin.The bound divides the joint error-and-answer probability by the pattern’s own answer rate, and the answer event is the singleton prediction-set event.
  • Failure mode: Reusing a calibration cutoff for routing can produce realized coverage 0, because conditioning on routing selects scores above the same threshold used for certification.The counterexample establishes the coupling exactly; the acquisition loop contains a milder version whenever test-time routing consults its calibration threshold.
  • Empirical counterexample: 0.888 ± 0.007 marginal coverage occurred for confidence-routed rules versus 0.901 ± 0.007 for threshold-free routing against the 0.90 target.The difference is attributed specifically to reusing the calibration threshold for routing; a confidence policy frozen on an independent split avoids that coupling.

S3.1 Proofs of the four structural results

The proofs establish when adaptive, pattern-conditional calibration is valid: policies must be fixed relative to calibration data or jointly certified, and uncertified patterns must decline rather than redirect cases. They also characterize finite-sample deployment limits and show pair look-ahead can strictly improve resolution over myopic acquisition on synergy-only inputs.

  • Sample splitting: Sample splitting restores validity by fitting the policy on one data split and calibrating on an independent split, but routing data then reduces calibration efficiency.K-fold routing uses the policy fitted without the held-out calibration fold, so the deployed guarantee cannot guide policy learning.
  • Calibration-aware acquisition: Calibration-aware acquisition is valid when a pre-fixed policy family is independent of calibration data, allowing terminal-pattern selective risk to be calibrated directly.For each candidate λ, the policy deterministically maps inputs to purchases, a terminal pattern, and answer-or-decline decisions.
  • Simultaneous certification: Joint certification can select richer policy families without sacrificing validity, but finite certification samples may force abstention on patterns or policy-pattern pairs.The single-cutoff family’s answer-rate loss reflects model restriction, while split selection and certification recovered answer fractions from 0.371 to 0.879 on CMU-MOSEI and 0.487 to 0.780 on IEMOCAP.
  • Deployment semantics: Certified restrictions must decline at uncertified patterns rather than continue purchasing, because redirection changes the population whose risk was certified.If continuation is desired, answer/continue/decline behavior must instead be included in the jointly certified policy family.
  • Finite-sample limits: At least 22 error-free answered cases are required for a Clopper–Pearson bound at δ=0.1 to reach α=0.10, so hard patterns correctly abstain.The deployment rule accepts only when the one-sided upper confidence bound is at most α; the bound is minimized when k=0 errors.
  • Pair look-ahead: At any budget ≥2, pair look-ahead resolves at least p more inputs than greedy acquisition when p is the mass of synergy-only inputs.The policies agree outside the synergy event; greedy resolves none there, whereas look-ahead resolves all of it.

S3.2 The deployed rule

The deployed rule answers a pattern only with a cutoff that passed an exact test on independent data, while data-poor patterns abstain rather than guess.

  • The deployed rule: Each pattern answers only with a cutoff that survived one exact test on data it was not chosen on.Proposition 2 states the deployed rule; its proof is in section S3.
  • The deployed rule: Data-poor patterns abstain rather than guess.

S3.3 The cost of a certificate

The section quantifies the calibration cost of certifying error-free patterns: the required number of cases scales logarithmically with the confidence correction and, for policy grids, with the number of candidate policies. Pointwise certification is cheapest, while family-size and proposition S4 corrections add corresponding logarithmic overhead.

  • Exact certification cost: n ≥ ln(1/δ′)/κ_α cases are necessary and sufficient to certify a pattern answering n calibration cases with no error at level δ′.This follows from the exact binomial test.
  • Exact certification cost: Under a pointwise certificate, the smallest certifiable pattern requires n_min = ⌈ln(1/δ)/κ_α⌉ cases.Corrections dividing δ by a family size F increase this to n_min + ⌈ln F/κ_α⌉.
  • Policy-family corrections: Applying proposition S4 to R × S requires n_min + ⌈ln |S|/κ_α⌉ cases, while a policy grid costs ln |R|/κ_α error-free cases per pattern, up to rounding.The grid incurs this generic cost without sequential correction.

S3.4 A sequential certificate for a scalar-indexed policy family … S8.1 Reproducibility

RouteCert extends finite-sample risk control to adaptive acquisition by certifying terminal patterns or complete policy-pattern pairs, with sequential scalar-family correction reducing calibration cost. Experiments show guaranteed adaptive routing can retain substantial answer rates across multimodal and ECG tasks, while simultaneous control, pattern fragmentation, and calibration-pool requirements determine the practical cost.

  • S3.4 A sequential certificate for a scalar-indexed policy family: 42 error-free answered cases suffice at α = δ = 0.1 for a 20 × 8 scalar-indexed family, versus 71 under generic correction.Sequential testing controls the family-wise error rate over R × S at δ, with correction scaling only with |S| when the family ordering is calibration-independent.
  • S3.5 Continuing instead of declining: 53 additional error-free answered cases per pattern price enlarging the family to policies that may continue buying at every pattern.Continuing is valid when the enlarged complete-policy family is fixed independently of certification; affordability depends on the calibration pool.
  • S4.1 Per-pattern answer rates: 0.99 answered fraction is obtained when nothing is missing and about 0.35 when audio or vision is missing; RouteCert abstains entirely on text-missing cases when validation fails.The guarantee-or-abstain behavior keeps each answered pattern at the risk cap rather than forcing answers in unsupported groups.
  • S4 Error-vs-coverage results and the safety attribution: 0.034 worst-pattern selective risk is achieved by per-pattern guarantees, whereas single global thresholds breach the cap at 0.108 to 0.128.Per-pattern calibration preserves safety under adaptive acquisition because the threshold-free buy signal prevents the policy from selecting its calibration group.
  • S5 Coalition look-ahead at M=5: 0.850±0.014 resolution from targeted pair rescue exceeds 0.786±0.023 from the strongest single-source comparison at M=5.The coalition reaches this result at mean acquisition cost 3.19 versus 2.80, while the single-source policy reaches 0.786 at the four-acquisition budget.
  • S6 Sensitivity, robustness, and ablations: 0.87 pointwise and 0.61 simultaneous answered fractions are retained by missing-count coarsening, compared with 0.51 pointwise and 0.10 under exact per-pattern conditioning at M=5.Coarsening trades granularity for data, reducing fragmentation from exponential pattern counts toward O(M) cells while preserving conditional guarantees.
  • S7 Clinical ECG case study (PTB-XL): 71.2% of held-out ECG patients are answered at 7.4% disagreement and attempted ordinal cost 2.442 of 5.0, with all three adaptive terminals certified.The guarantee concerns agreement with the cardiologist’s diagnostic superclass; acquisition is simulated by withholding simultaneously recorded leads, and costs are ordinal burden tiers.
  • S8 Setup and reproducibility: 50 resplits quantify calibration randomness only, while controlled missingness supplies complete source-and-label calibration pools required to score every terminal pattern.Natural-missingness deployments may lack this pool, and the reported code package does not claim exact regeneration of every number.
Loading 2608.15520v1…