PaperlayerView exact revision on arXiv

Source-linked AI summary

MECHVAR: Variance-Guided Mechanism Discrimination for Autonomous Machine Learning Experiment Selection

Yifan Guo

arXiv:2610.01819v1cs.LG

TL;DR

Benchmark gains can have several plausible mechanism-level explanations, so reproducing an improvement alone does not identify why it occurs. MECHVAR selects among finite candidate probes using posterior-weighted response variance; its primary comparisons with EIG are statistically unresolved, while EIG has a small numerical AUC advantage on Digits.

  • Problem

    A reproduced benchmark gain does not distinguish among plausible mechanisms, while autonomous ML workflows need to allocate limited experiments to separate explanations supported by the posterior.

    Source: §1 · p. 1

  • Method

    MECHVAR selects probes by posterior-weighted response variance, exactly proportional to the Box–Hill pairwise-KL criterion under shared Gaussian predictive variance.

    Source: §1 · pp. 1–2

  • Results

    Primary audit comparisons do not statistically distinguish MECHVAR and EIG; EIG has a small numerical AUC advantage on Digits.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Its O(KE) rescoring and pairwise audit provide a lightweight acquisition rule when candidate mechanisms, probes, and the shared-scale approximation are credible.

    Source: §1 · p. 1, §6 · p. 10

    The planner treats the finite mechanism set and response templates as given, and does not represent template-estimation uncertainty, recover from a missing true mechanism, generate hypotheses, or optimize non-myopic designs.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

Benchmark gains are often mechanism-ambiguous: reproducing an improvement does not by itself identify why it occurs. We study finite-library mechanism discrimination, where posterior-weighted candidate mechanisms, executable probes, and a limited experimental budget define a sequential experiment-selection problem. MECHVAR selects the next probe by maximizing the posterior-weighted variance of its predicted responses. Under a shared-Gaussian predictive model, this score is exactly proportional to the classical Box--Hill posterior-weighted pairwise-KL criterion, yet it admits O(KE) vectorized rescoring and a transparent additive audit over mechanism pairs. A local expansion further links the score to expected information gain (EIG) when predicted response separations are small. In a 25-block stress audit, MECHVAR outperforms confirmation-first in several moderate misspecification regimes, while its primary comparisons with EIG remain statistically unresolved. In a held-out Digits loop, normalized mechanism-identification AUC is 0.8975 for MECHVAR, 0.7825 for a score-greedy policy, and 0.9092 for EIG. At K = 100, E = 200, median single-thread full-library scoring is 10.36 microseconds for MECHVAR versus 57.69 ms for six-node quadrature EIG in the recorded environment. MECHVAR therefore provides a lightweight, auditable acquisition rule for finite-library experiment selection when a shared predictive scale is a defensible approximation.

1 Introduction

Benchmark gains can have multiple plausible explanations, and reproducing an improvement alone does not distinguish among them. MECHVAR selects probes by posterior-weighted response variance, which under shared Gaussian predictive variance exactly matches the Box–Hill criterion while enabling efficient rescoring and pairwise attribution.

1 Introduction

  • Reproducing a benchmark gain does not distinguish among explanations such as relational-structure use, effective-capacity changes, or identifier-like shortcuts.Semi-supervised gains can also reflect unlabeled data, augmentation, or optimization side effects.Source: §1 · p. 1
  • Sequential mechanism discrimination allocates a limited experiment budget among executable probes to separate explanations that remain plausible under the posterior.The objective differs from maximizing benchmark accuracy; in the held-out Digits experiment, score-greedy identifies the hidden mechanism less efficiently despite using the same posterior and calibration templates.Source: §1 · p. 1
  • The finite-library loop uses the posterior and response templates to score probes, records pairwise contributions, and reveals only the executed outcome before updating the posterior.The introduction presents this as a narrow acquisition role: MECHVAR neither generates hypotheses nor generates probes.Source: Figure 1 · p. 2, §1 · p. 2
  • The paper formalizes follow-up interventions as finite-library mechanism identification and distinguishes that objective empirically from maximizing the next experiment’s benchmark score.It also evaluates the acquisition rule across forecast and likelihood mismatch, scaling, structured forecast errors, and held-out execution.Source: §1 · p. 2
  • MECHVAR is exactly equivalent to the posterior-weighted pairwise-KL Box–Hill criterion under shared Gaussian predictive variance, while supporting O(KE) full-library rescoring and pairwise attribution.The introduction frames evaluation across forecast and likelihood mismatch, budget and mechanism-set scaling, structured forecast errors, and held-out execution.Source: §1 · pp. 1–2

2 Related Work

MECHVAR builds on Bayesian experimental design and model discrimination, making the classical Box–Hill criterion easier to rescore and audit for a finite probe library. It addresses experiment selection after candidate mechanisms and probes are available, rather than replacing broader design approaches that generate policies or hypotheses.

2 Related Work

  • Under shared-Gaussian predictive uncertainty, MECHVAR exposes an exact variance form of the posterior-weighted pairwise-KL Box–Hill objective, enabling vectorized library rescoring and pairwise attribution.The paper’s contribution is an operational form of the classical criterion, not its replacement.Source: §2 · p. 2
  • In the finite-mechanism setting, EIG is I(H; Y | e) and serves as the information-theoretic comparator to MECHVAR.BALD is described as using mutual information between predictions and model uncertainty.Source: §2 · p. 2
  • MECHVAR rescans an enumerated mechanism-and-probe library as the posterior changes, whereas adaptive-design approaches can learn policies mapping experiment histories to designs.MECHVAR addresses selection once candidate mechanisms and probes are available; it does not generate new hypotheses or experiments.Source: §2 · p. 3

3 Mechanism-Discriminating Experiment Planning

The planner represents a finite set of candidate mechanisms and executable probes, updates beliefs from observed outcomes, and selects probes using posterior-weighted response disagreement. Under a shared predictive variance, that score has the same probe ordering as the Box–Hill criterion and supports pairwise attribution and efficient library scoring.

3.1 Finite-Hypothesis Setting

  • The planner considers a finite mechanism set H and probe library E, with each mechanism predicting a scalar response for each probe; observational scores inform the initial distribution.In the stress audit, an incorrect decoy can appear favorable in the initial ranking, so the prior need not identify the true mechanism; execution is limited to B < E probes.Source: §3.1 · p. 3
  • After selecting a probe and observing its response, the planner updates its posterior over mechanisms.The experimental budget restricts which library probes can be executed.Source: §3.1 · p. 3

3.2 MECHVAR and the Box–Hill Equivalence

  • With predictive variance shared across mechanisms and probes, MECHVAR and the posterior-weighted ordered-pair Box–Hill KL criterion induce the same maximizing probe set.The equivalence requires a positive shared variance; under heteroscedastic uncertainty, KL- and variance-based acquisition can rank probes differently.Source: §3.2 · pp. 3–5
  • For a selected probe, pairwise attribution identifies which mechanism pairs contribute most by combining posterior weight with predicted response separation.In the worked example, one posterior-plausible pair contributes 79.9% of the selected probe’s MDV score.Source: §3.2 · pp. 4–5
  • The complete probe library can be scored jointly in O(KE) arithmetic, with the O(K^2) pairwise expansion needed only when selected-probe attribution is requested.This makes the variance score suitable for vectorized rescoring as the posterior changes.Source: §3.2 · p. 5
  • For sufficiently small response separations, MECHVAR and EIG induce the same ordering for probe pairs with a nonzero MDV gap.The agreement is local: higher-order terms can change the ordering when separations are no longer small compared with σb.Source: §3.2 · p. 5

4 Experimental Evaluation

The evaluation tests mechanism discrimination against alternative policies across misspecification, constrained budgets, larger mechanism libraries, and held-out executable interventions. MECHVAR exceeds confirmation-first in several misspecified conditions and outperforms score-greedy on held-out Digits, while primary comparisons with EIG remain unresolved.

4.1 Baselines and Statistical Protocol

  • The stress audit uses 25 paired blocks, paired Wilcoxon tests with Holm adjustment, and exploratory budget, mechanism-count, and structured-error sweeps.For Digits, each held-out seed is a paired block; its four MECHVAR AUC comparisons form a separate Holm family.Source: §4.1 · p. 6

4.2 Motivating Diagnostics

  • Three motivating diagnostics illustrate how similar benchmark observations can support different mechanism-level interpretations; they motivate, but do not evaluate, the planner.The examples concern a ColoredMNIST-style color swap, a NOTEARS permutation, and a global class permutation in CIFAR-10→SVHN.Source: §4.2 · p. 6

4.3 Robustness Under Misspecification

  • At σpred = 0.5, MECHVAR reaches 94.32% accuracy versus 91.12% for confirmation-first; under tripled likelihood noise, the respective accuracies are 94.72% and 92.08%.Both differences are statistically resolved, while the corresponding MECHVAR–EIG differences remain unresolved; the figure sweeps prediction misspecification and likelihood-noise mismatch.Source: §4.3 · p. 6, Figure 2 · p. 7

4.4 Budget, Scale, and Acquisition Cost

  • MECHVAR accuracy rises from 61.0% at B = 1 to 94.1% at B = 4 and 96.7% at B = 8; at K = 12 and E = 24, it reaches 93.6%.At K = 12, MECHVAR’s 7.6-point advantage over confirmation-first is exploratory, and its difference from EIG remains unresolved.Source: §4.4 · p. 7
  • At K = 100 and E = 200, median full-library scoring takes 10.36 µs for MECHVAR and 57.69 ms for six-node EIG in the recorded environment.The timing covers score computation only, excluding model fitting and probe execution.Source: §4.4 · p. 7, Figure 3 · p. 8

4.5 Held-Out Digits-Shortcut Validation

  • Over budgets 0–4, normalized mechanism-identification AUC is 0.8975 for MECHVAR and 0.7825 for score-greedy.MECHVAR also exceeds confirmation-first by +0.0217 and random by +0.0325; EIG is numerically higher by 0.0117, with the paired difference unresolved.Source: §4.5 · p. 8
  • A leave-one-seed-out replay across five measured intervention-response tables gives macro normalized AUCs of 0.6604 for MECHVAR, 0.6601 for EIG, 0.6586 for confirmation-first, and 0.6260 for random.Because profile labels and the intervention library are fixed before replay, this is a diagnostic of identification within that library.Source: §4.5 · p. 8

5 Discussion

In the held-out Digits loop, score-greedy identifies the hidden mechanism more slowly than MECHVAR and EIG, while the primary comparisons do not establish a winner between MECHVAR and EIG. The discussion distinguishes their computational and diagnostic trade-offs and sets boundaries on the shared-variance, finite-library evaluation.

5 Discussion

  • MECHVAR and EIG are numerically close, but primary audit comparisons do not distinguish them statistically, and the observations establish neither equivalence nor superiority.EIG has a small numerical AUC advantage on Digits.Source: §5 · p. 8
  • By budget B = 2, MECHVAR and EIG reach perfect mechanism identification, while score-greedy identifies the hidden mechanism more slowly.Score-greedy optimizes expected benchmark accuracy.Source: §5 · p. 8, Figure 4 · p. 9
  • MECHVAR uses matrix operations and exposes pairwise score contributions, whereas EIG evaluates a Gaussian-mixture integral for every candidate probe.Source: §5 · p. 9

5.1 Limitations

  • The exact Box–Hill reduction depends on shared predictive variance; MECHVAR and heteroscedastic Box–Hill select the same top probe in 81.2% of states at δ = 0.10 and 25.7% at δ = 0.50.Multi-metric outcomes require an explicit multivariate likelihood or scalarization.Source: §5.1 · p. 9
  • The planner assumes a given finite mechanism set and response templates; it does not model template-estimation uncertainty, recover a missing true mechanism, generate hypotheses, or optimize non-myopic designs.The discussion identifies these as boundaries for deployment in open-ended research agents.Source: §5.1 · p. 9
  • Six-node Gauss–Hermite quadrature matches the 50-node top probe in 99.7% of 1,000 diagnostic states, but the check does not cover every main-experiment posterior state.The 95th-percentile per-state maximum score error is 7.99 × 10−4 nats.Source: §5.1 · p. 9
  • The stress audit is synthetic, Digits uses planted mechanisms, and replay uses an existing intervention library, so the evidence supports finite-library discrimination rather than open-world discovery.Runtime measures score computation in one environment; end-to-end savings depend on model-fitting and probe-execution costs.Source: §5.1 · p. 9

6 Conclusion

The conclusion presents MECHVAR as a response-variance acquisition rule for selecting experiments that separate plausible mechanisms. Its shared-Gaussian reduction preserves Box–Hill probe ordering and supports an auditable, computationally lightweight approach within a finite-library setting.

6 Conclusion

  • Under a shared Gaussian predictive scale, the Box–Hill objective reduces to a response-variance score with the same probe ordering, an additive mechanism-pair audit, and O(KE) full-library evaluation.Source: §6 · p. 10
  • Across the stress audit and held-out Digits loop, MECHVAR is competitive with quadrature EIG while requiring much cheaper score computation in the measured implementation.Source: §6 · p. 10
  • Score-greedy optimizes benchmark performance rather than mechanism identification.Source: §6 · p. 10
  • MECHVAR is presented for settings where candidate mechanisms, probes, and the shared-scale approximation are credible.Source: §6 · p. 10

Reproducibility Statement

The statement documents manuscript-source availability and where evaluation, seed ranges, assumptions, proofs, diagnostics, and runtime measurements are recorded.

Reproducibility Statement

  • The arXiv source archive contains the manuscript source and figures required to compile the paper.Source: Reproducibility Statement · p. 10
  • Section 4 records the evaluation design and inferential unit, while Appendix G records the disjoint Digits calibration and held-out seed ranges.Source: Reproducibility Statement · p. 10
  • Section 3 and Appendix A state assumptions and proofs, and the corresponding experimental and appendix sections document numerical diagnostics and runtime measurements.Source: Reproducibility Statement · p. 10

AI Use Statement

The supplied material includes an AI-use disclosure, a local EIG ordering result, a worked pairwise audit, and diagnostic-study descriptions. These passages document distinct aspects of the paper rather than one shared result.

AI Use Statement

  • Generative AI tools supported language editing, code organization, and implementation; the author reports reviewing and verifying AI-assisted material and taking responsibility for the final content.Source: AI Use Statement · p. 10

A Local EIG Expansion

  • When two probes have a nonzero MDV gap, the leading O(λ^2) term fixes their ordering for sufficiently small λ.Source: A Local EIG Expansion · p. 12

B Worked Pairwise Audit Example

C Complete Diagnostic Case Summary

  • Figure 5 presents shortcut exposure, structural controls, and a semantic class permutation that leaves MSP OOD discrimination nearly unchanged.Source: Figure 5 · p. 12
  • Table 3 summarizes six diagnostic studies motivating mechanism identification; planner performance is evaluated separately in the main experiments.Source: Table 3 · p. 13

D.1 Structured Forecast Misspecification

Structured forecast-error rankings depended on error geometry, while Holm-adjusted MECHVAR–EIG comparisons were unresolved across the tested conditions. The additional diagnostics found close quadrature rankings at 20 nodes but a shared-scale ranking boundary under heteroscedasticity.

D.1 Structured Forecast Misspecification

  • MECHVAR ranked highest under IID Gaussian, probe-shared bias, and low-rank correlated error, while EIG was slightly higher under mechanism-shared bias and heavy-tailed error.The five-model, 20-block sweep was exploratory, with no confirmatory pairwise inference attached to the numerical ordering.Source: §D.1 · p. 14, Table 4 · p. 14

D.2 Multiplicity Control

  • Holm-adjusted MECHVAR–EIG p-values were 1.0 in all nine conditions; MECHVAR–confirmation-first values were 0.00196 at σpred = 0.5 and 0.0276/0.00800 at noise multipliers 2/3.The family comprised five prediction-misspecification levels and four non-duplicate likelihood-misspecification levels.Source: §D.2 · p. 14

E.1 Gauss–Hermite Convergence

  • 1.99 × 10−3 nats was the largest six-node score discrepancy; 20-node quadrature matched the 50-node top probe in all 1,000 states.The score errors were measured against 50-node quadrature.Source: §E.1 · p. 15, Table 5 · p. 15

E.2 Heteroscedastic Sensitivity

  • At δ = 0 the rankings coincide; disagreement grows with scale variation, and strong heteroscedasticity can materially change the selected experiment.MECHVAR ranks probes using response means, compared with the exact posterior-weighted ordered-pair KL score for heteroscedastic Gaussians.Source: §E.2 · p. 15

F Extended Score-Runtime Benchmark

The extended benchmark times single-thread acquisition-score calculations, using six-node Gauss–Hermite quadrature for EIG. Its reported timing ratio is specific to the recorded machine and implementations.

F Extended Score-Runtime Benchmark

  • Median timings used five posterior–template instances per K after warm-up on a five-vCPU KVM environment, with cached quadrature and no JIT compiler or GPU.Python, NumPy, SciPy, and thread settings were recorded for the run.Source: F Extended Score-Runtime Benchmark · p. 15
  • Table 7 reports single-thread acquisition-score timings, with six-node Gauss–Hermite quadrature for EIG; model fitting and probe execution are excluded.The EIG routine vectorizes over probes, quadrature nodes, and mixture components while looping over latent mechanisms.Source: F Extended Score-Runtime Benchmark · p. 15, Table 7 · p. 16

K E MECHVAR (µs) EIG (µs) EIG/MECHVAR

  • The EIG/MECHVAR ratio is a wall-clock measurement on the recorded machine, not a hardware-independent complexity constant.The table compares K, E, MECHVAR time, EIG time, and their ratio.Source: K E MECHVAR (µs) EIG (µs) EIG/MECHVAR · p. 16

G Online Digits-Shortcut Details

In the held-out Digits loop, MECHVAR’s normalized budget-AUC was below EIG’s but above score-greedy and confirmation-first. Retrospective replay found nearly equal macro normalized budget-AUC for MECHVAR and EIG across five intervention profiles.

G Online Digits-Shortcut Details

  • 0.8975 was MECHVAR’s normalized budget-AUC, versus 0.9092 for EIG, 0.8758 for confirmation-first, 0.7825 for score-greedy, and 0.8650 for random.Across 25 paired held-out blocks, MECHVAR’s difference from EIG was −0.0117 and statistically unresolved; it exceeded score-greedy by 0.1150 and confirmation-first by 0.0217.Source: G Online Digits-Shortcut Details · p. 16

H Retrospective Empirical Replay

  • Macro normalized budget-AUC across FB01, FB02, FB04, FB05, and FB06 was 0.6604 for MECHVAR, 0.6601 for EIG, 0.6586 for confirmation-first, and 0.6260 for random.Replay used leave-one-seed-out response profiles and a fixed intervention library, measuring sequential identification within that existing library.Source: H Retrospective Empirical Replay · p. 17, Figure 7 · p. 17

Related papers

Loading 2610.01819v1…