PaperlayerView exact revision on arXiv

Source-linked AI summary

HIFI: A Highly Efficient Framework for Statistical Inference with Foundation Models

Po-Wei Chen, Shan Yu, Chunlin Li

arXiv:2610.34550v1stat.MLcs.LG

TL;DR

Resampling-based inference for neural estimators is flexible but repeated model fitting is costly at foundation-model scale. HIFI separates task adaptation from residual resampling across two independent samples. Across vision and text tasks, it reports accurate point estimates and near-nominal coverage with substantially less training compute than direct resampling.

  • Problem

    Resampling can provide flexible inference for complex neural estimators, but repeated fits are costly at foundation-model scale and do not by themselves control systematic estimation error.

    Source: §1 · p. 1

  • Method

    HIFI adapts a foundation model once on a larger sample, then uses an independent smaller sample for resampling-based fits of lightweight PEFT estimators to the residual.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across vision and text tasks, HIFI achieves accurate point estimation and near-nominal coverage with 79.5–91.9% lower training EFLOPs than direct resampling.

    Source: §7 · pp. 9–10, §5.4 · p. 9

  • Takeaways & Limitations

    The Xenium application demonstrates HIFI’s utility for uncertainty-aware biological research.

    Source: §7 · p. 10

    The experiments focus on LoRA, PEFT effectiveness can be task-dependent—particularly for complex reasoning—and repeated Stage 2 fitting can remain costly for very large models.

    Source: §7 · p. 10

Abstract

from arXiv · show

Foundation models provide powerful representations for unstructured data, such as text and images, and parameter-efficient fine-tuning (PEFT) makes their adaptation to downstream tasks increasingly practical. For neural estimators that lack a tractable sampling distribution, resampling-based procedures offer a flexible route to statistical inference, but they often require hundreds of model fits, making them computationally prohibitive at foundation-model scale. We introduce HIFI, a two-stage fine-tuning framework for statistical inference on parameters such as conditional means. Stage 1 adapts a foundation model once on a larger sample, capturing the complex structure of the data and controlling estimation bias through a single fit. Stage 2 applies a resampling inference procedure to a smaller independent sample, concurrently fitting multiple lightweight PEFT estimators to the residual to quantify estimation variability without relearning the full task. HIFI accommodates a broad class of resampling procedures, including ensemble subsampling and the nonparametric bootstrap. Across vision and text experiments, HIFI scales to pretrained models ranging from tens of millions to more than ten billion parameters while providing accurate point estimates and well-calibrated confidence intervals across diverse architectures, and reduces training compute by up to 91.9% relative to direct resampling. An application to gene-expression inference illustrates the practical value of the framework for uncertainty-aware scientific inference.

1 INTRODUCTION

The introduction motivates confidence intervals for conditional means estimated with foundation models, where resampling is flexible but repeated fitting is costly and does not itself control estimation bias. HIFI addresses both concerns through two-stage adaptation and reports calibrated inference with reduced training computation.

1 INTRODUCTION

  • The inferential target is a confidence interval for the population conditional mean µ0(x) = E(Y | X = x) at a fixed input x, supporting scientific interpretation beyond point prediction.The motivation is to quantify uncertainty associated with an estimate, not only to obtain an accurate point estimate.Source: §1 · p. 1
  • Resampling offers flexible inference for complex neural estimators, but repeated fits are costly at foundation-model scale and do not by themselves eliminate systematic estimation error.Even when only a small parameter subset is updated, each additional fit still requires substantial forward–backward computation.Source: §1 · p. 1

Our contributions

  • HIFI uses a larger-sample, higher-capacity Stage 1 fit to capture task structure and control bias, then resamples lightweight residual PEFT estimators on an independent smaller sample.Batched Stage 2 fitting makes repeated estimators less expensive by sharing the Stage 1 adapted backbone.Source: §1 · p. 2
  • Across vision and text tasks, HIFI provides well-calibrated confidence intervals while reducing training computation by up to 92% at the same total sample size.The introduction also reports an application to gene-expression prediction from H&E images.Source: §1 · p. 2

2 RELATED WORK

Related work spans parameter-efficient adaptation, neural-network uncertainty quantification, frequentist inference, and efficient multi-adapter training. HIFI applies batched PEFT training to resampling-based frequentist inference for a conditional mean, rather than prediction uncertainty alone.

Parameter-efficient adaptation

  • PEFT methods make adaptation of large pretrained models practical, but primarily target downstream prediction rather than frequentist statistical inference.LoRA freezes pretrained weights and learns low-rank updates to selected matrices, reducing trainable parameters and adaptation memory.Source: §2 · p. 2

Uncertainty quantification for neural networks

  • HIFI targets a confidence interval for the population conditional mean at fixed x, unlike conformal methods that construct prediction sets for future responses.Validity for the conditional-mean interval requires estimation bias to be negligible relative to variability, beyond calibration.Source: §2 · p. 2

Frequentist inference for neural networks

  • Resampling methods offer flexible neural-network inference with little model-specific analytical structure, but existing approaches largely use conventional DNNs and parallelization or smaller subsamples for savings.Ensemble subsampling trains on multiple subsamples; nonparametric bootstrap methods refit on resampled datasets.Source: §2 · pp. 2–3

Efficient ensemble and multi-adapter training

  • Prior shared-backbone and multi-adapter methods focus on efficient ensembling, predictive uncertainty, or concurrent adapter training; HIFI brings batched PEFT training to resampling-based frequentist inference.The cited approaches include BatchEnsemble, LoRA ensembles, and concurrent LoRA-job systems.Source: §2 · p. 3

3 HIFI FRAMEWORK

HIFI targets pointwise inference for a conditional mean, separating one task-adaptation fit from repeated residual estimation on an independent sample. The framework combines resampling-based residual intervals with shared-backbone PEFT fits that can be batched.

3 HIFI FRAMEWORK

  • The target is the conditional mean µ0(x), with inference conducted for its transformed link-scale value θ0(x) and mapped back through g^-1.The link function may be identity, logit, or log for continuous, binary, or count outcomes, respectively.Source: §3 · p. 3

3.1 STAGE-1 TASK ADAPTATION

  • Stage 1 adapts the pretrained model once on D1, producing a task-specific predictor and adapted backbone that can use full fine-tuning or higher-capacity PEFT.Its training objective uses a loss whose conditional population risk is minimized at θ0(x).Source: §3.1 · p. 3

3.2 STAGE 2: UNCERTAINTY CALIBRATION

  • Conditional on the Stage-1 fit, Stage 2 estimates residual signal ρ0(x) = θ0(x) − bθ1(x) using B sampled or weighted versions of D2, with weights depending on inference method Iα.Source: §3.2 · p. 4

3.3 HIFI INSTANTIATION WITH LORA AND ESM

  • In the LoRA–ESM instantiation, Stage 2 draws size-r subsets and fits new LoRA factors and residual heads over the fixed Stage 1 matrices.The residual estimates are combined into a link-scale point estimate and confidence interval, then transformed to the response scale as needed.Source: §3.3 · p. 4

3.4 EFFICIENT BATCHED STAGE-2 TRAINING

  • Batched Stage 2 training stacks estimator-specific parameters and heads for GPU tensor operations while preserving each fit’s weights, minibatch, loss, and optimizer state.The batching is algebraically equivalent to separate fitting up to numerical differences; groups are sized according to GPU memory, and the construction also applies to other resampling procedures.Source: §3.4 · p. 5

4 THEORETICAL MOTIVATION

The theoretical motivation analyzes HIFI in a sparse ReLU-DNN setting, comparing its excess-risk order and computational cost with direct resampling. It also states conditions for conditional and unconditional ESM validity.

4 THEORETICAL MOTIVATION

  • The analysis uses a bounded sparse ReLU-DNN setting and derives risk bounds under explicit compatibility and learning assumptions.It also describes conditions for transferring conditional ESM validity to unconditional coverage.Source: §4 · p. 5
  • Under the stated γ-Hölder setting, HIFI and direct resampling have the same asymptotic risk-rate order in r when depth scales comparably.This comparison requires Stage 2 to retain sufficient approximation capacity for the residual target conditional on Stage 1.Source: §4 · p. 5
  • Conditional ESM inference is asymptotically valid when the fixed Stage 1 output leaves a Stage 2 problem satisfying the cited DNN–ESM conditions.Uniform conditional validity over high-probability sets for Stage 1 outputs and targets gives asymptotically nominal unconditional coverage for deterministic target sequences in those sets.Source: §4 · p. 5
  • With the stated parameter and sample-size allocations, HIFI’s training-cost ratio is 3.2/B + 0.05 and its cost is 5.64% of direct resampling at B = 500.The calculation assumes training cost scales as trainable parameters times sample size and comparable depth scaling for the risk order.Source: §4 · p. 6

5 SYNTHETIC EXPERIMENTS

Across vision and text tasks, HIFI provides near-nominal confidence-interval coverage across diverse pretrained backbones. The ablations show that Stage 1 adaptation improves estimation and that, at matched sample size, HIFI improves the statistical–computational trade-off over direct resampling.

5 SYNTHETIC EXPERIMENTS

  • The experiments compare LoRA-based HIFI using ESM and bootstrap with HulC and Bayesian-PEFT across vision regression and text probability estimation.They use three pretrained vision backbones and three multi-billion-parameter language backbones; the methods are evaluated on 100 replications.Source: §5 · p. 6

5.1 TASK 1: REGRESSION FROM SYNTHETIC IMAGES

  • 95% coverage is 0.937–0.955 for HIFI-ESM and 0.938–0.953 for HIFI-Bootstrap across three vision backbones; HIFI-Bootstrap has shorter intervals on all three.HulC coverage ranges from 0.906 to 0.965 and Bayesian-PEFT from 0.764 to 0.993.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 TASK 2: PROBABILITY PREDICTION FROM TEXT

  • 95% coverage is 0.926–0.945 for HIFI-ESM and 0.930–0.970 for HIFI-Bootstrap across three language backbones, versus 0.885–0.932 for HulC and 0.598–0.998 for Bayesian-PEFT.HIFI’s target-level coverage standard deviations are 0.02–0.05 across backbones.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 ABLATION STUDY 1: EFFECT OF STAGE-1 TASK-ADAPTATION

  • At n1 = 8,500, HIFI’s mean interval lengths are 51–57% smaller than corresponding single-stage baselines, with empirical coverage of 92.8–93.7%.As Stage 1 sample size increases, MAB decreases and 95% intervals narrow for both ESM and bootstrap, while coverage moves toward 95%.Source: Figure 2 · p. 8, §5.3 · p. 8

5.4 ABLATION STUDY 2: HIFI VS. DIRECT RESAMPLING AT MATCHED SAMPLE SIZE

  • 79.5–91.9% lower training EFLOPs accompany coverage of 92.8%–95.5% for HIFI versus direct resampling at the same total sample size.HIFI produces shorter ESM intervals across all four backbones; bootstrap intervals are shorter for DINOv2-B/14 and Qwen3-Embedding-8B and comparable for the others.Source: §5.4 · pp. 8–9, Figure 3 · p. 9

6 REAL DATA ANALYSIS

The real-data analysis applies HIFI to gene-expression prediction from H&E images in a two-section breast cancer dataset. HIFI provides point-prediction performance comparable to iStar and adds spatially varying uncertainty estimates.

6 REAL DATA ANALYSIS

  • Without uncertainty quantification, estimation error in spatial expression maps can be mistaken for biological structure and propagate into downstream analyses.Source: §6 · p. 9
  • HIFI with ESM performs comparably to iStar across RMSE, SSIM, and PCC while additionally providing uncertainty quantification.The analysis trains on Section 1 and predicts all 313 measured genes in held-out Section 2.Source: §6 · p. 9
  • For TACSTD2, confidence maps show spatially heterogeneous uncertainty, with some high-expression regions pronounced at the lower endpoint and other regions differing more between endpoints.Bootstrap-based HIFI likewise has point-prediction performance comparable to iStar and provides confidence interval maps.Source: §6 · p. 9

7 DISCUSSION

HIFI separates task adaptation from repeated uncertainty calibration to reduce resampling costs while retaining accurate point estimates and stable, near-nominal coverage. The Xenium application illustrates its use for uncertainty-aware biological research, alongside limitations in scope and scaling.

7 DISCUSSION

  • Figure 4 compares HIFI with ESM and iStar across 313 genes using RMSE, SSIM, and PCC, and shows TACSTD2 point estimates and pointwise confidence intervals.The caption specifies the figure’s scope but does not report which method performs better.Source: Figure 4 · p. 10
  • Across vision and text tasks, HIFI achieves accurate point estimation and near-nominal coverage with substantially less training compute than direct resampling.The Xenium application further demonstrates its utility for uncertainty-aware biological research.Source: §7 · pp. 9–10
  • The experiments focus on LoRA; PEFT effectiveness may depend on the task, and repeated Stage 2 fitting can remain costly for very large models.The authors identify extensions to other PEFT parameterizations, resampling methods, and more efficient or reasoning-oriented models as future directions.Source: §7 · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement describes the materials documented across the main text and appendix for reproducing theoretical and empirical results.

REPRODUCIBILITY STATEMENT

  • The main text specifies the framework, inference procedures, and experimental protocols, while the appendix provides theoretical assumptions and complete proofs.The paper also documents data procedures, model and training configurations, data splits, resampling schemes, evaluation protocols, and supplementary diagnostics and results.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The supplementary material records the authors’ AI use, theoretical conditions and results for HIFI, and implementation details for inference procedures, comparators, and synthetic vision data.

AI USE STATEMENT

  • Generative AI tools assisted with synthetic data, language editing, proof organization and formatting, methodology feedback, implementation, translation, and result interpretation; authors reviewed the work and take responsibility for it.The authors state that AI tools did not develop theoretical models or conceptual frameworks, formulate mathematical claims, supply critical proof ingredients, refine hypotheses, or clean and reformat datasets.Source: AI USE STATEMENT · p. 10

A.1 APPROXIMATION AND EXCESS RISK

  • The excess-risk comparison shows HIFI pays the Stage 1 estimation cost once and uses restricted Stage 2 fits, while retaining the full-class approximation term.The compatibility condition requires the loss in Stage 2 approximation capacity induced by the Stage 1 output to be controlled by Stage 1 estimation error.Source: §A.1 · pp. 14–15

A.2 INFERENTIAL VALIDITY

  • HIFI–ESM validity follows from uniform conditional normal-approximation and variance-consistency conditions, while bootstrap validity requires matching conditional error laws and symmetric limits.The validity argument also requires the chosen resampling method’s regularity conditions; the bootstrap construction assumes the number of independent replicates tends to infinity.Source: §A.2 · pp. 15–16

B IMPLEMENTATION DETAILS

  • The implementation section points to the ESM procedure and provides bootstrap construction and external-comparator details.The supplied comparator description adapts Bayesian neural-network inference to PEFT by sampling over LoRA parameters and a prediction head, with likelihoods differing by task.Source: B IMPLEMENTATION DETAILS · p. 17, §B.2 · p. 17

B.1 BOOTSTRAP

  • For bootstrap inference, the Stage 1 fit stays fixed while independent resamples of D2 generate Stage 2 residual fits that are averaged.Each bootstrap sample contains n2 observations drawn with replacement; the point estimate and interval are transformed to the response scale through g−1.Source: §B.1 · p. 17

B.2 EXTERNAL COMPARATORS

  • Bayesian-PEFT samples approximate posterior parameters, while HulC fits independent LoRA estimators on disjoint data subsets and uses their median as the point estimate.Bayesian-PEFT uses empirical equal-tailed quantiles for approximate credible intervals; HulC uses the Δ=0 construction with the fit count chosen for nominal confidence.Source: §B.2 · p. 17

C.1 SYNTHETIC VISION DATA

  • The synthetic vision task varies object shape, area, color, position, and orientation, and evaluates conditional-mean predictions on 500 fixed target images across 100 replications.Figure 5’s rows show circles, squares, and equilateral triangles.Source: §C.1 · p. 18, Figure 5 · p. 18

C.2 BILINGUAL TEXT DATA GENERATION

The bilingual simulation samples arrival lead times across success-probability strata and expresses them in English or Chinese text with added nuisance variation. In the vision task, HIFI shows stable target-wise calibration and estimates closer to the identity line than the external baselines.

C.2 BILINGUAL TEXT DATA GENERATION

  • The bilingual simulation samples one of 20 equal-width success-probability strata, then generates an English or Chinese arrival description with independently added auxiliary and irrelevant context.Auxiliary flight details are included with probability 0.50 and response-irrelevant context with probability 0.20.Source: §C.2 · p. 19

D MODEL ARCHITECTURE AND TRAINING SETTINGS

  • Stage 1 LoRA updates are merged into the backbone and frozen before Stage 2, which uses fresh LoRA adapters and a residual prediction head with a zero-initialized output layer.The passage reports this setting across both modalities.Source: D MODEL ARCHITECTURE AND TRAINING… · p. 19

D.3 ABLATION STUDY 1

  • The Stage 1 sample-size ablation uses Qwen3-Embedding-8B and varies Stage 1 data over 2,000, 4,500, and 8,500 observations while holding Stage 2 fixed.The single-stage reference uses the Stage 2 architecture and optimization settings with a standard randomly initialized prediction head.Source: §D.3 · p. 20

D.4 ABLATION STUDY 2

  • The second ablation allocates 8,000 training and 500 validation observations to HIFI Stage 1, then 1,000 training and 200 validation observations to Stage 2.Single-stage ESM uses 500 subsamples of size 3,621; single-stage bootstrap uses 400 resamples of size 9,000.Source: §D.4 · p. 20

E.1 TARGET-LEVEL RESULTS FOR THE VISION TASK

  • HIFI-ESM and HIFI-Bootstrap show stable target-wise calibration across three vision backbones, while Bayesian-PEFT under-covers for many ViT-B/32 targets and HulC has heavier lower tails.HIFI-Bootstrap generally has shorter intervals than HIFI-ESM.Source: §E.1 · p. 21
  • Across all three vision backbones, HIFI estimates closely track the identity line, while external baselines show target-specific bias for some targets.The comparison visualizes 50 targets selected across shape classes and true-mean bins.Source: §E.1 · p. 21

E.2 TARGET-LEVEL RESULTS FOR THE TEXT TASK

Across language backbones, HIFI methods show relatively stable target-wise calibration, while the external baselines vary by architecture and target. The additional analyses examine Stage 1 sample size, direct single-stage resampling, computational accounting, and the Xenium gene-expression evaluation.

E.2 TARGET-LEVEL RESULTS FOR THE TEXT TASK

  • HIFI-ESM and HIFI-Bootstrap show relatively stable target-wise calibration across language backbones, whereas HulC and Bayesian-PEFT have target-level under-coverage in some settings.HIFI-Bootstrap generally produces shorter intervals than HIFI-ESM; Bayesian-PEFT's shorter intervals under NV-Embed-v2 coincide with substantial under-coverage.Source: §E.2 · pp. 21–22

E.3 ABLATION STUDY 1

  • With Stage 2 fixed, increasing Stage 1 sample size moves coverage toward nominal levels and reduces mean interval length for ESM and bootstrap.The single-stage baselines have conservative coverage but substantially larger intervals.Source: §E.3 · p. 22

E.4 ABLATION STUDY 2

  • At 90% confidence, direct single-stage resampling remains strongly conservative, while HIFI has coverage closer to nominal and substantially shorter intervals in most settings.At 95%, single-stage coverage is concentrated near 100%; HIFI is closer to nominal, with shorter ESM intervals across all four backbones and method-dependent bootstrap interval lengths.Source: §E.4 · p. 23

E.5 COMPUTATIONAL COST

  • Training compute is reported as mean EFLOPs per replication, including Stage 1 once plus all Stage 2 members for HIFI, or all direct response-fitted members for single-stage configurations.The accounting includes forward and backward computation, loss computation, and AdamW updates.Source: §E.5 · p. 23

F.1 DATA PREPROCESSING AND SPLITTING

  • The Xenium analysis trains on one breast-cancer specimen section and evaluates on a separate held-out section, retaining 313 genes.Predictions are made at 8 µm resolution and aggregated to 16 µm blocks for reporting.Source: §F.1 · p. 24

F.2 MODEL IMPLEMENTATION AND INFERENCE

  • For Xenium, spot-level measurements provide weak supervision while the model retains gene-expression predictions on the finer native superpixel grid.Each pseudo-Visium spot is associated with 37 native superpixels.Source: §F.2 · p. 25

F.3 EVALUATION

  • Xenium point-prediction performance is evaluated on registered 16 µm blocks using RMSE, PCC, and SSIM.The compared observed and predicted values are block-total counts.Source: §F.3 · p. 26

F.4 ADDITIONAL RESULTS AND INTERPRETATION

On the Xenium gene-expression task, HIFI with bootstrap has point-prediction performance comparable to iStar across 313 genes. Additional ESM and bootstrap interval maps illustrate HIFI's uncertainty quantification.

F.4 ADDITIONAL RESULTS AND INTERPRETATION

  • HIFI with bootstrap shows point-prediction performance comparable to iStar across all 313 genes.The comparison uses RMSE, SSIM, and PCC, for which lower RMSE and higher SSIM and PCC indicate better agreement with observed expression.Source: §F.4 · p. 27, Figure 18 · p. 36
  • Additional spatial maps show ESM intervals for KRT7 and GATA3 and bootstrap intervals for KRT7, GATA3, and TACSTD2.The maps display observed expression, point estimates or bootstrap means, and interval endpoints.Source: §F.4 · p. 27, Figure 19 · p. 37, Figure 20 · p. 37
Loading 2610.34550v1…