PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Task Active Learning with Efficient Resource Allocation

Hanwen Ye, Jiuchen Zhang, Annie Qu

arXiv:2610.07045v1stat.MEstat.ML

TL;DR

Costly auxiliary information can be acquired during labeling but may be unavailable at deployment, creating a missing-by-design prediction problem. ALCATRAs combines cost-adaptive task selection with surrogate learning; theory, simulations, and a UCI heart disease application support improved resource efficiency under the studied settings.

  • Problem

    Auxiliary variables may be selectively acquired during labeling under a budget but are systematically unavailable at prediction time, coupling acquisition, surrogate construction, and prediction.

    Source: §1 · p. 3

  • Method

    ALCATRAs combines a task-selection policy with surrogate learning, using Streak Learning for acquisition decisions and Online-ELBO to correct selection bias in surrogate training.

    Source: §1 · pp. 3–4

  • Results

    A localized excess-risk bound contracts convergence by (N U)^-1/4, while simulations and the UCI heart disease application demonstrate improved resource efficiency.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Under budget constraints, selectively acquired auxiliary information can improve prediction efficiency rather than merely raw accuracy.

    Source: §7 · p. 31

    Gains depend on sufficiently informative auxiliary variables, and the framework assumes known, fixed acquisition costs.

    Source: §7 · p. 31

Abstract

from arXiv · show

Many scientific studies allow costly auxiliary information to be collected during data labeling but not at deployment. Examples include diagnostic tests, laboratory assays, and expert evaluations. We study prediction under this deployment asymmetry, where auxiliary variables are selectively acquired during labeling under a budget constraint but systematically unavailable at prediction time, creating a missing-by-design problem that couples data acquisition, surrogate construction, and prediction. In this work, we introduce Active Learning with Cost-Adaptive Task Resource Allocations (ALCATRAs), a unified framework for selectively acquiring auxiliary information under resource constraints and leveraging that information to improve downstream prediction. ALCATRAs consists of two main components: a task-selection policy which strategically selects a sequence of cost-effective tasks for unlabeled data to perform, and a surrogate learning procedure which transfers knowledge from completed tasks to enhance model predictions. In theory, we show the effectiveness of surrogate models and sample-efficient task policies in improving the model's prediction error bound. Simulation studies and an application to the UCI heart disease cohort demonstrate improved sample efficiency of the proposed ALCATRAs framework relative to baselines under the studied settings.

1 Introduction

ALCATRAs addresses prediction settings where costly auxiliary information can be collected during labeling but is unavailable at deployment. It combines budget-aware task acquisition with surrogate learning, supported by theoretical analysis and studies reporting improved resource efficiency.

1 Introduction

  • Auxiliary measurements may be selectively acquired during labeling under a budget, but are systematically unavailable to the deployed predictor.This missing-by-design setting couples acquisition, surrogate construction, and prediction.Source: §1 · pp. 2–4
  • ALCATRAs jointly formulates auxiliary-task acquisition and transfer of acquired information to a deployment-time predictor as one fixed-budget learning problem.The framework uses a task-selection policy and a model-agnostic surrogate learning procedure.Source: §1 · p. 3
  • Streak Learning targets continuing-optimal task decisions early in acquisition, while Online-ELBO corrects adaptive-selection bias in surrogate learning from partially observed auxiliary outcomes.Streak Learning is presented as avoiding the exponential variance growth of standard importance-weighted off-policy estimation.Source: §1 · p. 4
  • A localized excess-risk bound contracts convergence by (N U)^-1/4, and simulations plus the UCI heart disease application demonstrate improved resource efficiency.Finite-sample analyses of the acquisition rule and variational objective are deferred to supplementary materials.Source: §1 · p. 4

2.1 Notation and Multi-task Setting

  • Tasks have fixed, predetermined costs and are selected sequentially for each instance, without repeating a completed task, until acquisition stops or all tasks are completed.Instance selection uses a given uncertainty scoring function; choosing that function is not the paper's contribution.Source: §2.1 · pp. 5–6

2.2 Related Works: Feature Acquisition and Surrogate Learning

  • Feature-acquisition methods assume measurements remain available at prediction time, whereas existing surrogate methods assume auxiliary outcomes are observed for every labeled subject.ALCATRAs targets the joint setting where auxiliary outcomes are selectively acquired and missing at deployment.Source: §2.2 · pp. 7–8

3 Methodology

ALCATRAs couples budgeted task acquisition with surrogate learning in an iterative active-learning framework. Streak Learning guides task choices, while Online-ELBO adjusts surrogate learning to the evolving acquisition policy.

3.1 ALCATRAs: Cost-Adaptive Task Resource Allocations

  • ALCATRAs selects an unlabeled instance and a cost-effective sequence of supplementary tasks, then updates the predictor and task policy using the new labels and task outcomes.The task policy and surrogate learner are model-agnostic; reward feedback from model information gain guides later task selections.Source: §3.1 · pp. 9–11
  • Surrogate learning imputes task outcomes for unlabeled instances and uses them to produce informative surrogate labels without requiring tasks on the unlabeled population.The target predictor benefits from task information through surrogate labels rather than directly using task outcomes that are missing for unlabeled instances.Source: §3.1 · p. 10, §3.3 · p. 15

3.2 Streak Learning for Task Selections

  • Streak Learning reweights only the decision prefix matching the target policy, trading a small bias for reduced variance compared with full trajectory matching.It also accommodates variable-length task sequences and can stop early when further tasks offer minimal predictive benefit.Source: §3.2 · pp. 12–14

3.3 Surrogate Learning with Online Task Representations

  • Online-ELBO uses inverse-probability weighting to correct acquisition-policy selection bias when learning task representations, then imputes missing outcomes for surrogate labels.Weighting gives greater emphasis to less frequently selected task outcomes, improving reconstruction accuracy and robustness of the learned representation.Source: §3.3 · pp. 14–15
  • Under positivity and sequential ignorability, Online-ELBO is causally unbiased for the population evidence lower bound of the task-outcome distribution.The procedure adapts imputation to the evolving task-selection policy.Source: §3.3 · p. 16

4 Theoretical Results

The theoretical results analyze how surrogate learning bounds target-prediction risk and how Streak Learning improves sequential task-policy estimation. The guarantees rely on stated assumptions, including conditions for localized target realizability and causal regularity.

4.1 Surrogate Generalization and Localized Excess Risk Bound

  • N_U^-1/4 contracts the excess-risk bound beyond labeled-data rates when a stable surrogate and the stated assumptions hold.The bound combines localized function-class complexity, surrogate-learning error, and unlabeled-sample contraction; localization around the surrogate predictor can reduce effective complexity.Source: §4.1 · p. 18

4.2 Sample and Cost Efficiency of Streak Learning

  • Streak Learning is Fisher consistent when its weighting function is positive at every possible deviation depth under the stated causal regularity assumptions.Its Streak Value Function has the optimal policy as its unique maximizer.Source: §4.2 · p. 19
  • Streak Learning’s estimator variance is strictly dominated by standard IPW under the stated assumptions, avoiding probability divisions after the first non-optimal decision.Restricting reweighting to stages before the optimal streak breaks contracts the exponential penalty to the expected deviation depth.Source: §4.2 · pp. 19–20

5 Simulation

The simulations compare ALCATRAs with active-learning baselines under fixed budgets, then examine convergence, acquisition-policy cost efficiency, and task-representation quality.

5 Simulation

  • The simulation evaluates prediction under a fixed budget, using synthetic task features and an oracle task-acquisition priority to compare active-learning frameworks.The setup starts with labeled data whose task features follow a random behavior policy, then queries labels and task features from an oracle unlabeled dataset until the budget is exhausted.Source: §5 · pp. 20–22

5.1 Budget-Constrained Convergence of AL Frameworks

  • Across simulated scenarios, ALCATRAs converges fastest with the least budget and improves initial and overall predictive performance, with larger advantages as task features increase.Its surrogate labels raise initial performance, while cost-effective task selection and adaptive representation learning sustain improvement at low budget consumption.Source: §5.1 · pp. 22–23

5.2 Cost Efficiency of Task Acquisition Policies

  • Streak Learning has the lowest acquisition costs, selects the fewest tasks, and aligns most closely with the optimal task order among evaluated policies.It also maintains more consecutive optimal decisions; competing direct-search and feature-acquisition methods perform worse than random selection in this small-labeled-data setting.Source: §5.2 · pp. 24–25, Table 2 · p. 24

5.3 Adaptive Latent Task Representation Learning Performance

  • Online-ELBO achieves the lowest task-feature reconstruction MSE across studied configurations, with its reduction rate increasing as task count rises and labeled sample size falls.The method incorporates dynamic task-selection probabilities to emphasize underrepresented features and mitigate selection bias.Source: §5.3 · p. 26, Table 3 · p. 26

6 Data Application

The UCI heart-disease application evaluates prediction from routine covariates when costly diagnostic measurements are available during labeling. ALCATRAs improves predictive performance and yields cost-aware diagnostic rankings that the authors frame as interpretability evidence.

6 Data Application

  • ALCATRAs improves classification accuracy and sample efficiency over baselines, using informative task features to produce more accurate surrogate labels.The predictor uses basic covariates alone, without modeling separate belief states after each acquisition step.Source: §6 · p. 28, Figure 4 · p. 29

Interpretation of learned acquisition decisions

  • After accounting for test prices, Streak Learning and Random Forest both rank cholesterol as the most informative and cost-efficient diagnostic; without cost adjustment, Random Forest ranks stress testing highest.The authors describe the alignment with clinical practice as interpretability support, not independent clinical validation, given the small dataset and five diagnostics.Source: Interpretation of learned acquisition… · pp. 29–30, Table 5 · p. 30
  • The learned acquisition decisions concentrate budget on diagnostics with large marginal predictive contribution relative to cost, suggesting potential use in clinical research data-collection design.The application uses 303 UCI heart-disease patients, treating demographics and chest-pain descriptions as no-cost covariates and laboratory measurements as costly task features.Source: §6 · p. 27, Interpretation of learned acquisition… · p. 30

7 Discussion

The discussion presents ALCATRAs as a fixed-budget learning problem that jointly selects auxiliary measurements and transfers their information to deployment-time prediction. It identifies practical scope conditions and possible extensions.

7 Discussion

  • ALCATRAs jointly learns which auxiliary variables to acquire and how to transfer their information to a predictor that will not have them at deployment.The discussion situates the approach alongside semi-supervised learning, two-phase sampling, and missing-by-design observation.Source: §7 · pp. 30–31
  • Streak Learning adapts off-policy evaluation for small labeled samples, while Online-ELBO adjusts variational learning for selection bias in adaptively acquired data.The paper reports guarantees for excess risk, acquisition-rule efficiency, and causal identification of the variational objective.Source: §7 · p. 31
  • The framework depends on informative auxiliary variables and known, fixed acquisition costs; future work could consider individual budgets, confidence-filtered completions, uncertain labels, and costly tool calls.Under-informative auxiliaries reduce ALCATRAs to standard cost-sensitive active learning; cost-estimation error and budget uncertainty are outside scope.Source: §7 · pp. 31–32

9 Data Availability Statement

The paper identifies the UCI heart-disease dataset as publicly available and describes the contents of its online supplement.

9 Data Availability Statement

  • The analyzed UCI Heart Disease dataset is publicly available from the UCI Machine Learning Repository.The passage provides the dataset URL.Source: §9 · p. 32

SUPPLEMENTARY MATERIAL

  • The online supplement contains theorem and corollary proofs, additional regularity assumptions, implementation details, and a parallel FashionMNIST application.Source: SUPPLEMENTARY MATERIAL · p. 32

1 Derivation of Streak Learning estimator

The estimator derivation handles variable-length task trajectories and develops Streak Learning by relaxing full trajectory matching after the first decision error.

1 Derivation of Streak Learning estimator

  • The derivation assumes SUTVA, sequential ignorability, and positivity, and pads trajectories after each individual's stopping stage to handle unequal task counts.After stopping, auxiliary states are null and actions are uniform draws from the action space.Source: §1 · pp. 38–39
  • Standard inverse-propensity weighting remains sample-inefficient, so Streak Learning marks the first decision error with δ and seeks to preserve optimal choices beforehand.The derivation compares this structure with partial matching approaches that allow misalignment at any decision stage.Source: §1 · p. 39
  • After δ, the estimator relaxes matching by retaining K policy-aligned stages among the remaining decisions, then combines values across K using a weighting function ϕ(δ).The derivation assumes the random number of matched stages increases linearly.Source: §1 · p. 40

2 Theoretical proof

The proofs establish theoretical guarantees for the methods, including bounds and supporting results for surrogate learning, as well as consistency, variance, and unbiasedness properties under stated assumptions. They also characterize key mechanisms underlying those guarantees, such as truncation and inverse-propensity weighting.

2.1.1 Lemma 1 (Bartlett’s Localized Concentration Lemma)

  • Bartlett’s localized concentration result links empirical-process convergence to function-class variance, motivating the use of high-quality surrogates to reduce localized variance and tighten generalization bounds.The proof applies the lemma to the squared-loss class over the restricted target-function class.Source: §2.1.1 · pp. 42–43

2.1 Theorem 1: Localized Excess Risk for Surrogate Learning

  • Theorem 1 bounds the target predictor’s localized excess risk with high probability, with the bound depending on VAE reconstruction error, optimal surrogate error, task dimension, and finite capacity and tail constants.The theorem assumes boundedness, metric-entropy growth and stability, surrogate Lipschitz continuity, convexity, and localized target realizability.Source: §2.1.1 · p. 44

2.1.2 Lemma 2 (Lipschitz Property of Risks)

  • Under bounded predictions and labels, the squared-loss class is 4B-Lipschitz in the prediction argument, enabling a contraction-based risk bound.The lemma applies Talagrand’s contraction principle.Source: §2.1.2 · pp. 47–48

2.1.3 Lemma 3 (Target-Surrogate Discrepancy Bound)

  • Lemma 3 establishes a high-probability bound on the target–surrogate discrepancy under Assumptions 4, 5, 6, and 9.The proof evaluates Dudley’s entropy integral over the unlabeled sample using the capacity bound and empirical radius.Source: §2.1.3 · pp. 48–49

2.1.4 Lemma 4 (Completed Imputation Gap Bound)

  • Lemma 4 bounds the completed imputation gap by decomposing it into imputation error and surrogate generalization error.The bound is developed under boundedness, metric-entropy growth, and surrogate Lipschitz assumptions.Source: §2.1.4 · p. 49

2.2.1 Theorem2: Fisher Consistency of Streak Learning

  • Under Assumptions 1–3 and strictly positive weighting, the optimal Streak Learning policy uniquely maximizes the Streak Value Function.The proof establishes the result for any target policy that deviates from the optimal policy at any stage.Source: §2.2.1 · pp. 51–52

2.2.2 Theorem3: Variance Boundedness and Sample Efficiency

  • Under Assumptions 4 and 1–3, Streak Learning has strictly lower variance than standard IPW for any positive weighting function.For a deterministic target policy, the Streak importance-weighted sum truncates at the first stage where its action differs from the observed trajectory.Source: §2.2.2 · pp. 52–54

2.3 Causal Unbiasedness of Online-ELBO

  • Under Assumptions 1–3, Online-ELBO is causally unbiased for the population Evidence Lower Bound of the overall task-outcome distribution.Inverse-propensity weighting corrects the bias of naive variational training under adaptive task acquisition; the policy probabilities cancel in expectation.Source: §2.3 · pp. 55–56

3 Implementation and Algorithm

Algorithm 1 selects uncertain instances, acquires task outcomes, and iteratively updates the surrogate, target predictor, and task-selection policy until the budget is exhausted.

3 Implementation and Algorithm

  • Each iteration selects the most uncertain unlabeled instance, queries its label and selected task outcomes, and updates the labeled data and remaining budget.The loop returns the updated labeled set and final predictive model.Source: §3 · p. 57
  • The surrogate procedure imputes unobserved task features and uses them with observed task information to update the surrogate and target predictive models.Both target and surrogate models are parameterized as MLPs in the implementation.Source: §3 · pp. 56–57
  • The implementation updates variational parameters, surrogate parameters, target-model parameters, and then policy rewards and task-selection policy within each active-learning iteration.The variational model uses an encoder and decoder implemented as MLPs, while task selection uses an LSTM and greedy maximization.Source: §3 · pp. 57–58
  • Hyperparameters are selected by cross-validation on the initial labeled data and remain fixed during subsequent active-acquisition iterations.This choice is described as maximizing data efficiency.Source: §3 · p. 58

4 Closed-form simulation cost model

The simulation cost model combines base query costs with efficiency costs that penalize deviations from the optimal task sequence.

4 Closed-form simulation cost model

  • The full simulation cost model decomposes into base costs and efficiency costs for the acquired task sequence.The cost model is provided in closed form as a supplement to the verbal description in the main text.Source: §4 · p. 58
  • Base costs include a nonlinear basic query cost and a per-task acquisition cost c_q.The per-task cost is modeled as a uniform draw on [0, 1].Source: §4 · p. 58
  • Efficiency costs penalize acquiring lower-priority tasks before higher-priority ones and deviations in task order, scaled by α and β respectively.These components are labeled ranking costs and order costs.Source: §4 · pp. 58–59

5 FashionMNIST Synthetic Analysis

The FashionMNIST analysis uses synthetic patch-revealing tasks to examine ALCATRAs under a budget. ALCATRAs outperforms baselines at every observed budget stage, while its learned patch-selection order varies by item category.

5 FashionMNIST Synthetic Analysis

  • FashionMNIST images are divided into four synthetic patch-revealing tasks, and revealing all four recovers the original image.Figure 1 illustrates sequential selection of Task 1 followed by Task 4.Source: §5 · p. 59, Figure 1 · p. 59
  • ALCATRAs improves over baselines at every observed budget stage, with slower gains early and the best performance thereafter.The policy learns during the early 0%–20% budget phase, then adapts quickly.Source: Figure 2 · p. 60, §5 · p. 60
  • For boots the learned rule tends to reveal bottom-right then bottom-left, while for sandals it prioritizes top-right then bottom-left.These are anecdotal examples; the authors relate the selections to recognizable heels and sandal cut-out designs.Source: §5 · p. 61
Loading 2610.07045v1…