PaperlayerView exact revision on arXiv

Source-linked AI summary

Multitask Regression with Pairwise Fusion

Xiaodong Li, Zhentao Li

arXiv:2609.27280v1stat.MLcs.LG

TL;DR

Multitask regression must balance noisy separate task fits against bias from forcing tasks to share coefficients, while sharing can vary by predictor. The paper estimates this structure with pairwise fusion and finds that its benefit fades as coefficients become more task-specific; under explicit conditions, it can also pool a fully shared task subset exactly.

  • Problem

    Multitask regression must balance noisy separate task fits against bias from forcing tasks with genuinely different coefficients to share one vector.

    Source: §1 · p. 1

  • Method

    The estimator penalizes pairwise coefficient differences across tasks, with an additional group penalty when predictor selection is needed.

    Source: §1 · p. 1

  • Results

    The benefit of sharing is largest when few coefficients differ across tasks and fades continuously as coefficients become more task-specific.

    Source: §8 · p. 10

  • Takeaways & Limitations

    The household energy analysis shows climate and housing variables with large equality groups across states while some equipment coefficients are more dispersed.

    Source: §8 · p. 10

    Exact pooling assumes exact equality among shared tasks, and the theory does not cover approximate sharing.

    Source: §8 · p. 10

Abstract

from arXiv · show

We study multitask regression when coefficient sharing can differ by predictor. For a given predictor, many tasks may have the same coefficient while a few differ, and the exceptional tasks need not be the same for another predictor. We describe this structure by two quantities: the number of active predictors and the total number of task coefficients that differ from the most common value for their predictor. We estimate the coefficient matrix by penalizing all pairwise coefficient differences across tasks, with an additional group penalty when predictor selection is needed. The resulting upper and lower bounds have the same dependence on these two quantities. We also consider the stronger setting in which a large set of tasks shares one entire coefficient vector. Under explicit sample-size conditions, the same pairwise estimator pools those tasks exactly, while allowing the remaining tasks to differ. Simulations and household energy data illustrate the transition between broad sharing and task-specific coefficients.

1. INTRODUCTION

The paper studies multitask regression between separate task fits and one shared coefficient vector, allowing sharing patterns to differ by predictor. It estimates this structure with pairwise fusion, derives rates governed by active predictors and coefficient exceptions, and studies exact pooling of tasks that share a full vector.

1. INTRODUCTION

  • For each predictor, many tasks can share one coefficient while a few exceptions differ, and the exceptional tasks can vary across predictors.The structure is summarized by s, the number of active predictors, and r, the total number of coefficients differing from their predictor's most common value.Source: §1 · p. 1
  • Pairwise coefficient penalties fuse task coefficients separately for each predictor, without requiring task ordering or external similarity information.An additional group penalty selects predictors when needed.Source: §1 · p. 1
  • The sparse upper and expected-risk minimax lower bounds have the same dependence on the number of active predictors and predictor-specific coefficient exceptions.When coefficients vary freely across tasks, the exception-related term returns to the usual group-sparse multitask scale of order sT.Source: §1 · pp. 1–2
  • Under explicit sample-size conditions, pairwise fusion identifies and exactly pools a large subset of tasks sharing a full coefficient vector without knowing the subset in advance.The shared coefficient's error combines the usual pooled estimation error with a contribution from remaining tasks.Source: §1 · p. 2
  • Simulations examine both sharing settings, while the household energy example shows broad sharing and more dispersed coefficients across different predictors.The paper's focus differs from approaches requiring prespecified task relationships and centers rates on predictor-specific exceptions.Source: §1 · p. 2

2. MODEL AND METHODS

The model allows task-specific designs and predictor coefficients while representing repeated coefficients separately for each predictor. Its estimators use pairwise fusion, with a group penalty when predictor selection is required.

2.1 Multitask Regression

  • The model permits task-specific sample sizes and predictor distributions, with Gaussian designs and independent Gaussian errors under bounded covariance eigenvalues.Intercepts, when included, are treated separately and receive neither penalty.Source: §2.1 · pp. 2–3

2.2 Predictor-Specific Sharing

  • For each predictor, q_j is the smallest number of task coefficients differing from one shared value; the exceptional tasks may differ across predictors.The largest equality group for predictor j has size T − q_j, and total heterogeneity counts exceptional coefficients rather than unequal task pairs.Source: §2.2 · p. 3

2.3 Pairwise Fusion Estimators

  • Sparse pairwise fusion combines a group penalty for predictor selection with pairwise penalties encouraging task coefficients to agree.Setting the group-penalty parameter λ to zero gives Group Lasso.Source: §2.3 · p. 3
  • When predictor selection is unnecessary, pairwise fusion studies coefficient sharing without selection and is also used for exact pooling of a shared task subset.Tasks receive equal weight rather than weights based on their sample sizes.Source: §2.3 · p. 3

3. MINIMAX-OPTIMAL ESTIMATION

The section bounds estimation with at most s active predictors and r coefficient exceptions: pairwise fusion separates these costs, and a matching minimax lower bound shows both are unavoidable in the balanced isotropic setting.

3.1 Upper Bounds for Pairwise Fusion

  • Under Assumption 2.1, sparse pairwise fusion bounds every minimizer uniformly over matrices with at most s active predictors and r exceptions, with probability at least 1−δ.Source: §3.1 · p. 4
  • When each active predictor has about q exceptions (r=sq), the heterogeneity term is sq log(eT/q), reaching order sT when q is order T.Source: §3.1 · p. 4

3.2 Matching Minimax Lower Bound

  • In the balanced isotropic model, sparse upper and lower bounds have the same order, with separate costs for active predictors and coefficient exceptions.The upper bound is high-probability, while the lower bound is for expected risk; matching refers to dependence on the problem parameters.Source: §3 · p. 4, §3.2 · p. 5
  • Partial sharing can be substantially easier than unrestricted multitask regression when r≪sT, but that advantage shrinks as r approaches order sT.Source: §3.2 · p. 5

4. POOLING A SHARED SET OF TASKS

The paper asks whether pairwise fusion can identify a large set of tasks sharing a full coefficient vector without knowing that set. Under stated sample-size conditions, it pools those tasks exactly while permitting departures among the others.

4.1 A Shared Set of Tasks

  • The shared set contains T−q tasks with one full coefficient vector, which may be dense; the remaining q tasks may differ on predictor-specific subsets.The deviations need not be separated from the shared tasks, and the number of affected predictors is denoted s_Δ.Source: §4.1 · pp. 5–6

4.2 Exact Pooling of the Shared Tasks

  • Pairwise fusion recovers exact equality among the shared tasks without being given their identities.The theorem states that every minimizer gives tasks in C the same coefficient vector; outside tasks need not be separated from C.Source: §4.2 · p. 6
  • With comparable task sample sizes, the shared-vector error matches the known-set pooled estimation term, plus an extra contribution from estimating the remaining tasks.The extra term is no larger than the pooled term under the stated condition; when q and s_Δ are bounded, the sufficient conditions allow estimation even when n_min ≪ p.Source: §4.2 · p. 6
  • For an outside task, the error combines shared-vector error with error in that task’s sparse departure.The section also considers the stronger shared-task setting after a preceding theory that permits different exceptional tasks for each predictor.Source: §4 · p. 5, §4.2 · p. 6

5. COMPUTATION

The computation applies pairwise fusion separately to each predictor and adds group shrinkage when predictor selection is used. A sorting-based proximal step avoids explicitly forming every pairwise difference.

5. COMPUTATION

  • FISTA uses adaptive restart and step size η = 0.98/L, with L the largest eigenvalue of the smooth-loss Hessian.Convergence requires both relative iterate change and a normalized proximal fixed-point residual to fall below tolerance.Source: §5 · p. 7
  • Each predictor’s fusion update uses sorting and isotonic regression, followed by restoration of the original coefficient order.The proximal map handles the pairwise fusion penalty without explicitly forming all task pairs.Source: §5 · p. 7
  • The predictor update costs O(T log T), dominated by sorting, rather than explicitly manipulating T(T−1)/2 pairwise differences.When ν > 0, ordinary group shrinkage follows the fusion step; the composition is exact for computation.Source: §5 · p. 7

6. SIMULATIONS

The simulations vary how often active predictor coefficients depart from a shared value, comparing fused estimators with separate, pooled, and shared-support fits. Fusion performs best when exact sharing is common, while complete pooling deteriorates as coefficients differ.

6. SIMULATIONS

  • The experiment moves from full sharing at α = 0 to almost surely distinct active-predictor coefficients at α = 1, with departures selected independently across predictors.The random departure locations are nested across fractions, and designs and noise are held the same across fractions within each repetition.Source: §6 · p. 7
  • 0% departures: Pairwise Fusion has error 0.68 in low dimension and Sparse Pairwise Fusion 0.93 in high dimension; at 100%, their errors are 45.45 and 29.08, respectively.The comparison is total squared Frobenius error; Separate Ridge stays near 46 in low dimension and Group Lasso near 33 in high dimension.Source: §6 · p. 7, Table 1 · p. 8
  • The gain from fusion is largest when most coefficients retain their shared value and shrinks as exact sharing becomes less common.At 100%, where no true equalities remain, any remaining gain comes from finite-sample shrinkage rather than recovering shared coefficients.Source: §6 · pp. 7–8
  • At 100% departures, Pooled Ridge has error 277.27 in low dimension and Pooled Lasso has error 277.25 in high dimension.Complete pooling is competitive at 0% departures but deteriorates rapidly once task coefficients differ.Source: §6 · p. 8, Table 1 · p. 8

7. REAL DATA: HOUSEHOLD ELECTRICITY USE

The survey-weighted fit shows predictor-specific sharing: many state coefficients are tied for some predictors, while equipment coefficients are more dispersed. These fitted associations are conditional and do not establish causal effects or exact population equality.

7. REAL DATA: HOUSEHOLD ELECTRICITY USE

  • 14 of 16 predictors have a majority equality group, whose median largest size is 40 of 51 tasks; seven predictors share a coefficient in at least 80% of tasks.The fit retains all 16 predictors and all 51 state-level tasks.Source: §7 · p. 8
  • Figure 1 colors fitted coefficient departures in log-kWh units per common predictor standard deviation and brackets each predictor’s largest equality-group size.The 2020 RECS analysis uses 18,496 households across 50 states and DC, with 143–1,152 households per task.Source: §7 · p. 8, Figure 1 · p. 9
  • Heating degree days share one coefficient in 50 tasks with Alaska departing, while cooling degree days share one in 49 with Nevada and Utah departing.Floor area has a 40-task equality group; the exceptions are conditional on the other predictors.Source: §7 · p. 9
  • Electric-heating coefficients range from 0.139 in Florida and Mississippi to 0.547 in Montana, and the largest equality group has 11 states.Electric water heating has 25 coefficient groups, with two largest groups of six states each; air conditioning has a 27-task majority group.Source: §7 · p. 9
  • These fitted associations do not establish causal equipment effects or exact population equality, and some states have limited within-state contrasts.The survey weighting and imputation differ from the independent Gaussian design in the theory; observed departures are not claimed to be stable population features.Source: §7 · p. 9

8. DISCUSSION

The discussion considers how active predictors and task-specific coefficient exceptions shape multitask regression, including when pairwise fusion pools tasks and the stated limits on theory and interpretation of RECS equality groups. It also derives design and noise bounds and frames the minimax lower bound in terms of uncertainty about active predictors and exception locations.

8. DISCUSSION

  • The main rate depends on s active predictors and r task coefficients departing from predictor-specific shared values, with sharing benefits fading as more coefficients become task-specific.The sparse upper and lower bounds have a common scale, though the supplied passage does not state its formula.Source: §8 · p. 10
  • Pairwise fusion can exactly pool tasks sharing an entire coefficient vector without knowing which tasks they are, with shared-vector error combining pooled estimation and a controlled contribution from remaining tasks.This stronger-setting result requires explicit sample-size conditions, as described in the paper context.Source: §8 · p. 10
  • The theory uses tuning levels involving s and r, exact pooling assumes exact equality, and the Gaussian independent-task design excludes heavy tails, cross-task dependence, task ordering, and external similarity information.Approximate sharing is not covered.Source: §8 · p. 10
  • In RECS, climate and several housing variables have large equality groups across states, while some equipment coefficients are more dispersed.The analysis retains all predictors and states, and treats fitted equality groups as descriptive because survey weights and limited local contrasts preclude claims of exact population equality.Source: §8 · p. 10

A. PROOFS OF THE UPPER BOUNDS

  • The upper-bound proofs combine optimality, penalty restrictions, and design control to bound coefficients; Pairwise Fusion additionally requires handling its unpenalized common coefficient.The supporting bounds split noise into a penalty-absorbed part and a Frobenius-controlled remainder in coefficient coordinates.Source: A. PROOFS OF THE UPPER BOUNDS · p. 11, §A.1 · p. 14

A.1 Bounds Used in the Upper-Bound Proofs

  • The proof bounds include predictor-sparse errors and errors with a common vector plus a few varying predictors, where task-specific samples control variation and pooled samples control the shared component.The noise bounds split the noise into a part absorbed by penalties and a remainder controlled in Frobenius norm.Source: §A.1 · pp. 13–14

A.2 Proof of Theorem 3.1

  • Theorem 3.1’s proof controls centered task differences before recovering the shared-component error, and its bound holds for every minimizer.The pairwise penalty depends only on centered differences, so the proof separately fits the common component.Source: §A.2 · pp. 14–15

A.3 Proof of Theorem 3.2

  • Theorem 3.2’s proof uses both penalties to convert loss control into a coefficient bound, and the result holds uniformly over all minimizers on the joint design and noise event.Source: §A.3 · p. 15

A.4 Proofs of the Design and Noise Bounds

  • The design and noise bounds establish r log(e p T/r) ≲ λ_T √r and d ≲ ν√s, with events uniform over required supports, including those selected from realized noise.These bounds prove (A.12) and (A.13), respectively.Source: §A.4 · pp. 19–20

B. PROOF OF THE MINIMAX LOWER BOUND

  • The minimax lower bound treats unknown active predictors and unknown exception locations as separate sources of difficulty, bounds each, then combines their rates.The proof presents the theorem argument before the packing details.Source: B. PROOF OF THE MINIMAX LOWER BOUND · p. 20

B.1 Proof of Theorem 3.3

The proof builds lower bounds by packing models that vary active predictors and predictor-specific exception locations. The shared-task result uses a constrained fit and verifies that its equality constraint also holds for unrestricted minimizers.

B.1 Proof of Theorem 3.3

  • The lower-bound argument uses Assouad’s method for unknown active predictors and Fano’s inequality for unknown predictor-specific exception locations.The proof combines the resulting bounds to obtain the theorem’s lower bound.Source: §B.1 · pp. 21–22

B.2 Packing Details

  • Separated packings of predictor sets and task locations provide many candidate exception patterns for the lower-bound construction.The packing details use constant-size subset codes and finite-alphabet codes to control cardinality and separation.Source: §B.2 · p. 23

C. PROOF OF THE SHARED-TASK RESULT

  • For the shared-task result, the proof first constrains tasks in C to agree, then checks optimality conditions to transfer the bound to the original estimator.The proof fixes C, the true parameter, and tuning before sampling; it does not search over C.Source: C. PROOF OF THE SHARED-TASK RESULT · p. 24

C.1 Three Ingredients

  • With positive tuning and at least two tasks in C, the residual condition makes the constrained minimizer an unrestricted minimizer, and every unrestricted minimizer agrees on C.The design-and-noise ingredient assumes m ≥ T/2 and specified sample-size conditions.Source: §C.1 · pp. 24–25

C.2 Proof of Theorem 4.1

The shared-task proof transfers constrained estimation bounds to the unrestricted pairwise estimator by establishing exact agreement on the shared tasks. Supplementary simulations compare finite-sample behavior, but are not designed to verify the theorems’ sufficient sample-size conditions.

C.2 Proof of Theorem 4.1

  • The proof establishes the shared-vector bound for the unrestricted minimizer by removing the equality constraint after bounding the constrained fit.The argument requires the theorem’s stated lower bounds on T and n_min T and does not require similarity between task covariance matrices.Source: §C.2 · p. 27

C.3 Proofs of the Three Ingredients

  • Strict subgradient inequalities for pairs within C force every unrestricted minimizer to agree on that set.The proof’s design bound also states that n_min need not be at least p.Source: §C.3 · pp. 28–29

D. ADDITIONAL SIMULATIONS AND EXPERIMENTAL DETAILS

  • The additional simulations use 30 repetitions with matched data and report Monte Carlo standard errors, but do not test the theorems’ sufficient sample-size conditions.Tuning parameters are selected on independent pilot data and held fixed across reporting repetitions.Source: D. ADDITIONAL SIMULATIONS AND… · p. 30

D.1 Random Departures on the Active Support

  • The random-departure design varies predictor-specific exceptions across seven fractions, with nested masks and a fixed average population SNR of 4.Each active predictor has αT perturbed entries, and the exception count is s min{αT, T−1}.Source: §D.1 · p. 30

D.2 Error on the Shared Tasks

The experiment measures coefficient error on a shared task subset while varying the number of outside tasks and the distance between two coefficient groups. Pairwise Fusion performs best when the groups are sufficiently separated, while pooling all tasks is competitive at small separation.

D.2 Error on the Shared Tasks

  • Each setting fixes C at size T − q, assigns one coefficient vector to C and another to the q outside tasks, and varies their distance Δ.Panel (a) varies q with Δ = 4; panel (b) varies Δ with q = 15.Source: §D.2 · p. 31
  • The methods receive neither C nor either coefficient vector; Separate Ridge uses α = 0.316 selected at (q, Δ) = (15, 4), while other tuning is setting-specific.Selected parameters are evaluated on 30 fresh reporting repetitions.Source: §D.2 · p. 31
  • Pairwise Fusion stays below both alternatives as q rises, with error increasing from 0.414 at q = 10 to 1.717 at q = 25.Pooled-all deteriorates faster because its single coefficient vector averages two well-separated groups.Source: §D.2 · p. 31, Table 2 · p. 32
  • At Δ = 12, Pairwise Fusion and Pooled-all Ridge have errors 1.763 and 7.686, respectively; Pairwise Fusion is already better at Δ = 4.The gap widens at larger separations.Source: §D.2 · p. 31, Table 2 · p. 32
  • At Δ = 0.5, Pooled-all Ridge and Pairwise Fusion have errors 0.040 and 0.042, and Pooled-all remains better through Δ = 2.The two methods are competitive when the groups are close, before pooled-all's merging bias grows.Source: §D.2 · p. 31, Table 2 · p. 32
  • The two groups differ on all p coordinates, and q is nontrivial, so the settings probe a difficult finite-sample regime rather than isolating the asymptotic rate.For the shared-task evaluation, the paired experiment changes only the outside-task vector; Separate Ridge is evaluated on unchanged tasks in C.Source: §D.2 · p. 32

E. RECS ANALYSIS DETAILS

The RECS analysis fits state-specific intercepts and pairwise-fused slopes to household electricity-use data, using a fixed predictor set and survey-weighted validation. The fitted coefficients show majority equality groups for some predictors, but none is fully fused, and the analysis has explicit interpretive limits.

E. RECS ANALYSIS DETAILS

  • The analysis uses 18,496 household records from all 50 states and DC, with log annual KWH as the response and released imputed values retained.Task sizes range from 143 households in Delaware to 1,152 in California; outcomes are not trimmed and states are not removed.Source: E. RECS ANALYSIS DETAILS · p. 32
  • The fixed 16-column design includes heating, water-heating, and air-conditioning indicators; housing-type, income, and construction-period indicators; and four quantitative predictors.Encodings are fixed before fitting, with no interactions or added predictors.Source: E. RECS ANALYSIS DETAILS · p. 32
  • The estimator uses 1,275 equally weighted pairwise differences per predictor, leaves state intercepts unpenalized, and includes no group, lasso, or ridge penalty.State adjacency and Census regions do not enter the estimator; predictors and response use common scaling across states, with weighted state means profiled out.Source: E. RECS ANALYSIS DETAILS · p. 33
  • Five-fold validation selects ρ = 0.2 with mean validation MSE 0.214651; the resulting fit uses λ = 2.4421253128 × 10−5.The validation score is a tuning criterion, not an independent test-error estimate; there is no separate test split.Source: E. RECS ANALYSIS DETAILS · p. 33
  • The fit has 247 numerical exceptions among 816 entries, no completely fused predictor, and 14 predictors with a majority equality group.The median largest equality group has size 40; the display reference is not a unique common effect.Source: E. RECS ANALYSIS DETAILS · p. 33
  • Interpretation is constrained by survey weighting, imputation, the 2020 observation period, and limited within-state variation; the analysis makes no causal claim or survey-design confidence-interval report.Numerical fusion is not treated as verified equality of population coefficients.Source: E. RECS ANALYSIS DETAILS · p. 34
Loading 2609.27280v1…