PaperlayerView exact revision on arXiv

Source-linked AI summary

Benign Overfitting under Heterogeneous Input Fusion

Houzhen Liu, Xiaobo Xia

arXiv:2610.09340v1cs.LGstat.ML

TL;DR

The paper asks whether benign overfitting for individual inputs persists when dependent input blocks are fused, and analyzes this under a common population task. It identifies distinct fusion principles for regression and classification, including a protected regression regime and attainable classification interaction patterns.

  • Problem

    Prior benign-overfitting theory largely studies a single input, leaving open how dependence between multiple inputs affects generalization after fusion.

    Source: §1 · pp. 1–2

  • Method

    The paper compares minimum-norm interpolation on two dependent Gaussian input blocks separately and jointly while keeping the underlying population task fixed.

    Source: §1 · p. 2, §2.3 · p. 7

  • Results

    Regression's threshold-invariant full-spectrum covariance certificate is preserved under every compatible positive-semidefinite joint covariance, while classification admits every marginal-to-joint benign/non-benign pattern in the stated model class.

    Source: §1 · p. 2, §7 · p. 23

  • Takeaways & Limitations

    Fusion's effect depends on task-specific joint geometry: regression depends on the full covariance spectrum in its protected regime, while classification depends on surviving signal and nuisance contamination.

    Source: §1 · p. 2, §4 · p. 14, §7 · p. 23

    The regression certificate is sufficient rather than a characterization of all benign tasks; a counterexample to task-level closure lies outside its certified region, within the logarithmic gap between known spectral conditions.

    Source: §3.1 · p. 9, §3.6 · p. 13

Abstract

from arXiv · show

Benign overfitting is extensively studied when learning from a single high-dimensional input, but its behavior under heterogeneous input fusion remains largely unexplored. We study this question for minimum-norm linear interpolation under a heterogeneous Gaussian design, comparing two statistically dependent input blocks with their fusion while holding the underlying population task fixed. For regression, we identify a full-spectrum covariance certificate whose asymptotic status is independent of the cutoff threshold and prove that it is preserved by every positive-semidefinite joint covariance consistent with the two marginals. This protection is sharp, yet it does not extend to all benign regression problems: outside the certified regime, two benign marginals can have a harmful fusion. For one-sparse Gaussian classification, benignity in the regular regime is characterized by the balance between surviving predictive signal and nuisance contamination. Fusion can move these two quantities in opposite directions, and within this model class every marginal-to-joint benign/non-benign pattern is attainable. We further show that the same fused input can have qualitatively different effects on regression and classification. These results establish that benign overfitting under heterogeneous fusion is determined by the joint signal and spectral geometry created by input interaction, rather than by marginal benignity alone.

1 Introduction

This paper asks whether benign overfitting survives when statistically dependent inputs are fused, and studies the question while holding the underlying population task fixed. Its results show that fusion follows different composition principles for regression and classification.

1 Introduction

  • The paper examines whether conclusions about single-input benign overfitting survive fusion, since dependence can reshape both the joint covariance spectrum and predictive-signal placement.The analysis compares either marginal input with their concatenation under the same population task, using minimum-norm interpolation for regression and binary classification.Source: §1 · pp. 1–2

1.1 Related Work

  • Prior work studies benignity for individual inputs and multi-source learning, while this paper focuses on how dependent high-dimensional input blocks jointly change interpolation generalization.Unlike settings that learn features, design fusion architectures, or select sources, the input blocks and their joint distribution are given.Source: §1.1 · pp. 3–4

1.2 Overview of Results

  • Regression has a threshold-invariant full-spectrum certificate preserved by every positive-semidefinite joint covariance with certified marginals, but marginal benignity alone need not compose.For classification in the regular one-sparse regime, benignity balances surviving signal against nuisance contamination, and all marginal-to-joint benign/non-benign patterns are attainable.Source: §1.2 · p. 4
  • The same inputs and latent score can yield different fusion effects across tasks: classification can become benign while regression remains non-benign, or fusion can destroy benign regression while preserving benign classification.These examples show that fusion's effect depends on the task, not only on the input pair.Source: §1.2 · p. 4

1.3 Organization

  • The paper proceeds from the heterogeneous Gaussian model and regression theory to classification, then compares the tasks and illustrates interaction patterns numerically.Proofs are collected in the Appendix.Source: §1.3 · pp. 4–5

2 Preliminaries

The preliminaries define a dependent Gaussian input pair, compare its two marginal inputs with their fusion, and formulate regression and classification from a common latent score. They also specify the minimum-norm interpolators and the asymptotic notions of benign and harmful overfitting.

2.1 Notation

  • The preliminaries establish notation for matrix operations and asymptotic sequence comparisons, with limits taken as sample size n grows.Dimensions, covariances, maps, and signal vectors may vary with n.Source: §2.1 · p. 5
  • For regression, the cutoff is computed from the full covariance spectrum; classification's corresponding nuisance quantities first remove the true signal eigendirection.The setup defines tail mass and tail effective ranks, and the regression cutoff uses a fixed threshold.Source: §2.1 · pp. 5–6

2.2 Heterogeneous Gaussian input model

  • The setup uses two dependent Gaussian blocks: the second combines a component inherited from the first with an independent innovation, covering independent, redundant, and partially dependent inputs.The fused covariance is constrained to be positive semidefinite; no other restriction is placed on cross-block correlation for structural fusion results.Source: §2.2 · p. 6

2.3 Regression and classification from a common latent score

  • Both regression responses and binary labels derive from one latent Gaussian score, so fusion changes the observed input rather than the underlying population task.The input-specific oracle and excess risks are measured relative to the optimal predictor for that same observed input; the Bayes risks need not coincide across inputs.Source: §2.3 · pp. 6–7

2.4 Minimum-norm interpolating estimators

  • The regression and classification estimators use the minimum-Euclidean-norm interpolator, with classification applying the same least-squares rule directly to binary labels rather than using a hard-margin support-vector machine.The task-level results assume each covariance eventually has rank at least n, ensuring feasible interpolation almost surely under the Gaussian model.Source: §2.4 · pp. 7–8

2.5 Benign and non-benign overfitting

  • Benign overfitting asks whether population excess risk vanishes asymptotically despite interpolation; the paper defines this separately for regression and classification.It distinguishes non-benignity, which may hold along a subsequence, from harmfulness, which requires a persistent lower bound along the full sequence.Source: §2.5 · p. 8

3 Benign Overfitting in Regression under Heterogeneous Input Fusion

Regression fusion has a protected covariance-level regime, but this does not make task-level benignity universally compositional. The section identifies threshold-invariant spectral certification, sharp closure boundaries, and counterexamples beyond them.

3 Benign Overfitting in Regression under Heterogeneous Input Fusion

  • The section distinguishes task-level benign regression, Breg, from the covariance-level spectral certificate BFS.Breg means vanishing excess prediction risk for the minimum-norm interpolator; BFS is a sufficient covariance-level certificate.Source: §3 · p. 8

3.1 The benchmark and its spectral surrogate

  • Classical bounds distinguish variance from poor post-cutoff spectra and bias from signal orientation; BFS is a sufficient surrogate, with a logarithmic gap to a known necessary bias condition.The bias converse is worst-case over the signal and does not cover every sequence for which the sufficient bias ratio merely fails to vanish.Source: §3 · p. 8, §3.1 · p. 9

3.2 Threshold invariance of the full-spectrum certificate

  • Any fixed positive cutoff threshold yields the same asymptotic BFS/NFS state, making the certificate intrinsic to the covariance sequence.The post-cutoff condition supplies slack against knife-edge behavior at the cutoff.Source: §3.2 · p. 10

3.3 Fusion closure and its sharpness

  • If both marginals satisfy BFS, every positive-semidefinite joint covariance consistent with them also satisfies BFS; this closure is sharp.BFS + BFS → NFS is the sole forbidden full-spectrum transition, while every other spectral pattern is attainable.Source: §3.3 · p. 10, Table 1 · p. 11

3.4 Equivalence under well-conditioned forward and reverse maps

  • Uniformly well-conditioned forward or reverse conditional-mean maps make the corresponding marginal and fused BFS states equivalent.Without uniform conditioning, an arbitrary fusion map may discard or severely attenuate directions.Source: §3.4 · pp. 11–12

3.5 Case study I: vanishing innovation energy can still be harmful

  • With fixed-noise response, X1 remains benign while X2 and fusion are non-benign despite vanishing innovation energy, which may form an O(n)-wide spectral shoulder.The fused post-cutoff spectrum has n R1(ΣJ,n) → c/(c + 2)^2 > 0.Source: §3.5 · pp. 12–13

3.6 Case study II: task-level non-closure inside the logarithmic gap

  • Two benign marginal regression tasks can fuse into a non-benign task outside the certified region, even under well-conditioned forward and reverse maps.The counterexample lies in the logarithmic gap between known sufficient and necessary bias conditions.Source: §3.6 · p. 13

3.7 Scope of the closure law

  • Task-level closure requires certified marginals and signal-scale assumptions; signal-specific benignity may exist outside that protected regime.Fusion can turn a marginally negligible or conditionally invisible signal into a jointly visible low-variance direction with nonvanishing interpolation bias.Source: §3.7 · pp. 13–14

4 Benign Overfitting in Classification under Heterogeneous Input Fusion

Classification benignity depends on the balance between signal surviving minimum-norm interpolation and nuisance contamination, not on marginal benign/non-benign labels alone. Fusion changes both quantities, and all six patterns of marginal and fused benignity are attainable in the heterogeneous Gaussian model.

4.1 A spectral characterization through survival and contamination

  • In the regular one-sparse regime, classification is benign exactly when nuisance contamination is small relative to surviving predictive signal; outside that regime, a vanishing balance parameter remains sufficient.A positive limiting obstruction implies non-benignity in the regular regime, while a uniformly positive obstruction implies harmfulness.Source: §4.1 · p. 15

4.2 A survival-contamination decomposition for heterogeneous fusion

  • Fusion changes nuisance contamination and signal survival separately, and neither effect is monotone; the fused classifier is benign exactly when the transfer identity’s right-hand side tends to zero.The identity organizes the two spectral changes rather than serving by itself as a structural fusion theorem.Source: §4.2 · p. 15

4.3 The complete six-pattern classification law

  • All six unordered patterns of marginal and fused classification benignity are realized, so no Boolean closure rule based only on the two marginal states holds.The constructions include benign marginals fusing to a non-benign classifier.Source: §4.3 · p. 16

4.4 Discussion: a signal-to-noise interpretation

  • Benign classification corresponds to a diverging effective signal-to-noise ratio: minimum-norm interpolation retains predictive signal at scale s_n while introducing nuisance contamination at scale V_n.The effective ratio is induced by how the estimator transforms signal and nuisance through covariance geometry, not by the population problem alone.Source: §4.4 · p. 17
  • Fusion can improve population-level predictability while weakening signal survival or increasing nuisance contamination, reducing effective signal-to-noise even as the population prediction problem becomes easier.Regression, by contrast, depends on the full covariance spectrum and predictive-signal alignment rather than a single scalar signal-to-noise measure.Source: §4.4 · p. 17

5 Cross-Task Separation under Heterogeneous Fusion

The paper compares regression and classification on the same heterogeneous inputs while holding the latent score fixed. It constructs two transitions showing that fusion can affect the tasks differently, including preserving classification benignity while destroying regression benignity.

5 Cross-Task Separation under Heterogeneous Fusion

  • With non-benign marginals for both tasks, fusion makes classification benign while regression remains non-benign.Both transitions use the same latent label mechanism across the three inputs.Source: §5 · p. 17
  • With benign marginals for both tasks, fusion preserves classification benignity but destroys regression benignity.The inputs and latent score are held fixed across the tasks.Source: §5 · p. 17
  • The constructions use nonzero cross-block coupling and keep the target-generating mechanism unchanged across inputs, isolating task dependence from input or target changes.The classification separation is formal even though its marginal Bayes advantages vanish asymptotically.Source: §5 · pp. 17–18

6 Experiments

The experiments illustrate the paper’s regression and classification interaction patterns, their distinct governing mechanisms, and cases where the same fusion affects the two tasks differently. They also include a constant-factor robustness check and a limited out-of-model multi-view sanity check.

6 Experiments

  • The experiments are designed to illustrate proved interaction states and mechanisms, not as separately tuned finite-dimensional examples.The real-data check is explicitly limited and does not validate the Gaussian theory.Source: §6 · p. 18

6.1 Experimental protocol

  • The experiments use triangular Gaussian inputs and fit minimum-norm interpolants separately on each marginal and their concatenation.Synthetic runs cover n = 16 to 15000 with 50 repetitions; exact population risks are used rather than finite test sets.Source: §6.1 · p. 18

6.2 The five attainable regression interactions

  • The five regression interactions illustrate covariance-level closure alongside task-level benign and harmful fusion patterns.In one construction, marginal risks remain near 0.20 and 1 while joint risk falls below 10^-4; another has all three risks separated from zero.Source: §6.2 · p. 18, Figure 1 · p. 19

6.3 Regression case studies

  • The regression case studies illustrate both a vanishing-energy innovation associated with a variance obstruction and a logarithmic-gap task whose benign marginals fuse harmfully.In the latter construction, marginal risks decrease while fused risk stays separated; the marginal quantities r0/n and r0/(n log n) move in opposite directions.Source: §6.3 · p. 19, Figure 2 · p. 20

6.4 The complete classification interaction law

  • Across six classification interactions, finite-sample patterns illustrate all marginal-to-joint benignity outcomes, including transitions unavailable under the regression full-spectrum surrogate.The panels include fusion harm, rescue, and correlated rescue, consistent with classification being governed by contamination relative to surviving signal.Source: §6.4 · pp. 19–20

6.5 Cross-task separation under the same fusion

  • With the input distribution and latent score fixed, the two constructions display regression non-benignity with classification rescue, and regression degradation with preserved classification benignity.In the second construction, marginal classification excess risks decrease even though their Bayes advantages vanish.Source: §6.5 · pp. 20–21, Figure 4 · p. 22

6.6 Robustness of the small-innovation mechanism

  • Across five fixed shoulder multipliers, the second and joint regression risks persist while the first-input risk decreases, illustrating robustness to the chosen constant.This finite-sample sweep is not a uniform-in-a extension of the theorem.Source: §6.6 · p. 21

6.7 Out-of-model multi-view sanity check

  • On UCI Multiple Features, fusion has a negative median clean-test error gap throughout the limited protocol, which is presented as a sanity check rather than theorem validation.The views are neither Gaussian nor one-sparse eigensignal inputs.Source: §6.7 · p. 22

7 Conclusion

The conclusion distinguishes a regression regime protected by a threshold-invariant full-spectrum certificate from broader task-level benignity, which need not survive fusion. For one-sparse Gaussian classification, all marginal-to-joint benignity patterns are attainable, and the same fusion can affect regression and classification differently.

7 Conclusion

  • Regression’s full-spectrum covariance certificate is threshold-invariant and preserved by positive-semidefinite joint fusion, but task-level benign regression is not closed outside the certified regime.The appendix proof states that two certified marginals cannot fuse into a non-benign full-spectrum state.Source: §7 · pp. 22–23, §A.3 · p. 28
  • For one-sparse Gaussian classification, the regular-regime outcome depends on surviving predictive signal versus nuisance contamination, and every marginal-to-joint benign/non-benign pattern is attainable.This conclusion is scoped to the stated model class.Source: §7 · p. 23
  • The same fused input can have different effects across regression and classification, so marginal benignity alone does not determine fusion’s effect.The paper identifies joint spectral geometry for protected regression and the signal-contamination balance for classification.Source: §7 · pp. 22–23

A.1 Certificates and probabilistic consequences in Bartlett et al. [2020]

  • The appendix states that the full-spectrum certificate implies benign regression under its stated assumptions.The proof chooses a slowly diverging sequence and applies a probabilistic risk bound.Source: §A.1 · p. 25

A.1.1 A fixed-probability form of the variance lower bound

  • A nonvanishing variance obstruction along a subsequence is sufficient for regression benign overfitting to fail along a subsequence.A positive lim inf makes the lower bound persistent along the full sequence.Source: §A.1.1 · p. 25

A.2 Threshold invariance of the full-spectrum state

  • The full-spectrum state is independent of the fixed cutoff threshold used to define it.The appendix establishes equivalence across fixed thresholds.Source: §A.2 · p. 27

A.3 Fusion closure

  • If both marginal covariances satisfy the full-spectrum certificate, every consistent positive-semidefinite joint covariance satisfies it as well.The appendix also derives closure for any fixed number of sources.Source: §A.3 · p. 28

A.4 Five attainable regression interaction patterns

This section constructs five attainable regression interaction patterns to examine when benign full-spectrum status is preserved across fused and marginal covariances, showing that marginal behavior alone does not determine whether fusion is benign. One case has a first marginal in BFS and the joint covariance in NFS, with both reported cutoffs equal to 1 and joint r₀(ΣJ) = 1 + o(1).

A.4 Five attainable regression interaction patterns

  • A sufficiently wide tail can repair a defective marginal condition, but two variance-side shoulders cannot rescue each other; if both marginal tail effective ranks are O(n), the joint tail effective rank remains O(n).The NFS + NFS → BFS witness therefore pairs a variance-side defect with an explicit bias-side defect rather than treating failure of r0/n → 0 as sufficient for non-benign risk.Source: §A.4 · p. 29
  • Five task-level transitions are attained: Breg + Breg → Breg, Breg + Nreg → Breg or Nreg, and Nreg + Nreg → Breg or Nreg.The remaining Boolean pattern, BFS + BFS → NFS, is ruled out by Corollary 4.Source: §A.4 · pp. 29–32

A.5 Forward-reverse spectral equivalence

  • Under the stated forward and reverse map conditions, the fused covariance and both marginal covariances have equivalent BFS status.The proof uses uniform spectral comparison and sum–direct-sum equivalence to establish the three-way equivalence.Source: §A.5 · pp. 34–35

A.6 Case study I: vanishing innovation energy

  • The joint covariance has r0(ΣJ) = 1 + o(1); after removing its unique leading eigenvalue, its largest remaining eigenvalue is n−2(1 + o(1)).Its remaining-spectrum rank is r1(ΣJ) = (c + 2)n(1 + o(1)).Source: §A.6 · pp. 35–36
  • Although Σ1 ∈BFS, the joint covariance satisfies ΣJ ∈NFS.The reported cutoff equals 1 for both Σ1 and the joint covariance.Source: §A.6 · pp. 35–36

A.7 Case study II: task-level non-closure in the logarithmic gap

In the logarithmic gap, the two marginal regression tasks are benign, but their fused task is non-benign. The construction gives a task-level transition from benign marginals to harmful fusion.

A.7 Case study II: task-level non-closure in the logarithmic gap

  • For the considered spectrum, r0/[n log(1 + r0)] → 0 although r0/n → ∞; its leading eigenvalue is 1 + o(1), and its spectral sum is nh(1 + o(1)).The jointly Gaussian variables have zero cross-covariance, so Y ⊥ X2; its marginal coefficient is zero, giving zero bias and variance O(n^-1).Source: §A.7 · p. 36
  • The two marginal tasks satisfy the stated uniform two-sided bounds, while the proof separately establishes a high-probability lower bound for fusion.The passage reports this proof structure but does not give the bound's value.Source: §A.7 · p. 36
  • The fused task has a principal-direction amplitude error converging in probability to 1, which lower-bounds its noiseless excess risk away from zero.The proof establishes a high-probability lower bound for the fused task after verifying the marginal tasks.Source: §A.7 · pp. 36–37

B.1 The ∆-criterion and classification benignity

The section establishes a criterion for classification benignity using the balance between surviving signal and nuisance contamination. In the regular regime, benignity is equivalent to the criterion Δ_n vanishing.

B.1 The Δ-criterion and classification benignity

  • The criterion's proof controls signal survival at scale u ≍P nλ_t/S_n and bounds nuisance contamination using the variance-trace matrix C.The upper bound uses conditional Markov; the lower bound uses Paley–Zygmund without assuming relative concentration of C.Source: §B.1 · pp. 39–40
  • In the regular regime, classification is benign if and only if Δ_n → 0.The proof gives sufficiency and necessity, with the necessity argument using a fixed-positive-probability lower bound.Source: §B.1 · pp. 38, 40, 41
  • If Δ_n does not vanish, it stays bounded below along a subsequence, keeping Γ away from zero with fixed positive probability and preventing excess risk from converging to zero.This is the regular-regime necessity argument.Source: §B.1 · p. 41

B.2 Survival-contamination transfer identity

The transfer identity separates fusion's effects on nuisance contamination and surviving signal. Neither transfer is monotone, and in the regular regime the fused classifier is benign exactly when Δ_J,n vanishes.

B.2 Survival-contamination transfer identity

  • The identity separates fusion into a nuisance-contamination transfer and a signal-survival transfer; neither factor is monotone.A broader joint tail can reduce contamination, while increased nuisance mass can reduce signal survival.Source: §B.2 · p. 41
  • In the regular regime, the fused classifier is benign exactly when Δ_J,n → 0, equivalently when the product in the transfer identity tends to zero.The criterion applies input by input, with matching fixed-positive-probability contamination bounds.Source: §B.2 · p. 41
  • The identity's factors are deterministic spectral-scale transfers, not a claim about concentration of the random contamination ratio.Source: §B.2 · p. 41

B.3 Six classification interaction patterns

The proof constructs joint distributions matching desired marginal properties and interaction patterns, and completes six classification interaction patterns. The cases show that fusion can preserve benignity, rescue a classifier, or leave it non-benign.

B.3 Six classification interaction patterns

  • The construction realizes six classification patterns by choosing a joint distribution with specified marginal properties and interaction behavior.The six cases are completed using signal-deleted nuisance calculations and Theorem 8.Source: §B.3 · pp. 41, 45

Case 1: Bcls + Bcls → Bcls

  • Case 1, Bcls + Bcls → Bcls: both marginals and the fused classifier are benign, with interaction quantities ∆1, ∆2, ∆J = Θ(n−1).Source: §B.3 · p. 43

Case 2: Bcls + Ncls → Bcls

  • Case 2, Bcls + Ncls → Bcls: the first source’s wide nuisance tail rescues the fused classifier, whose interaction quantity is ∆J = Θ(n−1).The second marginal is non-benign, with ∆2 → 1/ρ.Source: §B.3 · p. 43

Case 3: Bcls + Ncls → Ncls

  • Case 3, Bcls + Ncls → Ncls: the second marginal and the fused classifier are both non-benign, with ∆2 and ∆J ≍ n.Source: §B.3 · p. 43

Case 4: Ncls + Ncls → Ncls

  • Case 4, Ncls + Ncls → Ncls: fusion preserves non-benignity, with ∆J → 1/(2ρ) > 0.Source: §B.3 · p. 43

Case 5: Bcls + Bcls → Ncls

  • Case 5, Bcls + Bcls → Ncls: both marginal interaction quantities vanish, but the joint quantity diverges; all cutoffs are o(n).Source: §B.3 · p. 43

Case 6: Ncls + Ncls → Bcls

  • Case 6, Ncls + Ncls → Bcls: both marginal interaction quantities stay away from zero, while the joint quantity vanishes.Source: §B.3 · pp. 44–45

C.1 Fusion rescue from two non-benign sources

The constructed inputs yield non-benign regression for both sources and their fusion, but classification changes from non-benign marginals to a benign fused classifier. The same common score supplies a one-sparse signal direction in all three inputs.

Classification states

  • The common score supplies a one-sparse eigendirection in all three inputs, with bounded normalized probit noise; the scalar rescaling leaves covariance construction and ∆ orders unchanged.Source: §C.1 · p. 45

C.1 Fusion rescue from two non-benign sources

  • For the common task, X1, X2, and their fusion are all non-benign for regression, whereas classification changes from two non-benign marginals to a benign fusion.Source: §C.1 · pp. 45–46

Regression states

  • The regression result holds because the fused signal has asymptotically unit bias, while the residual term vanishes.Source: §C.1 · p. 46

C.2 Task separation from two benign sources

Two benign marginals can produce a non-benign fused regression problem, while the same inputs remain benign for classification. In this example, marginal classification benignity is driven by an asymptotically uninformative task.

Regression states

  • Both marginal predictable signal variances are Θ(ε), while their residual variances converge to one.Source: §C.2 · p. 47

C.2 Task separation from two benign sources

  • Both marginals are benign for regression, but their fused regression problem is non-benign.The joint regression signal has asymptotically unit bias.Source: §C.2 · p. 47

Classification states

  • The same two marginals and their fusion are benign for classification, with marginal benignity driven by an asymptotically uninformative task.Source: §C.2 · pp. 47–48
Loading 2610.09340v1…