PaperlayerView exact revision on arXiv

Source-linked AI summary

From Common Information to Canonical Correlation Analysis and Beyond: Bridging Information Theory, Signal Processing, and Machine Learning

Paris A. Karakasis, Walid Saad

arXiv:2610.28978v1cs.ITcs.LGeess.SPmath.ITstat.ML

TL;DR

The paper addresses how to recover a factor shared across jointly observed views when each view also contains private variation. It brings together information-theoretic common information, CCA, receiver methods, and nonlinear formulations; for jointly Gaussian views, canonical correlations determine both mutual information and Wyner’s common information.

  • Problem

    The central question is how to recover a factor shared across jointly observed views when each view also contains private factors.

    Source: §1.1 · p. 1, §2.1 · p. 8

  • Method

    The tutorial relates information-theoretic common information to CCA and develops conditions for identifying the common subspace.

    Source: §3.7.3 · p. 47, §3.8 · p. 48, §7 · p. 92

  • Results

    For jointly Gaussian views, canonical correlations determine both mutual information and Wyner’s common information, while their link to Gács–Körner common information is one-sided.

    Source: §7 · p. 92

  • Takeaways & Limitations

    A designed second view lets a multiview receiver attain the best linear performance, beyond detecting a weak desired user.

    Source: §4.7 · p. 60

    With sufficiently rich function classes, finite-sample nonlinear objectives can overfit by indexing training samples, and kernel consistency need not identify a latent factor.

    Source: §5.7.6 · p. 84

Abstract

from arXiv · show

Many problems in signal processing, machine learning, and neuroimaging provide several jointly observed measurements of one process, each mixing a component common to all of them with a component private to itself, and require the common component to be recovered without supervision. This problem has been studied in several communities, largely independently. Statisticians proposed canonical correlation analysis (CCA), introduced by Hotelling in 1936. Information theorists proposed two notions of common information, due to G\'acs and K\"orner and to Wyner. Engineers developed subspace methods for sensor arrays and multiuser communications to design multiview receivers. This tutorial presents the main results and tools each community developed for this problem in a single exposition, and explains the connections that emerge where they meet, distinguishing exact equivalences from special cases and analogies. We develop the information-theoretic foundation from first principles. We then introduce CCA and its extension to several views, followed by the engineering thread from blind multiuser detection to CCA as a blind receiver. Next, we turn to nonlinear CCA, covering kernel, deep, and probabilistic formulations, the conditions under which the common factor is identifiable, and connections to self-supervised learning. We conclude with a discussion of further extensions and open directions. The exposition is self-contained, so that readers from each of these communities can follow the parts developed by the others.

1. Introduction

The introduction frames unsupervised common-factor recovery as a shared problem across disciplines, then positions the tutorial as a bridge among their theories and tools. It also explains why correlation alone does not establish that recovered structure is a real common factor.

1.1. A Question That Crosses Disciplines

  • Common factor recovery seeks shared structure across jointly observed views despite private variation, unknown mixtures, absent supervision, and potentially weak common signals.The introduction motivates the problem through wireless communications, neuroimaging, and multimodal learning.Source: §1.1 · pp. 2–3

1.2. Four Communities, One Problem

  • The tutorial connects statistics’ CCA, information theory’s common-information rates, engineering subspace receivers, and machine learning’s kernel and deep CCA, while emphasizing meaning and identifiability beyond correlation.A high correlation between transformed views does not by itself show that they share a concrete factor; transformations can manufacture agreement.Source: §1.2 · p. 4

1.3. Tutorial Objectives and Contributions

  • It develops common-information foundations, Gaussian and CCA relations, recovery conditions, blind-receiver methods, nonlinear CCA, self-supervision, and open directions.The tutorial presents the connections among these literatures while distinguishing their types.Source: §1.3 · p. 5

1.4. Relation to Existing Tutorials & Monographs

  • Earlier CCA tutorials and information-theory monographs cover different portions of the subject, and prior treatments generally leave the cross-field synthesis and common-factor identifiability unaddressed.The surveyed works variously omit information-theoretic foundations, CCA, identifiability, or the wireless communications setting.Source: §1.4 · p. 6

1.5. Organization

  • The tutorial proceeds from information theory through linear CCA and receivers to nonlinear CCA, then points to extensions and open problems, while distinguishing equivalences, consequences, and analogies.The sections cover continuous and multiview sources, generalized CCA, nonlinear formulations, and self-supervised learning.Source: §1.5 · pp. 7–8

2. The Information-Theoretic View of Common Information

The section distinguishes common information that the views can determine individually from a factor that explains their dependence, and develops the Gács–Körner and Wyner notions. It also covers limitations and extensions, including operational and multivariable formulations involving source coding and privacy constraints.

2.1. What Does “Shared” Mean?

  • Shared structure can be the richest labeling each view determines on its own, or a factor that explains dependence by rendering views conditionally independent.These interpretations motivate the Gács–Körner and Wyner quantities, respectively.Source: §2.1 · pp. 8, 10

2.2. Gács–Körner Common Information: The Highest-Entropy Shared Function

  • Gács–Körner common information is the maximum entropy of a variable computed deterministically from both views; it can vanish despite positive or arbitrarily large mutual information.The optimal common variable is the index of the finest shared partition, unique up to relabeling; equality with mutual information holds exactly when the views are conditionally independent given that common part.Source: §2.2 · pp. 10–11

2.2.1. Connection to Maximal Correlation

  • Maximal correlation characterizes existence of nontrivial common structure under mild conditions, but need not recover the full Gács–Körner common part; nondegenerate jointly Gaussian pairs have zero Gács–Körner information.For jointly Gaussian variables, maximal correlation is |ρ|, which is below one for nondegenerate pairs even when mutual information can be arbitrarily large.Source: §2.2.1 · p. 12

2.2.2. Extensions: Multiple Views and Relaxed Constraints

  • For K views, the maximum common function is constructed recursively, and positive Gács–Körner information corresponds to a common block structure across all views.Continuous-view methods [36] frame recovery as disentanglement with mutually independent shared and private components; other relaxations allow disagreement or stochastic shared variables optimized with variational autoencoders.Source: §2.2.2 · p. 13

2.2.3. Applications

  • Gács–Körner common information supports distributed source coding, secret-key capacity characterization, and multi-view representation learning [36].These applications use shared structure to reduce transmission, characterize secure computation, or separate shared from view-specific variation.Source: §2.2.3 · p. 13

2.2.4. Limitations and the Need for a Richer Notion

  • Because Gács–Körner captures only deterministic common structure and often vanishes despite dependence, the section motivates a probabilistic notion based on making views conditionally independent.Source: §2.2.4 · p. 14

2.3. Wyner’s Common Information

  • Wyner common information minimizes I(X, Y; W) over factors W that make X and Y conditionally independent, seeking a parsimonious explanation of their dependence.It equals mutual information in the special case where the Gács–Körner common part also makes the views conditionally independent; the doubly symmetric binary source instead has zero Gács–Körner information despite positive mutual information.Source: §2.3 · pp. 14–15, Example: the Doubly Symmetric Binary… · p. 16
  • For a doubly symmetric binary source, Wyner’s common information strictly exceeds mutual information for 0<a0<1/2, while Gács–Körner common information is zero.Wyner’s optimal W need not be a function of (X, Y).Source: Example: the Doubly Symmetric Binary… · p. 16
  • The quantities obey KGK ≤ I ≤ C ≤ GCE ≤ Gf, with equality throughout when Wyner’s factor explains all dependence.Common entropy GCE is within one bit of exact common information, measured by the minimum expected prefix-free code length for W.Source: Example: the Doubly Symmetric Binary… · p. 17

2.4. The Gray–Wyner Network: An Operational Meaning

  • In the Gray–Wyner network, C(X; Y) is the minimum common-link rate for lossless reconstruction at the minimum total rate H(X, Y).The minimum total rate remains H(X, Y), while the common link is shared and the two private links serve their respective receivers.Source: §2.4 · p. 18

2.5. Extension of Wyner’s Common Information to Multiple Variables

  • For K variables, Wyner’s conditional-independence factorization is a nonnegative tensor/CP decomposition (naive Bayes), with rank+ bounded by the product of all but the largest alphabet.CP decompositions can be essentially unique under mild sufficient conditions, including Kruskal-rank conditions.Source: §2.5 · pp. 19–20
  • Wyner’s K-variable common information is non-decreasing in K because full conditional independence makes the feasible W more demanding, and a three-variable example exceeds every pairwise value.By contrast, K-variable Gács–Körner common information is non-increasing in K [43].Source: §2.5 · p. 20

2.6. Common Information Under Multi-User Network Privacy

  • Under multi-user privacy, privacy-optimal C1 and efficiency-optimal C2 coincide as CTSP, the largest mutual information in W satisfying W−Xk−X̄\k for every k [6].This Markov condition means W can reveal information about other sources only through each individual Xk, so privacy and bandwidth efficiency coincide.Source: §2.6 · p. 22
  • CTSP is non-increasing in K, unlike Wyner’s K-variable quantity, and resembles K-variable Gács–Körner structurally rather than definitionally.CTSP arises from a rate–privacy formulation, whereas the Gács–Körner notion is based on deterministic recoverability.Source: §2.6 · p. 22

2.7. Extension of Wyner’s Common Information to Continuous Variables

This section develops common-information concepts across continuous and sample-based settings, covering Wyner’s extension to continuous sources, intersective and generative definitions, and their connections to PID and linear CCA. It also summarizes conditions and characterizations for these formulations, including the lossy-reconstruction interpretation of Wyner’s common information.

2.7. Extension of Wyner’s Common Information to Continuous Variables

  • The continuous extension requires finite mutual information, a finite common-information infimum, and an operational interpretation; exact lossless reconstruction of a real-valued source has no finite-rate meaning.Mutual information can be infinite when the joint law is not absolutely continuous relative to the product of marginals, as when Y = f(X) for continuous X and Y; finite mutual information alone does not ensure finite C(X; Y).Source: §2.7 · p. 23
  • For continuous sources with finite C(X; Y), C(X; Y) can be attained at positive distortion as the exact rate receivers share for lossy reconstruction over a whole range of fidelities.Source: §2.7 · pp. 24–25

2.7.1. Locating the Plateau: Binary and Gaussian Sources

  • The doubly symmetric binary source has a plateau through distortion a1, while the bivariate Gaussian plateau extends through distortion 1 − ρ.Both sources are successively refinable, so their edges meet the stated rate–distortion characterization; for the Gaussian, exact reconstruction requires infinite rate.Source: §2.7.1 · pp. 25–26

2.7.2. Going Beyond Wyner’s Common Information

  • Beyond Wyner’s common information, distributed channel synthesis, Rényi common information, and common information dimension address correlation generation, stronger divergence criteria, and infinite common information [61].Rényi common information interpolates from Wyner’s quantity for α ≤ 1 to exact common information at α = ∞.Source: §2.7.2 · p. 26

2.7.3. Extension to Many Continuous Random Variables

  • The K-variable Wyner common information extends to continuous sources under the same three conditions, and its lossy plateau and edge carry over to multiple sources [64].It is finite for zero-mean, unit-variance jointly Gaussian sources with common pairwise correlation ρ ∈ (0, 1).Source: §2.7.3 · pp. 26–27
  • Continuous density-factorization methods can learn a latent variable that renders views conditionally independent, but this structural connection does not generally recover Wyner’s mutual-information-minimizing variable.Amiridi et al. use Fourier expansions, while Karakasis and Sidiropoulos enforce valid density factors; the former can yield invalid densities, and the learned latent is restricted to be discrete.Source: §2.7.3 · pp. 27–28

2.7.4. Applications

  • Applications of common information include Gray–Wyner storage and retrieval trade-offs, secure source coding, caching, and learned shared representations [75] [76] [77].For sufficiently large caches, the smallest rate that makes two files conditionally independent is Wyner’s common information.Source: §2.7.4 · pp. 28–29

2.8. How Many Kinds of Common Information Are There After All?

  • Common-information quantities divide into intersective measures of what each source can recover and generative measures of what must be shared to explain dependence; the two families generally differ.Gács–Körner and Tandon–Sankar–Poor quantities do not increase with added sources, whereas K-variable Wyner common information does not decrease.Source: §2.8 · p. 29

2.9. Summary

  • Intersective common information asks what each source can recover alone, whereas generative common information asks what latent variable renders the sources conditionally independent.Gács–Körner and Tandon–Sankar–Poor exemplify the intersective family; Wyner’s quantity, exact common information, and common entropy exemplify the generative family.Source: §2.9 · p. 30
  • Intersective quantities are non-increasing with more sources, while the K-variable Wyner quantity is non-decreasing; for two finite discrete sources, KGK(X; Y) ≤ I(X; Y) ≤ C(X; Y).The inequalities are equalities exactly for joint PMFs that become block-diagonal with product blocks after relabeling; for jointly Gaussian variables with 0 < |ρ| < 1, KGK vanishes while C remains positive.Source: §2.9 · pp. 30–31
  • KGK(X; Y) is positive exactly when Hirschfeld–Gebelein–Rényi maximal correlation S(X, Y) equals 1, though maximizing functions generally recover only a coarsening of the optimal common variable.This criterion certifies existence of a nontrivial common variable but does not identify the optimal one.Source: §2.9 · p. 31
  • Wyner’s C(X; Y) is the minimum shared-link rate for lossless Gray–Wyner reconstruction at total rate H(X, Y), with a lossy continuous-source characterization under stated conditions.For full support and a normal distortion measure, C3(∆) equals C(X; Y) over a distortion range for successively refinable sources; CTSP also characterizes largest common rates under maximal privacy and source-entropy constraints.Source: §2.9 · p. 31
  • Wyner’s conditional-independence constraint becomes a nonnegative matrix factorization for two discrete sources and a tensor factorization for K sources, with uniqueness conditions supporting identifiability for three or more.The two-source matrix factorization is not unique in general, whereas tensor identifiability follows from conditions including Kruskal’s.Source: §2.9 · p. 31

2.8. How Many Kinds of Common Information Are There After All?

  • PID separates target-related information into redundant, unique, and synergistic parts; synergy is available only jointly and lies outside common-information notions, while interaction information is negative when synergy outweighs redundancy.The common-information notions retain the shared part and remain nonnegative.Source: §2.8 · p. 30, §2.9 · p. 31

2.9. Summary

  • The definitions specify what shared latent variable to recover but assume a known joint distribution; the next sections move to finite samples and linear combinations maximally correlated across views.Hirschfeld–Gebelein–Rényi maximal correlation formally links common-information theory to CCA, which restricts admissible functions to be linear for tractable, well-posed sample-based recovery.Source: §2.9 · p. 31

3. Canonical Correlation Analysis

This section develops canonical correlation analysis as a way to find maximally correlated linear combinations of two views and aligned representations of their shared structure, connecting its geometry to Gaussian information measures and latent-variable models. It examines when CCA recovers a shared subspace and extends the framework to multiple views, including generalized criteria, consensus formulations, and intersection conditions for recovery.

3.1. Three Problems to Set the Scene

  • Across sensor networks, fMRI neighborhoods, and cellular networks, CCA treats each measurement group as a multivariate view and finds correlated linear combinations to quantify and recover shared structure.Jointly processing channels within each view can reveal common signals that may be buried in individual measurements.Source: §3.1 · pp. 32–33

3.2. Hotelling’s Original Formulation

  • Hotelling introduced CCA in 1936, reducing each variable set to a linear combination chosen to maximize correlation and measure what the sets share.His examples concerned common reading and arithmetic abilities and how prices and quantities move together across related commodities.Source: §3.2 · p. 33

3.2.1. Problem Formulation and Lagrangian Derivation

  • Under unit-variance constraints, CCA maximizes cross-covariance; eliminating one direction yields an eigenvalue problem whose ordered canonical pairs have mutually uncorrelated variates.With invertible within-view covariance matrices, the canonical correlations are the nonnegative square roots of the eigenvalues, and there are at most min(M1, M2) correlated pairs.Source: §3.2.1 · pp. 34–35

3.3. Closed-Form Solution via SVD and the Geometry of Principal Angles

  • Whitening removes within-view scale and correlation, reducing CCA to an SVD of the cross-view coherence matrix; its singular values are canonical correlations and cosines of principal angles.The leading directions give the most-correlated projections, while the SVD directly returns the canonical correlations.Source: §3.3 · pp. 36–37
  • When view subspaces intersect, canonical variates with correlation one span their exact intersection; a population canonical correlation of one certifies positive Gács–Körner common information, but the converse can fail.CCA searches only among linear functions, so nonlinear shared random variables may exist even when every canonical correlation is below one.Source: §3.3 · pp. 38–39
  • Canonical correlations are invariant to independent invertible linear transformations of the two views, making them intrinsic to the relationship rather than to each view’s basis or scaling.The jointly Gaussian model is a natural choice for connecting CCA quantitatively to common-information measures.Source: Invariance of Canonical Correlations · p. 39, §3.3 · p. 39

3.4. Connection to Mutual Information and Wyner’s Common Information

  • For jointly Gaussian views, canonical correlations alone determine both mutual information and Wyner’s common information, so CCA computes both.For weakly correlated views, Huang, Wornell, and Zheng relate canonical directions to optimal features and approximate Wyner common information; Sula and Gastpar’s common information components analysis reduces exactly to CCA for Gaussian statistics, with γ setting the number of extracted components.Source: §3.4 · p. 40

3.5. Probabilistic Interpretation of CCA

  • In the Bach–Jordan probabilistic CCA model, a shared latent factor generates both views, and maximum-likelihood estimation recovers CCA parameters from canonical directions and correlations.The views are conditionally independent given the latent factor, while view-specific noise captures private variation. For jointly Gaussian views, the balanced Bach–Jordan latent with loadings scaled by square roots of the canonical correlations attains Wyner’s common information [69]; the balanced maximum-likelihood solution is its plug-in estimate.Source: §3.5 · pp. 40–41

3.6. Identifiability: When Does CCA Recover the Common Subspace?

  • In the noiseless two-view model, CCA recovers the common subspace when the common and private subspaces and each view’s mixing matrix satisfy the theorem’s full-column-rank conditions.The conditions are sufficient but not all necessary: with full-rank mixing, independent common and private subspaces are necessary, while private directions may be lost without preventing recovery; the common mixing must remain full rank.Source: §3.6 · pp. 42–43

3.7. Generalized CCA: From Two Views to K Views

  • Generalized CCA extends alignment to K ≥ 2 views, but for K > 2 its inequivalent criteria can produce different formulations and solutions.The five classical formulations collected by Kettenring all reduce to Hotelling’s CCA for two views.Source: §3.7 · p. 43

3.7.1. Kettenring’s Five Formulations

  • Kettenring’s five GCCA criteria are SUMCOR, MAXVAR, SSQCOR, MINVAR, and GENVAR, which measure alignment or dependence through distinct functions of the projected views’ correlation matrix.SUMCOR maximizes total pairwise correlation; MAXVAR seeks a consensus variable; SSQCOR maximizes squared pairwise correlations; MINVAR drives the smallest eigenvalue toward zero; and GENVAR minimizes the determinant. MAXVAR has a direct eigendecomposition, whereas SUMCOR requires an iterative solution.Source: §3.7.1 · pp. 44–45
  • Kettenring’s five criteria coincide for two views but diverge beyond; MAXVAR and MINVAR reduce to eigenvalue problems, whereas SUMCOR, SSQCOR, and GENVAR require iterative procedures.At perfect alignment, SUMCOR and SSQCOR attain K2, MAXVAR attains K, and det(Φ) and λmin vanish; unlike MINVAR, GENVAR penalizes the whole spectrum.Source: §3.7.1 · p. 45

3.7.2. Revisiting the MAXVAR Formulation

  • MAXVAR’s consensus formulation extracts D orthonormal components, with a closed-form eigenvector solution that can instead be computed by SVDs of the view bases and their concatenation.The columns of G are uncorrelated across successive components; the SVD route avoids forming an N × N matrix and scales linearly in the number of samples.Source: §3.7.2 · p. 46

3.7.3. Identifiability for K Views

  • K-view identifiability is governed by subspace intersection: adding views relaxes the dimension budget, and six views can identify a subspace that fails the stated budget with five.The stated budget is D + (1/(K−1))∑k Lk ≤ N, alongside D + Lk ≤ Mk for each view; with N = 1000, D = 10, and Lk = 800, it exceeds N at K = 5 but falls to 970 at K = 6.Source: §3.7.3 · pp. 46–47

3.8. Summary

  • CCA relates subspace geometry to common information: canonical correlations are cosines of principal angles, and in the Gaussian case CCA computes Wyner’s common information.Directions with canonical correlation one span the common subspace and play the role of the Gács–Körner common part; Wyner’s common information is finite and positive when correlations are nonzero but all below one.Source: §3.8 · pp. 47–48
  • In sensor networks, neuroimaging, and cell-edge detection, canonical correlations measure shared activity or signal strength, while extracted variates carry shared activity or support further decoding.Generalized CCA handles multiple network nodes or fMRI subjects jointly rather than pair by pair; high canonical correlation indicates one or more weak cell-edge users.Source: §3.8 · p. 48

4. The Signal Processing Thread, from Subspace Methods to CCA

This section traces the signal-processing thread from low-rank subspace receivers requiring prior information about target users to multiview CCA for recovering shared transmissions without signatures or pilots under identifiability conditions. It covers CCA applications to shared cell-edge structure, blind MMSE detection using a transmitter-designed second view, and recording synchronization.

4.1. The Signal Model and the Problem

  • When few sources mix across many channels, the noiseless recording is low rank, and its SVD exposes source-waveform and channel-vector subspaces.Recovering desired users’ waveforms depends on what the receiver knows about the sources or channels, or what repetition is built into transmission.Source: §4 · p. 49, §4.1 · p. 50

4.2. Multiuser Detection in CDMA: The Wang–Poor Approach

  • Classical CDMA zero-forcing requires every user’s signature, while MMSE additionally requires received amplitudes and noise variance; Wang–Poor’s subspace approach reduces this prior-information burden.Both classical detectors suppress interference, but zero-forcing amplifies noise, whereas MMSE balances interference suppression against noise enhancement.Source: §4.2 · pp. 51–52
  • Wang and Poor’s CDMA detectors can be built from the signal subspace, requiring only the desired user’s signature and timing—not interfering users’ signatures or received amplitudes.The signal eigenpairs supply amplitude information otherwise needed by the classical MMSE receiver; without the target signature, a single covariance cannot distinguish a weak user from interferers sharing the subspace.Source: What the Signal Subspace Buys · pp. 52–53

4.3. Multiple Antennas and Pilot Symbols

  • With multiple antennas, pilots provide the target channel direction, but target-only pilots let strong interferers corrupt its estimate, while pilots consume data-free snapshots and must be repeated as channels change.Knowing every user’s aligned pilots avoids interference in channel estimation but imposes strong coordination; a second view can later suppress interference with at most a few known symbols, or only their possible values.Source: §4.3 · pp. 53–54

4.4. The Multi-Array Leap from Signal Subspace to Common Subspace

  • Two stations share cell-edge transmissions but generally not cell-center users, so their common waveform subspace identifies the cell-edge users that a single station cannot isolate.The method addresses weak cell-edge signals for which signatures may be unknown and short pilots can be corrupted by strong cell-center users.Source: §4.4 · p. 54
  • Under Section 3.6’s identifiability conditions, CCA recovers the cell-edge users’ shared waveform subspace without signatures or pilots, regardless of cell-center users’ power.Interference power is removed because it acts through invertible mixing transformations; noise still affects canonical correlations, so shared signals must remain above the noise at both stations.Source: §4.4 · p. 55

4.5. Cell-Edge Detection

  • In the cell-edge model, both stations forward recordings to a central processor; the cell-edge symbol streams are common, while each station’s cell-center streams are private.The common subspace is col(G), where G contains the cell-edge symbol streams.Source: §4.5 · p. 55
  • In the noiseless full-rank regime, two-view MAXVAR CCA recovers the cell-edge subspace col(G) as GP, transferring the generative model’s identifiability conditions directly to system design.The rank conditions require N ≥ D + L1 + L2 and at least D + Lk antennas at station k. Recovering individual streams requires a second-stage unmixing step, using pilots or, in some cases, constant-modulus symbols; RACMA returns streams up to permutation and sign, resolved by identification sequences [8,19].Source: §4.5 · pp. 55–56

4.5.1. When Is a User Common?

  • A user has high canonical correlation only when both stations receive it well, so strong one-sided cell-center interference stays low-correlation regardless of its serving-station strength.For nearly orthogonal sequences and large arrays, each user's canonical correlation is the product of two station-specific factors, each between zero and one and increasing with that station's SNR.Source: §4.5.1 · p. 56

4.5.2. The Same Computation Synchronizes the Two Stations

  • Selecting the relative delay that maximizes canonical correlation synchronizes the stations without pilots, while CCA can also reveal cell-edge users and estimate their common subspace.Unlike correlation-based synchronization, this approach is not corrupted by strong cell-center interference.Source: §4.5.2 · pp. 56–57

4.6. A Designed Second View and Blind MMSE Detection

  • Repeating a desired user's data creates two views on which CCA yields its linear MMSE detector without channel estimation, at the cost of sending each symbol twice.With ensemble correlations, uncorrelated user transmissions, and full-column-rank symbol and channel matrices, the canonical directions are scaled MMSE detectors; canonical variates are linear MMSE estimates up to a scale resolvable by one pilot symbol. The channels may differ between views, and other users need not follow the protocol.Source: §4.6 · p. 58

4.7. Summary

  • The two CCA receivers trade prior information for different costs: cell-edge detection needs two stations and a central unit, while blind MMSE detection needs desired-user repetition and a single array [27].Cell-edge streams still require prior information for unmixing, whereas blind MMSE detection requires half the desired user's transmission rate and, when several users repeat, receiver-known interleavers. The second view replaces channel-estimation pilots in both approaches.Source: §4.7 · pp. 59–60
  • CCA's guarantees differ: noiseless identifiability recovers the cell-edge subspace, while ensemble-correlation optimality gives the linear MMSE detector; both routes assume paired views and linear mixtures.More views can relax identifiability conditions, but unknown groups of shared users require jointly estimating station membership and shared structure. Receiver nonlinearities break the common-subspace premise and motivate nonlinear CCA [116].Source: §4.7 · pp. 60–61

5. Beyond Linear: Nonlinear and Deep CCA

Nonlinear CCA addresses views whose shared factors are obscured by nonlinear mappings, extending correlation-based methods toward common-factor recovery. But maximizing correlation alone does not guarantee recovery of the intended common factor.

5. Beyond Linear: Nonlinear and Deep CCA

  • Different nonlinear mappings can make shared neural activity or speech factors difficult for linear CCA to recover.Hemodynamic responses vary across brain regions and subjects, while articulatory-to-acoustic mappings can be nonlinear and many-to-one.Source: §5 · pp. 61–62
  • Nonlinear CCA can measure view correlation or recover common factors for downstream tasks, especially when unlabeled paired samples are plentiful and labels scarce.Agreement on unlabeled samples restricts candidate view-function pairs in the stated low-label regime.Source: §5 · p. 62
  • The working model forms each view from a common factor and its own private factor through a possibly nonlinear function.The section focuses mostly on recovering the common factor, rather than disentangling both common and private factors.Source: §5 · p. 62
  • The section’s objective maximizes correlation over nonlinear transformations of multivariable views, meeting linear CCA and HGR maximal correlation at their respective settings.Linear transformations yield CCA; arbitrary functions of single variables yield HGR maximal correlation.Source: §5 · p. 62
  • Correlation can be inflated by degenerate or overfit transformations, and even perfect agreement may recover only a coarsening of the common part.Finite-sample learning sharpens both the degeneracy and coarsening problems.Source: §5 · p. 63

5.1. Optimal Scaling: The Pre-Neural Nonlinear CCA

Optimal scaling extends CCA by transforming individual variables within classes determined by their measurement types, then jointly optimizing those transformations and canonical weights. Its nonlinear extension is limited to ordinal and nominal variables, and learned transformations can create degenerate agreement.

5.1. Optimal Scaling: The Pre-Neural Nonlinear CCA

  • Optimal scaling jointly chooses variable transformations and canonical weights to maximize correlation, with admissible transformations determined by whether variables are numerical, ordinal, or nominal.The classes are affine, monotone nondecreasing, and arbitrary category assignments, respectively.Source: §5.1 · p. 63
  • Using transformed variables in MAXVAR yields CORALS for two views and OVERALS for K views; CANALS instead maximizes squared canonical correlations and matches CORALS for one component.With all variables numerical, optimal scaling reduces to ordinary CCA.Source: §5.1 · p. 63
  • Each encoder transforms coordinates separately before linearly mixing them, the reverse order of a post-nonlinear mixture.The transmitter nonlinearity precedes mixing, whereas a receiver nonlinearity follows it.Source: §5.1 · p. 64
  • Because learned nominal and monotone transformations can merge distinct values, agreement may come from the transformations rather than a shared factor.Under extreme nonlinearities and high noise, CORALS may yield degenerate results.Source: §5.1 · p. 64

5.2. Kernel CCA

Kernel CCA maps each view into a kernel-defined function space and applies CCA to transformations of the views as wholes. This richer transformation class can manufacture agreement, making norm regularization essential for interpreting the result.

5.2. Kernel CCA

  • Kernel CCA relaxes coordinate-wise transformations by learning one RKHS function per view, but its richer class can manufacture agreement without reflecting shared structure.Optimal scaling remains coordinate-wise and restricts each transformation to a class fixed by variable type.Source: §5.2 · p. 64
  • A positive-definite kernel computes inner products of mapped points without explicitly forming possibly infinite-dimensional feature maps.The candidate functions can be expanded using mapped training samples, making kernel evaluations sufficient for the computations.Source: §5.2 · p. 65
  • Kernel CCA maximizes the covariance-to-variance ratio over view functions, with the function class distinguishing it from linear CCA and HGR maximal correlation.A linear kernel reduces the problem to Hotelling’s CCA.Source: §5.2 · p. 65
  • For translation-invariant kernels, the first kernel canonical correlation increases as Gaussian-kernel bandwidth shrinks, so the value alone may not reliably characterize view dependence.The attainable value also cannot decrease when the feasible RKHS is enlarged.Source: §5.2 · p. 65
  • Without a norm penalty, direct kernel CCA is degenerate for commonly used kernels; penalizing RKHS norms bounds candidate functions.Norm-penalized formulations add a ridge to the Gram-matrix constraints.Source: §5.2 · pp. 65, 67

5.3. Why Regularization Is Not Optional

Kernel CCA can report perfect sample correlations when the two views’ effective feature spaces overlap too much. Regularization addresses this degeneracy, while consistency results and density-based alternatives come with their own conditions and practical constraints.

5.3. Why Regularization Is Not Optional

  • The effective feature space has dimension at most N, and unpenalized kernel CCA leaves transformation size unbounded, so correlations can reflect that freedom rather than the common factor.The failure can be diagnosed from the dimensions of the two effective feature spaces.Source: §5.3 · p. 68, §5.3.1 · p. 68

5.3.1. The Degeneracy Is a Dimension Count

  • When d1 + d2 > N, the views’ kernel subspaces intersect and yield unit canonical correlations; with Gaussian kernels, every correlation is one for N > 2.The Gaussian-kernel Gram spaces both equal 1⊥ for distinct training samples. As bandwidth shrinks, penalized correlation approaches 1/(1 + λ) regardless of data; reported values should be compared with unrelated views under the same settings.Source: §5.3.1 · p. 69

5.3.2. The Population Target, and Consistency

  • With Gaussian kernels, the unpenalized empirical correlation estimates no population quantity, while penalized estimators have consistency results under their respective conditions.Fukumizu, Bach and Gretton show convergence of estimated functions under regularity conditions when λ vanishes more slowly than N^-1/3.Source: §5.3.2 · p. 70

5.3.3. The Unrestricted Problem Has Its Own Lineage

  • Lancaster-based methods estimate transformations through the joint density, avoiding kernel-matrix inversion, but their safeguard against degeneracy is density-estimate resolution.Joint-density sample requirements grow exponentially with total view dimension, and the practical method uses a small number of nearest neighbors.Source: §5.3.3 · pp. 70–71

5.4. From Kernels to Networks

Kernel CCA can be computationally constrained by its sample-sized Gram matrices. Deep CCA replaces the chosen kernel with learned neural transformations, trading kernel CCA’s closed-form geometry for fixed-size models that scale better to large datasets and new points.

5.4. From Kernels to Networks

  • Kernel CCA’s N × N Gram-matrix decomposition costs O(N3) directly, motivating approximations such as incomplete Cholesky, Nyström, randomized nonlinear CCA, and incremental SVD.Incremental SVD reduces the decomposition to O(m2N) time and O(mN) space for truncation rank m ≪ N.Source: §5.4 · p. 71
  • Deep CCA learns neural transformations instead of choosing a kernel, using fixed architecture weights for training and evaluating new points without comparing them to the training set.This gives up kernel CCA’s closed-form geometry and its dimension-counting test for degeneracy.Source: §5.4 · p. 71

5.5. Deep CCA and the Trivial Solution Problem

Deep CCA learns nonlinear view transformations to maximize agreement, but finite-sample consensus constraints can admit data-independent trivial solutions. Reconstruction-based objectives address that degeneracy in part, while introducing a trade-off between alignment and retaining view-specific information.

5.5. Deep CCA and the Trivial Solution Problem

  • Deep CCA learns neural transformations that maximize the sum of canonical correlations, while constraining each view’s D variates to be centered, uncorrelated, and unit variance.For fixed networks, the consensus vectors and linear maps can be solved in closed form; the empirical procedure centers outputs and uses ridge-regularized covariance estimates.Source: §5.5 · pp. 71–72
  • Empirical covariance ridge regularizes raw network outputs but does not prevent degeneracy, and the coupled constraints do not directly permit one-sample-at-a-time stochastic optimization.The original work found full-batch L-BFGS performed much better than minibatches; later work found sufficiently large minibatches suffice.Source: §5.5 · p. 72
  • For K views, Benton et al. extend MAXVAR deep CCA by learning view networks and linear maps whose outputs agree on a consensus vector.With networks fixed, they alternate between solving an eigenproblem for the consensus and backpropagating its alignment loss through the networks.Source: §5.5 · p. 72

5.5.1. Trivial Solutions Can Emerge

  • With sufficiently rich transformations and finite training samples, all views can map to any feasible consensus matrix, making every term of the empirical objective zero.The constraints require at least D + 1 nonzero rows; this family of trivial solutions is not specific to MAXVAR.Source: §5.5.1 · p. 73
  • Unlike Gács–Körner common information, which constrains agreement almost surely under the joint law, empirical agreement at N points can leave the solution set independent of that law.Witsenhausen’s criterion can certify that a nontrivial common part exists under mild conditions, but does not identify it.Source: §5.5.1 · p. 73

5.5.2. Reconstruction as a Remedy

  • Reconstruction-based objectives target downstream representations available from one view at test time; correlation-based objectives performed better than reconstruction alone, while DCCAE performed best on most tasks.DCCAE combines cross-view correlation with reconstruction of each view from its own encoding, with λ setting the trade-off and larger values favoring reconstruction.Source: §5.5.2 · pp. 73–74
  • Reconstruction forces encodings to distinguish samples that a deterministic decoder would otherwise map identically, but may require private information that makes the encodings harder to align.That private information is called leakage; the formulation does not guarantee recovery of anything else because reconstruction is imposed only at finitely many points.Source: §5.5.2 · p. 74

5.6. Probabilistic and Variational Formulations

Probabilistic nonlinear CCA models views as conditionally independent given a common latent factor and fits the model by likelihood-based objectives. These formulations offer inference alternatives but leave unresolved which common factor is identified, and private-factor models add their own separation challenge.

5.6. Probabilistic and Variational Formulations

  • Probabilistic formulations assume a generative model with a common latent factor and fit it by maximizing likelihood or a lower bound, rather than by optimizing correlation.For the common-factor model, views are conditionally independent given the factor, with each observation model parameterized by a neural network.Source: §5.6 · p. 75, §5.6.1 · p. 75

5.6.1. The Common-Factor Model

  • Conditional independence is a feasibility condition, not a definition of the common factor: feasible choices range from minimally informative factors to the tuple of all views.Likelihood depends on the factor through the induced view marginal, so models with the same empirical likelihood can learn different factors.Source: §5.6.1 · pp. 75–76

5.6.2. Deep Variational CCA

  • Deep variational CCA uses a single-view approximate posterior and maximizes a lower bound containing expected log likelihoods for both views.The single-view posterior is deliberate when only one view will be available downstream; an alternative combines bounds for both views [155].Source: §5.6.2 · p. 76
  • The two view encoders approximate different posteriors, so their outputs for a paired observation need not agree and the formulation does not specify which representation to use.When both views are available at test time, the authors concatenate them.Source: §5.6.2 · p. 77

5.6.3. The Multimodal Variational Autoencoders

  • MVAE, MMVAE, and MoPoE-VAE share the same generative assumption but differ in how they assemble a joint posterior from per-view approximations.Their development concerns inference rather than the model; despite motivating shared-private separation, the cited formulations do not include private variables in their main models.Source: §5.6.3 · p. 77

5.6.4. Variational CCA with Private Factors

  • A common-factor-only model can limit reconstruction when views contain substantial variation unexplained by the common factor, motivating explicit private factors.With Gaussian observation models, the best reconstruction from g alone is E[xk | g], so observations sharing g receive the same reconstruction.Source: §5.6.4 · pp. 77–78
  • VCCA with private factors can reconstruct a view through its private latent while ignoring the common factor, and dropout’s empirical gains do not guarantee factor separation.The broader question of when common and private factors can be reliably separated remains open; the subsection’s methods assume conditional independence given the common factor.Source: §5.6.4 · p. 78

5.7. What Makes the Common Factor Recoverable

The section shows that recovering a common factor requires explicit generative assumptions, not agreement objectives alone. Under different assumptions, two-view and multiview results identify the common factor up to invertible reparameterization, while practical optimization and finite-sample guarantees remain distinct concerns.

5.7.1. The Generative Model

  • The shared model maps each view from a common factor and its own private factor, while the works differ in their assumptions on latent dependence and mixing functions.Both use the same agreement criterion; Lyu et al. assume mutually independent latent factors, whereas Karakasis and Sidiropoulos permit private factors to depend on the common factor.Source: §5.7.1 · pp. 78–79, §5.7.2 · p. 79, §5.7.3 · p. 80

5.7.2. Invertibility Recovers the Common Factor, Independence Adds the Private Ones

  • Under the stated generative models, Lyu et al. identify the common factor for two views, while Karakasis and Sidiropoulos identify it for K views under weaker assumptions.The former also recovers private factors when its independence constraint is enforced; the latter allows private factors to depend on the common factor.Source: §5.7.2 · p. 80, §5.7.3 · pp. 80, 82

5.7.3. Partial Invertibility and Conditional Independence Recover the Common Factor

  • A representation shared by all views and rendering them conditionally independent is the common factor up to an invertible map, under partial invertibility of each mixing function.The common-factor model places it in both Gács–Körner and Wyner feasible sets, but their information values are infinite for continuous recoverable factors.Source: §5.7.3 · p. 82

5.7.4. A Tractable Relaxation

  • The practical relaxation combines cross-reconstruction with a penalty for imperfect agreement, excluding each view from reconstructing itself to prevent private-information leakage.Cross-reconstruction coincides with the original reconstruction objective at feasible points; optimization alternates encoder-decoder updates with a constrained consensus update.Source: §5.7.4 · pp. 83–84, §5.7.5 · p. 84

5.7.5. Algorithm

  • The consensus update reduces to an orthogonal Procrustes problem, while encoder and decoder updates use stochastic optimization.The consensus matrix is updated under orthogonality and centering constraints.Source: §5.7.5 · p. 84

5.7.6. Trivial Solutions

  • The identifiability theorems are population results, and the empirical objectives can still admit overfitting solutions unrelated to the common factor.The cited neural formulations do not establish convergence of finite-sample learned encoders to the factor.Source: §5.7.6 · p. 84

5.8. Connection to Self-Supervised Learning

Self-supervised learning can be viewed through the multiview lens: augmentations create views whose agreement should retain shared content without collapse. Its guarantees depend on the loss and the assumptions governing how views are generated or constructed.

5.8.1. The Original Argument

  • Becker and Hinton’s normalized covariation objective maximizes output correlation and avoids the constant-output optimum of a squared-difference agreement loss.For equal output variances, the ratio is (1 + ρ)/(1 − ρ).Source: §5.8.1 · p. 85

5.8.2. How Collapse Can Be Prevented

  • Self-supervised losses pair agreement with anti-collapse mechanisms: negatives in SimCLR, cross-correlation constraints in Barlow Twins, and variance and covariance penalties in VICReg.Barlow Twins’ off-diagonal constraint compares coordinates across views, not within a view; Barlow Twins and VICReg also admit trivial empirical solutions with sufficiently rich encoders.Source: §5.8.2 · pp. 85–86

5.8.3. When the Views Are Given After All

  • CLIP aligns image and text pairs, while ImageBind aligns each of six modalities to images and reports emergent alignment between modality pairs never seen together.ImageBind’s image-centered alignment resembles fitting all views to a shared representation.Source: §5.8.3 · p. 87

5.8.4. What Is Recovered

  • Identifiability results depend on the view-generating assumptions: contrastive learning recovers modeled latents, while content-style augmentations can identify content up to an invertible map.The latter result requires known content dimensionality; guarantees for designed augmentations do not automatically transfer to given views.Source: §5.8.4 · pp. 87–88

5.9. Summary

The summary highlights the limits of agreement-based learning and the role of explicit assumptions in identifying a common factor.

5.9. Summary

  • Rich function classes can overfit finite-sample objectives, and probabilistic conditional-independence models do not by themselves single out the common factor.Private factors may reconstruct individual views while the shared factor is ignored.Source: §5.9 · p. 88
  • Agreement alone permits representations that discard common information; reconstructing other views can discourage private-information leakage while retaining predictive shared content.Self-reconstruction can reward private variation, whereas cross-view reconstruction avoids that route under the stated model.Source: §5.9 · p. 88
  • Under explicit generative models, Lyu et al. and Karakasis and Sidiropoulos establish population identifiability up to an invertible map, but trivial solutions remain at fixed sample size.The multiview result allows dependent private factors when they are conditionally independent given the common factor and requires at least one view to be mean-dependent on it.Source: §5.9 · pp. 88–89
  • Self-supervised losses seek nontrivial agreement, and their identifiability assumptions depend on augmentations preserving content while changing style.The summary identifies negatives, soft whitening, and variance floors as different anti-collapse mechanisms.Source: §5.9 · p. 89

6. Further Extensions and Open Problems

The section surveys extensions of multiview recovery and identifies open problems in identifiability, clustering, finite-sample guarantees, and causal interpretation. Existing extensions address settings such as unpaired data, unknown latent dimensions, and nonlinear or adaptive estimation.

6.1.1. Unifying Generalized CCA with Subspace Clustering

  • SCoS jointly estimates view-cluster assignments and each cluster’s common subspace, with GCCA as its single-cluster case.It extends subspace clustering from vector-valued samples to matrix-valued ones, unifying the two formulations.Source: §6.1.1 · p. 89

6.1.2. Tensor CCA and the High-Order Correlation Criterion

  • High-order tensor CCA replaces pairwise criteria with a K-view covariance-tensor criterion equivalent to its best rank-one approximation.Deep high-order correlation and tensor-valued views are also considered, but the cited approaches lack identifiability results.Source: §6.1.2 · p. 89

6.1.3. Identifiability Without Paired Samples

  • Identifiability of shared components extends to linear mixtures without paired samples by matching cross-modality distributions.Content-style identifiability without known latent dimensions is also established for unaligned domains when learned representations are sparse.Source: §6.1.3 · p. 89, §6.1.4 · p. 90

6.1.5. Disentangling the Private Factors from the Common One

  • Multi-region neural models separate orthogonal shared and private subspaces using coupled transformer encoders and decoders that handle non-stationary dynamics.The approach is related to prior independence-based recovery of private factors.Source: §6.1.5 · p. 90

6.1.6. Preventing Collapse in Deep CCA

  • A noise regularizer is proposed to keep deep CCA network weights full rank, addressing performance drops associated with low-rank weights during training.The regularizer also applies to deep GCCA.Source: §6.1.6 · p. 90

6.1.7. Adaptive Estimation

  • Adaptive CCA updates canonical directions as samples arrive, complementing blind subspace tracking and enabling time synchronization for signal alignment.The standard CCA estimators discussed earlier require the full record before processing.Source: §6.1.7 · p. 90

6.1.8. Applications Beyond the Three Examples

  • Extensions include neural response separation, graph alignment, underlay communications, and packet alignment across frequency hops or multicarrier waveforms.The cited applications use GCCA, subspace intersection, or CCA with repetition coding, depending on the setting.Source: §6.1.8 · p. 90

6.2.1. Relaxing the Invertibility Condition

  • Open problems include joint-only recovery without single-view invertibility, private-factor identification under conditional independence, unknown view clustering, and finite-sample minimax limits.A further open question is whether view-invariant factors can support domain generalization: shared factors may be confounders and need not be stable across environments.Source: §6.2.1 · p. 91, §6.2.2 · p. 91, §6.2.3 · p. 91

7. Conclusions

The conclusion synthesizes information-theoretic, statistical, engineering, and machine-learning approaches to recovering what multiple views share. It emphasizes that agreement alone is insufficient: recovery depends on identifiable formulations, and shared variation need not be causal.

7. Conclusions

  • For scalar views, Gács–Körner and Wyner common information bound mutual information from below and above, and coincide with it under a shared conditionally independent common part.For vector views, CCA links canonical correlations to these information measures for jointly Gaussian data, while its relation to Gács–Körner common information is weaker.Source: §7 · p. 92
  • CCA recovers the common subspace exactly in the noiseless case under rank conditions, while generalized CCA’s intersection-based identifiability generically improves as views are added.The tutorial also connects blind multiuser detection to CCA receivers, including recovery of cell-edge users and convergence to the linear MMSE detector under repetition.Source: §7 · p. 92
  • Nonlinear CCA can produce agreement unrelated to the views, while explicit generative models establish common-factor identifiability up to an invertible map under stated assumptions.In the cited self-supervised result, the recovered factor is whatever the augmentations leave unchanged.Source: §7 · pp. 92–93
  • High correlation or learned agreement alone does not establish shared structure; the recovered factor may be a confounder and is not necessarily causal.The conclusion recommends checking that the learning criterion is well-posed and its identifiability conditions fit the application.Source: §7 · p. 93

Abbreviations

The section lists abbreviations used in the manuscript.

Abbreviations

  • CCA means Canonical Correlation Analysis, and GCCA means Generalized Canonical Correlation Analysis.The list also includes DCCA for Deep Canonical Correlation Analysis and KCCA for Kernel Canonical Correlation Analysis.Source: Abbreviations · pp. 93–94
  • CDMA means Code Division Multiple Access, and MMSE means Minimum Mean-Square Error.The list also defines MUD as Multiuser Detection and SVD as Singular Value Decomposition.Source: Abbreviations · p. 94
  • RKHS means Reproducing Kernel Hilbert Space, and SCoS means Subspace Clustering of Subspaces.The abbreviation list also includes VAE for Variational Autoencoder and IoT for Internet of Things.Source: Abbreviations · p. 94

Related papers

Loading 2610.28978v1…