Source-linked AI summary
Group Component Analysis for Multiblock Data: Common and Individual Feature Extraction
Guoxu Zhou, Andrzej Cichocki, Yu Zhang, Danilo Mandic
TL;DR
Linked multi-block data contain both shared and individual features, but existing methods may focus only on correlated structure. The paper proposes CIFE and COBE to separate these subspaces, then applies task-dependent extraction; experiments support classification and clustering applications, while broader machine-learning potential remains open.
Problem
Multi-block data contain shared and individual features, motivating methods that separate both rather than modeling only correlated structure.
Method
CIFE uses COBE algorithms to extract common bases with known or unknown component counts, then applies factorization methods separately in common and individual subspaces.
Results
Experiments on synthetic and real data provide evidence that CIFE supports classification and clustering using separated common and individual features.
Takeaways & Limitations
The framework provides a linked way to exploit common and individual features for multi-block data analysis and task-dependent feature extraction.
Abstract
from arXiv · showhide
Very often data we encounter in practice is a collection of matrices rather than a single matrix. These multi-block data are naturally linked and hence often share some common features and at the same time they have their own individual features, due to the background in which they are measured and collected. In this study we proposed a new scheme of common and individual feature analysis (CIFA) that processes multi-block data in a linked way aiming at discovering and separating their common and individual features. According to whether the number of common features is given or not, two efficient algorithms were proposed to extract the common basis which is shared by all data. Then feature extraction is performed on the common and the individual spaces separately by incorporating the techniques such as dimensionality reduction and blind source separation. We also discussed how the proposed CIFA can significantly improve the performance of classification and clustering tasks by exploiting common and individual features of samples respectively. Our experimental results show some encouraging features of the proposed methods in comparison to the state-of-the-art methods on synthetic and real data.
I. INTRODUCTION AND MOTIVATION
The paper addresses linked multi-block data that contain both shared and block-specific features. It proposes CIFE, including COBE algorithms and separate analysis of common and individual subspaces for feature extraction and learning tasks.
- Multi-block measurements share information while retaining individual features useful for characterizing the ensemble and identifying its members.
- Existing multiblock methods mainly model correlated features, whereas CIFE targets both common and individual variations.
- COBE provides efficient common orthogonal basis extraction algorithms for known or unknown numbers of common components.
- CIFE applies suitable factorization methods, including BSS and NMF, separately to common and individual subspaces rather than the global space.
- The model represents each block through shared and individual component matrices with corresponding coefficient matrices.
- The two-step procedure first reduces each block and then separates common and individual components without introducing additional factorization error.
A. The COBE Algorithm: The Number of Common Components C is Unknown
COBE extracts common orthogonal basis vectors sequentially when the number of common components is unknown. It uses alternating optimization, residual-space updates, and a threshold-based stopping rule.
- COBE estimates common basis vectors by alternating optimization over basis vectors and transformed block coefficients.
- A candidate basis vector is accepted when its minimized objective satisfies min f_k ≤ ϵ; otherwise extraction stops.
- After each accepted vector, COBE updates the residual spaces to prevent repeated common basis vectors.
- The algorithm returns C common orthogonal basis vectors after no further vector meets the threshold.
- The threshold ϵ controls whether extracted components are identical or approximately highly correlated.
- COBE can overestimate the number of components with a small threshold and then use SORTE to detect the common–individual subspace gap.
B. The COBE Algorithm When The Number of Common Components C is Given
When the number of common components is given, COBEC estimates the common basis through alternating optimization and a truncated SVD.
- COBEC alternates between optimizing transformed coefficients and the common basis for a specified component count C.
- The coefficient update aggregates the transformed block representations, while the basis update uses a truncated SVD of their sum.
- The optimal basis is formed from the leading singular-vector factors, yielding the COBEC algorithm.
C. Pre-processing: Dimensionality Reduction
Dimensionality reduction is required when observed dimension exceeds latent rank and also helps reduce complexity and data artifacts before COBE.
- COBE requires every block’s latent rank R_n to be smaller than the observed dimension D.
- If R_n < D is not satisfied, dimensionality reduction such as PCA should precede COBE.
- Preprocessing reduces computational complexity, noise, outliers, and artifacts while producing cleaned low-rank data for COBE.
- After preprocessing, COBE transforms the reduced representation so common and individual subspaces are completely disjoint.
- PCA suits i.i.d. Gaussian noise, whereas robust PCA is suggested for sparse distributions; SORTE can estimate latent component counts.
D. Relation to Other Methods
COBE extracts highly correlated common components across data sets, distinguishing them from PCA’s global components and JIVE’s joint variation. Its correlation-based formulation also offers efficient optimization and flexible estimation of the number of common components.
- COBE and CCA: COBE extracts components whose projected variables have correlations above a specified threshold, unlike CCA, which maximizes correlation.As the approximation error ϵ approaches zero, the projected-variable correlation approaches one, motivating COBE as higher-order correlation analysis.
- COBE and CCA: In a simulation, CCA produced canonical variables with correlation 0.8867, while COBE extracted common components that matched the highly correlated projected components.The common component was compared with components reconstructed from each data matrix and with the canonical variables.
- COBE, JIVE, and PCA: COBE extracts principal components of normalized common columns, whereas PCA uses all columns and JIVE captures joint variation.The comparison highlights that JIVE may produce principal components rather than highly correlated common components when those components have relatively weak energy.
- Computational comparison: Compared with JIVE, COBE is more efficient in optimization and more physically intuitive and flexible for estimating the number of common components.The computational comparison concerns JIVE’s repeated SVDs of large matrices containing all data.
E. Scalability For Large-Scale Problems
For large-scale data, COBE reduces computational demands through dimensionality reduction and random projection while preserving the common basis under the stated exact-recovery condition.
- Dimensionality reduction: Dimensionality-reduced matrices Q with R_n < D reduce the effective dimensionality used by COBE, so the value of J_n is generally not problematic.This addresses large indices D and J_n in the original data matrices.
- Random projection: A random projection P ∈ R^(D_P×D), with max_n(R_n) < D_P ≪ D, reduces time and memory requirements for very large D.The projected model is solved first in the lower-dimensional space.
- Basis recovery: After estimating W_n from the projected data, the original common basis is recovered as Ā = Y_nW_n, and no common basis vectors are lost when Y_nW_n = Ā.The projected relation P Y_nW_n = P Ā follows under the exact common-basis condition.
- Caveat: Random projection can create fake common components when residual errors occasionally lie in P’s null space, but these components can be detected in practice.The passage identifies this as a worst-case issue rather than the normal operating condition.
III. COMMON AND INDIVIDUAL FEATURE EXTRACTION (CIFE)
CIFE provides a general group component-analysis framework for extracting common and individual features from multi-block data. It is positioned as a linked alternative to methods that extract common information or joint variation differently.
- CIFE framework: CIFE is presented as a general framework for common and individual feature extraction from multi-block data.The framework is also described as group component analysis.
- Relation to existing methods: COBE extracts principal components of normalized common columns, while JIVE captures joint variation and may miss highly correlated components with relatively weak energy.This distinction motivates separating common-feature extraction from generic joint-variation extraction.
A. Linked BSS with Pre-whitening
Linked BSS applies source separation to the common subspace extracted from linked multi-block data. This enables common features with desired properties and supports more flexible component models than global-space analysis.
- Common-space source separation: BSS is applied after common-component extraction to resolve nonunique orthogonal bases and estimate latent common sources.The common basis may be rotated by any orthogonal matrix, so BSS imposes source-separation structure.
- Feature properties: Linked BSS can impose sparsity, independence, or nonnegativity on common features through suitable constraints and penalties.The procedure may also extract more than C common signals using underdetermined BSS methods.
- Comparison with JBSS: Unlike JBSS, linked BSS performs separation on the common subspace rather than the global data space, enabling more reliable group variables when blocks also contain individual components.JBSS extracts groups with the highest correlations and requires distinct correlations among extracted groups.
- Method flexibility: Linked BSS supports ICA, NMF, and other component-analysis methods, allowing components with varied properties and more physically meaningful interpretations.This flexibility is identified as a distinguishing property of the linked approach.
B. Common Nonnegative Features Extraction (CNFE)
CNFE first extracts a common subspace, then applies low-rank NMF or semiNMF to obtain common nonnegative components. Additional constraints can be imposed to improve component uniqueness.
- Common subspace extraction: CNFE first extracts the common subspace before factorizing the data into common nonnegative components.The common subspace is obtained using equation (11), after which a low-rank approximation is applied.
- Factorization model: Low-rank NMF is used when the coefficient matrix is nonnegative, whereas semiNMF is used when it is real-valued.
- Optimization: Iterative multiplicative updates produce nonnegative component and coefficient matrices.
- Identifiability: Additional constraints may be imposed, as in BSS, to extract unique nonnegative components.
C. Individual Feature Extraction (IFE)
IFE removes common components from each data matrix and analyzes the resulting rank-deficient individual spaces separately. The extracted individual features support task-specific visualization, discrimination, clustering, and recognition.
- Individual feature spaces: Each data matrix retains individual features after the common features are extracted.
- Individual feature spaces: The residual matrix is rank deficient, so its dimensionality reduction must be handled carefully before further analysis.Its rank satisfies rank(˘Yn) + rank(¯Y) = Rn.
- Individual feature extraction: IFE applies dimensionality reduction separately to each residual matrix, followed by BSS or related methods to extract individual features.
- Task-specific analysis: The dimensionality-reduction objective can be task-specific, such as visualization, discriminative feature extraction, or establishing neighbor relationships.
- Classification: For classification, test samples are matched against common-feature spaces extracted from each class, using scores based on distance or correlation.The score can be computed through least squares or CCA, respectively.
- Clustering and recognition: In the PIE database, COBE with CNFE produced common faces containing basic facial features and individual faces showing varied local features.The individual features were reported as helpful for clustering and recognition accuracy.
- Clustering and recognition: The individual-feature clustering procedure splits samples into groups, removes common components, extracts residual features, and clusters the resulting representations.
V. SIMULATIONS AND VALIDATION
Simulations on linked speech, image decomposition, clustering, and classification evaluate COBE and downstream common or individual feature extraction. Results indicate that COBE identifies common structure efficiently, while separating common and individual features supports clustering and classification.
- Linked BSS: COBE and JIVE achieved higher SIRs than JBSS and PCA, while COBE was much faster than JIVE in linked BSS experiments.
- Linked BSS: COBE identifies true common subspaces even when common components are relatively weak.
- Linked BSS: COBE’s computational complexity depends on data size and the number of common components, making it more efficient than JIVE.
- Linked BSS: Running time was almost linear in the projected dimension Dp, indicating that random projections can improve COBE efficiency for very large observation dimensions.
- Feature extraction: For dual-energy X-ray decomposition, COBE extracted common soft-tissue and bone sources before CNFE separated those nonnegative components.
- Clustering: Clustering performance improved significantly after removing the first 2–3 common components, although selecting the optimal number C remains challenging.
VI. CONCLUSIONS AND FUTURE WORK
The paper presents CIFE and COBE algorithms for separating common and individual features in linked multi-block data, with applications and experiments supporting their effectiveness. Future work concerns parameter selection, data grouping, higher-order extensions, and realizing CIFE’s broader machine-learning potential.
- CIFE provides a scheme for extracting common and individual features from naturally linked multi-block data.
- Two COBE algorithms extract common orthogonal bases depending on whether the number of common components is known.
- CIFE performs task-dependent feature extraction in common and individual subspaces rather than a single global space.
- The parameter ϵ controls the number of common features and their cross-dataset similarity, but its optimal theoretical value remains challenging to determine.
- Practical performance may depend on manually grouping data, while extending CIFE from one-dimensional common features to tensor data remains future work.
- Two example applications provide experimental evidence for CIFE’s validity and performance, while its full potential in machine learning remains open.