Source-linked AI summary
Building Deep Networks on Grassmann Manifolds
Zhiwu Huang, Jiqing Wu, Luc Van Gool
TL;DR
Deep learning on Grassmann manifolds was limited despite their use in visual recognition and prior shallow or embedded learning approaches. The paper develops a Grassmannian deep network with geometry-aware layers and manifold optimization, achieving stronger performance than existing Grassmann methods and results comparable to state-of-the-art approaches. The study also leaves end-to-end ConvNet+GrNet training for future work.
Problem
Grassmannian representations are useful in visual recognition, but prior Grassmann learning included shallow schemes, motivating deep representation learning directly on these nonlinear manifolds.
Method
The paper builds a deep Grassmann network using full-rank mapping, re-orthonormalization, projection pooling, projection mapping, manifold SGD, and generalized backpropagation.
Results
In three visual classification evaluations, Grassmann networks significantly outperformed existing Grassmann learning methods and performed comparably with state-of-the-art methods.
Takeaways & Limitations
The work demonstrates a deep network architecture for learning representations directly over Grassmann manifolds.
Takeaways & Limitations
The study leaves end-to-end ConvNet+GrNet training for future work.
Abstract
from arXiv · showhide
Learning representations on Grassmann manifolds is popular in quite a few visual recognition tasks. In order to enable deep learning on Grassmann manifolds, this paper proposes a deep network architecture by generalizing the Euclidean network paradigm to Grassmann manifolds. In particular, we design full rank mapping layers to transform input Grassmannian data to more desirable ones, exploit re-orthonormalization layers to normalize the resulting matrices, study projection pooling layers to reduce the model complexity in the Grassmannian context, and devise projection mapping layers to respect Grassmannian geometry and meanwhile achieve Euclidean forms for regular output layers. To train the Grassmann networks, we exploit a stochastic gradient descent setting on manifolds of the connection weights, and study a matrix generalization of backpropagation to update the structured data. The evaluations on three visual recognition tasks show that our Grassmann networks have clear advantages over existing Grassmann learning methods, and achieve results comparable with state-of-the-art approaches.
Introduction
The paper introduces a deep network architecture that accepts Grassmannian data directly and learns favorable representations for visual recognition. It also generalizes backpropagation to train network parameters and structured data on manifolds.
- Introduction: Grassmannian data are motivated by visual applications including face verification, emotion estimation, and activity recognition.These applications commonly model image sets using low-dimensional linear subspaces compared on Grassmannians.
- Introduction: The proposed architecture performs end-to-end deep representation learning directly on Grassmann manifolds.It is designed to learn new Grassmannian representations that improve final visual recognition tasks.
- Introduction: The paper generalizes backpropagation with a connection-weight update rule on a Riemannian manifold.It also incorporates QR decomposition into backpropagation.
Background
Grassmann manifolds represent linear subspaces with intrinsic Riemannian geometry. Prior learning methods commonly embed them into tangent or Hilbert spaces, while this paper targets deep learning directly on Grassmannians.
- Background: A Grassmann manifold Gr(q, D) is the compact Riemannian manifold of q-dimensional linear subspaces of R^D.Each point is represented by an orthonormal basis matrix X satisfying X^T X = I_q.
- Background: Projection mapping Φ(X) = XX^T represents Grassmannian subspaces as symmetric matrices and induces the projection metric.The associated inner product is defined using the trace of projected matrices.
- Background: Existing discriminant methods typically use tangent-space approximations or positive-definite kernels to apply Euclidean techniques after embedding.These approaches rely on tangent spaces or reproducing kernel Hilbert spaces.
- Background: Prior geometry-aware dimensionality reduction on Grassmannians remained shallow, motivating deep representation learning on nonlinear manifolds.The cited scheme maps data to a lower-dimensional, more discriminative Grassmann manifold while preserving its Riemannian structure.
- Background: Before this work, manifold networks had been developed for other domains, including shape manifolds and symmetric positive-definite matrices.The paper identifies its architecture as the first deep network studied specifically on Grassmann manifolds.
Grassmann Network Architecture
GrNet generalizes neural-network blocks to Grassmannian data through full-rank mappings, re-orthonormalization, projection pooling, and projection mapping to Euclidean outputs.
- Projection block: ReOrth layers apply QR decomposition to normalize mapped matrices into orthonormal basis matrices on the Stiefel manifold.The QR factors are used to recover an orthonormal matrix, and the operation also supplies a nonlinear activation mechanism.
- Projection block: FRMap layers linearly transform input orthonormal subspace matrices into new matrices using full-rank connection weights.The transformation is followed by ReOrth normalization because the mapped result is generally not an orthonormal basis.
- Overall architecture: The architecture uses three basic block types—Projection, Pooling, and Output—to build Grassmann network configurations.The Projection block contains FRMap and ReOrth layers, while the Output block follows projection mapping.
- Output block: ProjMap converts orthonormal matrices into projection-matrix representations in a flat space where Euclidean layers and softmax can be applied.The architecture can therefore place fully connected and other regular output layers above the Grassmannian computation.
- Pooling block: Projection pooling maps Grassmannian data to projection matrices, performs Euclidean mean pooling, and maps the result back to orthonormal data.A-ProjPooling pools across projection matrices, whereas W-ProjPooling applies a sliding mean filter within square patches.
- Pooling block: A-ProjPooling with OrthMap outputs m n orthonormal matrices of size dk−1 × q, while W-ProjPooling with OrthMap outputs m orthonormal matrices of size dk−1 √n × q.The two pooling variants differ in whether they pool across matrices or within patches of one projection matrix.
Training Grassmann Network
Training GrNet requires manifold-aware optimization for structured weights and matrix-specific backpropagation for layers built from matrix factorizations.
- Matrix backpropagation: Traditional element-wise backpropagation is insufficient because many GrNet layers use complex matrix factorization functions.The paper therefore adopts matrix backpropagation based on variations of upper-layer variables with respect to lower-layer variables.
- Training formulation: GrNet is represented as successive function compositions with a weight matrix assigned to each layer and a loss propagated from the final output.The layerwise loss is defined by composing the remaining network functions with the final loss function.
- Matrix backpropagation: The matrix backpropagation chain rule uses a nonlinear adjoint operator F* under the matrix inner product to propagate derivatives.Here, F maps input variations dXk−1 to output variations dXk, while F* defines the corresponding adjoint relation.
- Manifold optimization: FRMap weights are updated by stochastic gradient descent on PSD manifolds using tangent-space transport, removal of the normal component, and retraction.This procedure preserves the manifold structure while updating the connection weights.
- Layer-specific derivatives: The paper derives structured-data derivatives for ReOrth layers by treating QR decomposition as a virtual tuple-output layer.Variations of Q and R are combined with the matrix chain rule to obtain the ReOrth update rule.
- Layer-specific derivatives: OrthMap derivatives are obtained using the stated proposition for eigenvalue decomposition, including derivatives with respect to eigenvectors and eigenvalues.The proposition assumes a symmetric decomposition Xk−1 = UΣU^T and specifies the eigenvalue-gap-dependent matrix used in the derivative.
Empirical Evaluation
GrNet is evaluated against general manifold-learning methods, Grassmann methods, standard ConvNets, and SPD-based manifold networks across emotion recognition, action recognition, and face verification. It generally improves over several baselines, with performance gains linked to geometry-aware layers, deeper stacking, and compact matrix representations, while some comparisons remain unfavorable.
- Evaluation setup: GrNet is evaluated on AFEW, HDM05, and PaSC for emotion recognition, action recognition, and face verification.The comparison includes general manifold-learning methods, Grassmann learning methods, and standard ConvNets.
- Results: More than 11%: GrNet outperforms state-of-the-art Grassmann learning methods on HDM05.The paper attributes this improvement to the larger amount of training data available for that task.
- Results: GrNet with 2 blocks achieves the highest reported PaSC performance despite the softmax output layer being poorly suited to verification.The result is presented as evidence that the model learns more favorable Grassmannian representations.
- Ablations: 17.62% and 26.15%: removing ProjMap and ReOrth, respectively, produces very poor AFEW performance.These ablations correspond to removing Riemannian computing and geometry-aware learning components.
- Ablations: Stacking more GrNet blocks consistently improves classification performance by learning more discriminative Grassmannian representations.The paper uses this trend to support deeper GrNet architectures.
- Architectural comparisons: W-ProjPooling typically outperforms A-ProjPooling, while GrNet surpasses DeepO2P but does not surpass SPDNets.A-ProjPooling’s weaker performance is associated with extrinsic Grassmannian means and weak relationships across multiple projection matrices.
- Efficiency: GrNet processes d × q orthonormal matrices, usually with q set to 10, whereas SPDNet processes d × d SPD matrices.The paper argues that this lower-dimensional representation makes GrNet theoretically faster, although reported per-epoch times vary across datasets and methods.
- Convergence: Up to 40%: validation improvement is reported for GrNet on HDM05 after training, compared with near 12% on AFEW.The paper also reports that GrNet can converge in fewer epochs than SPDNet on AFEW.
Conclusion
The paper presents the first deep-learning architecture for Grassmann manifolds and reports strong performance across visual classification evaluations, with comparisons summarized in Figure 2.
- Conclusion: The paper introduces the first network architecture for deep learning over Grassmann manifolds.
- Conclusion: Grassmann networks significantly outperform existing Grassmann learning methods across three typical visual classification evaluations.
- Conclusion: The networks perform comparably with state-of-the-art methods in the reported evaluations.
- Conclusion: Figure 2 compares single versus multiple FRMap, pooling across versus within projections, and SPDNet versus GrNet convergence and accuracy on AFEW.
Appendix
The appendix derives gradients for QR decomposition by differentiating its implicit system and manipulating triangular and orthogonality relations to obtain gradients with respect to R and Q.
- Appendix: The derivation begins by differentiating the implicit system of the QR decomposition.
- Appendix: The appendix uses Rtril to isolate elements below R's main diagonal before deriving intermediate relations.
- Appendix: Multiplication by Q^T and R^-1 transforms the QR differential equations into forms used for gradient derivation.
- Appendix: The antisymmetry of Q^T dQ determines its lower-triangular relation and supports the gradient with respect to R.
- Appendix: Substituting the derived expressions yields the QR gradients with respect to R and Q.
- Appendix: Matrix inner-product identities and lower-triangular symmetric and antisymmetric components produce an equivalent gradient equation.