Classification and Geometry of General Perceptual Manifolds

SueYeon Chung; Daniel D. Lee; Haim Sompolinsky

Source-linked AI summary

Classification and Geometry of General Perceptual Manifolds

SueYeon Chung, Daniel D. Lee, Haim Sompolinsky

arXiv:1710.06487v3Oct 17, 2017cond-mat.dis-nncond-mat.stat-mechcs.NEq-bio.NCstat.ML

TL;DR

The paper addresses how neural systems can classify objects despite continuous within-object variability represented by perceptual manifolds. It develops a statistical mechanical theory for linear separation of general manifold geometries, introducing geometry measures and analyzing representative manifolds and sparse labels. The theory relates classification capacity to manifold radius, dimension, geometry, and label sparsity, with predictions corroborated by numerical evaluations.

Problem
Invariant object recognition requires separating neural manifolds that contain substantial variability, while prior perceptron theories addressed finite random points rather than geometrically organized manifolds.
Method
The paper derives mean-field and self-consistent KKT equations for linear classification of general finite-dimensional manifolds, using anchor-point statistics to define effective geometric measures.
Results
The theory explains capacity across ℓ2 ellipsoids, ℓ1 polytopes, ring manifolds, and sparse-label settings through manifold radius, dimension, and label sparsity.
Takeaways & Limitations
The framework provides geometry-based measures for assessing linear separability and how perceptual manifolds are reformatted in biological and artificial systems.
Takeaways & Limitations
Application to real data may require further extensions of the theory.

Abstract

from arXiv · show

Perceptual manifolds arise when a neural population responds to an ensemble of sensory signals associated with different physical features (e.g., orientation, pose, scale, location, and intensity) of the same perceptual object. Object recognition and discrimination requires classifying the manifolds in a manner that is insensitive to variability within a manifold. How neuronal systems give rise to invariant object classification and recognition is a fundamental problem in brain theory as well as in machine learning. Here we study the ability of a readout network to classify objects from their perceptual manifold representations. We develop a statistical mechanical theory for the linear classification of manifolds with arbitrary geometry revealing a remarkable relation to the mathematics of conic decomposition. Novel geometrical measures of manifold radius and manifold dimension are introduced which can explain the classification capacity for manifolds of various geometries. The general theory is demonstrated on a number of representative manifolds, including L2 ellipsoids prototypical of strictly convex manifolds, L1 balls representing polytopes consisting of finite sample points, and orientation manifolds which arise from neurons tuned to respond to a continuous angle variable, such as object orientation. The effects of label sparsity on the classification capacity of manifolds are elucidated, revealing a scaling relation between label sparsity and manifold radius. Theoretical predictions are corroborated by numerical simulations using recently developed algorithms to compute maximum margin solutions for manifold dichotomies. Our theory and its extensions provide a powerful and rich framework for applying statistical mechanics of linear classification to data arising from neuronal responses to object stimuli, as well as to artificial deep networks trained for object recognition tasks.

I. INTRODUCTION

The paper asks how linear classifiers can separate perceptual manifolds—neural representations of objects varying in physical features—and develops a general theory linking separability to manifold geometry. It introduces geometric measures, analyzes representative manifold classes and sparse labels, and derives capacity bounds and transitions.

Motivation: Perceptual manifolds represent within-object variability in neural state space, making invariant object recognition equivalent to discriminating manifolds with a linear decoder.The motivating variability includes orientation, position, pose, lighting, background, acoustic variation, and odor concentration.
Problem: Existing perceptron theories classify finite point sets and do not address infinitely many inputs organized geometrically as manifolds.The paper extends earlier analyses of simple ball geometries to general finite-dimensional manifolds.
Capacity: In high ambient dimension, the maximal number of separable finite-dimensional manifolds is proportional to N despite each manifold containing infinitely many points.The theory bounds capacity between isolated-point and full-affine-subspace limits and defines αM(κ) as the maximal separable load at margin κ.
Method: The theory uses replica-derived mean-field equations and self-consistent KKT conditions based on manifold anchor points to characterize linear-separation capacity and optimal separating weights.Anchor-point statistics describe support vectors and the dimensions of the sets intersected by the optimal separating plane.
Geometric measures: Anchor radius and dimension, together with Gaussian radius and dimension, approximate manifold capacity through equivalent ℓ2-ball geometry and quantify representation changes in neural and artificial systems.For many realistic cases below the crossover, the manifold dimensionality is substantially smaller than estimates from naive second-order statistics.
Representative geometries: The theory applies to smooth convex, convex-polytope, and smooth nonconvex ring manifolds, whose capacity and geometric measures smoothly cross over from high-capacity small manifolds to low-capacity large manifolds.The examples include ℓ2 ellipsoids, ℓ1 polytopes, and orientation-like ring manifolds.
Sparse labels: Sparse labels enhance manifold capacity only when fR_g^2 ≪ 1; when fR_g^2 ≫ 1, capacity remains low and approaches 1/D even for extremely small f.Over a broad parameter regime, sparsely labeled manifolds are approximately described by sparsely labeled ℓ2 balls with radius R_g and dimension D_g.
Scope: The theory yields quantitative and qualitative predictions for perceptron classification of realistic data structures, while applications to real data may require further extensions.Numerical evaluations corroborate the theoretical predictions for the analyzed manifold classes and sparse-label regimes.

III. STATISTICAL MECHANICAL THEORY

The paper extends statistical-mechanical perceptron theory to linearly separating finite-dimensional manifolds under Gaussian and random-label assumptions. Replica analysis yields an exact thermodynamic-limit inverse-capacity expression based on manifold support functions, anchor points, and maximum-margin constraints.

Statistical assumptions: Gaussian manifold bases and independently random labels define the thermodynamic-limit model with fixed affine dimension and finite load.The components of each affine basis vector are i.i.d. Gaussian, labels are equally likely ±1, and N,P→∞ with α=P/N finite.
Motivation: Finite-dimensional manifolds require extending point-classification theories because their infinitely many points possess geometric organization.The resulting capacity depends on finite manifold size and geometry, while large affine dimension can make simple bounds loose.
Mean-field derivation: Replica theory derives mean-field equations for manifold separation capacity and the optimal separating vector.The calculation follows Gardner’s framework by averaging log Z, the volume of admissible solution vectors.
Capacity expression: The inverse capacity is expressed as an average over Gaussian vectors of a per-manifold optimization problem involving the support function.The Gaussian vector represents quenched variability from manifold bases and labels, while F imposes the minimum-projection constraint.
KKT characterization: KKT conditions characterize the unique optimal field vector through a scale factor and an anchor-point subgradient of the support function.The scale factor is zero when the margin constraint is inactive and positive when the optimal field differs from the Gaussian input vector.

B. Mean field interpretation of the KKT relations

The KKT solution admits a mean-field interpretation in which each manifold contributes an anchor point to the maximum-margin classifier. Conic decomposition links these anchors and support dimensions to the position of a random field relative to manifold-associated cones.

Mean field interpretation: The maximum-margin weight vector is represented as a linear combination of one convex-hull support vector per manifold.In the large-N limit, these manifold contributions are treated as mutually uncorrelated.
Anchor points: Each anchor point depends on the orientations of all other manifolds through a random Gaussian field, not solely on its associated manifold.For a fixed manifold, changing the other manifolds changes the anchor location within its convex hull.
Support regimes: Support manifolds are classified by support dimension: touching manifolds have k=1, whereas fully supporting manifolds have k=D+1.Touching manifolds meet the margin hyperplane only at their anchor point; fully supporting manifolds lie entirely in it.
Conic decomposition: The KKT relations decompose −T into a component in a shifted polar cone and a component λS̃ in the manifold cone.At zero margin the components are orthogonal by Moreau decomposition, whereas nonzero margin removes that orthogonality requirement.
Geometric regimes: The cone location determines support structure: −T in the shifted polar cone gives k=0, while −T in the manifold cone gives full support k=D+1.These cases correspond respectively to λ=0 and V=κc.

E. Numerical solution of the mean field equations

The mean-field equations are solved by computing an anchor point for each Gaussian field and averaging its capacity contribution, analytically for simple geometries and numerically for complex ones. Varying manifold size and margin produces distinct interior, touching, partially supporting, and fully supporting regimes.

Numerical solution: Mean-field evaluation has two stages: solve for V and S̃ at fixed T, then average the inverse-capacity contribution over Gaussian T.The first stage is analytic for simple geometries such as ℓ2 ellipsoids and numerical for more complicated ones.
Numerical solution: Finite-manifold calculations use quadratic semi-infinite programming because each manifold may contain infinitely many points.The authors compare mean-field predictions with maximum-margin simulations using a specialized manifold-classification algorithm.
Support regimes: For sufficiently positive t0, manifolds are interior with λ=0; as t0 decreases, touching and partially supporting regimes can emerge.Partially supporting manifolds satisfy tfs(t)≤t0−κ≤ttouch(t) and have support dimension 1≤k≤D.
Support regimes: At sufficiently negative t0, manifolds become fully supporting with v0=κ, v=0, and anchor points inside the convex hull.This regime occurs when t0−κ<tfs and has support dimension D+1.
Size effects: As manifold size approaches zero, capacity approaches isolated-point classification, whereas large size approaches affine-subspace separation.For large size, support structures include fully supporting and near-parallel touching or partially supporting regimes.
Margin effects: For κ≫1, capacity approaches that of P random points with αM≈κ^-2, independent of manifold geometry.Increasing κ raises the probability of supporting regimes and decreases the anchor-point magnitude.

D. Manifold anchor geometry

The paper defines manifold anchor geometry from anchor-point statistics induced by Gaussian vectors, yielding radius and dimension measures relevant to linear classification. It contrasts this geometry with Gaussian geometry and illustrates how shape affects anchor locations and angular distributions.

Anchor geometry: Manifold anchor geometry uses anchor-point statistics induced by Gaussian vectors to characterize classification-relevant manifold structure.These statistics determine classification properties and supporting structures associated with the maximum-margin solution.
Geometric measures: The manifold anchor radius RM is defined from the mean squared length of the anchor points, while DM measures their angular spread.DM is bounded by the affine dimension D.
Geometric measures: Unlike Gaussian geometry, RM and DM generally depend on the margin and on the distribution of both Gaussian vectors and the scalar variable t0.The dependence arises because the optimal vector and anchor point depend on t0 − κ.
Gaussian geometry: For small manifolds, the Gaussian anchor is the unique boundary point touched by a hyperplane normal to −t, except for measure-zero directions.For polytopes, the anchor is usually a vertex rather than a point along an edge.
Gaussian geometry: Gaussian geometry uses Rg and Dg, which measure Gaussian anchor amplitude and angular spread; for general manifolds, Dg can be much smaller than affine dimension D.For an ℓ2 ball, Rg equals its radius and Dg equals D.
Gaussian geometry: For ellipsoids, manifold-anchor norms can extend below the minor radius and the angle between −t and the anchor varies, unlike the Gaussian geometry.For the D = 2 ellipsoid example, anchor-geometry angles are more concentrated near zero because of fully supporting configurations.

F. Geometry and classification of high dimensional manifolds

In high dimensions, manifold classification capacity is governed by the anchor radius RM and dimension DM, often through an equivalent ℓ2-ball approximation. Scaling and support regimes determine when this approximation and the Gaussian limit apply.

High-dimensional approximation: For high-dimensional manifolds, classification capacity can be described using the anchor radius RM and dimension DM alone.The paper defines high-dimensional manifolds by DM ≫ 1 while DM remains finite in the thermodynamic limit.
High-dimensional approximation: The capacity of a general high-dimensional manifold is well approximated by that of an ℓ2 ball with dimension DM and radius RM.The effective margin is approximately RM√DM, with input-norm corrections included in the full expression.
Scaling regime: Finite capacity in the high-dimensional regime requires the effective margin κM = RM√DM to remain order unity, implying a small-radius scaling with increasing DM.In this scaling regime, Gaussian statistics can replace the full anchor geometry to leading order.
Support regimes: For strictly convex manifolds with RM = O(1), touching supports dominate, whereas non-strictly convex manifolds can contribute through partial supports and large-RM manifolds through fully supporting regimes.The relevant support structure changes as manifold radius and geometry vary.

B. Convex polytopes: ℓ1 ellipsoids

ℓ1 balls model convex polytopes whose support structure changes across radius scales. Their classification capacity is captured well by equivalent-ball parameters, while their effective dimension grows from logarithmic to affine-dimensional values.

Geometry: ℓ1 ellipsoids are convex polytopes formed as convex hulls of finitely many points, with the equal-radius case corresponding to ℓ1 balls.The high-dimensional analysis considers equal principal radii Ri = R.
Scaling regime: In the small-radius scaling regime, the ℓ1-ball anchor is usually a vertex selected by the largest-magnitude Gaussian component.For large D, the maximum component concentrates near √(2 log D).
Scaling regime: Dg = 2 log D in the scaling regime, much smaller than affine dimension D, while Rg equals the common radius R.The resulting effective margin is κg = R√(2 log D).
Support regimes: As radius increases, support dimensions shift from touching solutions toward intermediate faces and eventually nearly fully supporting configurations.All support dimensions 1 ≤ k ≤ D + 1 can occur at intermediate sizes.
Classification capacity: For D = 100, capacity approaches 2 as r → 0 and 1/D = 0.01 as r → ∞, while simulations resemble an ℓ2 ball with RM and DM.RM ≈ r for r < 0.1 but becomes much smaller than r at large radius.

C. Smooth nonconvex manifolds: Ring manifolds

Ring manifolds model neural responses to periodic variables such as orientation as smooth, non-convex curves in neural state space. Their classification geometry changes systematically with scale, including a logarithmic-to-linear growth in manifold dimension.

Ring manifold construction: Ring manifolds represent smooth periodic neural responses to a continuous variable θ, such as object orientation, and generally span multiple linear dimensions.They are non-convex curves whose convex hulls have complex geometry.
Ring manifold construction: Each ring manifold is a closed, non-intersecting smooth curve on the surface of a D-dimensional sphere, reducing to a circle when D = 2.The model uses Fourier components to parameterize the neural response.
Convex-hull geometry: The ring manifold’s convex hull contains support faces of varying dimensions, including partially supporting solutions rather than only point-, line-, or fully supporting faces.This geometry links ring manifolds to properties studied for trigonometric moment curves and polytopes.
Classification geometry: In the scaling regime, the Gaussian manifold dimension grows roughly as Dg ≈ 2 log D, then increases dramatically toward D as r grows.This logarithmic scaling resembles that of ℓ1-ball polytopes.
Classification geometry: As manifold size increases, classification capacity decreases smoothly while radius and dimension increase, producing a crossover near Rg ∝ 1/√Dg.For smaller manifolds, dimensionality can remain substantially below naive second-order estimates.

VI. MANIFOLDS WITH SPARSE LABELS

Sparse-label classification is governed by the interaction between label fraction, manifold size, and geometry. The theory predicts distinct regimes in which sparsity can strongly increase capacity for small manifolds but large manifolds constrain capacity toward 1/D.

Theory and setup: The sparsity parameter f is the fraction of positively labeled manifolds, with f = 0.5 representing balanced labels.Sparse labels require optimizing a nonzero bias because the bias affects the two classes differently.
Theory and setup: Optimizing the bias converts sparse-label classification into a capacity problem αM(κ, f) = max_b αM(κ, f, b).The bias contributes positively to the margin for positive manifolds and negatively for negative manifolds.
Sparsity regimes: For small manifolds, capacity increases as sparsity decreases, following αM(0, f) ∝ 1/(f|log f|), similar to uncorrelated points.Large manifold directions constrain this gain because the separating solution must become orthogonal to them.
Sparsity regimes: The scaled sparsity f̄ = f(1 + Rg^2) organizes the crossover between sparsity and manifold size, with capacity roughly proportional to f̄^-1 at small f̄.The proportionality constant depends on the Gaussian manifold dimension Dg.
Sparsity regimes: When f̄ exceeds 1, capacity becomes small and approaches 1/D for sufficiently large manifolds, rather than the 1/Dg limit of the spherical approximation.This large-size regime depends on the detailed manifold geometry.
Numerical validation: Across ℓ2 balls, ℓ1 ellipsoids, and ring manifolds, simulations agree well with mean-field theory and the spherical approximation for f̄ < 1.The similar capacity decline across examples supports scaled sparsity as a common organizing variable.
Class-dependent geometry: Sparsity and bias alter anchor geometry differently for majority and minority classes even when the underlying manifold shape is unchanged.Figure 12 tracks class-specific changes in manifold radius and dimension as scaled sparsity varies.

VII. SUMMARY AND DISCUSSION

The paper develops a statistical-mechanical theory for linear classification of general perceptual manifolds and introduces geometry-based measures tied to separability. It extends prior point-based perceptron theory toward invariant discrimination and analyzes implications for neural and deep-network representations.

The theory applies to compact subsets of any finite-dimensional affine subspace, including continuously varying responses and finite sampled stimuli.
Universal mean-field equations characterize manifold classification capacity at a given margin, with iterative algorithms for complex geometries.The algorithms solve only O(D) variables for a single manifold rather than simulating a full system of P manifolds embedded in R^N.
The framework addresses the gap between point-based perceptron theory and classification of geometrically organized, potentially infinite manifolds.
The geometric measures are designed to determine linear separability and assess the quality of neural representations.
Across deep layers, manifold geometry can quantify representation untangling and compare linear readout performance across biological and artificial networks.
The theory assumes uncorrelated affine-subspace directions and may require extensions for correlations, nonlinear transformations, noise, and nonzero training error.

Appendix A: Replica Theory of Manifold Capacity

Appendix A derives the replica-theory formulation of manifold capacity from the volume of separating solutions. The derivation uses Gaussian manifold directions, random labels, margin-scaled constraints, and a replica-symmetric saddle-point analysis.

Capacity α_M(κ) is the maximal load α = P/N for which a margin-κ separating weight vector exists with high probability.
The analysis assumes independently Gaussian manifold directions, equally likely binary labels, and the thermodynamic limit N, P → ∞ with fixed load α.
The thermodynamic margin must be scaled as κ′ = ||x||κ/√N; under ||x^μ|| = O(1), the constraint uses y^μw·x^μ ≥ κ.
Manifold separability is expressed through signed projections of affine directions and the support function g_S of the coordinate shape S.
Replica averaging converts the typical solution-volume calculation into a replica-symmetric saddle-point problem with overlap order parameter q.
The fields contain quenched randomness from Gaussian manifold directions and thermal variability from the solution space.
At capacity, the solution volume vanishes, solution overlap approaches unity, and the capacity follows from the leading-order saddle-point expression.

Appendix B: Strictly Convex Manifolds

For strictly convex manifolds, the general capacity expression simplifies because the boundary lacks edges or higher-dimensional flats. The resulting capacity is expressed through touching and fully supporting regimes.

Strict convexity means every nontrivial line segment between manifold points lies in the interior, eliminating boundary edges or flats of spanning dimension k > 1.
When t_0 < κ + t_fs(t), the manifold is fully embedded, the anchor displacement v is zero, and the integrand reduces to the convex-manifold expression.
The convex-manifold capacity is written using the touching and fully supporting thresholds together with the minimizing anchor point.

B.2. ℓ2 Balls

The ℓ2-ball case yields explicit contact geometry and recovers earlier ball-capacity results. In the large-size limit, manifold separation approaches separation of affine subspaces with dimension D.

ℓ2 Balls: For an ℓ2 ball of radius R, the support function is g(v) = −R||v||, giving an explicit touching threshold.
ℓ2 Balls: The general convex-manifold expression reduces to the previously established capacity of balls.
Ellipsoids: For ellipsoids, the support function is evaluated on the boundary, with principal radii determining the anchor point across interior, touching, and fully supporting regimes.
Ellipsoids: In the fully supporting regime, the center and entire ellipsoid support the max-margin solution, while the anchor lies at an interior point antiparallel to t.
Large-size limit: As manifold size grows, linear separation approaches separation of P random D-dimensional affine subspaces, which must be fully embedded in the margin plane.
Large-size limit: At large sizes, the interior regime has negligible fractional volume, while touching and partially embedded regimes dominate the support structure.

Appendix D: High Dimensional Manifolds

High-dimensional manifold capacity can be approximated using effective anchor radius and dimension, while support structure changes across manifold-size regimes. The approximations rely on self-averaging and may require retaining longitudinal variability for greater accuracy.

Approximation limits: The simplified formulas should retain dependence on the longitudinal Gaussian variable t0 when anchor radius and dimension vary with t0.A more accurate treatment substitutes RM(t0) and DM(t0) and averages only over intrinsic coordinates.
High-dimensional balls: For high-dimensional balls, support behavior varies with radius: large-radius touching balls become nearly parallel to margin planes, causing capacity to reach its lower bound.The fully supporting fraction becomes negligible in this regime.
High-dimensional balls: In the large-manifold limit, systems are either touching or fully supporting, with probabilities H(κ) and H(−κ), respectively.This regime is realized when R is much smaller than the relevant dimensional scale.
General manifolds: For large ambient dimension, self-averaging reduces intrinsic-coordinate sums and yields simplified capacity expressions for general manifolds.The touching-point condition is approximated by ttouch ≈ t·s ≈ κM.

Appendix E: Capacity of ℓ2 Balls with Sparse Labels

Sparse-label capacity is analyzed for ℓ2 balls across radius and dimensionality regimes. The analysis identifies scaled sparsity as the combined parameter controlling how radius and label frequency affect capacity.

Sparse-label limit: As f → 0, the optimal bias diverges and the inverse capacity scales as α_0^-1 ≈ 2f|log f|.This asymptotic result applies in the extreme sparse-label limit.
General formulation: Capacity for sparse labels is obtained by optimizing the bias b in the ℓ2-ball capacity equations.The analysis assumes zero margin offset, κ = 0.
Radius regimes: For small-radius balls, capacity matches that of points unless the dimensionality is high.The induced margin becomes noticeable only for moderate sparsity and radius regimes.
Scaled sparsity: R and f affect capacity only through scaled sparsity f̄ = fR^2.For small f̄, capacity scales approximately as (f̄|log f̄|)^−1; in a realistic small-f̄ regime it decreases roughly as 1/f̄.
Scaled sparsity: When f̄ > 1 at sufficiently large R, the second term dominates and contributes ⟨t^2⟩ = D to the inverse-capacity expression.This is the large-scaled-sparsity regime.