PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Polynomial Neural Networks

Grigorios Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Jiankang Deng, Yannis Panagakis, Stefanos Zafeiriou

arXiv:2006.13026v2cs.LGcs.CVstat.ML

TL;DR

DCNNs dominate generative and discriminative learning, but their linear-and-nonlinear compositional structure has known limitations. The paper introduces Π-Nets, which use shared-factor tensor decompositions to implement polynomial function approximators. It reports improved baseline performance across multiple tasks and signals, while noting potential overfitting without activation functions.

  • Problem

    DCNNs rely on compositions of linear and nonlinear operators, a structure that theoretical and empirical studies identify as limited.

    Source: §1 · p. 1

  • Method

    Π-Nets approximate functions with high-order polynomials and use coupled tensor factorization with shared factors to estimate their parameters.

    Source: §1 · p. 1

  • Results

    The authors report improved baseline performance across generation, classification, and face verification or identification, with architectures that outperform prior art on signals including images, meshes, and audio.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The polynomial architectures are used to learn high-dimensional distributions without nonlinear activation functions.

    Source: §1 · p. 2

    Without activation functions, polynomials might be prone to overfitting; the authors did not observe the same consistent pattern when activations were added.

    Source: §5 · p. 10

Abstract

from arXiv · show

Deep Convolutional Neural Networks (DCNNs) are currently the method of choice both for generative, as well as for discriminative learning in computer vision and machine learning. The success of DCNNs can be attributed to the careful selection of their building blocks (e.g., residual blocks, rectifiers, sophisticated normalization schemes, to mention but a few). In this paper, we propose $\Pi$-Nets, a new class of function approximators based on polynomial expansions. $\Pi$-Nets are polynomial neural networks, i.e., the output is a high-order polynomial of the input. The unknown parameters, which are naturally represented by high-order tensors, are estimated through a collective tensor factorization with factors sharing. We introduce three tensor decompositions that significantly reduce the number of parameters and show how they can be efficiently implemented by hierarchical neural networks. We empirically demonstrate that $\Pi$-Nets are very expressive and they even produce good results without the use of non-linear activation functions in a large battery of tasks and signals, i.e., images, graphs, and audio. When used in conjunction with activation functions, $\Pi$-Nets produce state-of-the-art results in three challenging tasks, i.e. image generation, face verification and 3D mesh representation learning. The source code is available at \url{https://github.com/grigorisg9gr/polynomial_nets}.

1 INTRODUCTION

The paper challenges the established composition of linear and nonlinear operators in DCNNs and introduces Π´nets, whose outputs are high-order polynomials of their inputs. Shared-factor tensor decompositions make these models practical, and the paper evaluates them across generative and discriminative tasks.

1 INTRODUCTION

  • The motivation is that DCNNs rely on compositions of linear and nonlinear operators, a structure that theoretical and empirical studies identify as limited.Source: §1 · p. 1
  • The authors extend earlier higher-order expansions with task-tailored models, expanded face verification and identification experiments, and discussion of open challenges.The earlier work covered generative and discriminative networks.Source: §1 · p. 1
  • Π´nets model outputs as high-order polynomials and estimate their tensor-valued parameters through coupled tensor factorization with shared factors.The factorization addresses the combinatorial growth in parameters as polynomial order increases.Source: §1 · p. 1
  • The polynomial architectures are used to learn high-dimensional distributions without nonlinear activation functions.Source: §1 · p. 2
  • The paper reports improved baseline performance across generation, classification, and face verification or identification, using signals including images, meshes, and audio.It also reports that the architectures outperform prior art.Source: §1 · p. 2

2 RELATED WORK AND NOTATION

The related work situates Π´nets among polynomial networks, hierarchical generative models, and multiplicative-interaction approaches, distinguishing their high-order function-approximation goal. The notation section introduces tensor operations and the CP decomposition used in the analysis.

2 RELATED WORK AND NOTATION

  • Π´nets are positioned as a polynomial-expansion approach to function approximation, alongside hierarchical generative models and work on multiplicative interactions.The related work distinguishes this goal from studies characterizing depth efficiency in convolutional networks.Source: §2 · p. 2
  • Earlier polynomial networks use predefined input subsets or product units, and the paper says these approaches do not scale well to high-dimensional signals.Their evaluations are also described as limited to known ground-truth distributions and, in one case, inputs and outputs of up to three dimensions.Source: §2 · p. 2

2.1 Notation

  • The notation section defines tensor mode-vector products and identifies CP decomposition as a factorization into rank-one component tensors.It also introduces the Hadamard and Khatri-Rao matrix products.Source: §2.1 · pp. 2–3

3 METHOD

The method represents outputs as high-order polynomials and controls parameter growth through shared tensor factorizations and products of lower-degree polynomials. It presents three decompositions, compares their activation-free image generation, and describes task-dependent inputs and outputs.

3 METHOD

  • The method models each output as a polynomial of the input, while parameter growth with expansion order motivates efficient implementations.The parameters for unrestricted high-order correlations are represented by high-order tensors.Source: §3 · p. 3

3.1 Single polynomial

  • Tensor decomposition reduces polynomial parameters and supports neural-network implementations; the paper presents three such decompositions.The method also proposes products of lower-degree polynomials as a principled route to efficient implementation.Source: §3.1 · p. 3, §3 · p. 3

Model 1: CCP (Coupled CP decomposition)

  • CCP jointly factorizes polynomial parameter tensors with lower-order factors shared across approximation levels.The paper illustrates the factorization for a third-order approximation and gives a recursive form for arbitrary order.Source: Model 1: CCP (Coupled CP decomposition) · p. 3

Model 2: NCP (Nested coupled CP decomposition)

Model 3: NCP-Skip (Nested coupled CP decomposition with skip)

Comparison between the models

  • CCP serves as a proof of concept for learning high-dimensional distributions, while NCP and NCP-Skip convert CNN/linear and residual networks into polynomial forms.The three models use different recursive forms and decompositions.Source: Comparison between the models · p. 4
  • All three decompositions generate images with similar properties, and all generate faces without inter-layer activation functions at similar quality.The comparison covers fashion images and facial images.Source: Comparison between the models · p. 5

3.2 Product of polynomials

  • A product of polynomials can combine different decompositions and expansion orders, and needs fewer parameters than a single polynomial of the same approximation order.The total order is the product of the orders of the constituent polynomials.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments convert established baselines into polynomial networks and evaluate them across image generation, classification, face verification and identification, and 3D mesh representation learning. Results include activation-free generation and mesh experiments, comparable classification accuracy with fewer parameters, and improvements on several benchmarks.

4.1 Image generation

  • Figures 5 and 6 compare the proposed models in fashion image [54] and facial image [55] generation, respectively, without activation functions.Source: Figure 5 · p. 6, Figure 6 · p. 6
  • On FFHQ, the polynomial mapping-network modification improved StyleGAN’s best FID from 7.15 to 6.82, a reported 5% improvement; on CIFAR10, ProdPoly further boosted SNGAN beyond the single-polynomial variant.The StyleGAN change converts its mapping network into an NCP, making the generator a product of two polynomials.Source: §4.1 · pp. 5–6

4.2 Classification

  • On Speech Commands, CIFAR10, and CIFAR100, Prodpoly-ResNet matched ResNet accuracy with 38%, about 40%, and about 30% fewer parameters, respectively.The CIFAR10 comparison reports the same accuracy for the ResNet baselines and their corresponding polynomial models.Source: §4.2 · p. 7, Table 4 · p. 7, Table 5 · p. 7
  • Prodpoly-ResNet50 reached 22.827% Top-1 and 6.431% Top-5 single-crop validation error on ImageNet, beating ResNet50 by 0.719% and 0.473%, respectively.The comparison reports negligible additional computational complexity.Source: §4.2 · p. 7, Table 7 · p. 8, Figure 10 · p. 8

4.3 Face verification and identification

  • On face-verification benchmarks CPLFW, AgeDB-30, CALFW, and RFW, Prodpoly-ResNet50 significantly outperforms ResNet50, indicating that the method can improve embedding-feature robustness under pose variations and age.Source: §4.3 · p. 8
  • On IJB-B and IJB-C, Prodpoly-ResNet50 improved 1:1 TAR (@FAR=1e-4) by 0.46% and 0.41%, and 1:N rank-1 identification by 0.23% and 0.34%, respectively.The 1:1 results are shown in Fig. 11 and Table 10; the 1:N mixed-media results are in Table 11.Source: §4.3 · p. 9, Figure 11 · p. 9, Table 10 · p. 10
  • On MegaFace, Prodpoly-ResNet50 improved rank-1 identification by 0.50% and verification TPR@FAR=1e-6 by 0.31% over ResNet50.The evaluation uses the refined MegaFace dataset; Fig. 12 compares the models’ curves.Source: §4.3 · p. 9, Figure 12 · p. 9, Table 12 · p. 10

4.4 3D Mesh representation learning

  • On COMA and DFAUST mesh autoencoding, Π-nets outperformed published methods even without network-wide activation functions; increasing polynomial order generally improved performance, though simple-model results declined at orders three and four.The authors suggest overfitting may account for the simple model’s higher-order decline; Fig. 13 compares reconstruction errors.Source: §4.4 · p. 10, Figure 13 · p. 10

5 FUTURE DIRECTIONS

The authors identify generalization, architecture design, redundancy reduction, and normalization as open directions for Π-nets.

5 FUTURE DIRECTIONS

  • Without activation functions, Π-nets may be prone to overfitting in classification; the authors did not observe a consistent pattern when nonlinear activations were added.The overfitting observation comes from evaluation, including classification experiments in the supplementary material.Source: §5 · p. 10
  • The relationship between tensor decompositions and neural architecture search remains an opportunity for further work.The three decompositions in this work result in three different architectures.Source: §5 · p. 11
  • Polynomial neural networks may reduce network redundancy, and pruning or low-rank constraints could further help.The authors point to theoretical properties of multiplicative interactions and their experiments as evidence for the potential to reduce redundancy.Source: §5 · p. 11
  • Higher-order terms may produce unbounded gradients, making normalization important; alternatives to the employed batch or instance normalization might perform better.The authors identify the study of normalization schemes as a future direction.Source: §5 · p. 11

6 CONCLUSION

The conclusion presents Π-nets as polynomial function approximators that extend compositional networks and report consistent performance improvements across several tasks.

6 CONCLUSION

  • Performance consistently improves when state-of-the-art architectures are modified for image generation, image and audio classification, face verification and identification, and mesh representation learning.The conclusion reports results across both discriminative and generative tasks.Source: §6 · p. 11
  • Π-nets approximate functions with high-order polynomials, efficiently implemented through skip connections and tensorial factors.The formulation extends the standard compositional paradigm of linear operations overlaid with activation functions.Source: §6 · p. 11
  • Experiments without activation functions showcase polynomial expressivity, while Π-nets are effective in both discriminative and generative tasks.Source: §6 · p. 11
Loading 2006.13026v2…