PaperlayerView exact revision on arXiv

Source-linked AI summary

You Only Learn One Representation: Unified Network for Multiple Tasks

Chien-Yao Wang, I-Hau Yeh, Hong-Yuan Mark Liao

arXiv:2105.04206v1cs.CV

TL;DR

CNN models often specialize in one objective because they underuse implicit knowledge, limiting feature reuse across tasks. The paper proposes a unified network that combines implicit and explicit knowledge into a general representation and reports benefits across tasks, including comparable object-detection accuracy with faster inference. It concludes that implicit representations can support multi-task learning and correspond to physical characteristics of tasks.

  • Problem

    CNN models typically fulfill a single objective, and their extracted features are poorly adaptable to other problems because implicit knowledge is not used.

  • Method

    The unified network integrates implicit and explicit knowledge into a general representation whose sub-representations serve various tasks, using operations including kernel space alignment, prediction refinement, and multi-task learning.

  • Results

    The method benefits the reported tasks; combined with state-of-the-art methods, it achieves comparable object-detection accuracy to Scaled-YOLOv4-P7 while increasing inference speed by 88%.

  • Takeaways & Limitations

    Implicit representations can support a unified single-model architecture for multi-task learning and can correspond to specific physical characteristics of tasks.

  • Takeaways & Limitations

    The initial implicit prior is assumed to have no effect on the explicit representation, with a small variance near zero, and is trained jointly by backpropagation.

Abstract

from arXiv · show

People ``understand'' the world via vision, hearing, tactile, and also the past experience. Human experience can be learned through normal learning (we call it explicit knowledge), or subconsciously (we call it implicit knowledge). These experiences learned through normal learning or subconsciously will be encoded and stored in the brain. Using these abundant experience as a huge database, human beings can effectively process data, even they were unseen beforehand. In this paper, we propose a unified network to encode implicit knowledge and explicit knowledge together, just like the human brain can learn knowledge from normal learning as well as subconsciousness learning. The unified network can generate a unified representation to simultaneously serve various tasks. We can perform kernel space alignment, prediction refinement, and multi-task learning in a convolutional neural network. The results demonstrate that when implicit knowledge is introduced into the neural network, it benefits the performance of all tasks. We further analyze the implicit representation learnt from the proposed unified network, and it shows great capability on catching the physical meaning of different tasks. The source code of this work is at : https://github.com/WongKinYiu/yolor.

1. Introduction

The paper argues that CNN features are usually specialized to one objective because implicit knowledge is not used, and proposes a unified network combining implicit and explicit knowledge for multiple tasks.

  • CNN features are usually poorly adaptable to other problems because conventional models extract features mainly from neurons while leaving abundant implicit knowledge unused.
  • The network targets one deep neural network that can serve many tasks, rather than distinct models for distinct tasks.
  • Implicit knowledge is defined as model knowledge unrelated to direct observation, contrasting with explicit knowledge that corresponds directly to observation.
  • The proposed unified network integrates implicit and explicit knowledge into a general representation with task-suitable sub-representations.
  • The network improves performance with additional parameters and calculations totaling less than one ten-thousandth of the original amount.
  • The authors introduce kernel space alignment, prediction refinement, and multi-task learning, and report comparable object-detection accuracy to Scaled-YOLOv4-P7 with 88% faster inference.

2. Related work

The related work is organized around explicit deep learning, implicit deep learning, and knowledge modeling approaches for learning or integrating representations.

  • Explicit deep learning: Explicit deep learning includes attention-based methods and input-dependent kernel selection.
  • Implicit deep learning: Implicit deep learning includes implicit neural representations and deep equilibrium models.
  • Implicit deep learning: Implicit neural representations map discrete inputs to parameterized continuous representations, whereas deep equilibrium models formulate implicit learning through residual networks and equilibrium-point computation.
  • Knowledge modeling: Knowledge modeling includes sparse representation using dictionaries and memory networks combining embeddings into dynamically changeable memory.

3. How implicit knowledge works?

The paper models implicit knowledge as task-usable latent information that can reshape representations, align kernel spaces, and support additional task-specific functions.

  • 3.1. Manifold space reduction: Implicit representations are modeled as constant tensors independent of observation and applied to different tasks.
  • 3.1. Manifold space reduction: Taking the inner product of a projection vector and implicit representation can reduce manifold-space dimensionality and support task performance.
  • 3.2. Kernel space alignment: Addition and multiplication of output features with implicit representations can translate, rotate, and scale kernel spaces for alignment across neural-network outputs.
  • 3.2. Kernel space alignment: Kernel space alignment is presented for feature-pyramid alignment, knowledge distillation, and zero-shot domain transfer.
  • 3.3. More functions: Addition can predict center-coordinate offsets, multiplication can search anchor hyperparameters, and dot multiplication or concatenation can support feature selection or preconditions.

4. Implicit knowledge in our unified networks

The unified network models task-dependent error by combining explicit and implicit knowledge, allowing a shared representation to support multiple tasks. Implicit knowledge can be modeled through task-specific representations and integrated with neural-network features using several operators.

  • Formulation of implicit knowledge: Conventional training minimizes error so representations are discriminative for one task but invariant to other potential tasks.The representation maps observations with the same target toward a single point in the task-specific subspace.
  • Formulation of implicit knowledge: A general-purpose network relaxes the error constraint so solutions for multiple tasks can coexist on a manifold.Because task solutions are not isolated by simple selection rules, the error term must be modeled explicitly.
  • Formulation of implicit knowledge: Explicit and implicit knowledge jointly model the error term and guide training of the multi-purpose network.The explicit error comes from observation x, while the implicit error comes from task latent code z.
  • Training and inference: The combination operator can be addition, multiplication, or concatenation, while the initial latent prior is chosen near zero or one depending on the operator.The latent code and its parameters are trained by backpropagation; implicit representations can later be reduced to constant tensors before inference.
  • Formulation of implicit knowledge: The network starts from a common representation, passes through task-specific implicit representations, and uses task-specific discriminators to produce predictions.Z contains latent codes for different tasks; Φ generates implicit representations, and Ψ computes final output parameters from explicit and implicit representations.
  • Modeling implicit knowledge: Implicit knowledge can be represented directly by vectors, generated through neural networks, or formed by matrix factorization.These choices differ in basis structure and whether dimensions are treated as independent or dependent.

5. Experiments

Experiments use MSCOCO and a YOLOv4-CSP baseline to test implicit knowledge in feature alignment, prediction refinement, and multi-task learning. The section also evaluates alternative operators and modeling approaches.

  • Experimental setup: MSCOCO supplies annotations for eight vision and language-related tasks, enabling evaluation of a unified network across diverse objectives.The tasks include detection, segmentation, keypoint detection, captioning, classification, and long-tail recognition.
  • Experimental setup: The experiments introduce implicit knowledge into FPN feature alignment, prediction refinement, and multi-task learning using YOLOv4-CSP as the baseline.The multi-task setting covers object detection, multi-label image classification, and feature embedding.
  • 5.2. Feature alignment for FPN: About 0.5% improvement in APS, APM, and APL follows feature-space alignment with implicit representations.The implicit representation is added to each FPN feature map.
  • Experimental setup: The experiments compare the proposed setup against YOLOv4-CSP-fast at 640×640 resolution, with FA denoting feature alignment.These baseline and abbreviation definitions correspond to the feature-alignment ablation.

5.3. Prediction refinement for object detection

Prediction refinement adds implicit representations to YOLO output layers for object detection. The mechanism improves nearly all reported indicators and learns anchor-specific prediction patterns without supplied prior knowledge.

  • Prediction refinement: Implicit representations are added to YOLO output layers to refine object-detection predictions.The experiment reports results in Table 2 and visualizes detection effects in Figure 9.
  • Prediction refinement: Nearly all indicator scores improve after implicit representations are introduced for prediction refinement.The passage reports the overall direction of the Table 2 results without specifying individual metric values.
  • Prediction refinement: The learning mechanism automatically learns (x, y), (w, h), objectness, and class patterns for each anchor without provided prior knowledge.These learned patterns correspond to the object-detection output components described in the experiment.
  • Prediction refinement: Prediction refinement is evaluated against YOLOv4-CSP-fast at 640×640 input resolution.PR denotes prediction refinement in the reported ablation.
  • Multi-task motivation: Joint optimization can reduce overall multi-task performance relative to separately trained models, motivating a canonical shared representation.The stated issue concerns training one model across multiple tasks.

5.5. Implicit modeling with different operators

The operator study finds that the best combination rule depends on the task’s physical decoding structure. Addition and concatenation help feature alignment, whereas multiplication is more effective for prediction refinement.

  • Feature alignment: Addition and concatenation improve feature-alignment performance, while multiplication degrades it.The authors relate feature alignment to scaling global shifts and individual clusters.
  • Prediction refinement: Multiplication outperforms addition for prediction refinement, while concatenation is excluded because it changes output dimensionality.The reported explanation links multiplication to anchor-scale decoding and addition to center-shift decoding.
  • Prediction refinement: Prediction refinement benefits more from multiplication because anchor scale has a larger optimization space than bounded center coordinates.The passage states that center coordinates are grid-bounded and anchor-scale decoding is multiplicative.
  • Operator design: The operator ablation compares addition, multiplication, and concatenation as combination rules for explicit and implicit representations.The notation is {+, ×, ⊕} for those three operators.
  • Modified operators: The modified experiments split feature space by anchor cluster for multiplication and restrict multiplicative refinement to prediction width and height.These variants are denoted ×iFA* and ×iPR*.

5.6. Modeling implicit knowledge in different ways

Implicit knowledge can be modeled with vectors, neural networks, or matrix factorization, and all tested approaches improve the overall effect. Matrix factorization gives the strongest reported gains while adding very little computational overhead.

  • Modeling approaches: 0.2%, 0.4%, and 0.5% are the AP, AP50, and AP75 gains, respectively, achieved by matrix-factorization modeling.Matrix factorization produces the best results among the tested implicit-representation models.
  • Modeling approaches: Vector, neural-network, and matrix-factorization models are all evaluated as ways to represent implicit knowledge.For neural networks and matrix factorization, the default implicit-prior dimension is twice the explicit-representation dimension.
  • Modeling approaches: Neural-network and matrix-factorization modeling both improve the overall effect, with matrix factorization achieving the best result.The passage presents this as evidence for the potential of alternative implicit-representation models.
  • Model analysis: Less than one ten-thousandth of additional parameters and calculations accompanies the implicit-knowledge experiments.The model with implicit knowledge also converges quickly and correctly during training.
  • Model analysis: The overhead and learning-process comparison uses parameter counts, FLOPs, and learning curves for models with and without implicit knowledge.The corresponding results are reported in Table 7 and Figure 11.

5.8. Implicit knowledge for object detection

The proposed implicit-knowledge network is evaluated for object detection against state-of-the-art methods. Without additional training data or annotations, it achieves results sufficient to match state-of-the-art methods.

  • The method is compared with object-detection state-of-the-art methods using the scaled-YOLOv4 training process.Training consists of 300 epochs from scratch followed by 150 fine-tuning epochs.
  • The comparison uses no additional training data or annotations.
  • The unified network with implicit knowledge achieves results sufficient to match state-of-the-art object-detection methods.

6. Conclusions

The paper concludes that integrating implicit and explicit knowledge can produce a unified network effective for multi-task learning under a single model architecture.

  • The unified network integrates implicit and explicit knowledge and remains effective for multi-task learning under a single model architecture.

A. Appendix

The appendix documents alternative down-sampling and stem modules, model topology assignments, and comparisons for lightweight and large implicit-knowledge models.

  • Architecture modules: Four down-sampling modules are used: discrete wavelet transform, re-organization, convolution, and CSP convolution.
  • Stem blocks: Stem blocks differ across YOLOv4-CSP and YOLOR model variants, with Stem D used in YOLOv4-P6-light, YOLOR-P6, and YOLOR-W6.
  • Architecture topology: Models are mapped to architecture topologies, and stems containing two down-sampling modules omit corresponding early backbone stages.
  • Comparison tables: The appendix includes lightweight-model, large-model, and additional comparison tables for models with implicit knowledge.
  • Model comparisons: YOLOR-Y4-SSSS obtains 0.1% better AP than U5R5-S with 39% faster inference speed.
  • Model comparisons: YOLOR-D6 obtains 0.9% better AP than Y4-P6 at almost the same inference speed.
Loading 2105.04206v1…