Source-linked AI summary
Feedback Networks
Amir R. Zamir, Te-Lin Wu, Lin Sun, William Shen, Jitendra Malik, Silvio Savarese
TL;DR
Computer-vision models commonly use feedforward successive representations, motivating an iterative feedback alternative. The paper develops and evaluates a general recurrent, weight-shared architecture that predicts at each iteration, finding early prediction, taxonomy compliance, episodic curriculum learning, and distinct coarse-to-fine representations while retaining competitive endpoint performance.
Problem
Most successful computer-vision learning models use feedforward successive representations, leaving iterative feedback as an alternative approach to investigate.
Method
The paper uses a recurrent weight-shared operation that passes each iteration’s output through a hidden state, trains losses at every iteration, and studies architectural design choices.
Results
Feedback networks provide early predictions, taxonomy-compliant outputs, episodic curriculum learning, and considerably different coarse-to-fine representations, with endpoint performance on par or better than feedforward baselines.
Takeaways & Limitations
Feedback-based learning is presented as a worthwhile alternative with practical early outputs and a representation aligned with hierarchical prediction and curriculum learning.
Takeaways & Limitations
Some compared methods use additional mechanisms, such as stochasticity in depth, that the feedback model does not implement and that could be combined with feedback in future work.
Abstract
from arXiv · showhide
Currently, the most successful learning models in computer vision are based on learning successive representations followed by a decision layer. This is usually actualized through feedforward multilayer neural networks, e.g. ConvNets, where each layer forms one of such successive representations. However, an alternative that can achieve the same goal is a feedback based approach in which the representation is formed in an iterative manner based on a feedback received from previous iteration's output. We establish that a feedback based approach has several fundamental advantages over feedforward: it enables making early predictions at the query time, its output naturally conforms to a hierarchical structure in the label space (e.g. a taxonomy), and it provides a new basis for Curriculum Learning. We observe that feedback networks develop a considerably different representation compared to feedforward counterparts, in line with the aforementioned advantages. We put forth a general feedback based learning architecture with the endpoint results on par or better than existing feedforward networks with the addition of the above advantages. We also investigate several mechanisms in feedback architectures (e.g. skip connections in time) and design choices (e.g. feedback length). We hope this study offers new perspectives in quest for more natural and practical learning models.
1. Introduction
The paper presents feedback-based learning as an iterative alternative to feedforward prediction, with early outputs, taxonomy-compliant decisions, and episodic curriculum learning. It defines a recurrent architecture that predicts at every iteration and reports a coarse-to-fine representation.
- Core idea: Feedback-based learning forms predictions iteratively from the thus-far output, rather than producing a one-time final output.The authors attribute its advantages to iterative prediction coupled with an explicit notion of the output at each iteration.
- Advantages: Early predictions provide output estimates during fractions of the total inference time, unlike feedforward outputs available only at the network end.The paper highlights robotics and autonomous driving as settings where limited time and computation make early estimates useful.
- Advantages: Feedback predictions naturally follow a taxonomy: early iterations produce coarse classes, while later iterations decompose them into finer classes.The paper links this behavior to iterative inference and a representation that develops from coarse to fine.
- Advantages: Episodic Curriculum Learning enforces sequential easy-to-hard decisions across prediction episodes for a single datapoint, allowing a taxonomy to serve as a curriculum.This contrasts with feedforward curriculum strategies that order training examples by complexity across epochs.
- Architecture: The model uses a weight-shared recurrent operation whose hidden state carries the output notion, and trains prediction losses at every iteration.The paper instantiates the generic architecture with existing RNNs and emphasizes the conceptual properties over endpoint optimization.
2. Related Work
Related work spans machine learning and neuroscience, including recurrence-inspired feedforward methods, explicit feedback systems, spatial-attention mechanisms, curriculum learning, and taxonomic prediction. The paper distinguishes its general feedback-based inference focus from task-specific or temporally oriented prior methods.
- Overview: Prior research in both machine learning and neuroscience contains substantial work related to feedback-based learning.The paper provides a categorized overview of selected related studies.
- Feedforward and recurrence-inspired methods: Residual, highway, hypernetwork, stochastic-depth, RCNN, and GoogLeNet methods remain feedforward because they do not iteratively inject the thus-far output.The paper states that recurrence alone is insufficient for proper feedback and reports this requirement as empirically critical.
- Explicit feedback methods: Existing explicit feedback methods often target specific tasks or temporal problems, whereas this paper investigates general feedback-based inference with feedback in hidden space.Hidden-space feedback avoids requiring task-specific error-to-input functions.
- Related mechanisms: Other feedback-like mechanisms address spatial attention, long-term dependencies, computational efficiency, or localization rather than the paper’s central inference focus.Curriculum learning and taxonomic prediction are established areas, but the paper notes that prior work did not provide its feedback-based approach.
3. Feedback Networks
Feedback networks repeatedly apply shared convolutional operations while routing hidden-state information and predictions across iterations. The architecture supports iteration-level losses, temporal skip connections, taxonomic outputs, and shorter computation graphs than feedforward networks.
- Core feedback formulation: Feedback-based prediction repeatedly applies a shared convolutional operation, producing an output at each iteration from a hidden state carrying the prior output.Losses are connected to every iteration so the network learns to perform the task throughout the recurrent process.
- Core feedback formulation: ConvLSTM modules use convolutional operations with temporal hidden states, shared filter weights across time, and gates controlling information flow.The architecture permits multilayer convolutional operations and residual connections within a module.
- Temporal skip connections: Feedback networks are grouped by the number of feedforward layers inside each ConvLSTM module: Stack-1, Stack-2, and Stack-All.This local feedback length determines how distributed hidden-state propagation is across the physical network.
- Temporal skip connections: Identity skip connections regulate signal flow, with experiments using skip length n = 2 and evaluating their endpoint impact on CIFAR100.The connections are illustrated as temporal links in the unrolled feedback model.
- Taxonomic prediction: Taxonomic prediction sums fine-class probabilities into higher-level coarse classes, allowing feedback outputs to conform to a label taxonomy.The paper reports that this conformity is especially evident in early predictions.
- Computation graph analysis: For equal virtual depth D = m × n, feedforward and Stack-1 feedback graph depths are dff = mn − 1 and dfb = m + n − 1, respectively.Under sufficiently parallel hardware, the smaller feedback depth corresponds to shorter prediction time than the feedforward graph.
4. Experimental Results
The experimental evaluation uses three benchmarks: CIFAR100, Stanford Cars, and MPII Human Pose.
- Benchmarks: Experiments are conducted on CIFAR100, Stanford Cars, and MPII Human Pose.These benchmarks provide the evaluation settings described for the paper’s experimental section.
- Benchmarks: Results for the three benchmarks are provided in the experimental-results section.The passage identifies the evaluation scope without giving numerical outcomes.
4.1. Baselines and Terminology
The paper distinguishes physical and virtual depth and compares feedback networks with feedforward baselines using matched module architectures and established models.
- Terminology: Physical depth is the number of convolutional layers from input to output, while virtual depth multiplies physical depth by the number of iterations.Virtual depth is defined for feedback networks and represents effective spatial-temporal depth.
- Baselines: ResNet and VGG serve as feedforward baselines with the same physical module architecture as the proposed method.The baselines differ in their residual connections, while kernel sizes and filter-number transitions follow the original paper settings.
- Baselines: The evaluation also compares against the original ResNet architecture and a feedback version of the feedforward Hourglass model.The Hourglass comparison is made in Section 4.4.
4.2. CIFAR-100 and Analysis
On CIFAR100, feedback networks provide useful early and taxonomic predictions while developing coarse-to-fine representations, without sacrificing endpoint performance. Their results depend on feedback-module length and improve with episodic curriculum learning.
- 4.2.1 Feedback Module Length: The best feedback-module performance occurs at an intermediate local feedback length, with the optimal value varying across tests and architectures.Physical depth and iteration count remain fixed at 4 and 4 in the study summarized by Table 2; later experiments often use lengths 2 or 3.
- 4.2.2 Early Prediction: Feedback networks outperform a computation-time-matched ensemble of ResNets for early prediction.The comparison matches feedback outputs at 12T, 15T, 18T, and 21T with ResNet depths 12, 15, 18, and 21.
- 4.2 Analysis: Removing loss connections from earlier iterations makes the model recurrent feedforward and removes its early- and taxonomic-prediction abilities.This experiment isolates feedback from recurrence alone.
- 4.2.3 Taxonomic Prediction: Feedback predictions comply with the CIFAR100 taxonomy at shallow virtual depths, while feedforward models reach comparable compliance only near the final layer.Early results are described as more relevant and interpretable, and feedback representations are disentangled earlier.
- 4.2.3 Taxonomic Prediction: Feedback representations are relatively disentangled throughout processing, whereas feedforward representations disentangle classes only toward the end.The feedback updates mainly form finer separation regions after early coarse organization.
- 4.2.4 Curriculum Learning: Episodic curriculum learning gives the feedback network the highest performance boost and improves taxonomic prediction.The feedback network and auxiliary-loss baselines use episodic curriculum training, while no-auxiliary-loss baselines use conventional datapoint sorting.
- 4.2.5 Endpoint Performance Comparison: Feedback networks outperform same-physical-depth feedforward baselines and match or exceed baselines with the same virtual depth or greater depth at endpoint evaluation.The authors report that early and taxonomic advantages do not require sacrificing endpoint performance.
4.3. Stanford Cars Dataset
Experiments on Stanford Cars reproduce the CIFAR100 trends for feedback networks. The study uses shallower models trained from scratch without pretrained ImageNet models or additional images.
- 4.3. Stanford Cars Dataset: Stanford Cars experiments show trends similar to CIFAR100, including the same endpoint and curriculum-learning observations.Early- and taxonomic-prediction curves are provided in the supplementary material.
- 4.3. Stanford Cars Dataset: All networks are trained from scratch without fine-tuning pretrained ImageNet models or augmenting the dataset with additional images.The smaller training set motivates using shallower models.
- 4.3. Stanford Cars Dataset: The feedforward baselines use depth 24, while the feedback network uses physical depth 6 and 4 iterations.This follows the design used in the corresponding CIFAR100 experiments.
4.4. Human Pose Estimation
The paper also evaluates feedback on MPII Human Pose estimation by adapting a state-of-the-art Hourglass model. Table 8 reports performance using the standard PCKh localization metric.
- 4.4. Human Pose Estimation: The MPII Human Pose benchmark contains 40k samples, with 28k for training and 11k for testing.The paper applies feedback to the state-of-the-art Hourglass model.
- 4.4. Human Pose Estimation: The feedback adaptation replaces one Hourglass stack’s ResNet-like convolutional sequence with ConvLSTM, replacing physical depth with virtual depth.Table 8 evaluates the resulting models using PCKh, the standard body-joint localization accuracy metric.
5. Conclusion
The study argues that feedback-based learning is a worthwhile alternative to feedforward models, offering early prediction, taxonomy compliance, and Episodic Curriculum Learning. It also develops a meaningfully different coarse-to-fine representation.
- Feedback-based learning offers early prediction, taxonomy compliance, and Episodic Curriculum Learning as alternatives to feedforward processing.
- Feedback networks develop a coarse-to-fine representation that is meaningfully and considerably different from feedforward representations.