Source-linked AI summary
DAG-Recurrent Neural Networks For Scene Labeling
Bing Shuai, Zhen Zuo, Gang Wang, Bing Wang
TL;DR
Scene labeling often relies on local image patches, leaving long-range contextual dependencies insufficiently encoded. The paper introduces DAG-RNNs within a convolutional-deconvolutional labeling network and adds rare-class weighting, achieving state-of-the-art results on SiftFlow, CamVid, and Barcelona.
Problem
Local image representations generated from surrounding patches do not effectively encode long-range contextual information needed for scene labeling.
Method
DAG-RNNs process DAG-structured image units within a convolutional-deconvolutional network, alongside a class weighting function that attends to rare classes.
Results
The integrated network achieves state-of-the-art results on the SiftFlow, CamVid, and Barcelona scene labeling benchmarks.
Takeaways & Limitations
DAG-RNNs enhance local representations by modeling long-range contextual dependencies, while class weighting improves recognition of rare classes.
Takeaways & Limitations
Larger hidden representations did not yield much additional SiftFlow improvement and imposed substantially heavier computation burdens.
Abstract
from arXiv · showhide
In image labeling, local representations for image units are usually generated from their surrounding image patches, thus long-range contextual information is not effectively encoded. In this paper, we introduce recurrent neural networks (RNNs) to address this issue. Specifically, directed acyclic graph RNNs (DAG-RNNs) are proposed to process DAG-structured images, which enables the network to model long-range semantic dependencies among image units. Our DAG-RNNs are capable of tremendously enhancing the discriminative power of local representations, which significantly benefits the local classification. Meanwhile, we propose a novel class weighting function that attends to rare classes, which phenomenally boosts the recognition accuracy for non-frequent classes. Integrating with convolution and deconvolution layers, our DAG-RNNs achieve new state-of-the-art results on the challenging SiftFlow, CamVid and Barcelona benchmarks.
1. Introduction
The paper addresses limited long-range context in local image representations by introducing DAG-RNNs and integrating them into an end-to-end labeling network. A rare-class weighting function and benchmark results further support the proposed approach.
- 1. Introduction: DAG-RNNs model long-range contextual dependencies among image units, enhancing local representations for scene labeling.The method decomposes undirected cyclic image structures into directed acyclic graphs for recurrent processing.
- 1. Introduction: The full network combines convolution, DAG-RNN, and deconvolution layers to produce dense label maps in one feed-forward pass.Convolution extracts representations, DAG-RNNs add context, and deconvolution restores output dimensionality.
- 1. Introduction: A class weighting function attends to rare classes in the highly imbalanced class distribution of natural scene images.The weighting function is designed to improve recognition of infrequent categories.
- 1. Introduction: The network is evaluated on SiftFlow, CamVid, and Barcelona, where the authors report improvements over CNN baselines and state-of-the-art methods.The reported gains include stronger local representations, better rare-class accuracy, and superior overall benchmark performance.
2. Related Work
Related work includes contextual models, recurrent approaches, and structured prediction methods for scene labeling and sequential data. The paper distinguishes DAG-RNNs by explicitly modeling image-unit structure and combining recurrent context modeling with a full labeling network.
- 2. Related Work: Earlier contextual models enlarge or stack input windows but do not explicitly represent correlations among image units.The paper contrasts these approaches with graph-structured modeling.
- 2. Related Work: Compared with probabilistic graphical models, the paper uses recurrent weight matrices and single-pass information propagation rather than iterative belief convergence.The comparison emphasizes a difference in dependency representation and inference procedure.
- 2. Related Work: The paper distinguishes RNN recurrence across structured image units from methods that reuse an identical model across iterations or layers.Examples include recurrent CNN or classifier applications with shared parameters across iterations.
- 2. Related Work: DAG-RNN generalizes chain-, tree-, and two-dimensional RNNs to model long-range semantic dependencies in graph-structured images.This positions DAG-RNNs as a structured extension of recurrent modeling beyond temporal chains.
3. Approach
The approach uses DAG-RNNs to propagate contextual information through image-structured graphs, then integrates them with convolution and deconvolution layers for dense labeling. It also introduces a class-weighting function that emphasizes rare classes.
- 3.4. Full Labeling Network: The network sequentially uses convolution to produce local feature maps, DAG-RNNs to encode contextual dependencies, and deconvolution to generate dense label predictions.The hidden representation embeds information from local inputs and their graph predecessors.
- 3.2. DAG-RNNs: DAG-RNNs decompose an image’s undirected cyclic graph into directed acyclic graphs, enabling recurrent contextual modeling while preserving spatial relationships.Several DAGs are processed independently, and their hidden representations are aggregated.
- 3.2. DAG-RNNs: For each DAG vertex, the hidden state combines its local input with information from all direct predecessors, allowing local information to propagate through the graph.The recurrent weight is shared across predecessor vertices unless a finer-grained variant is learned.
- 3.2. DAG-RNNs: DAG-RNN backpropagation accumulates direct output errors and indirect errors propagated from successor vertices, resembling message passing in graphical models.Vertices are processed in reverse topological order during the backward pass.
- 3.5. Attention to Rare Classes: The class-weighting function assigns higher weights to rare classes and increases their attention exponentially according to their frequency ratio relative to a threshold.The threshold follows an 85%-15% rule, and k=2 in the experiments.
4. Experiments
Across SiftFlow, CamVid, and Barcelona, DAG-RNNs improve context-aware scene labeling over CNN and recurrent-free baselines, while the full network reaches state-of-the-art performance. Experiments also show gains from shorter propagation paths, rare-class weighting, and contextual dependency modeling.
- SiftFlow Dataset: DAG-RNN(8) raises SiftFlow average class accuracy by 6.9% with VGG-conv5 features, while remaining comparable to FCNs globally and significantly better class-wise.The full labeling network also achieves new state-of-the-art performance under this setting.
- Training Analysis: The full network jointly optimizes convolutional and DAG-RNN components; independently optimizing them yields 80.1% and 42.7% accuracies, indicating the value of joint training.The comparison comes from disassembling the full labeling network into CNN-65 and DAG-RNN(8) parts.
- CamVid Dataset: 4.6% global and 10.2% class-accuracy gains on CamVid show that DAG-RNN(8) improves CNN representations and outperforms competing methods by 4.8% and 5.8%.The recurrent model continues improving performance even when VGG-conv5-ENN is already highly competitive.
- Per-class Accuracy: Contextual modeling improves almost all SiftFlow classes by 11.2% with CNN-65 and raises rare-class accuracy by 8.6% with VGG-conv5.The authors report that dependency modeling mitigates local ambiguities, but extremely small and rare object classes do not show commensurate gains.
5. Conclusion
The paper proposes DAG-RNNs to encode contextual dependencies among local features, enhancing their discriminative power for scene labeling. Combined with class weighting and convolution/deconvolution layers, the method achieves state-of-the-art results on three challenging benchmarks.
- DAG-RNNs model interactions among local features in DAG-structured data, encoding long-range contextual dependencies that enhance local representations.The paper frames these dependencies as useful for producing smooth and semantically sensible labeling maps.
- A novel class weighting function addresses imbalanced class distributions and improves recognition of rare classes.
- Integrating DAG-RNNs with convolution and deconvolution layers yields state-of-the-art results on SiftFlow, CamVid, and Barcelona.