DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks

I. INTRODUCTION

• Existing dense reconstruction methods track cameras and reconstruct scenes but lack semantics, while many recognition methods focus on specific objects or individual 2D images.Earlier approaches that incorporate frame recognition into 3D models still couple frame reasoning and information accumulation only loosely.Source: §I · p. 1
• DA-RNNs connect frame-level RGB-D analysis over time through recurrent units whose cross-frame links are determined by mapping data associations.Recurrence is limited to currently observed map regions, avoiding a fixed unit for every surface element.Source: §I · p. 1
• The RNN’s pixel labels are integrated into KinectFusion’s 3D voxels to produce a consistent semantic labeling of the reconstructed scene.The overview reports an overall labeling and reconstruction rate of 5 fps.Source: §I · p. 1, Figure 1 · p. 1
• Experiments on the RGB-D Scene and a generated synthetic dataset demonstrate the method’s ability to produce semantically labeled 3D scenes from RGB-D videos.The paper also contributes pixel-wise labels for RGB-D Scene and a new synthetic dataset.Source: §I · p. 2
• The paper contributes a recurrent network with a Data Associated Recurrent Unit, a weighted-moving-average update rule, and integration with KinectFusion for semantic 3D reconstruction.The recurrent unit captures dependencies across video frames.Source: §I · p. 2

A. Dense 3D Scene Reconstruction

• Dense reconstruction methods represent scenes with points, voxels, or surfels; DA-RNNs use KinectFusion’s volumetric representation but require only dense associations between consecutive frames.The framework is therefore described as independent of the underlying reconstruction representation.Source: A. Dense 3D Scene Reconstruction · p. 2

B. Semantic Labeling

• Image-labeling methods include graphical models and CNNs, while video and mapping approaches combine multiple views or fuse frame labels into 3D maps, sometimes with 3D inference or object detection.These approaches extend single-image labeling with temporal information or reconstructed scene structure.Source: B. Semantic Labeling · p. 2
• DA-RNNs learn individual-frame and across-frame parameters in one network, generating recurrent connections on the fly from data associations rather than assuming fixed image-to-network links.The recurrent layer can also be used in existing CNN-based methods for semantic video labeling.Source: B. Semantic Labeling · p. 2

III. METHOD

• The framework first labels individual frames with a convolutional network, extends it to recurrent video labeling, and integrates the recurrent output with KinectFusion for semantic reconstruction.This describes the method’s progression from frame-level labeling to 3D mapping.Source: §III · p. 2

A. Single Frame Labeling with Fully Convolutional Networks

• A Fully Convolutional Network provides the single-frame labeling basis, with a design inspired by converting classification networks to convolutional form and increasing output resolution.The architecture is illustrated in Fig. 2.Source: A. Single Frame Labeling with Fully… · p. 2, Figure 2 · p. 3

1) Single Stream Network:

• The single-stream network extracts image features, embeds them at higher resolution, and classifies pixels; its deconvolutional layers restore output to input resolution for dense labeling.The described network has 16 convolutional layers, four pooling layers, two deconvolutional layers, and one addition layer.Source: 1) Single Stream Network · pp. 2–3

2) Double Stream Network:

• RGB and depth are processed separately, then their 512-dimensional features are concatenated into 1024-dimensional features before embedding and classification.The network uses late fusion for the two modalities.Source: 2) Double Stream Network · p. 3

B. Video Semantic Labeling with DA-RNNs

• DA-RNN stores and passes information across video frames to use temporal information for semantic labeling.The recurrent layer combines features from previous frames with current-frame features.Source: B. Video Semantic Labeling with DA-RNNs · p. 3, 1) DA-RNN Architecture · p. 3

1) DA-RNN Architecture:

• Each pixel location has a recurrent unit that maintains and updates hidden states; DA-RNN uses ReLU in its new unit for more efficient training.The architecture is illustrated in Fig. 4; the DA-RU is diagrammed in Fig. 5.Source: 1) DA-RNN Architecture · pp. 3–4, Figure 4 · p. 4, Figure 5 · p. 4

2) Data Associated Recurrent Unit (DA-RU):

• Each recurrent unit stores a hidden-state vector and a weight vector; the weight vector measures the importance of hidden-state elements.The vectors have the same dimension, 64-D in this RNN.Source: 2) Data Associated Recurrent Unit… · p. 4
• Data association passes hidden state and weight vectors between corresponding units in consecutive frames; units without an association, and all first-frame units, start at zero..Source: 2) Data Associated Recurrent Unit… · p. 4
• The new hidden state is a weighted sum of the previous hidden state and current input, with weights accumulated over time and ReLU used as the activation.The weight vector at the next time step sums the accumulated previous-frame weight and the current-input weight.Source: 2) Data Associated Recurrent Unit… · pp. 4–5
• DA-RU performs a data-associated weighted moving average of its input over time, with weights dynamically generated from unit parameters and received data and learned during training.In DA-RNN, DA-RUs combine features from previous and current frames for semantic labeling.Source: 2) Data Associated Recurrent Unit… · p. 5

C. Joint 3D Mapping and Semantic Labeling

• KinectFusion estimates video-frame camera poses, which the system uses to compute data association for the recurrent layer..Source: C. Joint 3D Mapping and Semantic… · p. 5
• The system fuses pixel semantic labels into KinectFusion’s volumetric space..Source: C. Joint 3D Mapping and Semantic… · p. 5
• Combining DA-RNN with 3D mapping reconstructs and semantically labels the scene..Source: C. Joint 3D Mapping and Semantic… · p. 5

1) Data Association with KinectFusion:

• Camera poses are estimated by ICP between current-frame 3D points and points on the KinectFusion map; consecutive-frame association projects 3D points between frames using those poses.The map represents the scene as a voxel space storing TSDF values updated per frame.Source: 1) Data Association with KinectFusion · p. 5

2) Semantic Fusion:

• Each voxel stores a semantic-label probability vector alongside its TSDF value; near-surface voxels update those probabilities from RNN predictions and running averaging reduces fusion noise.A voxel’s label is the class with the maximum stored probability.Source: 2) Semantic Fusion · p. 5

IV. EXPERIMENTS

• The experiments evaluate the proposed system for 3D scene mapping and semantic labeling..Source: §IV · p. 5

1) RGB-D Scene Dataset:

• The RGB-D Scene dataset contains 14 Kinect indoor videos with 3D scenes labeled across nine object classes plus background, but lacks dense pixel labels for every frame.The paper obtains dense labels by annotating 3D object boxes and projecting depth-image points into the reconstructed space; missing-depth pixels remain unlabeled. Seven videos (5,808 frames) train the model and seven (5,619 frames) test it.Source: 1) RGB-D Scene Dataset · p. 5

2) ShapeNet Scene Dataset:

• The synthetic ShapeNet dataset contains 100 rendered RGB-D videos, split evenly into 50 training and 50 test videos with no object instances shared across the split.Each scene contains a table and five tabletop objects; 100 frames are rendered per video, and labels are generated from a second rendering with class-colored shapes.Source: 2) ShapeNet Scene Dataset · pp. 5–6

B. Evaluation Metrics

• Pixel labeling is evaluated with per-class pixel IoU over the dataset, while 3D point labeling uses precision and recall for comparison with prior work.The 3D metrics follow.Source: B. Evaluation Metrics · p. 6

C. Implementation Details

• DA-RNN training uses mini-batches of three consecutive frames, and sequential testing passes hidden states to the next frame via data association.The first 13 convolutional layers are initialized from VGG16 trained on ImageNet; training uses SGD with momentum and softmax cross-entropy loss.Source: C. Implementation Details · p. 6

D. Comparison on Network Architectures

• The proposed single-stream FCN significantly outperforms the FCN in [19] on RGB-D Scene pixel labeling.The cited comparison uses the same experimental setup; the table covers four networks with RGB input.Source: D. Comparison on Network Architectures · p. 6, Table I · p. 6
• DA-RU achieves better labeling performance than GRU and is designed as a weighted moving average, with ReLU updates reported as more efficient to train.The comparison is reported in Table I for RGB-input pixel labeling.Source: D. Comparison on Network Architectures · p. 6, Table I · p. 6
• DA-RNN achieves better labeling accuracy than the FCN, which the authors attribute to capturing temporal information across video frames.The comparison is for pixel-wise semantic labeling with RGB input on the RGB-D Scene dataset.Source: D. Comparison on Network Architectures · p. 6, Table I · p. 6

E. Analysis on Network Inputs

• On RGB-D Scene, RGB performs better than depth or normals, while the double-stream RGB-D FCN improves labeling accuracy over any single input.The authors attribute RGB's advantage to the limited set of object instances, for which color is more discriminative than depth and normals.Source: E. Analysis on Network Inputs · p. 6, Table II · p. 7
• DA-RNN consistently improves over its FCN counterpart across input types on the ShapeNet and RGB-D Scene pixel-labeling evaluations.Table II covers RGB-D Scene inputs, and Table III covers ShapeNet Scene inputs.Source: E. Analysis on Network Inputs · p. 7, Table II · p. 7, Table III · p. 7
• On ShapeNet Scene, depth is more discriminative than color, combining RGB and depth does not improve over depth alone, and DA-RNNs outperform FCNs across input types.The authors relate the depth advantage to varied colors and unseen test-set objects with more discriminative general shapes.Source: E. Analysis on Network Inputs · p. 7, Table III · p. 7

F. 3D Scene Labeling Results

• 3D point-labeling precision and recall are comparable to HMP2D+3D, and the method processes RGB-D video automatically without that system's ground-plane and tabletop heuristics.The comparison uses two-fold cross-validation on all 14 RGB-D Scene videos; Table IV reports 3D point-labeling precision and recall.Source: F. 3D Scene Labeling Results · p. 7, Table IV · p. 7
• Labeling errors often involve similar-shaped classes such as mugs and cans, while incorrect data association can label an object's bottom as table.The latter occurs when the system associates the object region with table pixels in a previous frame.Source: F. 3D Scene Labeling Results · p. 8

• DA-RNNs perform recurrent reasoning over the currently visible environment, using data association to connect units, then fuse predicted labels into a KinectFusion map.Source: §V · p. 8
• Experiments on real-world and synthetic RGB-D video datasets demonstrate advantages in video semantic labeling and 3D scene mapping.Source: §V · p. 8
• The architecture focuses on object-class labeling but could extend to instance and pose detection, material recognition, and physical support estimation.Data association could also use optical flow, and incorporating 3D-map shape information is identified as a possible improvement.Source: §V · p. 8