AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures

1 INTRODUCTION

• Appearance and motion representations must be combined, but how and where modalities should exchange information across temporal resolutions remains an open problem.This matters for activity recognition, which requires understanding detailed semantic video content.Source: §1 · p. 1
• AssembleNet searches connectivity among multi-stream blocks that process different input modalities and temporal resolutions.Its blocks learn spatio-temporal representations, and the search can merge or split blocks.Source: §1 · p. 1
• The method represents multi-stream CNNs as directed graphs and uses learned connection weights to guide evolution alongside random block operations.The search explores combining, splitting, or connecting sub-network blocks.Source: §1 · p. 2
• AssembleNet outperforms prior work and baselines on two challenging benchmarks, with an equivalent number of parameters to standard two-stream (2+1)D ResNet models.The example learned architecture is shown in Figure 1.Source: §1 · p. 2, Figure 1 · p. 2

2 PREVIOUS WORK

• Two-stream models combine motion and appearance inputs (Simonyan & Zisserman, 2014; Feichtenhofer et al., 2016a; 2017; 2016b), with later work exploring multiple temporal resolutions and intermediate or late fusion (Feichtenhofer et al., 2018; Diba et al., 2019).These video CNN architectures were carefully designed by human experts.Source: §2 · p. 2
• Neural architecture search (NAS) automatically finds better architectures based on data, using reinforcement-learning controllers (Zoph & Le, 2017; Zoph et al., 2018) or evolutionary algorithms (Real et al., 2019), and these methods meaningfully outperform handcrafted architectures.Source: §2 · p. 2
• Image NAS typically searches modules for a fixed single-stream meta-architecture, while video search must handle many blocks at different resolutions and their possible interactions.Fully connecting those blocks requires substantial memory and training data, making direct extension challenging.Source: §2 · p. 3
• Prior learnable gating and RandWire work connects branches or layers within fixed architectures, whereas this work targets data-driven connectivity between multi-stream video blocks.The paper reports that randomly connecting blocks is insufficient for multi-stream video CNNs.Source: §2 · p. 3

3.1 GRAPH FORMULATION

• Each AssembleNet architecture is a directed acyclic graph whose nodes are convolutional blocks and whose edges connect lower-level blocks to higher-level blocks.Nodes can run in parallel at the same level, and each has its own temporal resolution and channel count.Source: §3.1 · pp. 3–4
• Temporally dilated 1D convolution lets blocks focus on different temporal resolutions without losing temporal granularity.The dilation rate sets the temporal resolution; rate 1 is standard 1D temporal convolution.Source: §3.1 · p. 4
• Learned weighted summation combines connected nodes’ outputs while keeping the input tensor size independent of the number of connections.Connection weights are trained by backpropagation and can guide architecture evolution.Source: §3.1 · p. 4

3.2 EVOLUTION

• Figure 2’s third model has lower fitness than the second after random mutations, yet survives and later evolves into a better model.The illustrated architectures have parent-child relationships.Source: Figure 2 · p. 5
• Evolution selects parent architectures by proxy-task fitness, inherits high-weight connections, replaces low-weight connections with random ones, and applies node and resolution mutations.Splitting or merging blocks preserves the total number of parameters.Source: §3.2 · pp. 5–6
• 51.4% mAP on Charades was achieved by the simpler two-stem AssembleNet-50 shown in Figure 3.The figure presents diverse architectures with higher-than-50% mean-average precision on Charades.Source: Figure 3 · p. 6

4.1 DATASETS

• On Charades, evaluation used long videos with overlapping activities; Moments in Time provided more than 800K videos averaging about 3 seconds each.Charades contains 9,848 videos across 157 classes, while Moments in Time was chosen for architecture evolution and comparison with prior models.Source: §4.1 · pp. 6–7

4.2 RESULTS

• AssembleNet set new state-of-the-art results on both datasets, with 58.6 on Charades versus prior known bests of 42.5 and 45.2.The architectures searched on Moments in Time were trained and tested on both datasets, demonstrating their usefulness across datasets.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · pp. 7–8
• Connection-learning-guided evolution found better architectures more quickly than random search; standard evolution performed similarly to random search.The authors attribute standard evolution's weakness to the exponential size of the connectivity search space.Source: §4.2 · p. 8

4.3 ABLATION STUDIES

• With similar parameter counts, connection-learning architectures were powerful and evolution further improved performance; a flow-to-RGB model scored 30.2% accuracy on MiT and 49.5% on Charades.The models were trained from scratch on MiT and pretrained on MiT for Charades.Source: §4.3 · p. 8, Table 3 · p. 8
• The search-space ablation compared the full space with constraints including two stems only and fixed temporal resolution.The models in Table 4 were trained from scratch.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 GENERAL FINDINGS

• Learned architectures often use more than two streams, including four at many levels; they also favor one highest-level block and a simple final block gathering prior-level outputs.Random-connectivity models perform poorly compared with AssembleNet, and multiple temporal resolutions often appear at the same level.Source: §4.4 · pp. 8–9

• AssembleNet searches for multi-stream video architectures with improved connectivity and temporal resolutions through connection-learning-guided evolution.Source: §5 · p. 9
• The learned models significantly outperform previous models on two challenging benchmarks.Source: §5 · p. 9

• Figure 5 visualizes the possible connections and channel and temporal-resolution options considered during architecture evolution.Source: §A.1 · p. 12, Figure 5 · p. 12
• The evolutionary objective is to find a high-performing super-graph subgraph while maintaining total parameters; fitting the full super-graph in memory was infeasible in these experiments.Source: §A.1 · p. 12

• Channel size C sets the widths of 1x1 and temporal convolutional layers, while 2D convolution widths are fixed by level: D1=64, D2=128, D3=256, and D4=512.Stem layers have 64 channels with two stems and 32 with four stems.Source: §A.2 · p. 12
• Splitting a node halves the resulting nodes’ channel sizes, keeping the total model parameter count identical before and after the split.Source: §A.2 · p. 12
• Merging nodes unions their input and output connections, sums their channel sizes, and randomly selects the merged node’s temporal dilation rate from the original two.Source: §A.2 · p. 12

• Figure 7 illustrates hand-designed (2+1)D CNN ablation models and their learned connection weights.The weights indicate which connections the model used or did not use.Source: §A.3 · p. 12, Figure 7 · p. 13
• In two-stream architectures, stronger connections tend to form later; in four-stream architectures, they form early and connect across modalities at the next level.At least one RGB node connects to a flow node, and vice versa.Source: §A.3 · pp. 12–13
• The ablation variants include late fusion, fusion at level 4, fully connected two-stream nodes, and Flow→RGB, where only RGB nodes combine both streams’ outputs.Source: §A.3 · p. 13

A.4 ASSEMBLENET MODEL/LAYER DETAILS

• The final model's block descriptions specify input connections, with 2D and (2+1)D residual modules repeated across four levels.Repetition counts are 1.5, 2, 3, and 1.5; AssembleNet-101 instead uses 11.5 at level 3.Source: §A.4 · p. 14
• AssembleNet has 50 layers, with block depths of 9, 12, 18, and 9 convolutional layers across levels 1–4.The stem has at most two convolutional layers; the reported depth is similar to a conventional (2+1)D ResNet-50.Source: §A.4 · p. 14
• Spatial stride 2 is placed at a block's first 2D convolution; the stem's stride 4 uses stride 2 at both its 2D convolution and max pooling.Every convolution is followed by batch normalization and ReLU; 2D filters are 3x3 and 1D filters are 3.Source: §A.4 · p. 14

A.5 SINK NODE DETAILS

• The sink node spatially pools the final 5D features, then temporally pools them into a 2D representation before a fully connected and softmax layer produces class outputs.MiT uses temporal average pooling and Charades uses max pooling; multiple level 4 nodes, when present, are averaged or concatenated, with little empirical difference.Source: §A.5 · p. 14

A.6 TRAINING DETAILS

• Training uses 224x224 inputs, momentum optimization, 12k warmup iterations, cosine decay, and dataset-specific batches and frame counts.MiT uses batch size 512 and 32 frames per video; Charades uses batch size 128 and 128 frames. Flow input is extracted with TV-L1; evolution training takes 3–5 hours for 10k iterations, while full 50k-iteration training takes about 24 hours per dataset.Source: §A.6 · pp. 14–15

A.7 EVALUATION DETAILS

• MiT evaluation uses 36 frames per video, while Charades combines predictions from five random 128-frame crops using max pooling to score average precision.Testing uses 256x256 resolution; 36 frames roughly cover MiT's three-second videos, while Charades videos are roughly 30 seconds long.Source: §A.7 · p. 15