Broadcasting Convolutional Network for Visual Relational Reasoning

• Conventional RNs incur O(n^2) cost because they compute pairwise relations over grid positions, including locations without objects.Source: §1 · p. 1
• CNNs struggle with geometric variation because fixed local receptive fields discard spatial locations and poorly handle large deformations.Source: §1 · p. 2
• BCN embeds absolute positional information, broadcasts spatial features globally, and analyzes object relations within CNNs.Source: §1 · p. 2
• MultiRN uses BCN to improve relational reasoning performance and computational efficiency, achieving state-of-the-art performance on CLEVR.Source: §1 · p. 2
• The paper presents BCN, multiRN, related-work comparisons, experiments, and conclusions across Sections 2–6.Source: §1 · p. 2

• BCN combines coordinate channel embedding, spatial feature encoding, and global broadcasting through expansion.Source: §2 · p. 2
• After coordinate concatenation and lightweight convolutions, BCN max-pools each filter into a 1 × 1 × n vector and expands it across the original feature maps.Source: §2 · pp. 2–3, Figure 1 · p. 3
• The broadcast features are merged with original maps so later convolutions correlate objective positional features with relative visuospatial features.Source: §2 · p. 3
• BCN’s post-CCE convolutions generate location-based abstractions that let later mappings infer relative positions with limited additional computation.Source: §2 · pp. 3–4
• Coordinate embedding preserves feature positions before pooling and supports comparisons against spatial features from other locations.Source: §2 · p. 4
• Coordinate planes encode normalized x-axis, y-axis, and radial positions while adapting values to input aspect ratios.Source: §2 · p. 4

• RNs compute relations over all pairwise object combinations, producing O(n^2) computation through gθ.Source: §3 · p. 5
• BCN lets each object feature pair with multiple object-location features at once, enabling multiwise relations computed in O(n).Source: §3 · p. 5
• MultiRN feeds BCN’s multi-location representation into gθ so relation outputs convey how object features at particular locations should contribute to reasoning.Source: §3 · p. 6
• For pixel-based inputs, the object count is n = h × w, allowing RN’s gθ MLP to be replaced by 1 × 1 convolution after channel concatenation.Source: §3 · p. 6
• Question vectors can be expanded across channels and processed with the BCN-derived tensor for visual question-answering relational reasoning.Source: §3 · p. 6

• BCN extracts dynamic spatial features and globally projects them onto feature maps to widen receptive fields for multi-scale and dense relational reasoning.Source: §4 · p. 6
• Unlike hand-crafted spatial pooling methods, BCN learns its spatial feature extraction end-to-end.Source: §4 · p. 6
• ROI pooling aggregates channels but does not represent relations among feature maps.Source: §4 · p. 6
• Deformable and atrous convolutions enlarge or adapt receptive fields, but their local or sparse sampling can limit complete spatial support.Source: §4 · p. 7
• BCN differs from skip-connections by convolving each feature with broadcast features from varied locations to analyze spatial relationships.Source: §4 · p. 7
• Compared with RN and FiLM, multiRN uses BCN-based multiple relations and achieves similar performance with smaller networks than FiLM’s larger architecture.Source: §4 · p. 7

Scaled-MNIST:

• BCN outperformed the five-layer baseline on Scaled-MNIST by 0.3% in classification and with three times less localization error.The BCN model used a shallower three-layer configuration, whereas the stronger baseline used twice as many filters.Source: Scaled-MNIST · p. 9, Table 1 · p. 9
• BCN achieved 77.0% accuracy on STL-10, 1.24% higher than the single-model result of Tho17-2 [32].The model was trained from scratch and established a new state-of-the-art result on a single-network basis.Source: Table 2 · p. 10, Scaled-MNIST · p. 11
• BCN improved STL-10 performance by 3.4% over the baseline and exceeded the same convolutional layers used as a skip-connection.The result is attributed to extending the receptive field to whole image grids.Source: Scaled-MNIST · p. 11

Sort-of-CLEVR:

• MultiRN was 3% better than a similarly structured RN on Sort-of-CLEVR while using comparably less computational cost.Removing BCN reduced performance by 4.2%, indicating that the module supplies correlations among multiple objects.Source: Sort-of-CLEVR · pp. 11–12, Table 3 · p. 11

5.3 Pixel-based Relational Reasoning Problem

• MultiRN scales as O(n), so increasing objects from 25 to 100 quadrupled its computation cost, whereas RN with O(n2) took 16 times greater computation loads, from 625 to 10000.Source: Sort-of-CLEVR · p. 12

Sort-of-CLEVR:

• On CLEVR, multiRN improved over RN* by 1.4%, while CNNh+multiRN improved by 6.3% and achieved 97.7% test accuracy at 224×224 input size.The paper reports this as state-of-the-art performance on CLEVR with raw pixels, compatible with FiLM using ResNet-101.Source: Sort-of-CLEVR · pp. 12–13, Table 4 · p. 12

5.4 Can BCN Extend the Receptive Field Globally and Represent Multiple Objects?

• Activation maps showed BCN extracting important locations and multiple objects, then broadcasting those features globally to expand receptive fields.The visualizations covered Scaled-MNIST, STL-10, Sort-of-CLEVR, and CLEVR; activation maps represent max-pooled features rather than necessarily all objects.Source: §5.4 · p. 13, Figure 4 · p. 13

5.5 Study on Coordinate Embedding Methods

• Coordinate channels substantially improved performance, with zero-centered coordinates outperforming top-left-zero coordinates and radial channels slightly improving relational accuracy further.The coordinate-plane comparison also showed that radial channels reduced convolutional deflection, while their performance gain was mainly in localization.Source: §5.5 · p. 14, Table 5 · p. 14, Figure 5 · p. 14

6 Conclusion

• BCN efficiently collects and represents spatial information by extending receptive fields, producing strong localization performance.Source: §6 · p. 15
• MultiRN improves RN through computational gains while achieving state-of-the-art performance on pixel-based relational reasoning problems.Source: §6 · p. 15
• Future work will test BCN for object detection and semantic segmentation and apply multiRN to additional visual relational reasoning problems.Source: §6 · p. 15

A Experiment Details

• Scaled-MNIST models use staged SGD learning rates, 30-epoch training with early stopping, and jointly weighted classification and localization losses.Source: A Experiment Details · p. 18
• STL-10 training uses SGD with scheduled learning-rate decay, padded random crops, channel normalization, and ResNet-18 input-layer modifications.Source: A Experiment Details · p. 18
• Sort-of-CLEVR uses unaugmented 75 × 75 images, ADAM optimization, staged learning rates, and early stopping at 50 epochs.Source: A Experiment Details · p. 18
• Table 6 cautions that dilated- and deformable-convolution results may not represent their original algorithms because the tested models were modularized versions.Source: Table 6 · p. 19

CLEVR

• CLEVR requires jointly training an LSTM question encoder with ADAM, weight decay, GloVe embeddings, and a single-GPU batch size of 64.Source: CLEVR · p. 19

B.1 Robustness against Deformations

• The classification-only repeat isolates BCN’s classification performance from the earlier hybrid classification-and-localization training objective.Source: §B.1 · p. 19
• BCN is intended to provide spatial invariance against scale and rotation deformations by globally broadcasting spatial features across receptive fields.Source: §B.1 · p. 19
• Figure 6 visualizes broadcast information for randomly rotated MoreScaled-MNIST images and marks misclassified digits with red boxes.Source: Figure 6 · p. 20, §B.1 · p. 20
• MoreScaled-MNIST evaluates unseen scale and aspect-ratio changes, rotations up to ±45°, and affine transformations using models trained only on Scaled-MNIST.Source: §B.1 · pp. 19–20

B.2 Sectional Runtimes

• RN’s MLP computation grows 16-fold when feature-map area increases fourfold, whereas post-broadcast convolution grows only fourfold.Source: §B.2 · p. 20
• The FLOP comparison treats input convolutions and fφ as fixed while varying BCN and gθ implementations in multiRN.Source: Table 7 · p. 21
• Table 8 evaluates how changing BCN size affects CNNh+multiRN validation accuracy on Sort-of-CLEVR, averaging three validation results.Source: Table 8 · p. 21
• Table 7 reports substantially lower total FLOPs for multiRN than RN under the Sort-of-CLEVR runtime comparison.Source: §B.2 · p. 21, Table 7 · p. 21

B.3 Effect of the size of BCN on MultiRN.

• The [128, 128, 256] three-layer BCN achieves the best relational accuracy in Sort-of-CLEVR.This configuration is the bottleneck design used in the paper.Source: §B.3 · pp. 21–22
• BCN influences relational queries more strongly than non-relational queries, for which removing BCN still reaches 99.3% accuracy.Larger final BCN layers broadcast more information because max-pooling determines the output dimension.Source: §B.3 · pp. 21–22
• Three-layer BCN models outperform two-layer models, while [128, 128, 128] exceeds [256, 256] with fewer parameters and computations.Bottleneck designs improve performance while reducing parameter count.Source: §B.3 · p. 21

B.4 More Activation Maps

• Randomly selected activation maps show how BCN broadcasts effective spatial features across the tested tasks.The samples were presented without manual selection.Source: §B.4 · p. 22
• Figure 7 presents activation maps for image-classification tasks, with odd rows showing original images and even rows showing maps.The examples are randomly selected data.Source: Figure 7 · p. 23
• Figure 8 presents activation maps for relational-reasoning tasks, using odd rows for original images and even rows for maps.The examples are randomly selected data.Source: Figure 8 · p. 24