PaperlayerView exact revision on arXiv

Source-linked AI summary

Fully Convolutional Neural Networks for Crowd Segmentation

Kai Kang, Xiaogang Wang

arXiv:1411.4464v1cs.CV

TL;DR

Crowd segmentation must handle varied scenes, occlusion, small pedestrians, stationary groups, and misleading motion. The paper proposes a fully convolutional network with multi-stage appearance, motion, and structure fusion, evaluated on two datasets spanning 235 and 11 scenes. FCNN provides one-pass, translation-invariant segmentation for arbitrary image sizes, while the created datasets support evaluation and component analysis.

  • Problem

    Crowd segmentation is difficult because scene appearance varies, pedestrians are occluded and small, stationary groups evade motion cues, and moving non-crowd objects create false positives.

    Source: §1 · pp. 1–2

  • Method

    The paper replaces CNN fully connected layers with 1 × 1 convolutions and combines appearance, motion, and structure cues through multi-stage pretraining followed by joint fine-tuning.

    Source: §1.2 · p. 2

  • Results

    FCNN segments whole images in one forward pass, preserves translation invariance, accepts arbitrary image sizes, and is evaluated on datasets containing 235 and 11 scenes.

    Source: §1.2 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The approach provides a fast full-frame crowd-segmentation framework that integrates complementary visual cues and supports cross-scene dataset evaluation.

    Source: §1.2 · p. 2

    The model requires substantial labeled video data with distinct camera views, localization annotations, and ideally pixel-level segmentation ground truth.

    Source: §5.1 · p. 5

Abstract

from arXiv · show

In this paper, we propose a fast fully convolutional neural network (FCNN) for crowd segmentation. By replacing the fully connected layers in CNN with 1 by 1 convolution kernels, FCNN takes whole images as inputs and directly outputs segmentation maps by one pass of forward propagation. It has the property of translation invariance like patch-by-patch scanning but with much lower computation cost. Once FCNN is learned, it can process input images of any sizes without warping them to a standard size. These attractive properties make it extendable to other general image segmentation problems. Based on FCNN, a multi-stage deep learning is proposed to integrate appearance and motion cues for crowd segmentation. Both appearance filters and motion filers are pretrained stage-by-stage and then jointly optimized. Different combination methods are investigated. The effectiveness of our approach and component-wise analysis are evaluated on two crowd segmentation datasets created by us, which include image frames from 235 and 11 scenes, respectively. They are currently the largest crowd segmentation datasets and will be released to the public.

1. Introduction

Crowd segmentation is difficult because scenes vary substantially and motion cues miss stationary groups while producing false positives. The paper proposes FCNN-based appearance and motion modeling to provide fast, translation-invariant, full-frame segmentation across varied scenes.

1. Introduction

  • Crowd segmentation supports counting, tracking, behavior analysis, and abnormality detection, making accurate crowd-region localization foundational for surveillance applications.Counting methods may require segmented crowd regions, while tracking and behavior analysis require known crowd locations.Source: §1 · p. 1
  • Motion segmentation misses stationary crowds and can falsely detect moving objects, while appearance-based methods face occlusion, small pedestrians, and cross-scene visual variation.Handcrafted HOG, SIFT, and LBP features are described as unable to handle these complex appearance changes well.Source: §1 · pp. 1–2, Figure 1 · p. 1

1.1. CNN for segmentation

  • Patch-by-patch CNN segmentation is translation-invariant but computationally impractical, taking five minutes for a 576 × 720 frame in the reported GPU experiment.Whole-image CNN regression reduces computation but loses translation invariance and requires fixed-size inputs because of fully connected layers.Source: §1.1 · p. 2

1.2. Our method

  • FCNN replaces fully connected layers with 1 × 1 convolutions, producing whole-image segmentation maps in one pass while supporting translation invariance, large receptive fields, and arbitrary input sizes.The reported design avoids image-size normalization and captures contextual information through six convolutional and two pooling layers.Source: §1.2 · p. 2, §1.1 · p. 2, Figure 2 · p. 3
  • Multi-stage deep learning combines appearance, motion, and structure cues by pretraining filters stage-by-stage and then jointly fine-tuning them.Motion filters are pretrained on samples that appearance features cannot confidently classify, with appearance outputs supplied as contextual information.Source: §1.2 · p. 2
  • The approach is evaluated on two created datasets containing 235 and 11 scenes, with detailed component-wise analysis.The supplied passages state that the datasets include manually annotated frames, while the conclusion describes them as the largest crowd segmentation datasets up to that date.Source: §1.2 · p. 3

2. Related Work

Prior crowd-segmentation methods commonly rely on motion, background subtraction, detection, tracking, or combined cues, but often depend on restrictive scene assumptions. Related general image-segmentation methods also face fixed-size or computational limitations.

2. Related Work

  • Background subtraction and motion segmentation often require static cameras or fixed pedestrian motion patterns, while detection and tracking degrade under severe crowd occlusion.Most reviewed methods also train and test on the same scene, limiting cross-scene applicability.Source: §2 · p. 3
  • Traditional CNN segmentation uses patch-by-patch scanning or fully connected regression, with the latter requiring fixed-size inputs and outputs.These approaches are presented as the conventional alternatives for general image segmentation.Source: §2 · p. 3
  • The cited mlpconv approach uses feature maps instead of fully connected layers, but its layers behave similarly to patch-by-patch scanning.The passage distinguishes this behavior from the paper’s fully convolutional approach.Source: §2 · p. 3
  • 1 × 1 convolutional kernels were also used in GoogLeNet [32] to reduce computation cost and support greater network depth and width.This related use concerns network scaling rather than the paper’s crowd-segmentation pipeline.Source: §2 · p. 3

3. Fully Convolutional Neural Networks

This section replaces fixed-size fully connected prediction with fully convolutional processing, enabling spatially aligned segmentation outputs while preserving translation-related properties. It also defines the receptive-field behavior used to incorporate context into crowd predictions.

3.1. Convolutional Neural Networks (CNN)

  • CNNs combine convolution, pooling, nonlinear neuron activations, and fully connected layers, with pooling improving translation robustness and reducing parameters.Fully connected layers instead use fixed weight matrices and biases tied to fixed input and output sizes.Source: §3.1 · pp. 3–4
  • Fully connected layers can be rewritten as 1 × 1 convolutions, allowing the network to preserve spatial feature-map dimensions during fusion.The 1 × 1 operation treats the fully connected computation as convolution over 1 × 1 feature maps.Source: §3.1 · p. 4, §3.2 · p. 4

3.2. Fully-convolutional Neural Networks (FCNN)

  • Unlike fully connected segmentation, convolutional and pooling operations preserve local spatial relationships and are independent of input-map shape.This supports applying learned filters to images of varying sizes rather than requiring fixed input dimensions.Source: §3.2 · p. 4, §3.1 · p. 4
  • FCNN takes whole images and directly predicts down-sampled crowd probabilities in one forward pass, using pooled labels and cross-entropy training.Padding preserves convolutional dimensions, while two 2 × 2 pooling layers reduce the output resolution by a factor of four.Source: §3.2 · p. 4

3.3. Receptive Fields of FCNN

  • Receptive fields specify the input regions influencing outputs; pooling shifts neighboring fields by the pooling size, while 1 × 1 layers leave them unchanged.The model uses 3 × 3 fusion kernels to increase receptive fields and incorporate more context into crowd predictions.Source: §3.3 · p. 4

4. Multi-stage FCNN

The multi-stage FCNN combines appearance, motion, and structure cues because each supplies distinct evidence for crowd segmentation. It compares input, feature, and decision fusion, then trains added branches progressively before global fine-tuning.

4.1. Visual Cues for Crowd Segmentation

  • Appearance detects stationary crowds, motion helps identify moving pedestrians and suppress background structures, and structure cues encode crowd-like scene geometry.The structure branch uses edge-detection results to represent scene dimensions and perspective-related information.Source: §4.1 · pp. 4–5
  • Figure 3 pre-trains appearance, motion, and structure branches successively, fixing earlier branches before finally fine-tuning the complete network.The sequence is appearance fixed while motion is tuned, then both fixed while structure is tuned, followed by global optimization.Source: Figure 3 · p. 5

4.2. Fusion Schemes and Extensible Architecture

  • The three fusion schemes are input fusion, feature fusion, and decision fusion, differing in whether they combine input maps, intermediate features, or separately trained outputs.Decision fusion combines three independently trained networks in a manner similar to learning from multiple experts.Source: §4.2 · p. 5
  • Cascaded training fixes previous branches while tuning each new branch, preserving the original network if the new branch is removed and encouraging complementary information.This architecture is intended to remain extensible as additional information sources are added.Source: §4.2 · p. 5
  • The labeling scheme uses region-level crowd polygons for training and validation, and pixel-level ground truth for test evaluation.The region labels cover crowd or pedestrian areas, whereas test labels support accurate full-frame evaluation.Source: Figure 4 · p. 5

5. Experiments

The experiments introduce two cross-scene video datasets and compare FCNN components against conventional motion and HOG-based baselines. Training uses augmentation, a shared network design, layer-wise pre-training, and global fine-tuning.

5.1. Datasets

  • The datasets were built because no public dataset satisfied the requirements for distinct camera views, labeled frames, pixel-level testing, and video-based motion cues.These requirements target cross-scene evaluation and integration of appearance with motion information.Source: §5.1 · p. 5
  • The Shanghai World Expo dataset contains 235 camera views, with 184 for training and 51 for testing, while the City Dataset contains 11 test scenes.Training labels are polygon-based, whereas both test sets use pixel-level annotations.Source: §5.1 · p. 5
  • Training spans 184 scenes to cover broad perspective variation, motivated by the strong effect of perspective distortion on crowd-patch appearance.Insufficient scene diversity is described as limiting generalization capability.Source: Figure 5 · p. 6, §5.1 · p. 6

5.2. Baseline Models

  • Baselines include Gaussian Mixture background subtraction on original video clips and patch-based HOG features with a linear SVM.The HOG-SVM baseline trains on 72 × 72 patches and samples test patches every 10 pixels.Source: §5.2 · p. 6
  • Component analysis trains separate FCNNs on original frames, background-subtracted frames, and edge detections, using 256 × 256 crops and random horizontal flipping.The validation scenes are disjoint from the training scenes: 161 training scenes versus 23 validation scenes.Source: §5.2 · p. 6, §5.3 · p. 6

5.3. System Settings

  • The FCNN uses padded convolutions, two pooling stages, a sigmoid output, cross-entropy loss, and the same architecture for appearance, motion, and structure branches.Layer-wise pre-training adds convolutional layers progressively before globally fine-tuning all parameters.Source: §5.3 · p. 6

6. Results

Across the Shanghai World Expo and City datasets, FCNN variants and fusion schemes are evaluated against baseline methods using ROC/AUC results. Fusion improves segmentation by combining complementary appearance, motion, and structure cues, although the best combination depends on the dataset.

6. Results

  • The evaluation uses ROC curves and AUC values on two custom test datasets containing 580 pixel-level annotated frames from 47 and 11 scenes.Figure 6 and Tables 1–2 report the Shanghai World Expo and City Dataset results.Source: §6 · p. 6, Figure 6 · p. 7, Table 1 · p. 7
  • FCNN models outperform the baseline methods across the evaluated datasets, demonstrating stronger generalization than handcrafted or background-modeling approaches.On Shanghai, the authors report a large margin over other methods; on City, all three single FCNN models improve over baselines.Source: §6.1 · p. 7, §6.2 · p. 7

6.1. Shanghai World Expo Dataset

  • On Shanghai World Expo, all three fusion schemes improve single-model performance, with Feature Fusion slightly outperforming Decision Fusion.Input Fusion provides only a slight improvement, while multi-stage combinations add motion and structure information to an appearance-based main branch.Source: §6.1 · p. 7, Table 1 · p. 7

6.2. City Dataset

  • City results show that FCNN structure models outperform HOG models by a large margin, while HOG generalizes poorly across substantially different appearances.The City videos also contain more frames and fewer stationary crowds, which helps motion-based models relative to appearance models.Source: §6.2 · p. 7
  • On the City Dataset, Input Fusion performs best, while Feature Fusion and Decision Fusion are comparable but lower than the single motion model.The authors attribute this pattern to the appearance branch being the main branch in multi-stage fusion, whereas motion can dominate in the single-stage Input Fusion model.Source: §6.2 · p. 7, Table 2 · p. 7

6.3. Discussions

  • Fusion corrects complementary errors by removing appearance false positives, recovering distant moving crowds, and restoring stationary pedestrians missed by motion.These improvements combine motion cues with appearance cues and are illustrated on both datasets in Figure 7.Source: §6.3 · p. 8, Figure 7 · p. 8

7. Conclusion

The paper proposes FCNN-based full-frame crowd segmentation with integrated appearance, motion, and structure information. It also introduces two large crowd-segmentation datasets for training and evaluation, which the authors plan to release publicly.

7. Conclusion

  • The paper proposes a fully convolutional model for full-frame training and testing in crowd segmentation.Source: §7 · p. 8
  • The approach incorporates appearance, motion, and structure information through three proposed fusion schemes.Source: §7 · p. 8
  • Two datasets contain 235 and 11 distinct camera views, are described as the largest crowd-segmentation datasets at the time, and will be released publicly.Source: §7 · p. 8
Loading 1411.4464v1…