Real-time Semantic Segmentation of Crop and Weed for Precision Agriculture Robots Leveraging Background Knowledge in CNNs

Motivation

• Per-plant weed control requires real-time image classification that labels individual plants as crop or weed.Selective spraying and mechanical control are presented as alternatives to uniform field treatment.Source: §I · p. 1

Problem

• The paper addresses crop-field classification from standard RGB images at roughly the camera framerate.The output can support targeted weed control or field monitoring without human interaction.Source: §I · p. 1

Deployment

• The system is intended for real-time operation on an agricultural robot deployed in field conditions.The figure shows the Bonirob platform and a classification result with crops colored green and weeds red.Source: Figure 1 · p. 1, §I · p. 1

Contribution

• The contribution is a CNN-based crop–weed classifier that feeds task-relevant background knowledge to improve training speed and generalization to new fields.The added knowledge is supplied through additional input channels derived from representations used in handcrafted classification features.Source: §I · p. 1

Claims

• The approach performs pixel-wise segmentation of crops, weeds, and soil without expensive near-infrared information.The authors also claim robustness to unseen lighting, soil, and weather conditions with little adaptation data.Source: §I · p. 1

Generalization

• Prior RGB+NIR classification reached up to 96% pixel accuracy, but performance decreased when plant appearance changed substantially.This limited suitability for weed-control applications under changing conditions.Source: §II · p. 2

Adaptation

• Other work uses crop arrangement priors or data augmentation to address changing feature distributions and adaptation to new fields.These approaches incorporate geometric information or simulated real-world variation.Source: §II · p. 2

This work

• The paper proposes an RGB CNN feature extractor and classifier that uses no geometric prior and combines efficient segmentation with vegetation indices and preprocessing mappings.Its stated target is generalization across soil, weather, and illumination conditions.Source: §II · p. 2

Labeling cost

• Supervised classification approaches require labeled data for retraining when adapting to a new field.Prior work explored training-data generation and related adaptation strategies.Source: §II · p. 2

Vegetation detection

• Threshold-based vegetation detection can fail across fields because soil, growth stage, illumination, and weather alter index distributions.Global thresholding misses underrepresented vegetation, while adaptive thresholding can oversegment large plants or split connected components.Source: §II · p. 2, Figure 2 · p. 2

III. APPROACH

• The paper proposes a purely visual, pixel-wise semantic-segmentation CNN to distinguish weeds, value crops, and soil in real time.The design explicitly considers accuracy, processing time, hardware efficiency, and generalization.Source: §III · p. 3
• The pipeline first computes vegetation indices and alternate representations, then feeds them as additional inputs to a self-designed semantic-segmentation network.The second stage produces per-pixel semantic labels.Source: §III · p. 3
• Additional input information is motivated by the limited amount of labeled agricultural-field data available for training.The pipeline is structured to use task-relevant representations before CNN-based segmentation.Source: §III · p. 3

Input motivation

• Raw RGB inputs alone would require broad training data covering fields, lighting, weather, growth stages, and soil types.The paper therefore introduces input assumptions to improve generalization with limited training data.Source: A. Input Representations · p. 3

Vegetation indices

• Four RGB-derived vegetation indices—ExG, ExR, CIVE, and NDI—provide additional cues for vegetation segmentation.These indices are selected because they have been used for plant classification and vegetation detection.Source: A. Input Representations · p. 3

Alternate representations

• The network also receives HSV representations and Sobel, Laplacian, and Canny-derived operators on the indices.These alternate representations complement the four vegetation indices.Source: A. Input Representations · p. 3

Input volume

• 14 channels are formed by concatenating the representations with the channel-wise normalized RGB image before CNN processing.The resulting input volume is illustrated through alternate representations and listed in Table I.Source: A. Input Representations · p. 3, Figure 3 · p. 3, Table I · p. 3

Reported effects

• The extra representations improve generalization and vegetation separation while speeding training convergence.The paper reports these effects relative to using the raw inputs alone in its experiments.Source: A. Input Representations · p. 3

• The encoder-decoder network performs pixel-wise prediction while running at 20+ Hz and using fewer than 30,000 parameters.Source: B. Network Architecture · p. 4
• Separating each 5×5 convolution into 5×1 and 1×5 operations reduces computation from 1,856 FLOPs to 896 FLOPs.Source: B. Network Architecture · p. 4
• Shared pooling indexes let decoder unpooling preserve encoder activation positions without using comparatively expensive transposed convolutions.Source: B. Network Architecture · p. 4
• The final softmax predicts a three-element pseudo-probability vector per pixel for background, weed, and crop.Source: B. Network Architecture · p. 4
• The encoder compresses representations through 13 convolutional layers and four pooling layers, while the decoder reconstructs same-sized semantic maps using 12 convolutional and four unpooling layers.Source: B. Network Architecture · p. 4
• Residual separable bottlenecks replace 24 of 25 convolutional layers to make the network smaller, easier to train, and more efficient while preserving a 200×200-pixel receptive field.Source: B. Network Architecture · pp. 4–5

• The experiments are designed to assess pixel-wise classification accuracy, generalization, and real-time operation.Source: §IV · p. 5
• The complete system was implemented using TensorFlow and OpenCV and tested on a Bosch Deepfield Robotics BoniRob UGV.Source: §IV · p. 5

• Three datasets were captured in Bonn, Stuttgart, and Zurich, with plants and weeds across growth stages and differing soil, weather, and illumination conditions.Source: A. Training and Testing Data · p. 5, Figure 6 · p. 5
• The network is trained only on Bonn images and evaluated on a held-out Bonn test split plus the complete Zurich and Stuttgart datasets, including data from different years.Source: A. Training and Testing Data · p. 5
• Bonn data are partitioned into 70% training, 15% validation, and 15% testing subsets.Source: A. Training and Testing Data · p. 5
• Training uses random geometric perturbations, weighted cross-entropy to address soil-dominated class imbalance, and Adam-based gradient updates.Source: A. Training and Testing Data · p. 5
• The input-channel experiment compares RGB-only, RGB with extra representations, and an RGB+NIR baseline network.Source: A. Training and Testing Data · p. 5

• The experiment evaluates pixel-wise segmentation of crops, weeds, and soil, including heavy plant overlap across growth stages.Source: B. Performance of the Semantic… · p. 5
• The RGB-plus-representations network significantly outperforms RGB alone across all evaluated categories and performs comparably to the RGB+NIR network.The comparison covers held-out Bonn data and complete Stuttgart and Zurich datasets.Source: B. Performance of the Semantic… · p. 5, Table III · p. 6
• Object-wise performance is higher with all representations than with RGB alone, with the difference becoming critical when generalizing to Zurich and Stuttgart.The RGB network becomes unusable for most tasks on those datasets.Source: B. Performance of the Semantic… · p. 5, Table IV · p. 6
• 30% faster convergence to 95% of final accuracy is achieved with RGB plus extra representations than with RGB alone, and 15% faster than with RGB+NIR.Source: B. Performance of the Semantic… · p. 5

• The adaptation experiment measures how much labeled data from Zurich and Stuttgart is needed to reach strong performance in a new field.The datasets are split into training and testing halves, with 10, 20, 50, or 100 retraining images sampled from the training halves.Source: C. Labeling Cost for Adaptation to New… · p. 6
• 100 Zurich images for RGB retraining achieve roughly the same performance as 10 images with all input representations, substantially reducing relabeling effort.Source: C. Labeling Cost for Adaptation to New… · p. 6, Table V · p. 6
• With 100 Zurich retraining images, the representation-based network reaches precision and recall values in the order of 90%.Source: C. Labeling Cost for Adaptation to New… · p. 6, Table V · p. 6
• On Stuttgart, RGB fails to reach acceptable performance, whereas the representation-based network improves approximately linearly as more images are added.Source: C. Labeling Cost for Adaptation to New… · p. 6, Table VI · p. 6

• The runtime experiment evaluates whether the approach supports real-time online operation on hardware suitable for mobile robots.Source: D. Runtime · p. 6
• More than 20 frames per second is achieved on an Intel i7 CPU with an NVIDIA GTX1080Ti GPU, despite the speed penalty from all extra representations.Source: D. Runtime · p. 6, Table VII · p. 6
• Almost 5 Hz is obtained on the compact 15 W peak-power Jetson TX2 platform.The platform is described as suitable for operation on a flying vehicle.Source: D. Runtime · p. 6, Table VII · p. 6
• Figure 7 reports object-wise mean accuracy as a function of relabeling effort, with zero images representing no retraining.Source: Figure 7 · p. 7

• A deep encoder-decoder CNN segments crops, weeds, and background pixel-wise from RGB data using a 14-channel input of vegetation indexes and related representations.Source: §V · p. 7
• The additional task-relevant background knowledge speeds training and improves generalization to new crop fields, especially with limited training data.Source: §V · p. 7
• Evaluations on data from three cities in Germany and Switzerland suggest that the system generalizes well, runs at around 20 Hz, and supports online field operation.Source: §V · p. 7