Source-linked AI summary
Real-time Semantic Segmentation of Crop and Weed for Precision Agriculture Robots Leveraging Background Knowledge in CNNs
Andres Milioto, Philipp Lottes, Cyrill Stachniss
TL;DR
The paper addresses real-time RGB-only segmentation of crops, weeds, and background for robotic weed control, where changing fields and limited labeled data challenge conventional approaches. It augments an efficient encoder–decoder CNN with vegetation indices and related representations, achieving strong cross-field performance, reduced adaptation effort, and real-time operation on agricultural-robot data.
Problem
Real-time crop–weed segmentation from RGB images must support targeted weed control while handling changing field conditions and limited labeled data.
Method
The authors use a deep encoder–decoder CNN with a 14-channel input combining normalized RGB, vegetation indices, and other task-relevant representations.
Results
The representation-augmented network significantly outperforms RGB-only segmentation across test categories, matches NIR-assisted performance comparably, and reduces relabeling effort for new fields.
Takeaways & Limitations
The system supports real-time agricultural-robot operation using RGB data while adapting to unseen fields with comparatively little training data.
Abstract
from arXiv · showhide
Precision farming robots, which target to reduce the amount of herbicides that need to be brought out in the fields, must have the ability to identify crops and weeds in real time to trigger weeding actions. In this paper, we address the problem of CNN-based semantic segmentation of crop fields separating sugar beet plants, weeds, and background solely based on RGB data. We propose a CNN that exploits existing vegetation indexes and provides a classification in real time. Furthermore, it can be effectively re-trained to so far unseen fields with a comparably small amount of training data. We implemented and thoroughly evaluated our system on a real agricultural robot operating in different fields in Germany and Switzerland. The results show that our system generalizes well, can operate at around 20Hz, and is suitable for online operation in the fields.
I. INTRODUCTION
The paper targets real-time RGB-based crop–weed classification for precision weeding, reducing reliance on uniform herbicide treatment. Its CNN adds task-relevant background knowledge to support accurate segmentation, generalization, and near-camera-rate operation.
- Motivation: Per-plant weed control requires real-time image classification that labels individual plants as crop or weed.Selective spraying and mechanical control are presented as alternatives to uniform field treatment.
- Problem: The paper addresses crop-field classification from standard RGB images at roughly the camera framerate.The output can support targeted weed control or field monitoring without human interaction.
- Deployment: The system is intended for real-time operation on an agricultural robot deployed in field conditions.The figure shows the Bonirob platform and a classification result with crops colored green and weeds red.
- Contribution: The contribution is a CNN-based crop–weed classifier that feeds task-relevant background knowledge to improve training speed and generalization to new fields.The added knowledge is supplied through additional input channels derived from representations used in handcrafted classification features.
- Claims: The approach performs pixel-wise segmentation of crops, weeds, and soil without expensive near-infrared information.The authors also claim robustness to unseen lighting, soil, and weather conditions with little adaptation data.
II. RELATED WORK
Prior crop–weed systems use supervised vision methods, often with NIR, handcrafted features, or geometric priors, but changing field conditions and adaptation data remain challenges. The paper addresses these issues with an RGB CNN using vegetation indices and preprocessing mappings.
- Generalization: Prior RGB+NIR classification reached up to 96% pixel accuracy, but performance decreased when plant appearance changed substantially.This limited suitability for weed-control applications under changing conditions.
- Adaptation: Other work uses crop arrangement priors or data augmentation to address changing feature distributions and adaptation to new fields.These approaches incorporate geometric information or simulated real-world variation.
- This work: The paper proposes an RGB CNN feature extractor and classifier that uses no geometric prior and combines efficient segmentation with vegetation indices and preprocessing mappings.Its stated target is generalization across soil, weather, and illumination conditions.
- Labeling cost: Supervised classification approaches require labeled data for retraining when adapting to a new field.Prior work explored training-data generation and related adaptation strategies.
- Vegetation detection: Threshold-based vegetation detection can fail across fields because soil, growth stage, illumination, and weather alter index distributions.Global thresholding misses underrepresented vegetation, while adaptive thresholding can oversegment large plants or split connected components.
III. APPROACH
The approach is an end-to-end semantic-segmentation CNN designed for agricultural robots, combining visual preprocessing with per-pixel classification. It emphasizes accuracy, computational efficiency, and generalization for real-time weed control.
- III. APPROACH: The paper proposes a purely visual, pixel-wise semantic-segmentation CNN to distinguish weeds, value crops, and soil in real time.The design explicitly considers accuracy, processing time, hardware efficiency, and generalization.
- III. APPROACH: The pipeline first computes vegetation indices and alternate representations, then feeds them as additional inputs to a self-designed semantic-segmentation network.The second stage produces per-pixel semantic labels.
- III. APPROACH: Additional input information is motivated by the limited amount of labeled agricultural-field data available for training.The pipeline is structured to use task-relevant representations before CNN-based segmentation.
A. Input Representations
The input representation augments normalized RGB with vegetation indices, color-space channels, and image operators. These 14 channels provide task-relevant cues intended to improve learning, generalization, vegetation separation, and convergence.
- Input motivation: Raw RGB inputs alone would require broad training data covering fields, lighting, weather, growth stages, and soil types.The paper therefore introduces input assumptions to improve generalization with limited training data.
- Vegetation indices: Four RGB-derived vegetation indices—ExG, ExR, CIVE, and NDI—provide additional cues for vegetation segmentation.These indices are selected because they have been used for plant classification and vegetation detection.
- Alternate representations: The network also receives HSV representations and Sobel, Laplacian, and Canny-derived operators on the indices.These alternate representations complement the four vegetation indices.
- Input volume: 14 channels are formed by concatenating the representations with the channel-wise normalized RGB image before CNN processing.The resulting input volume is illustrated through alternate representations and listed in Table I.
- Reported effects: The extra representations improve generalization and vegetation separation while speeding training convergence.The paper reports these effects relative to using the raw inputs alone in its experiments.
B. Network Architecture
The network uses a compact encoder-decoder CNN for pixel-wise crop, weed, and background segmentation, combining task-specific building blocks with RGB-derived input representations. Residual separable bottlenecks, shared-index unpooling, and a narrow three-class output support efficient inference.
- The encoder-decoder network performs pixel-wise prediction while running at 20+ Hz and using fewer than 30,000 parameters.
- Separating each 5×5 convolution into 5×1 and 1×5 operations reduces computation from 1,856 FLOPs to 896 FLOPs.
- Shared pooling indexes let decoder unpooling preserve encoder activation positions without using comparatively expensive transposed convolutions.
- The final softmax predicts a three-element pseudo-probability vector per pixel for background, weed, and crop.
- The encoder compresses representations through 13 convolutional layers and four pooling layers, while the decoder reconstructs same-sized semantic maps using 12 convolutional and four unpooling layers.
- Residual separable bottlenecks replace 24 of 25 convolutional layers to make the network smaller, easier to train, and more efficient while preserving a 200×200-pixel receptive field.
IV. EXPERIMENTAL EVALUATION
The experiments evaluate whether the method provides accurate pixel-wise crop-weed classification, generalizes across conditions, and operates in real time. The approach was implemented with TensorFlow and OpenCV and tested on a BoniRob agricultural robot.
- The experiments are designed to assess pixel-wise classification accuracy, generalization, and real-time operation.
- The complete system was implemented using TensorFlow and OpenCV and tested on a Bosch Deepfield Robotics BoniRob UGV.
A. Training and Testing Data
The evaluation uses three datasets from Germany and Switzerland spanning varied growth, soil, weather, and illumination conditions. Generalization is tested by training on Bonn data and evaluating both held-out Bonn images and the Zurich and Stuttgart datasets.
- Three datasets were captured in Bonn, Stuttgart, and Zurich, with plants and weeds across growth stages and differing soil, weather, and illumination conditions.
- The network is trained only on Bonn images and evaluated on a held-out Bonn test split plus the complete Zurich and Stuttgart datasets, including data from different years.
- Bonn data are partitioned into 70% training, 15% validation, and 15% testing subsets.
- Training uses random geometric perturbations, weighted cross-entropy to address soil-dominated class imbalance, and Adam-based gradient updates.
- The input-channel experiment compares RGB-only, RGB with extra representations, and an RGB+NIR baseline network.
B. Performance of the Semantic Segmentation
The proposed network accurately segments crops, weeds, and soil at pixel and object levels, including across fields with different conditions. Extra representations improve performance over RGB alone and converge faster than both RGB and RGB+NIR alternatives.
- The experiment evaluates pixel-wise segmentation of crops, weeds, and soil, including heavy plant overlap across growth stages.
- The RGB-plus-representations network significantly outperforms RGB alone across all evaluated categories and performs comparably to the RGB+NIR network.The comparison covers held-out Bonn data and complete Stuttgart and Zurich datasets.
- Object-wise performance is higher with all representations than with RGB alone, with the difference becoming critical when generalizing to Zurich and Stuttgart.The RGB network becomes unusable for most tasks on those datasets.
- 30% faster convergence to 95% of final accuracy is achieved with RGB plus extra representations than with RGB alone, and 15% faster than with RGB+NIR.
C. Labeling Cost for Adaptation to New Fields
The study tests adaptation by retraining on small labeled subsets from new Zurich and Stuttgart fields. Extra representations reduce labeling effort and enable stronger, more reliable cross-field performance than RGB alone.
- The adaptation experiment measures how much labeled data from Zurich and Stuttgart is needed to reach strong performance in a new field.The datasets are split into training and testing halves, with 10, 20, 50, or 100 retraining images sampled from the training halves.
- 100 Zurich images for RGB retraining achieve roughly the same performance as 10 images with all input representations, substantially reducing relabeling effort.
- With 100 Zurich retraining images, the representation-based network reaches precision and recall values in the order of 90%.
- On Stuttgart, RGB fails to reach acceptable performance, whereas the representation-based network improves approximately linearly as more images are added.
D. Runtime
The network is designed for real-time field operation on mobile-robot hardware. It exceeds 20 frames per second on the robot platform and remains operational at almost 5 Hz on a low-power Jetson TX2.
- The runtime experiment evaluates whether the approach supports real-time online operation on hardware suitable for mobile robots.
- More than 20 frames per second is achieved on an Intel i7 CPU with an NVIDIA GTX1080Ti GPU, despite the speed penalty from all extra representations.
- Almost 5 Hz is obtained on the compact 15 W peak-power Jetson TX2 platform.The platform is described as suitable for operation on a flying vehicle.
- Figure 7 reports object-wise mean accuracy as a function of relabeling effort, with zero images representing no retraining.
V. CONCLUSION
The paper presents an RGB-only, real-time CNN for pixel-wise crop-field segmentation using vegetation indexes and other background knowledge. Evaluations across German and Swiss fields indicate strong generalization and suitability for online agricultural-robot operation.
- A deep encoder-decoder CNN segments crops, weeds, and background pixel-wise from RGB data using a 14-channel input of vegetation indexes and related representations.
- The additional task-relevant background knowledge speeds training and improves generalization to new crop fields, especially with limited training data.
- Evaluations on data from three cities in Germany and Switzerland suggest that the system generalizes well, runs at around 20 Hz, and supports online field operation.