PaperlayerView exact revision on arXiv

Source-linked AI summary

Fully Convolutional Networks with Sequential Information for Robust Crop and Weed Detection in Precision Farming

Philipp Lottes, Jens Behley, Andres Milioto, Cyrill Stachniss

arXiv:1806.03412v1cs.CV

TL;DR

Changing weed, crop, plant-appearance, and soil conditions make robust crop-weed classification difficult for precision-farming robots. The paper introduces an end-to-end encoder-decoder FCN with a sequential module that learns crop-row arrangement from RGB+NIR image sequences. Experiments report better generalization to unseen fields and different growth stages without retraining.

  • Problem

    Crop-weed classifiers must remain reliable across fields with changing weed pressure, plant appearance, growth stages, and soil conditions because robotic targeted treatment depends on plant identification.

  • Method

    An end-to-end encoder-decoder FCN uses a sequential module and 3D convolutions over RGB+NIR image sequences to encode row-plant spatial arrangement for pixel-wise segmentation.

  • Results

    The system generalizes better to unseen fields than other state-of-the-art approaches and robustly classifies crops at different growth stages without retraining.

  • Takeaways & Limitations

    Exploiting repetitive crop arrangement helps improve generalization of sequential FCN-based crop-weed classification under changing environmental conditions.

Abstract

from arXiv · show

Reducing the use of agrochemicals is an important component towards sustainable agriculture. Robots that can perform targeted weed control offer the potential to contribute to this goal, for example, through specialized weeding actions such as selective spraying or mechanical weed removal. A prerequisite of such systems is a reliable and robust plant classification system that is able to distinguish crop and weed in the field. A major challenge in this context is the fact that different fields show a large variability. Thus, classification systems have to robustly cope with substantial environmental changes with respect to weed pressure and weed types, growth stages of the crop, visual appearance, and soil conditions. In this paper, we propose a novel crop-weed classification system that relies on a fully convolutional network with an encoder-decoder structure and incorporates spatial information by considering image sequences. Exploiting the crop arrangement information that is observable from the image sequences enables our system to robustly estimate a pixel-wise labeling of the images into crop and weed, i.e., a semantic segmentation. We provide a thorough experimental evaluation, which shows that our system generalizes well to previously unseen fields under varying environmental conditions --- a key capability to actually use such systems in precision framing. We provide comparisons to other state-of-the-art approaches and show that our system substantially improves the accuracy of crop-weed classification without requiring a retraining of the model.

I. INTRODUCTION

The paper targets robust crop-weed identification for robotic, plant-specific interventions despite changing field conditions. It proposes an image-sequence FCN that exploits crop-row geometry for pixel-wise segmentation and reports strong generalization without retraining.

  • Motivation: Robotic selective spraying or mechanical weed stamping requires reliable pixel-wise identification of individual crops and weeds.Such classification is presented as the prerequisite for targeted treatment actions.
  • Challenge: Crop-weed classifiers must handle changes in weed pressure, weed types, plant appearance, growth stages, and soil conditions across fields.Purely visual systems can degrade when these environmental factors shift between training and testing.
  • Approach: Crop rows provide a geometric signal because crops share similar lattice spacing while weeds appear more randomly, making arrangement less sensitive to visual changes.The method analyzes image sequences covering a local strip of field surface to exploit this signal.
  • Approach: The proposed end-to-end FCN uses a sequential module to segment soil, crop, and weed pixels from image sequences without pre-segmentation or handcrafted features.The input images contain RGB and NIR channels.
  • Results: The system generalizes to unseen fields and different crop growth stages without retraining, achieving average recall above 94% for crops and above 91% for weeds.The claims are experimentally validated on real-world datasets against state-of-the-art and nonsequential FCN approaches.

II. RELATED WORK

Prior crop-weed systems include handcrafted pipelines, patch-based CNNs, and FCN semantic segmentation, but generalization often requires new labeling. This work combines FCN segmentation with sequential plant-arrangement information to improve generalization without relabeling.

  • Existing approaches: Earlier crop-weed methods commonly used handcrafted features, while newer approaches increasingly use end-to-end CNNs.The shift toward CNNs addresses inflexibility and limitations associated with handcrafted vision pipelines.
  • Existing approaches: Patch-based CNN systems include vegetation-first cascades and compressed deep networks designed to retain practical processing times.These approaches classify image patches or vegetation pixels rather than directly segmenting the complete image.
  • FCN segmentation: FCNs directly estimate pixel-wise segmentation for complete images, and encoder-decoder architectures are common in semantic segmentation.Related crop-weed systems apply SegNet-like architectures to multispectral imagery or add background-knowledge channels.
  • Generalization: Prior systems share a need for tedious relabeling or labeling effort when adapting to different fields.Related work reduces this burden through self-supervised, semi-supervised, or unsupervised use of plant arrangement and image similarity.
  • Contribution: The proposed approach combines an FCN with sequential information to exploit repetitive row structure and improve generalization without relabeling effort.The authors identify it as an end-to-end learned semantic segmentation approach using row-plant spatial arrangement.

III. SEQUENTIAL CROP-WEED CLASSIFICATION SYSTEM

The system performs pixel-wise classification of background, crop, and weed using RGB+NIR image sequences from a crop row. Its sequential FCN learns plant arrangement while preprocessing reduces input-distribution diversity.

  • System objective: The model segments background, sugar beet crops, and weeds by learning planting-arrangement information from image sequences.The primary objective is robust operation when plant and soil appearance changes.
  • Sequence input: Five RGB+NIR frames are selected from the robot’s trajectory, with historical images chosen to avoid object-space overlap.Odometry and camera calibration determine the selection, while the current frame is paired with four historical frames.
  • Output: The network outputs a per-pixel probability distribution over class labels as a label mask.The example visualization compares predicted crop and weed masks with ground truth on a Stuttgart field using a model trained only in Bonn.
  • Architecture: The architecture contains preprocessing, an encoder-decoder FCN, and a sequential module, extending a single-image FCN into a sequence-to-sequence model.The visual encoder shares weights across the time axis and produces compressed visual codes for each image.
  • Preprocessing: Preprocessing independently blurs, standardizes, and scales each RGB and NIR channel to [-1, 1] to reduce input diversity.The Gaussian blur uses a [5 × 5] kernel with µ = 0 and σ^2 = 1.

C. Encoder-Decoder FCN

The encoder-decoder FCN converts multispectral images into full-resolution pixel labels through downsampling and upsampling. It uses FC-DenseNet blocks, bottlenecks, strided convolutions, and skip-like feature concatenation to preserve spatial information.

  • Architecture: The FCN follows an hourglass structure that downsamples resolution in the encoder and restores it in the decoder for pixel-wise segmentation.This structure allows the output to regain the input image’s full resolution.
  • Dense connectivity: The encoder-decoder uses FC-DenseNet blocks, combining densely connected convolutional blocks with fully convolutional networks.Dense connections concatenate earlier feature maps, encouraging feature reuse and supporting gradient flow.
  • Sequential extension: The architecture processes sequential context through a spatio-temporal pathway alongside visual features before producing the final label mask.The figure traces visual encoding and decoding, spatio-temporal fusion and decoding, then feature merging.
  • Encoder: The first encoder layer expands 4-channel images with 32 [5 × 5] kernels, followed by dense blocks, bottlenecks, and stride-2 downsampling.Bottleneck layers compress feature volumes between dense blocks.
  • Decoder: The decoder reverses downsampling with stride-2 transposed convolutions and concatenates encoder and decoder feature maps to recover spatial information.The concatenated features are processed by bottleneck and dense blocks.

D. Sequential Module

The sequential module adds a parallel pathway that processes image sequences to encode spatio-temporal plant-arrangement information, then merges it with visual features for pixel-wise labeling.

  • Sequential module: The sequential module is a parallel information pathway comprising spatio-temporal fusion, spatio-temporal decoding, and a merge layer.These components enable sequential data processing alongside the visual pathway.
  • Spatio-temporal fusion: Spatio-temporal fusion concatenates visual code volumes across the sequence and processes them with three-dimensional convolutions.The 3D kernels operate across the full sequence dimension, allowing joint sequence processing.
  • Spatio-temporal fusion: Increasing spatial kernel size and dilation in later 3D convolutions expands the receptive field for extracting plant-arrangement context.The design targets geometric patterns that extend across a larger spatial neighborhood.
  • Spatio-temporal decoder: The spatio-temporal decoder upsamples the sequence code to the output resolution through recurrent upsampling, bottlenecks, and dense blocks.Its pathway remains independent from the visual decoder because the two decoders do not share weights or connections.
  • Merge layer: The merge layer concatenates visual and sequence features, processes them through convolutional layers, and computes the output label mask.The merge therefore combines appearance and sequential features before final prediction.

IV. EXPERIMENTAL EVALUATION

The evaluation uses multiple sugar beet datasets and compares the proposed sequential FCN with a matched baseline and prior visual or geometry-based classifiers under varied field conditions.

  • Experimental setup: The experiments assess the method on sugar beet fields near Bonn and Stuttgart recorded with variants of the BoniRob platform.Datasets vary in lighting setups, crop growth stages, weed types, and soil types while using RGB+NIR cameras.
  • Evaluation protocol: Performance is evaluated object-wise by comparing predicted label masks with crop and weed objects obtained from connected components in ground-truth segments.Plant segments smaller than 0.5 cm^2 are treated as noise, and training, validation, and test images do not overlap.
  • Compared methods: The baseline is an encoder-decoder FCN with the proposed preprocessing but without the sequential module, using a comparable parameter count.Comparisons also include a random-forest vision-and-geometry method and a purely visual FCN using RGB data and vegetation features.

B. Parameters

The networks are trained from scratch on downsampled RGB+NIR sequences using a short sequence length, weighted cross-entropy, and RMSPROP optimization.

  • Parameters: Training uses images of W = 512 and H = 384, corresponding to a ground resolution of approximately 1 mm per pixel.All networks are trained from scratch on the downsampled images.
  • Parameters: The sequence length is S = 5, and dense blocks use grow rate G = 4.These values are selected as part of the reported architecture configuration.
  • Training: Optimization uses RMSPROP with a mini-batch size of B = 2, weighted cross-entropy, initial learning rate 0.01, and tenfold crop-and-weed error penalties.The learning rate is divided by 10 after the reported schedule milestones.
  • Training: The selected sequence length S = 5 balances available spatial information against computational cost, while inference runs at approximately 5 Hz on a GTX 1080 Ti.The hyperparameter combination was selected through a Bonn2016 validation search.

C. Performance Under Changing Environmental Conditions

Cross-field and cross-condition tests examine generalization without retraining, showing strong crop-weed performance and benefits from sequential information when visual conditions change.

  • Cross-field generalization: The first experiment evaluates a model trained on one robotic setup against previously unseen fields with changed visual appearance and no retraining.This setting is intended to represent practical deployment conditions.
  • Cross-field generalization: More than 95% of crops and around 88% of weeds are detected on Stuttgart, with roughly a 10% average F1-score gain over the second-best method.On Bonn2017, the method has an approximately 5% average F1-score margin and recalls 91% of crops and 95% of weeds.
  • Cross-field generalization: The method precisely separates crop and weed pixels from soil across the Stuttgart sequence, supporting high crop recall and vegetation separation.Predictions are presented over the whole sequence despite sequence-to-one classification.
  • Cross-field generalization: Visual baselines suffer substantial performance decreases under changed appearance, whereas exploiting plant arrangement improves generalization to other field environments.The reported comparisons find the proposed approach significantly better than baselines at the 99% confidence level.

D. Performance Under Changing Growth Stage

The approach maintains strong crop-weed classification performance when tested across changing sugar beet growth stages. Sequential information helps compensate for visual changes that reduce the baseline’s performance.

  • 91% average F1-score demonstrates solid performance under changing growth stages.The test images were captured later in the same field, with sugar beets about six times larger than in training.
  • Sequential data exploits the repetitive sugar beet arrangement and generalizes better to other growth stages than the baseline.The baseline relies solely on visual cues, whose effectiveness decreases when plant appearance changes.

E. Ablation Study

The ablation study evaluates preprocessing, sequential modeling, and expanded spatial context across held-out and cross-field data. Preprocessing and sequential architectural components improve generalization, while larger receptive fields provide further gains.

  • Ablation Study: The study compares architectural configurations using Bonn2016 training data and tests on held-out Bonn2016, Stuttgart, and Bonn2017.The evaluation uses 75% of Bonn2016 for training and assesses generalization across three test settings.
  • Ablation Study: Preprocessing reduces the effect of different lighting conditions with only a negligible Bonn2016 performance decrease.This design choice is intended to improve generalization across fields.
  • Ablation Study: Adding the sequential module on top of preprocessing further improves generalization capabilities.
  • Ablation Study: Larger kernels and dilated convolutions further improve performance by increasing the sequential module’s receptive field and spatial context.
  • Ablation Study: High performance on held-out Bonn2016 data indicates stable FCN performance when training and test distributions have comparably low diversity.The authors conclude that distributional diversity is crucial for practical evaluation and that preprocessing plus crop-arrangement modeling improves generalization.
  • Ablation Study: On simulated data, the sequential approach exploits crop-arrangement patterns, whereas the baseline cannot properly identify crops and weeds without shape information.The simulations encode arrangement using uniform-sized blobs without spectral or shape cues and include intra-row weeds.

V. CONCLUSION

The paper presents a fully convolutional crop-weed segmentation system that integrates sequential information to encode plant arrangement. Real-world experiments show improved generalization to unseen fields and growth stages, while simulations demonstrate that the sequential module captures spatial arrangement.

  • The system performs pixel-wise semantic segmentation into crop and weed for precision agriculture robots.
  • The method integrates sequential information into a fully convolutional network using 3D convolutions over image sequences to encode plant arrangement.
  • Real-world experiments demonstrate better generalization to unseen fields and robust crop classification across different growth stages.
  • Simulation results show that the sequential module encodes the spatial arrangement of plants.
Loading 1806.03412v1…