Source-linked AI summary
weedNet: Dense Semantic Weed Classification Using Multispectral Images and MAV for Smart Farming
Inkyu Sa, Zetao Chen, Marija Popovic, Raghav Khanna, Frank Liebisch, Juan Nieto, Roland Siegwart
TL;DR
Reliable weed detection is needed for selective treatment, but dense crop–weed annotation is difficult because plant boundaries are hard to distinguish. The paper combines SegNet-based dense semantic segmentation with multispectral MAV imagery and controlled datasets, achieving about 0.8 F1-score for weed detection and 1.8 Hz embedded inference while releasing the data for future work.
Problem
Reliable and precise weed detection is needed for selective treatment, while dense crop–weed annotation is difficult because plant boundaries are hard to distinguish.
Method
The paper uses an encoder–decoder SegNet CNN with multispectral MAV imagery, NDVI-based ground truth for homogeneous plots, and manual annotation for mixed imagery.
Results
About 0.8 F1-score was reported for weed detection, and the model ran at 1.8 Hz on an embedded platform deployable on a MAV.
Takeaways & Limitations
The released multispectral weed and crop images with ground truth support future agricultural robotics research.
Takeaways & Limitations
Performance showed spatio-temporal inconsistencies, including crops classified as weeds in an independent field image, indicating a need for more temporal training data and coverage of different weeds.
Abstract
from arXiv · showhide
Selective weed treatment is a critical step in autonomous crop management as related to crop health and yield. However, a key challenge is reliable, and accurate weed detection to minimize damage to surrounding plants. In this paper, we present an approach for dense semantic weed classification with multispectral images collected by a micro aerial vehicle (MAV). We use the recently developed encoder-decoder cascaded Convolutional Neural Network (CNN), Segnet, that infers dense semantic classes while allowing any number of input image channels and class balancing with our sugar beet and weed datasets. To obtain training datasets, we established an experimental field with varying herbicide levels resulting in field plots containing only either crop or weed, enabling us to use the Normalized Difference Vegetation Index (NDVI) as a distinguishable feature for automatic ground truth generation. We train 6 models with different numbers of input channels and condition (fine-tune) it to achieve about 0.8 F1-score and 0.78 Area Under the Curve (AUC) classification metrics. For model deployment, an embedded GPU system (Jetson TX2) is tested for MAV integration. Dataset used in this paper is released to support the community and future work.
I. INTRODUCTION
Smart farming requires spatially targeted treatment, but reliable pixel-level weed detection remains difficult because crop–weed boundaries are unclear. This paper addresses the challenge with dense semantic CNN segmentation, multispectral MAV imagery, controlled datasets, and released pixel-wise labels.
- Precision agriculture surveys crop-health indicators and applies herbicides, pesticides, and fertilizers only to relevant areas.
- Reliable weed detection supports selective stamping, spot spraying, and mechanical tillage while minimizing damage to surrounding vegetation.
- Dense semantic CNN segmentation is used to predict crop and weed classes at the pixel level.
- A downward-facing multispectral camera on a MAV collects crop-only, weed-only, and mixed imagery, with NDVI enabling automatic ground-truth generation for homogeneous plots.Mixed crop–weed images still require manual labeling, taking about 60 minutes per image in the described dataset.
- The paper contributes released pixel-wise sugar beet/weed datasets and a study of dense semantic segmentation with varying multispectral input channels.
II. RELATED WORK
Prior work spans CNN-based pixel-wise segmentation, vegetation detection, and crop–weed classification using handcrafted features or geometric information. SegNet addresses reduced-resolution prediction through paired encoder and decoder networks, while deep learning motivates end-to-end multispectral classification.
- Image segmentation assigns a human-readable class label to each pixel, but sequential pooling and down-sampling can produce low-resolution predictions.
- SegNet uses an encoder to compress images into lower-resolution features and a corresponding decoder to up-sample them for full-resolution per-pixel segmentation.
- Existing vegetation methods use features such as NDVI, Excess Green Index, spectral values, crop-row geometry, and handcrafted learning algorithms.
- Earlier crop–weed approaches classify image patches or detected vegetation using multispectral or spatial features combined with crop-row geometry and Random Forests.
- Deep learning research has shown end-to-end approaches outperform traditional handcrafted feature learning, motivating CNN-based semantic segmentation for crop classification and biomass estimation.
C. Applications using Multispectral Images
Multispectral imagery supports vegetation-specific indices that are more robust under changing illumination and has been applied to plant and weed segmentation. The paper extends this direction with MAV-based real-time dense semantic segmentation and dataset release.
- Multispectral images enable vegetation-specific indices based on radiance ratios that are more robust under varying lighting conditions.
- Prior applications use multispectral imagery to segment sweet pepper plant parts and compute weed maps in maize fields.
- The paper presents a real-time, approximately 2 Hz CNN-based dense semantic segmentation system using multispectral MAV imagery for agricultural robotics.
- The authors release the dataset because publicly available weed datasets are insufficient.
A. Dataset Acquisition
The dataset uses a controlled sugar beet field with varying herbicide levels to produce crop-only, mixed, and weed-only imagery. Multispectral acquisition and NDVI-based processing support automated vegetation ground-truth generation, under stated alignment assumptions.
- A. Dataset Acquisition: Dense manual crop/weed annotation is difficult because plant boundaries require pixel-level tools and domain-specific knowledge.
- A. Dataset Acquisition: A 40 m×40 m field with maximum, medium, and minimum herbicide levels yields crop-only, mixed, and weed-only image regions.
- A. Dataset Acquisition: The Sequoia sensor combines four narrow-band multispectral imagers with a 16 MP RGB camera for image acquisition.
- A. Dataset Acquisition: NDVI = (NIR−Red)/(NIR+Red) distinguishes soil from plant using corresponding NIR and Red images.
- A. Dataset Acquisition: Image alignment uses undistortion, geometric transformation estimation, and cropping, while Green and Red Edge channels remain unaligned.
- A. Dataset Acquisition: The method assumes a camera baseline roughly two orders of magnitude smaller than the ground distance and applies no radiometric or atmospheric corrections.
2) NDVI extraction:
The segmentation pipeline preprocesses multispectral imagery, assigns three pixel classes, and trains SegNet with class balancing and flexible channel input. Its architecture combines convolutional encoding, indexed up-sampling, and channel concatenation.
- 2) NDVI extraction:: NDVI preprocessing applies Gaussian blur, sharpening, Otsu thresholding, and blob detection before labeling background, crop, and weed classes.
- C. Dense Semantic Segmentation Framework: The framework contains 26 convolution layers, five max-pooling stages, five up-sampling stages, and outputs probabilities for three classes.
- C. Dense Semantic Segmentation Framework: SegNet uses a modified VGG16 encoder with corresponding decoder up-sampling layers for dense semantic segmentation.
- C. Dense Semantic Segmentation Framework: Class-frequency weighting penalizes weed misclassification more because weed pixels appear less frequently than background and crop pixels.
- C. Dense Semantic Segmentation Framework: The input/output layer feeds any number of input images into the subsequent concatenation layer.
IV. EXPERIMENTAL RESULTS
The experimental section covers qualitative and quantitative evaluation of the approach and includes a preliminary deployment test on an embedded computer.
- IV. EXPERIMENTAL RESULTS: The experiments include both qualitative and quantitative evaluation of the proposed approach.
- IV. EXPERIMENTAL RESULTS: The study also demonstrates a preliminary performance evaluation of the model on an embedded computer.
- IV. EXPERIMENTAL RESULTS: The experimental setup precedes evaluation and deployment assessment of the model.
A. Experimental Setup
The study uses a MAV-mounted multispectral camera and SegNet-based evaluation to classify sugar beet, weed, and background pixels across models with varying input channels and fine-tuning conditions.
- A 40 m×40 m sugar beet field received varying herbicide levels for automated ground-truth acquisition.
- Only NIR and Red channels were considered because of difficulties registering the other multispectral bands.
- The DJI Mavic carried a downward-facing multispectral camera, while training used a Titan X GPU and inference used an embedded Tegra TX2 module.
- Models were trained for up to 40,000 iterations over 12 hours, although 10,000–20,000 iterations appeared sufficient because later improvements were subtle.
- Six models evaluated background, crop, and weed classes while varying fine-tuning and the number of input channels; all classes exceeded 80% F1-score.
- The two-channel NIR+Red model outperformed separate one-channel NIR and Red models for crop and weed classification, while more input data did not always improve performance.
- The NIR+Red model surpassed the three-channel model for weed classification, possibly because NDVI depends on NIR and Red and channel alignment can be inaccurate near image edges.
- AUC was also evaluated, but its values were not directly comparable with F1-scores because the measures capture different classifier properties.
C. Qualitative Results
Qualitative evaluation compares multispectral inputs, ground truth, and probability outputs, revealing localized crop and weed misclassifications, especially where plants surround one another.
- Seven instances compare the three-channel inputs, NDVI heat maps, annotated ground truth, and color-coded network probability outputs.
- Misclassification appears in crop and weed regions, mostly when crops or weeds are surrounded by each other.
- The qualitative errors imply that the network captures object shapes and textures in addition to low-level edges and intensities.
D. Discussion, Limitations, and Outlook
The pipeline performs well under consistent environmental conditions but shows poor generalization across growth stages and weed types. Broader temporal and multi-scale training data are needed to improve spatio-temporal consistency.
- Limitations: The pipeline was trained and tested under consistent environmental conditions, limiting validation of scale variance and spatio-temporal consistency.The authors identify these as important unresolved validation aspects.
- Limitations: Most crops were classified as weeds in an independent image from a different sugar beet field and month.The image used the same altitude and sensor but represented a different temporal setting.
- Limitations: The model was trained on larger crops and weeds than those in the independent image, and may not cover its weed type.These dataset differences are identified as likely sources of the observed false positives.
- Outlook: More multi-scale, diverse weed data collected over longer periods are required for spatio-temporally consistent detection.The authors also propose smarter data augmentation and incremental dataset construction despite labor-intensive annotation.
E. Inference on an Embedded Platform
Onboard inference addresses delays and wireless-coverage concerns associated with ground-station processing. The Jetson TX2 provides a compact embedded deployment option, although the reported inference is offline because the camera stores images.
- Motivation: 2–5 Hz operation on physically constrained mobile robots remains challenging for trained deep CNN models.Ground-station processing can introduce time delays and depend on wireless communication coverage.
- Embedded Platform: The Jetson TX2 offers onboard processing with 2 GHz hexa-CPU cores, 1.3 GHz 256 GPU cores, and 7.5–14 W power use.The cited power range spans idle to maximum utilization.
- Performance: Titan X processes the tested models 3.6 times faster than TX2, while TX2 supports onboard deployment.The comparison used 300 images and four models.
- Deployment Constraint: The embedded evaluation uses offline inference because the multispectral camera saves images to storage rather than providing an online stream.Online real-time detection is planned with another hyperspectral camera.
V. CONCLUSIONS
The paper demonstrates dense semantic crop–weed classification from MAV-mounted multispectral imagery using an encoder–decoder CNN and automatically generated homogeneous-image labels. Six models achieve acceptable weed-detection performance, while embedded deployment reaches 1.8 Hz and the dataset is released.
- Conclusions: The encoder–decoder network uses MAV-acquired multispectral imagery and herbicide-controlled sugar beet plots for crop–weed classification.The dataset includes crop-only, weed-only, and mixed imagery.
- Conclusions: NDVI and classic image processing automatically generate labels for homogeneous imagery, while mixed imagery requires approximately 30 hours of manual annotation.This labeling strategy addresses the labor involved in creating training data.
- Conclusions: ∼0.8 F1-score is reported for weed detection on the mixed test dataset across models trained with varying input channels and conditions.The models were also evaluated using AUC and qualitative probability-prediction comparisons.
- Conclusions: The model runs the high-level perception task at 1.8 Hz on an embedded platform deployable on a MAV.Performance was compared with a high-performance desktop GPU in inference speed and accuracy.
- Conclusions: The paper releases multispectral weed and crop images with corresponding ground truth to support agricultural robotics research.The release is intended for the robotics community and future work.