Source-linked AI summary
Vision Transformers For Weeds and Crops Classification Of High Resolution UAV Images
Reenul Reedha, Eric Dericquebourg, Raphael Canals, Adel Hafiane
TL;DR
Crop and weed monitoring requires image classification that can support precision agriculture while addressing the limitations of CNN-based approaches. The paper applies self-attention through convolution-free Vision Transformers to UAV plant images and compares them with ResNet and EfficientNet. ViT models achieve the strongest reported performance, including a best F1-Score of 99.8% and high performance with reduced training data.
Problem
UAV-based crop and weed classification is important for monitoring, but CNN approaches face high computation costs and require large labelled datasets.
Method
The study applies convolution-free Vision Transformers using self-attention to classify UAV images of beet, parsley, and spinach plants.
Results
Vision Transformers outperform ResNet and EfficientNet, with the ViT B-16 model achieving a best F1-Score of 99.8%.
Takeaways & Limitations
The results indicate that Vision Transformers are a promising approach for agricultural plant classification, particularly with small training datasets.
Abstract
from arXiv · showhide
Crop and weed monitoring is an important challenge for agriculture and food production nowadays. Thanks to recent advances in data acquisition and computation technologies, agriculture is evolving to a more smart and precision farming to meet with the high yield and high quality crop production. Classification and recognition in Unmanned Aerial Vehicles (UAV) images are important phases for crop monitoring. Advances in deep learning models relying on Convolutional Neural Network (CNN) have achieved high performances in image classification in the agricultural domain. Despite the success of this architecture, CNN still faces many challenges such as high computation cost, the need of large labelled datasets, ... Natural language processing's transformer architecture can be an alternative approach to deal with CNN's limitations. Making use of the self-attention paradigm, Vision Transformer (ViT) models can achieve competitive or better results without applying any convolution operations. In this paper, we adopt the self-attention mechanism via the ViT models for plant classification of weeds and crops: red beet, off-type beet (green leaves), parsley and spinach. Our experiments show that with small set of labelled training data, ViT models perform better compared to state-of-the-art CNN-based models EfficientNet and ResNet, with a top accuracy of 99.8\% achieved by the ViT model.
I. INTRODUCTION
The paper motivates automated UAV-based crop and weed monitoring because weeds threaten production and herbicide use creates environmental and resistance concerns. It introduces convolution-free Vision Transformers as a self-attention approach for classifying high-resolution UAV plant images, including with limited labelled data.
- Weeds compete with crops for water, nutrients, and sunlight, while extensive herbicide use raises pollution and resistance concerns.
- Automated monitoring can reduce labour costs and herbicide use, but distinguishing crops from weeds is difficult because their visual characteristics often overlap.
- UAV images cover large field areas and can be divided into smaller tiles before learning algorithms classify crops and weeds.
- Vision Transformers represent image patches in a shared space and learn relations among them through self-attention rather than convolution operations.
- The study applies Vision Transformers to UAV images of red-leaf beet, green-leaf beet, parsley, and spinach, comparing them with ResNet and EfficientNet using small labelled datasets.
A. Image collection and annotation
The dataset was collected over beet, parsley, and spinach fields in France using a stabilized high-resolution UAV camera and planned overlapping RGB flights. Images were manually annotated into five classes for patch-based classification.
- The study area comprised beet, parsley, and spinach fields in France, with irrigation available on 95% of the plots.
- A Starfury UAV carried a 36-megapixel Sony camera on a three-axis stabilized gimbal for aerial surveying.
- Flights occurred at 30 m over beet and 20 m over parsley and spinach, capturing RGB images at regular intervals.
- Image overlap was at least 70% longitudinally and 50–60% laterally, covering the 4-ha fields and supporting orthophoto generation.
- Images were manually labelled with bounding boxes and organized into five classes, producing 19,265 patches of size 64 x 64 pixels.
B. Image preprocessing
The preprocessing pipeline uses augmentation to expand the agricultural image dataset and improve robustness to varying field conditions. Images are normalized and augmented during training with geometric, crop, colour, and RandAugment transformations.
- Data augmentation addresses insufficient agricultural training data and the high labour cost of supervised labelling.
- Augmentation is intended to cover variation caused by soil, environment, seasons, and climate conditions while improving model generalization.
- After normalization, preprocessing applies random rotation, random resized cropping, horizontal flipping, colour jitter, and RandAugment.
- The transformations are generated on the fly during training with Keras ImageDataGenerator.
C. Self attention for weeds detection
Self-attention lets the model relate image-patch positions globally and emphasize plant regions that distinguish weeds from crops. In ViT-B16 attention maps, the model focuses on weed objects and plant differences.
- Self-attention mechanism: The attention matrix scores which parts of the input sequence each query should focus on.The values are weighted by these scores to capture relations among the input sequence.
- Transformer attention: Multi-head attention runs several self-attention operations in parallel and combines their outputs to attend jointly to different information.The concatenated head outputs are projected using W O.
- Self-attention mechanism: Self-attention models interactions among all image-patch positions by aggregating global information into each sequence component.The mechanism projects inputs into queries, keys, and values, then weights values according to attention scores.
- Attention visualizations: ViT-B16 attention maps show attention concentrated on weed regions and differentiated from beet-plant regions.In the image-patch visualization, dark blue and purple pixels represent attention on weeds, while light blue pixels represent attention on beet plants.
- Self-attention for plant recognition: Attention selectively emphasizes image locations, helping the model learn contrasts between weeds and crops.The paper also states that this mechanism helps the model learn features faster and decreases training cost.
D. Vision Transformers
Vision Transformers apply transformer processing directly to sequences of image patches rather than relying on convolutional operations. The described architecture embeds flattened patches, adds positional information, and processes them through transformer layers.
- Vision Transformer motivation: Vision Transformers adapt transformer models to computer vision by processing image information as sequences of patches.The paper presents ViT as an approach showing that reliance on CNNs is not necessary for image classification.
- Patch representation: The ViT architecture first splits an image into fixed-size patches and flattens each 16×16 patch into a one-dimensional vector.These vectors form the input sequence of image-patch tokens.
- Patch representation: The number of image patches is determined by the image resolution, patch size P, and channel structure, with N representing sequence length.The paper describes N as analogous to the number of words in a sentence.
- Embedding and position: Each flattened patch is linearly projected into a patch embedding and combined with positional embeddings before entering the transformer.The embedding matrix E transforms the unrolled patch vector, while positional embeddings encode patch positions.
III. PERFORMANCE EVALUATION
The evaluation compares ViT-B16 and ViT-B32 with EfficientNet and ResNet implementations under shared training conditions. Models use common optimization settings and pretrained library weights to support comparison.
- Training protocol: All models use the same training parameters for an unbiased and reliable performance comparison.Training uses a 0.0001 initial learning rate, batch size 8, 100 epochs, and early stopping after 10 epochs without improved scores.
A. Cross Validation
The experiments use stratified five-fold cross-validation while varying the number of validation and testing folds. This design preserves class proportions and evaluates performance with progressively smaller training sets.
- Cross-validation design: Stratified K-Fold divides the dataset into folds while maintaining the same proportion of each label in every fold.The data are shuffled and divided into five equally sized class-balanced folders before validation and training splits are formed.
- Cross-validation design: One-fold-out validation uses the remaining four folds for training and a separate testing fold containing unprocessed images.This procedure generates five trained models.
- Cross-validation design: With five folders and k validation folds, the remaining folders are used for training, and increasing k reduces the training data.For n = 5 and k = 2, the procedure produces 10 training models.
- Experiment sets: The three experiments increase testing images and consequently reduce training and validation images.Training sizes decrease from 12,844 images in Experiment 1 to 7,706 in Experiment 2 and 4,535 in Experiment 3.
B. Evaluation metrics
The study evaluates plant-classification performance using ground-truth and predicted labels, precision, recall, F1-Score, cross-validation averages, standard deviation, and cross-entropy loss.
- Ground-truth labels are compared with predicted classes to obtain true positives, false positives, and false negatives.
- The evaluation metrics are precision, recall, and F1-Score, with F1-Score weighting precision and recall while accounting for false positives and false negatives.
- Mean and standard deviation of F1-scores summarize model performance across cross-validation splits.
- Cross-entropy loss measures the difference between true and predicted classes.
IV. RESULTS AND DISCUSSION
The experiments compare ViT and CNN models on a five-class agricultural image dataset using five-fold cross-validation. ViT models achieve the strongest reported classification performance.
- Five-class classification experiments compare ViT-B16 and ViT-B32 with ResNet and EfficientNet using five-fold cross-validation.
- 99.8% is the best F1-Score, achieved by ViT B-16, with a minimum loss of 0.656.
- The CNN models also obtain high scores but perform below the ViT models in these experiments.
A. Influence of the training set size
The experiments test how changing training and testing set sizes affects model performance. ViT B-16 maintains higher F1-Scores than the CNN baselines as available training data decreases.
- Influence of the training set size: The study varies the number of validation folds from 1 to 4 while keeping the testing set at 3211 images to evaluate reduced training sets.
- Influence of the training set size: 99.7% mean F1-Score with a 0.1% standard deviation is obtained by ViT B-16 when k=2.
- Influence of the training set size: 99.63% overall accuracy is achieved by ViT B-16 with 3211 training images, a 75% dataset reduction and only a 0.44% decrease.
- Influence of the training set size: 99.75% F1-Score is retained by ViT B-16 after a 25% training-image reduction, compared with 99.80% previously.
- Influence of the training set size: 99.14% is achieved by ViT B-16 with 13 596 testing images and 4535 training images, despite performance decreases across models.
V. CONCLUSION
The study applies convolution-free vision transformers with self-attention to UAV crop and weed images. ViT models outperform ResNet and EfficientNet, particularly with small training datasets.
- Vision transformers classify UAV images of weeds, beet, parsley, and spinach using self-attention.
- ViT models outperform ResNet and EfficientNet and are described as having high accuracy and low calculation cost.
- ViT B-16 performs especially well with small training datasets where the other models do not achieve similarly high accuracy.
- The convolution-free ViT processes images as sequences of patches through a standard transformer encoder.