PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Generation Optimization of YOLOv26, YOLOv11, and YOLOv8 for Fine-Grained Small-Object Detection and Instance Segmentation in Complex Orchards

Ranjan Sapkota, Manoj Karkee

arXiv:2608.23636v1cs.CV

TL;DR

Small-object detection and instance segmentation in orchards are challenged by green-on-green appearance, occlusion, and loss of fine spatial detail. The study benchmarks YOLOv8, YOLOv11, and YOLOv26 across model scales and training resolutions, finding that newer generations and larger capacity do not consistently improve performance. Compact models with 960×960 training provide competitive accuracy, while peduncle remains the main difficulty.

  • Problem

    Small orchard structures are difficult to detect and segment because resizing and downsampling reduce spatial detail amid green-on-green appearance and occlusion.

  • Method

    The study compares YOLOv8, YOLOv11, and YOLOv26 across five scales and conventional 640×640 versus small-object-focused 960×960 training configurations.

  • Results

    Neither newer YOLO generations nor larger capacity consistently improved performance; YOLOv11s-960 achieved the highest observed aggregate mask mAP50:95 of 0.402, while YOLOv26s-960 reached 0.397 with 10.37 M parameters and 34.1 GFLOPs.

  • Takeaways & Limitations

    Compact-to-moderate YOLO models with small-object-focused training offer favorable accuracy–efficiency trade-offs for fine-grained orchard perception.

  • Takeaways & Limitations

    The findings are scoped to 600 images from one orchard, acquisition period, phenological stage, and imaging platform, with single-run evaluation and workstation-based computational measurements.

Abstract

from arXiv · show

Small-object detection and instance segmentation remain challenging in orchard environments because of green-on-green similarity, occlusion, and limited pixel representation of fine fruit anatomy. This study presents a cross-generation benchmark of Ultralytics YOLOv8, YOLOv11, and YOLOv26 for detecting and segmenting apple fruitlet, calyx, and peduncle structures for robotic orchard perception. Five model scales (n, s, m, l, and x) were evaluated under conventional 640 x 640 and small-object focused 960 x 960 training configurations, yielding 30 experiments. Increasing model capacity did not consistently improve accuracy. YOLOv11s-960 achieved the highest observed mask mAP@50:95 (0.402) and box mAP@50:95 (0.426), while YOLOv26s-960 achieved comparable values of 0.397 and 0.425 with only 10.37 M parameters and 34.1 GFLOPs. Peduncle remained the most challenging class. Overall, compact-to-moderate YOLO models with small-object-focused training provided favorable accuracy efficiency trade-offs, establishing a practical benchmark for fine-grained agricultural robotics and orchard perception. Github Link: https://github.com/rnjnspkt/Optimizing-and-Comparing-Ultralytics-YOLOv26-YOLOv11-and-YOLOv8-for-Small-Object-Detection-and-Seg

1. Introduction

Fine-grained orchard perception is limited by small, low-contrast anatomical targets and complex canopy conditions. The study addresses these gaps through a controlled cross-generation benchmark that varies YOLO generation, model capacity, and training resolution.

  • Motivation: Small targets lose spatial detail through image resizing and network downsampling, hindering recognition and accurate mask-boundary reconstruction.The problem is especially consequential for thin, narrow, irregular, or partially visible structures.
  • Motivation: Green-on-green coloration, dense foliage, illumination variation, occlusion, and clustering reduce foreground–background separability and challenge instance separation.Dense clusters can contain approximately ten nearby fruitlets whose boundaries overlap or visually merge.
  • Research gap: Calyx, fruitlet body, and peduncle differ in size, geometry, visibility, and functional significance, making anatomical-level perception harder than whole-fruit recognition.The peduncle is particularly difficult because it is narrow, small, and often occluded.
  • Research gap: Existing evidence insufficiently separates improvements from YOLO generation, model scaling, and training configuration, while fine-grained anatomical segmentation remains less studied than whole-fruit detection.The benchmark is designed to test whether higher-resolution training benefits small structures across the full capacity spectrum.
  • Objective and contribution: The study compares YOLOv8, YOLOv11, and YOLOv26 across five scales and conventional 640×640 versus small-object-focused 960×960 training configurations.This produces 15 architecture–scale combinations and 30 model configuration experiments for detection and instance segmentation.

2. Methods

The methodology combines orchard data preparation, anatomical instance annotation, systematic evaluation of three YOLO generations and five scales, and comparison under two training configurations.

  • Data and evaluation: A structured workflow collected and curated orchard images, annotated calyx, fruitlet, and peduncle instances, and organized fixed training, validation, and test subsets.The annotations target three anatomically distinct classes under challenging green-on-green conditions.
  • Model comparison: Each YOLO generation was evaluated at nano, small, medium, large, and extra-large scales under two training configurations.The design yields 15 architecture–scale combinations evaluated with common data organization and experimental procedures.

2.1. Data collection and Preparation

The dataset comprises early-stage apple fruitlet imagery collected in a commercial orchard, manually annotated for three anatomical classes, and partitioned for controlled model development and evaluation.

  • Data collection: Images were acquired on May 6, 2024, in a commercial apple orchard in Prosser, Washington, using an iPhone 14 Pro.Acquisition targeted the early fruit-development stage, when fruitlets are small and have minimal contrast with the canopy.
  • Class definitions: Each image was annotated for calyx, fruitlet main body, and peduncle, which are compact, frequently occluded, and visually similar to surrounding vegetation.These annotations support robotic fruitlet thinning requiring attachment-point and anatomical-boundary information.
  • Annotation: All 600 images were manually labeled by four annotators over approximately two months and cross-verified by a horticulture expert.The annotation process required careful inspection because of small, closely spaced, multi-class objects.
  • Dataset partitioning: The dataset was split into 503 training, 49 validation, and 48 test images.The split was intended to support model learning, hyperparameter tuning, and independent performance evaluation.

2.2. Cross-Generation YOLO Architectures for Small-Object Detection and Instance Segmentation

The benchmark compares YOLOv8, YOLOv11, and YOLOv26 as successive Ultralytics generations, each evaluated across five segmentation scales with shared task outputs and training objectives.

  • Cross-generation design: YOLOv8, YOLOv11, and YOLOv26 were selected for a consistent cross-generation benchmark of fine-grained detection and instance segmentation.The families differ in feature extraction, multi-scale fusion, prediction components, and deployment characteristics.
  • Experimental workflow: The workflow evaluates architecture variants through orchard data preparation, training and validation, and comparison of detection, segmentation, efficiency, and deployment suitability.Both conventional and small-object-focused configurations are included.
  • YOLOv26: YOLOv26 emphasizes simplified inference, end-to-end prediction, and deployment efficiency while retaining hierarchical multi-scale feature extraction.Its one-to-one prediction pathway supports NMS-free end-to-end deployment, while one-to-many prediction provides denser training supervision.
  • YOLOv11: YOLOv11 combines updated feature-processing and fusion components with C3k2, SPPF, and C2PSA mechanisms before instance-mask prediction.Five capacities from YOLOv11n through YOLOv11x were evaluated under the common training framework.
  • YOLOv8: YOLOv8 provides an earlier-generation reference using anchor-free detection, decoupled prediction, multi-scale aggregation, and learned prototype masks.Its five model capacities establish a reference for evaluating later architectural and computational progression.
  • Training design: The study evaluates 15 generation–scale combinations, with each network fine-tuned for calyx, fruitlet, and peduncle outputs under unchanged Ultralytics composite loss formulations.The comparison focuses on publicly available pretrained segmentation architectures rather than modifying internal losses or prediction heads.

2.3. Training Environment and Cross-Generation Model Configuration

The benchmark used a common GPU-enabled environment and pretrained Ultralytics segmentation models across three YOLO generations, five scales, and two input-resolution configurations. Training and augmentation settings were standardized to support controlled comparison.

  • Training environment: All 15 models were trained on the same GPU-enabled computational platform with fixed hardware and software conditions.The environment included an RTX 5000 Ada GPU, Ubuntu 22.04.5, Python 3.10.12, PyTorch 2.1.0+cu121, and Ultralytics.
  • Model configuration: The benchmark covered YOLOv8, YOLOv11, and YOLOv26 segmentation variants at nano, small, medium, large, and extra-large scales.Each model used its corresponding pretrained Ultralytics segmentation checkpoint and the same three-class orchard dataset.
  • Training configurations: Each architecture–scale combination was independently trained at 640×640 and 960×960 input resolutions, producing 30 training runs.Validation monitored optimization, checkpoints were retained, and training curves were generated for each run.
  • Training configurations: The small-object-focused configuration used scale=0.5, mosaic=1.0, and close_mosaic=10, while conventional settings followed the standard Ultralytics workflow.These settings were paired with the shared augmentation and optimization framework.
  • Reproducibility controls: Common seeds, validation-based checkpoint selection, AMP-enabled GPU training, and standardized augmentation supported reproducible comparison across generations and capacities.The recorded framework used a fixed random seed of 42, disabled image caching, and four data-loading workers.

2.4. Implementation Workflow for Cross-Generation YOLO Benchmarking

The workflow controlled dataset partitions, pretrained initialization, optimization, validation, and evaluation while comparing YOLO generations, model scales, and training resolutions. Accuracy and computational costs were analyzed jointly to identify deployment-relevant trade-offs.

  • Experimental design: The experiment compared YOLOv8, YOLOv11, and YOLOv26 across nano, small, medium, large, and extra-large scales using fixed orchard data partitions.The dataset contained instance-level annotations for calyx, fruitlet, and peduncle.
  • Optimization workflow: The common training schedule used up to 200 epochs, batch size 8, pretrained initialization, automatic optimizer selection, AMP-enabled GPU execution, and checkpoint saving.Early stopping used a patience of 100 epochs, and the best validation checkpoint was retained.
  • Evaluation workflow: Performance evaluation combined precision, recall, F1-score, box and mask mAP, model complexity, processing times, convergence, and wall-clock training duration.This protocol enabled direct comparison of predictive and computational behavior.
  • Comparative analysis: The analysis examined architectural progression, capacity scaling, and conventional versus small-object-focused resolution training.Predictive gains were interpreted jointly with computational and training costs.

2.5. Performance Evaluation Across YOLO Generations and Model Scales

The evaluation framework compared 30 YOLO model–resolution experiments using accuracy, class-wise behavior, computational complexity, processing times, and convergence measures. It separated localization from mask quality and included a single-run scope limitation.

  • Benchmark scope: The benchmark evaluated 15 YOLOv8, YOLOv11, and YOLOv26 segmentation architectures at 640×640 and 960×960 resolution, yielding 30 experiments.The model scales were nano, small, medium, large, and extra-large.
  • Evaluation protocol: The test results used an independent held-out 48-image set, while the 49-image validation set supported convergence monitoring, early stopping, and checkpoint selection.The test set contained calyx, fruitlet, and peduncle annotations.
  • Limitation: Each of the 30 configurations was evaluated using a single training run, so the results describe the standardized setting rather than across-run variability.Repeated multi-seed experiments were identified as useful for quantifying performance variance and statistical robustness.
  • Detection metrics: Precision and recall captured positive-prediction reliability and ground-truth recovery, while F1 summarized their balance for robotic perception.False positives may arise from visually similar vegetation, whereas false negatives are consequential for small or occluded structures.
  • Overlap metrics: IoU measured overlap between predicted and reference boxes or masks, with object-level detection statistics kept separate from pixel-level mask overlap.This separation avoided conflating localization performance with segmentation-boundary quality.
  • Aggregate accuracy: mAP50 used a 0.50 IoU threshold, whereas mAP50:95 averaged AP across ten thresholds from 0.50 to 0.95 in 0.05 increments.The study reported box and mask versions of both measures, along with class-wise AP and recall.
  • Efficiency metrics: Complexity and deployment behavior were assessed using layers, trainable parameters, GFLOPs, checkpoint size, preprocessing, inference, postprocessing, and training times.These measures supported comparisons among model capacities and YOLO generations.
  • Scope boundary: SAHI was treated as a separate deployment strategy and did not change the evaluated YOLO models’ learned parameters, architecture, GFLOPs, or checkpoint size.The principal results were interpreted in terms of YOLO generation, scale, resolution, accuracy, convergence, and computation.

3. Results and Discussion

Qualitative outputs suggest that higher-resolution training can preserve spatial information for small fruitlets, but improvements were not uniform across structures or model generations. YOLOv26 at 640×640 recovered one especially small peduncle missed by the other illustrated configurations.

  • Qualitative comparison: Higher-resolution training qualitatively improved the representation of fruitlets that were less distinguishable at lower resolution.The passage attributes this pattern to preservation of additional spatial information for small targets.
  • Qualitative comparison: The resolution-related improvement was not uniform across every anatomical structure or YOLO generation.The qualitative comparison included YOLOv26, YOLOv11, and YOLOv8 at both 640×640 and small-object-focused resolution.
  • Peduncle example: At 640×640, YOLOv26 successfully detected and segmented one very small peduncle that the other illustrated model–resolution configurations failed to identify.The example involved a narrow, elongated, weakly contrasted structure with limited pixel footprint and fewer training annotations.

3.1. Cross-Generation Detection and Instance Segmentation Performance

Across YOLOv26 configurations, higher input resolution and larger model capacity produced metric-specific, non-monotonic effects rather than universally higher accuracy. YOLOv26s-960 offered the strongest balance of segmentation quality and computational efficiency.

  • Performance across configurations: YOLOv26x-640 achieved the highest recall (R = 0.765), whereas YOLOv26x-960 achieved the highest precision (P = 0.689).The extra-large model therefore shifted from a recall-dominant operating point at 640 pixels to a precision-dominant point at 960 pixels.
  • Performance across configurations: YOLOv26l-640 achieved the highest box mAP50 of 0.696, while YOLOv26l-960 achieved the highest box mAP50:95 of 0.428.The higher-resolution large model improved stricter localization despite lower mAP50 than its 640-pixel counterpart.
  • Resolution effects: Higher resolution benefited YOLOv26s most consistently, increasing all six aggregate metrics, including recall by 0.105 and mask mAP50:95 by 0.056.This pattern supports simultaneous gains in object recovery, localization, and pixel-level segmentation for the small model.
  • Scaling effects: YOLOv26 model scaling did not consistently improve accuracy: at 960 pixels, different scales led box mAP50:95, mask mAP50:95, and precision.At 640 pixels, box mAP50 rose from 0.661 for YOLOv26n to 0.696 for YOLOv26l before declining to 0.669 for YOLOv26x.
  • Efficiency: YOLOv26x-960 required 12.8 ms inference latency and more than four times the inference time of YOLOv26s-960 without exceeding its mask mAP50 or mask mAP50:95.The comparison underscores the computational cost of extra-large capacity when it does not produce higher segmentation accuracy.
  • Performance across configurations: YOLOv26s-960 achieved the highest mask mAP50 (0.686) and mask mAP50:95 (0.397) among the ten YOLOv26 configurations.It also achieved box mAP50 = 0.687 and box mAP50:95 = 0.425.
  • Efficiency: YOLOv26s-960 represented the most balanced configuration because it combined near-leading box localization with the highest mask accuracy and lower complexity than larger variants.The study therefore favors compact-to-moderate models with greater spatial resolution over simply increasing network capacity.

3.2. Class-Wise Performance for Calyx, Fruitlet, and Peduncle

Across YOLOv26, YOLOv11, and YOLOv8, fruitlet was comparatively easiest, calyx intermediate, and peduncle most difficult. Increasing resolution most consistently benefited small anatomical structures, especially peduncles, while larger models showed mixed or diminishing returns.

  • Cross-generation pattern: Peduncle remained the most challenging class across YOLOv26, YOLOv11, and YOLOv8, whereas fruitlet was comparatively easiest and calyx intermediate.This hierarchy was attributed to peduncle’s thin geometry, limited pixel coverage, partial visibility, low contrast, and similarity to neighboring structures.
  • YOLOv26: YOLOv26s increased peduncle box recall from 0.365 to 0.635 and mask mAP50:95 from 0.163 to 0.290 when training resolution rose from 640×640 to 960×960.YOLOv26m likewise increased peduncle mask recall from 0.346 to 0.673.
  • Resolution and scale interaction: Higher resolution produced mixed returns for large YOLOv26 configurations, including a YOLOv26x precision increase accompanied by recall decreasing from 0.654 to 0.362.The strongest and most consistent benefits occurred in nano, small, and medium configurations rather than the largest models.
  • YOLOv26: 0.077 was the mean YOLOv26 improvement in peduncle mask mAP50 with 960×960 training, compared with essentially no mean fruitlet change (−0.001).Mean peduncle mask precision and recall also increased by 0.106 and 0.069, respectively, while strict mask mAP50:95 improved by 0.044.
  • YOLOv11: YOLOv11s increased peduncle box mAP50 from 0.324 to 0.521 and mask mAP50:95 from 0.164 to 0.297 with higher-resolution training.Across YOLOv11 scales, peduncle had the largest mean gains in box and mask AP metrics, while fruitlet gains remained modest.

3.3. Quantitative Evaluation of Detection and Segmentation Performance

Paired 640×640 and 960×960 experiments show that resolution effects depend on YOLO generation and model scale. Higher resolution most consistently improves strict mAP50:95 measures, with the strongest and most consistent AP gains concentrated in small models.

  • Paired evaluation: The paired comparison defined resolution-induced change as ∆M = M960 − M640 and also evaluated precision, recall, AP, and post hoc F1.Training duration and inference latency were included to separate accuracy changes from computational cost.
  • Strict spatial accuracy: YOLOv8 and YOLOv11 improved box and mask mAP50:95 at every scale, while YOLOv26 improved box mAP50:95 at every scale and mask mAP50:95 at four scales.YOLOv26 mask mAP50:95 decreased marginally only for the extra-large model (∆= −0.003).
  • Small-scale response: YOLOv11s improved box mAP50 by +0.071, box mAP50:95 by +0.062, mask mAP50 by +0.071, and mask mAP50:95 by +0.068.This was the strongest individual response among the reported small-scale configurations.
  • Small-scale response: YOLOv26s improved precision by +0.053, recall by +0.105, F1 by +0.075, box mAP50 by +0.060, and mask mAP50 by +0.079.Its mask mAP50:95 also increased by +0.056.
  • Diminishing returns: Large and extra-large models showed diminishing returns, including YOLOv26x’s precision increase of +0.111 alongside recall decrease of −0.134 and essentially unchanged F1.This pattern rejects a universal advantage for either higher resolution or greater model size.
  • Scale interaction: Across generations, small models had the largest mean gains in box mAP50 (+0.052), box mAP50:95 (+0.046), mask mAP50 (+0.064), mask mAP50:95 (+0.053), F1 (+0.049), and recall (+0.064).Medium and large models instead showed mean F1 changes of −0.013 and −0.002.
  • Generation interaction: YOLOv11 had the largest mean gains in box mAP50 (+0.022), box mAP50:95 (+0.034), and mask mAP50:95 (+0.045), while YOLOv26 had the largest average precision (+0.032) and F1 (+0.020) improvements.YOLOv26’s larger variants exhibited substantial saturation.

3.4. Model Complexity, Processing Speed, and Training Efficiency

Model complexity, latency, and training duration scale differently across YOLO generations and sizes. The benchmark shows that greater capacity can substantially increase computation without reliably improving predictive performance, while inference increasingly dominates runtime for larger models.

  • Model complexity: YOLOv8n to YOLOv8x increases parameters from 3.259 M to 71.724 M and GFLOPs from 11.3 to 328.0, approximately 22-fold and 29-fold increases.Scaling patterns differ among generations.
  • Model complexity: YOLOv26n-seg is the smallest configuration at 2.689 M parameters and 9.0 GFLOPs, whereas YOLOv8x-seg is the largest at 71.724 M parameters and 328.0 GFLOPs.Lower complexity implies lower weight-storage and forward-pass requirements.
  • Cross-generation scaling: At extra-large scale, YOLOv26x uses 62.730 M parameters and 313.0 GFLOPs, below YOLOv8x’s 71.724 M parameters and 328.0 GFLOPs.Architectural recency therefore does not necessarily imply greater computational complexity.
  • Processing speed: Inference ranges from 1.0 ms for YOLOv11n at 640 pixels to 12.9 ms for YOLOv11x at 960 pixels, while preprocessing ranges from 0.1 to 1.9 ms.Forward-pass optimization becomes increasingly important for larger models.
  • Processing speed: YOLOv26n, YOLOv26s, and YOLOv26m require approximately 2.3, 2.5, and 3.5 ms total at 640 pixels, corresponding to isolated theoretical throughputs of approximately 435, 400, and 286 images s−1.These are not complete robotic-system frame rates because acquisition and transfer costs are excluded.
  • Resolution cost: Increasing YOLOv26 resolution from 640 to 960 pixels raises inference from 3.0 to 5.9 ms for m, 3.9 to 7.7 ms for l, and 6.3 to 12.8 ms for x.The corresponding total processing times are 6.9, 8.5, and 13.7 ms.
  • Accuracy–cost trade-off: YOLOv26m to YOLOv26x increases parameters by 166.8%, GFLOPs by 158.3%, and 960-pixel latency by 116.9%, while box mAP50 decreases slightly.This demonstrates a non-monotonic complexity–performance relationship.
  • Training efficiency: YOLOv26x-960 requires 2.124 h and completes all 200 epochs, whereas early-stopped runs can have lower wall-clock durations without representing fixed-workload speed.The lowest duration, 0.332 h for YOLOv8l-640, ended at epoch 131.

4. Conclusion

The study benchmarks YOLOv8, YOLOv11, and YOLOv26 across five scales and two resolutions for fine-grained orchard detection and segmentation. Results favor compact-to-moderate models with 960×960 training, while dataset scope and single-run evaluation limit broader conclusions.

  • Conclusion: The benchmark evaluates 15 architectures across five scales and two training configurations, producing 30 model experiments on an early-stage apple fruitlet dataset.The task covers fine-grained detection and instance segmentation in green-on-green orchard conditions.
  • Conclusion: Neither newer YOLO generations nor greater network capacity consistently produced higher predictive performance; outcomes reflected generation, scale, training configuration, and target morphology.This is the paper’s central cross-generation conclusion.
  • Conclusion: YOLOv11s-960 achieved the highest observed aggregate mask mAP50:95 of 0.402, while YOLOv26s-960 reached 0.397 with 10.37 M parameters and 34.1 GFLOPs.YOLOv26x-960 required 62.73 M parameters, 313 GFLOPs, and 12.8 ms without exceeding smaller configurations’ aggregate segmentation performance.
  • Conclusion: Fruitlet generally performed best, whereas peduncle remained the principal bottleneck because of its small pixel footprint, elongated geometry, occlusion, weak contrast, and limited representation.The 960×960 configuration generally produced stronger stringent box and mask mAP50:95 values.
  • Scope and limitations: The dataset contains 600 images from one orchard, date, phenological stage, and imaging platform, with class imbalance and single-run experiments limiting generalization and statistical certainty.Computational performance was measured on a GPU workstation rather than onboard robotic hardware.
Loading 2608.23636v1…