Source-linked AI summary
A Lightweight Multimodal Vision-Language Framework for Early-Stage Anatomical Green Fruit Classification in Commercial Orchards
Ranjan Sapkota, William Bu, Chen Chen, Yunjun Xu, Manoj Karkee
TL;DR
Robotic thinning needs reliable recognition of small, camouflaged fruitlet anatomy in cluttered orchards, while conventional high-capacity vision-language models are difficult to deploy on embedded hardware. This paper adapts TinyCLIP for patch-based, text-guided anatomical classification and heatmap localization, achieving strong patch-level recognition and efficient Jetson deployment. The study supports interpretable lightweight perception for automated fruitlet analysis, while comprehensive quantitative localization remains outside its proof-of-concept scope.
Problem
Automated thinning requires fine-grained recognition of calyxes, fruitlets, and peduncles despite occlusion, canopy clutter, and variable illumination.
Method
The framework fine-tunes TinyCLIP on overlapping image patches aligned with class-specific language prompts, then aggregates predictions into whole-image anatomical heatmaps.
Results
The framework achieves strong anatomical classification and rapid TensorRT inference suitable for real-time field deployment on NVIDIA Jetson hardware.
Takeaways & Limitations
Interpretable heatmaps provide presence and spatial-likelihood information relevant to early-season robotic thinning, particularly for peduncle detection.
Takeaways & Limitations
Comprehensive quantitative localization evaluation was beyond the scope of this proof-of-concept study, and heatmaps do not provide instance-level segmentation or exact counting.
Abstract
from arXiv · showhide
Accurate identification of early-stage apple fruitlet anatomical structures, including the calyx, fruitlet body, and peduncle, is essential for robotic thinning, crop-load management, and other precision orchard operations. This study presents a lightweight multimodal vision-language framework that adapts TinyCLIP for fine-grained fruitlet anatomy classification in complex orchard environments. A dataset of 600 high-resolution RGB images collected from Scilate and Scifresh apple orchards was converted into 224 x 224 image patches and annotated for three anatomical classes. Domain-specific language prompts, such as ``a photo of a class,'' were used to guide multimodal alignment between orchard imagery and horticultural structures. A sliding-window inference strategy with a stride of 112 pixels aggregates patch-level predictions into spatial heatmaps, enabling interpretable whole-image localization of fruitlet components relevant to robotic thinning. Patch-level evaluation on an NVIDIA T4 GPU achieved F1-scores of 0.95 for calyx, 0.98 for fruitlet, and 0.85 for peduncle, with a macro-F1 score of 0.93. Deployment-oriented optimization using ONNX and TensorRT enabled efficient inference on NVIDIA Jetson hardware, preserved accuracy under INT8 quantization, and supported model sizes of approximately 127-137 MB with millisecond-level patch inference. These results demonstrate that lightweight vision-language models can provide interpretable and edge-deployable perception for automated fruitlet analysis and future robotic thinning systems. The source code and implementation details are publicly available at https://github.com/WilliamBu1/A-Lightweight-Vision-Language-Model-for-Early-Stage-Fruitlet-Classification-in-Apple-Orchards.
1 INTRODUCTION
Early-stage fruitlet thinning is labor-intensive and difficult to automate because anatomical structures are small, camouflaged, occluded, and exposed to variable illumination. The study adapts TinyCLIP into a lightweight patch-based multimodal framework that produces anatomical classifications and interpretable heatmaps for orchard robotics.
- Motivation: Manual thinning is labor-intensive, physically demanding, and constrained by agricultural labor shortages and repetitive overhead work.These conditions contribute to high labor costs, operational bottlenecks, and worker health risks.
- Perception challenge: Small green fruitlets resemble foliage and are frequently occluded by leaves and branches in cluttered orchard canopies.Fruitlets also vary in developmental stage, complicating fine-grained anatomical recognition.
- Perception challenge: Real-time robotic thinning requires fine-resolution anatomical localization under variable environmental conditions and on embedded hardware.The target structures are the calyx, fruitlet, and peduncle.
- Proposed framework: TinyCLIP jointly aligns overlapping 224 × 224 orchard patches with class-specific prompts to predict calyx, fruitlet, and peduncle presence.Patch predictions are aggregated into class-specific heatmaps for interpretable whole-image localization.
- Reported performance: Macro F1 = 0.93 while remaining deployable on NVIDIA Jetson edge hardware.The study also evaluates FP16 and INT8 TensorRT optimization for real-time orchard perception.
- Reported performance: Multimodal semantic grounding improves performance for challenging structures such as fruitlet peduncles compared with purely visual baselines.This supports lightweight vision-language perception for precision agriculture and orchard robotics.
2 METHODOLOGY
The study adapts TinyCLIP into a patch-based multimodal framework for classifying calyx, fruitlet, and peduncle structures in commercial orchard imagery. Overlapping patch predictions are trained with horticultural prompts and aggregated into interpretable heatmaps for anatomical localization and edge deployment.
- Data Acquisition: 600 high-resolution RGB images from Scifresh and Scilate orchards were annotated for calyx, fruitlet, and peduncle structures.Images were collected under natural daylight, and each instance was bounding-box annotated and cross-verified by a horticulture expert.
- Patch Extraction & Label Generation: 224 × 224 patches were generated with a 112-pixel stride, producing 900 overlapping patches per image for complete coverage.Patch labels were assigned from overlap with class-specific bounding boxes, while background-only crops supplied [0, 0, 0] negatives.
- Multimodal Model: TinyCLIP jointly embeds orchard patches and prompts such as “a photo of a calyx” in a shared space for multi-label anatomical prediction.Cosine similarity produces class-alignment scores, and a sigmoid head converts them into independent class probabilities.
- Fine-Tuning: The model was fine-tuned with binary cross-entropy using AdamW for five epochs without augmentation, with balanced batches for peduncle representation.The final vision-encoder layers and full text encoder were unfrozen during NVIDIA T4 training.
- Heatmap Localization: Class-specific heatmaps average overlapping patch probabilities over their spatial supports to localize anatomical likelihoods across whole orchard images.This aggregation supports interpretable decision-making for robotic thinning and dense sliding-window evaluation.
- Deployment: ONNX conversion, TensorRT optimization, batch processing, and deployment-oriented software produced a lightweight edge-deployable multimodal perception system.The pipeline supports parallel inference on T4 GPUs and Jetson hardware.
3 RESULTS AND DISCUSSION
TinyCLIP shows reliable fine-grained classification of calyx, fruitlet, and peduncle in cluttered orchard imagery, with strong FP16 performance and competitive INT8 robustness. Quantitative and qualitative analyses identify peduncle detection as the main challenge, while quantization preserves overall class separability and deployment utility.
- Patch-Level Multi-Label Classification Performance: Representative 224 × 224 patches closely align ground-truth and predicted labels for calyx, fruitlet, and peduncle in a cluttered early-season orchard scene.The examples illustrate recognition from morphology, texture, petal remnants, and star-shaped calyx geometry despite occlusion and low color contrast.
- Patch-Level Multi-Label Classification Performance: The FP16 engine achieved a macro-averaged F1-score of 0.9115 and overall accuracy of 91.15% across anatomical classes.Precision ranged from 0.964–0.988, while peduncle recall was 0.750.
- Patch-Level Multi-Label Classification Performance: INT8 inference achieved an accuracy of 88.79% and macro F1-score of 0.8881, with reduced performance concentrated mainly in recall.Peduncle recall decreased to 0.678, whereas precision remained consistently high.
- Confusion Matrix Analysis for FP16 and INT8 Deployment: FP16 confusion matrices show strong diagonal dominance, although peduncle predictions are sometimes confused with the negative class.This reflects the peduncle’s small spatial footprint and anatomical subtlety.
- Confusion Matrix Analysis for FP16 and INT8 Deployment: INT8 confusion matrices show modest degradation, particularly increased peduncle-to-negative misclassification, while calyx and fruitlet separability remains strong.The overall classification structure remains consistent with FP16, supporting functional utility under quantization.
- Precision–Recall Curve Analysis: Precision–recall curves remain qualitatively similar after INT8 quantization, but peduncle precision declines more than calyx or fruitlet precision.Peduncles are thin, low-contrast, and spatially small, making them intrinsically difficult for patch-based lightweight classification.
3.2 Whole-Image Localization and Heatmap Analysis
The framework aggregates overlapping patch predictions into interpretable heatmaps that localize fruitlet anatomy across complex orchard scenes. Heatmaps provide approximate spatial guidance for robotic thinning, while peduncle localization remains more difficult and comprehensive quantitative localization evaluation is limited.
- Whole-image heatmap generation: Sliding-window inference converts patch classifications into class-specific heatmaps that highlight calyx, fruitlet, peduncle, and negative regions.The heatmaps provide interpretable spatial likelihood maps across whole orchard images.
- Anatomical localization patterns: Calyx and fruitlet heatmaps form concentrated clusters around fruitlets, while peduncle responses appear as sparser elongated traces aligned with stem locations.These patterns remain visible despite occlusion, variable lighting, and canopy clutter.
- Background suppression: Negative-class heatmaps saturate background regions, supporting suppression of non-fruitlet areas and reducing false activations in foliage-heavy scenes.The negative category is used to distinguish background regions from anatomical targets.
- Robotic utility: Heatmaps provide soft spatial priors for cluster detection, region-of-interest selection, temporal smoothing, multi-view fusion, and potential peduncle cut-point localization.The continuous probability fields can guide downstream robotic perception and manipulation.
- Error modes: Systematic errors include calyxlike leaf-tip false positives, reduced peduncle activation under shadow or leaf venation, and confidence bleeding from sliding-window overlap.Region clustering or stem-line tracking are identified as possible complementary methods for improving precision.
- Scope and limitations: The heatmap representation targets approximate localization rather than precise boundaries or instance-level segmentation.A comprehensive quantitative localization evaluation was beyond the scope of this proof-of-concept study; limited IoU inspection indicated stronger fruitlet alignment than peduncle alignment.
3.3 Ablation Studies and Design Choices
The ablation studies examine text prompts, negative patches, stride size, and quantization to assess their contributions to TinyCLIP performance and deployability.
- Ablation design: Ablation studies evaluated text prompts, negative patch inclusion, stride size, and quantization as key TinyCLIP design components.The study reports these experiments as tests of methodological justification and deployability.
Effect of Text Prompts
The multimodal TinyCLIP model outperformed a visual-only baseline, with the largest gains occurring in peduncle classification.
- Effect of text prompts: +6–10% macro F1-score improvement was achieved by multimodal TinyCLIP over a visual-only classifier across different training runs.The largest gains occurred in peduncle classification, where text embeddings provided semantic anchors for fine-grained recognition.
Negative Patch Inclusion
Including negative patches reduced false positives, particularly in leaf-dense regions where foliage resembles fruitlet anatomy.
- Negative patch inclusion: 20–30% fewer false positives resulted from training with additional negative patches, especially in leaf-dense regions.Without negative samples, calyx or fruitlet labels were often assigned to visually similar leaf textures.
Patch Stride
The study selects a 112-pixel sliding-window stride to balance localization detail and inference speed. Quantization further improves deployment efficiency, with accuracy trade-offs concentrated in peduncle recognition.
- 112 px balanced localization detail and inference speed, whereas 224 px reduced localization robustness and smaller strides slowed inference 2–3×.
- FP16 quantization caused negligible accuracy loss and nearly doubled throughput.
- INT8 quantization reduced peduncle recall but substantially improved speed.
- Ablations identified multimodal alignment, negative sampling, and optimized stride as necessary for robust classification.
3.4 Computational Efficiency and Edge Deployment
TinyCLIP was evaluated for computational efficiency on NVIDIA T4 and Jetson Nano platforms using latency, full-image time, throughput, and memory measures. TensorRT deployment supports fast edge inference, with FP16 favoring balance and INT8 favoring speed despite peduncle-recall losses.
- FP16 processed approximately 96 patches per second on the NVIDIA T4 GPU, while full-image processing with overlapping patches required approximately 6 seconds.
- INT8 increased inference speed to 112 FPS but measurably reduced anatomical recall, particularly for thin peduncles.
- On the Jetson Nano, batches of eight patches completed inference within 5–6 seconds per orchard image.
- TensorRT-optimized TinyCLIP satisfies field-deployment requirements for latency, memory, and throughput.
- FP16 provides the best balance between accuracy and speed, whereas INT8 enables ultra-fast scanning with modest accuracy loss.
3.5 Peduncle Identification for Autonomous Fruit Thinning
Peduncle localization is treated as a central perception requirement because peduncles are the cutting points for scissor-type robotic thinning. The TinyCLIP framework combines domain-specific prompts and patch analysis to identify these thin structures under difficult canopy conditions.
- Peduncles are the actual cutting points for scissor-type end-effectors, making reliable localization essential for safe automated thinning.
- Correctly classified examples show peduncle recognition under occlusion, variable illumination, and canopy clutter.
- Slender peduncles blend with stems and foliage, while illumination, occlusion, and orientation variation create substantial intra-class variability.
- Domain-specific prompts such as “a photo of a peduncle” guide the visual encoder toward semantically meaningful plant structures.
- The pipeline isolates peduncles under foliage occlusion, low contrast, and diverse orientations, providing anatomical cues for downstream robotic manipulation.
- The proposed framework combines semantic grounding, patch-based analysis, and deployment optimization to support practical robotic thinning in real orchard environments.
4 CONCLUSION
The study concludes that TinyCLIP provides lightweight, interpretable, and deployable multimodal perception for fine-grained apple fruitlet anatomy classification and localization. TensorRT deployment supports field use, while heatmaps provide spatial likelihood information without instance-level segmentation or exact counting.
- TinyCLIP effectively addresses fine-grained classification and localization of early-stage apple fruitlet anatomy under complex orchard conditions.
- The optimized TensorRT pipeline achieves rapid inference speeds suitable for real-time field deployment.
- Heatmaps provide robust presence and spatial likelihood information but do not provide instance-level segmentation or exact counting.
- Reliable peduncle detection has direct operational consequences for safe and effective fruit removal.
STATEMENT ON AI WRITING ASSISTANCE
The authors used ChatGPT and Perplexity for grammatical and sentence-level editing, with human review of all AI-generated revisions. ChatGPT-4o also generated a visualization for Figure 11.
- ChatGPT and Perplexity were used to improve grammatical accuracy and sentence structure.
- All AI-generated revisions were reviewed and edited for relevance.
- ChatGPT-4o generated a realistic visualization for Figure 11.