Source-linked AI summary
How Merge-Tolerant Are Vision Transformers for Wheat Phenotyping?
Simon Ravé, Pejman Rasti, David Rousseau
TL;DR
Wheat phenotyping needs repeated, efficient measurements, but high-resolution ViTs incur costly attention and must preserve task-specific biological evidence. This paper benchmarks inference-only ToMe and MPM across classification, detection, and segmentation, finding that classification is most merge-tolerant while dense tasks and deployment overhead constrain practical gains.
Problem
Wheat phenotyping lacks evidence on when token sequences in already trained ViTs can be compressed without losing task-relevant biological signal under deployment constraints.
Method
The benchmark inserts ToMe or MPM only at inference into frozen plain ViT-based models and measures task quality, throughput, token count, latency, and peak GPU memory across three tasks and Raspberry Pi 5.
Results
Classification is highly merge-tolerant, while detection provides limited practical gains and segmentation degrades under aggressive merging, dense reconstruction, and runtime overhead.
Takeaways & Limitations
Token merging should be selected per task and runtime, with measured deployment behavior profiled on the target path rather than inferred from token count.
Takeaways & Limitations
The benchmark uses plain ViT classifiers, Segmenter, and YOLOS, compares two training-free mergers, and does not establish a biological mechanism for observed associations.
Abstract
from arXiv · showhide
Vision-based wheat phenotyping requires repeated measurements under deployment constraints, from growth-stage recognition to wheat-head counting and organ segmentation. Plain Vision Transformers (ViTs) provide a common architecture for these tasks, but quadratic attention limits high-throughput and edge inference. Training-free token merging is attractive because it can be inserted into trained models without retraining. We provide a systematic benchmark of ToMe and Mutual Pair Merging across growth-stage classification, wheat-head detection, and wheat-organ segmentation, measuring task quality, throughput, token count, and peak GPU memory, with additional Raspberry Pi 5 measurements. The benchmark reveals a clear hierarchy: classification is highly merge-tolerant, while detection and segmentation are constrained by repeated instances, thin organs, dense boundaries, reconstruction, and runtime overhead. Optimized attention backends can erase apparent speedups, so deployment value must be profiled on the target runtime rather than inferred from token count.
1 Introduction
Wheat phenotyping demands efficient repeated measurements across tasks with different spatial evidence, while ViT attention becomes costly at high resolution. This benchmark tests whether inference-only token merging preserves task quality and deployment utility across classification, detection, and segmentation.
- Motivation: Field phenotyping spans growth-stage recognition, wheat-head counting, and pixel-level organ separation under varying device budgets.Deployment may involve fixed stations, UAVs, robots, and low-cost edge devices.
- Motivation: Wheat’s visual redundancy can enable compression, but merging may remove small instances and boundary information.The same repeated structures that offer compression opportunities can also carry phenotype-relevant evidence.
- Approach: The benchmark applies ToMe and MPM at inference across multiple ViT backbone sizes, reporting task quality, token count, latency, throughput, and GPU or Raspberry Pi 5 behavior.Models are trained and frozen before merging is inserted.
- Research question: Training-free merging aggregates similar tokens in frozen transformers, but its benefit may differ between global classification and spatially precise plant tasks.ToMe targets throughput without retraining, while MPM emphasizes end-to-end latency and dense-feature reconstruction.
- Headline findings: Classification tolerates substantial token reduction, whereas detection offers limited practical gains and segmentation degrades under aggressive merging and dense reconstruction.These outcomes motivate task- and runtime-specific deployment guidance rather than a universal merging setting.
2 Related Work
Related work frames token merging as aggregation within ViT sequences and positions wheat phenotyping as a set of tasks with distinct spatial evidence requirements. These differences motivate evaluating merge behavior across global, instance-level, and dense predictions.
- Vision Transformers: ViTs process images as patch-token sequences, and Segmenter and YOLOS extend this interface to segmentation and detection.The shared sequence interface allows token merging to be studied across task types.
- Token Merging: Token merging aggregates similar tokens rather than deleting them; ToMe uses paired tokens, whereas MPM uses mutual nearest-neighbor pairs and dense reconstruction mappings.Prior comparisons indicate that accuracy–speed behavior cannot be inferred from token count alone.
- Wheat Phenotyping: Wheat benchmarks differ from global canopy recognition to repeated-head detection and dense separation of leaves, stems, and spikes.These tasks expose distinct requirements for global structure, small repeated objects, and boundary-sensitive organs.
3 Method
The method evaluates inference-only token merging on frozen plain ViT-based models across three wheat tasks, using common quality and deployment measurements. It explicitly accounts for task-specific sequence structure, reconstruction, tiling, padding, and runtime overhead.
- Benchmark Design: Each task is trained without merging, then evaluated with frozen weights after inserting a training-free merger at inference.This isolates changes caused by the modified token sequence and merge-operator overhead.
- Benchmark Design: Merge tolerance combines task-performance drop relative to the unmerged checkpoint with measured speedup under matched execution conditions.A setting is deployment-useful only when it mostly preserves the task metric and achieves s > 1.
- Tasks and Datasets: The benchmark covers global growth-stage classification, instance-level wheat-head detection, and dense organ segmentation with task-specific dataset splits and metrics.WGSP and GWHD use training, validation, and test splits, while GWFSS uses training and evaluation splits.
- Tasks and Datasets: Input resolutions produce 196, 1024, and 1600 image tokens for classification, segmentation, and detection, respectively; YOLOS also uses 100 detection tokens and nine tiled passes.These resolution and tiling differences confound direct cross-task throughput comparisons.
- Merging Methods: ToMe removes a fixed number r of image tokens per selected block, while MPM averages input-dependent mutual nearest-neighbor pairs controlled by a discrete layer schedule L.MPM reports effective final tokens and padding because its retained count varies by input.
- Task Integration: Segmentation reconstructs the dense grid before the unchanged decoder, while YOLOS protects its 100 detection tokens and merges only image patches.Reconstruction duplicates merged features at source positions but does not restore removed spatial detail.
- Evaluation and Profiling: Latency includes merging, aggregation, required reconstruction, and the unchanged head, measured on GPU and Raspberry Pi 5 paths with peak CUDA memory reported for GPU runs.The profiling scope differs by task, and adaptive merging may require padding for batched execution.
4 Results
Across tasks, classification is most merge-tolerant, segmentation occupies an intermediate regime, and detection is most constrained by localization and pipeline costs. Merge behavior also depends on task structure, reconstruction, padding, and the deployed attention backend.
- Matched token comparisons: At approximately 101 tokens, ToMe and MPM classification differ by 0.02 Top-1 points, with MPM faster; at approximately 450 tokens, segmentation differs by 0.06 mIoU points, again with higher MPM throughput.At approximately 1,310 tokens, detection methods differ by 4.42 mAP points, showing that equal token counts do not ensure equal information retention.
- GPU operating points: 1.59× and 1.47× speedups are achieved for classification with Top-1 losses of 0.13 and 0.26 points, ending with 11 and 79 tokens.These are measured operating points rather than universal loss tolerances.
- GPU operating points: Segmentation reaches 1.33× and 1.16× speedups at costs of 0.82 and 0.36 mIoU points, while merging reduces all class IoUs and boundary F1.Peak memory is higher for segmentation because merge maps are stored for dense reconstruction.
- GPU operating points: Detection mAP increases numerically by 1.27 and 1.21 points, but speedups are only 1.11× and 1.02×; count MAE rises and count R2 falls.High-resolution images are divided into nine tiles, limiting end-to-end throughput gains.
- Edge deployment: On Raspberry Pi 5, classification remains most merge-tolerant while detection and segmentation degrade faster; aggressive settings reach about 2× CPU speedup under reported thermal throttling.Compact detectors have low absolute mAP in this edge setting.
- Backend dependence: With FlashAttention-2, classification ToMe and aggressive MPM reach only 0.92× and 0.35× baseline throughput, while tiled YOLOS remains slower at 0.64× and 0.85×.Merge construction, reconstruction, padding, and task-specific costs can dominate savings from shorter sequences.
- Merge locality: Classification merges are more spatially dispersed, whereas detection pairs are local: classification ToMe has median distance 4.29, while both detection methods have median distance 1.00.The distributions align with canopy-level classification versus instance- and boundary-sensitive dense tasks.
- Adaptive padding: MPM adaptive padding averages 20.3, 70.7, and 62.9 padded tokens per classification, segmentation, and detection image, making mean retained count incomplete for efficiency assessment.The 95th-percentile padding costs are 25.5, 91.6, and 86.7 tokens, respectively.
5 Discussion and Limitations
Token merging is useful only when task requirements and runtime conditions align: classification tolerates substantial reduction, whereas detection and segmentation expose losses and limited deployment gains. The benchmark therefore supports task- and runtime-specific selection rather than default merging.
- Plant-Specific Interpretation: Classification tolerates substantial token reduction, while repeated heads, thin organs, and boundaries constrain detection and segmentation.Secondary metrics can reveal degradation even when AP50 or mIoU changes modestly.
- Deployment Guidance: 1.59× and 1.47× speedups with losses below 0.3 Top-1 points are measured classification operating points, not universal tolerances.These correspond to ToMe r = 16 without proportional attention and MPM L = {0, 2, 4, 6, 8}.
- Deployment Guidance: 1.33× and 1.16× throughput gains in segmentation reduce mIoU, every class IoU, and boundary F1.Region and boundary losses must be assessed against application-specific tolerance.
- Deployment Guidance: Detection provides limited practical gains: mild ToMe approximately preserves mAP but increases count MAE, while MPM is nearly throughput-neutral.With FlashAttention-2, both tested detection mergers are slower than Full because the tiled pipeline may dominate runtime.
- Deployment Guidance: Deployment usefulness must be evaluated on the complete target path, since shorter sequences help only when they improve measured performance under the intended settings.The guidance applies across GPU and Raspberry Pi 5 environments, where CPU gains were more consistent but throttling was reported.
- Scope and Limitations: The benchmark isolates plain ViT classifiers, Segmenter, and YOLOS and does not establish that its conclusions transfer to hierarchical backbones, modern detectors, or foundation-model decoders.It compares two training-free mergers rather than task-trained reduction; joint training may improve robustness but removes fixed-checkpoint control.
- Scope and Limitations: Energy cost was not profiled, and the benchmark is limited to wheat phenotyping; other crops and plant organs may have different self-similarity and spatial requirements.These boundaries are especially relevant when assessing edge deployment or generalization beyond wheat.