PaperlayerView exact revision on arXiv

Source-linked AI summary

ReLEAP: Upcycling Vision Datasets through Cross-Task Unification and Semi-Supervised Annotation

Julia Simon, Daniel Steininger, Andreas Trondl

arXiv:2610.32289v1cs.CVcs.LG

TL;DR

Fine-grained vision tasks require extensive annotation, while differences among existing datasets make them difficult to combine and reuse. ReLEAP unifies datasets and annotations across tasks, then evaluates strategies for generating higher-level annotations. The experiments report improved mask quality from source aggregation and comparable performance from lightweight domain-specific models given the same prior information.

  • Problem

    Fine-grained vision models need large annotated datasets, but heterogeneous formats, label definitions, and learning tasks hinder reuse and integration.

    Source: §1 · pp. 1–2

  • Method

    ReLEAP abstracts data sources, maps labels through a shared hierarchical taxonomy, and integrates datasets and annotation recycling across tasks.

    Source: §3 · p. 3, Taxonomy-Based Label Mapping · p. 4, §5 · p. 14

  • Results

    Aggregating sources consistently improves class-agnostic mask quality; separating mask generation from classification further benefits fine-grained assignment, and lightweight domain-specific models perform comparably to larger foundation models given the same prior information.

    Source: §5 · p. 14

  • Takeaways & Limitations

    Integrating and recycling annotations across 15 agricultural datasets expands reusable training data and supports combining data across tasks and domains with minimal manual effort.

    Source: §5 · p. 14

Abstract

from arXiv · show

Developing robust vision models requires large amounts of annotated data, especially for tasks involving fine-grained class distinctions under variable environmental conditions. Even in domains with numerous existing datasets, differences in annotation formats, label definitions and targeted learning tasks hinder data reuse and integration. To address this challenge, we introduce ReLEAP, a scalable Python framework for unifying heterogeneous image datasets through data-source abstraction, systematic reuse of annotations across learning tasks, and mapping of labels to a shared hierarchical taxonomy. We demonstrate its applicability in precision agriculture by integrating fifteen diverse datasets covering a broad range of plant species and annotation types. Beyond unification, ReLEAP enables systematic recycling of existing data and provides a robust foundation for learning-based annotation upcycling. Leveraging this foundation, we conduct extensive cross-dataset and fine-tuning experiments to investigate how domain-specific models trained on unified data and foundation models can be used to automatically generate higher-level annotations under unsupervised, weakly supervised and semi-supervised settings, and how dataset characteristics influence their effectiveness. Our results show that unified data access and annotation upcycling substantially expand the pool of reusable training data and annotations while improving data transparency, enabling more sustainable and resource-efficient use of existing datasets. We release the framework and trained models for academic use to support accessible and collaborative research, particularly in data-scarce and heterogeneous domains.

1 Introduction

Fine-grained vision tasks need substantial expert annotation, yet existing datasets are difficult to combine because their formats, labels, and learning tasks differ. ReLEAP addresses this gap by unifying datasets and reusing annotations across tasks.

1 Introduction

  • Fine-grained distinctions across variable environmental conditions make annotation time-consuming and resource-intensive, especially for rare plant species.Precision agriculture requires distinguishing visually similar plants across growth stages, weather conditions, and seasons.Source: §1 · pp. 1–2
  • Existing datasets rarely capture all the variability a new application needs, while heterogeneous protocols, formats, labels, and tasks complicate integration.Fragmentation can leave too little annotated data for particular task and label combinations and add suitability-assessment and conversion effort.Source: §1 · p. 2
  • The task hierarchy distinguishes direct recycling to a lower-level task from upcycling annotations to higher-level tasks using specialized models.Only task connections shown by black lines are suitable for direct recycling.Source: Figure 1 · p. 2
  • ReLEAP is a framework for unifying heterogeneous image datasets and reusing annotations across learning tasks and label spaces, demonstrated in precision agriculture.Source: §1 · p. 2

1 Available for academic use at https://github.com/releap-vision/releap

The paper evaluates annotation upcycling and automatic labeling across supervision settings, model choices, dataset characteristics, and domain shifts. It also integrates diverse datasets and releases the framework and trained models for academic use.

1 Available for academic use at https://github.com/releap-vision/releap

  • Cross-dataset and fine-tuning experiments evaluate annotation upcycling and automatic labeling under varying supervision, model choices, dataset characteristics, and domain shifts.Source: §1 · p. 3
  • An extensive survey supports integration of 15 diverse datasets, and the framework and trained models are released for academic use.Source: §1 · p. 3
  • The release is intended to support accessible and sustainable vision research.Source: §1 · p. 3

2 Related Work

Prior work addresses dataset harmonization, annotation reuse, and semi-supervised learning, but these efforts are commonly bounded by individual tasks or datasets. ReLEAP combines these aspects in a pipeline for cross-task unification and annotation upcycling.

2 Related Work

  • Prior dataset-integration work adapts architectures to annotation inconsistencies or harmonizes labels using hand-crafted or learned ontologies.ReLEAP complements ontology work by integrating taxonomies for cross-task label harmonization and annotation upcycling.Source: §2 · p. 3
  • Annotation-reuse methods include label propagation, Label-Aligned Transfer, and taxonomy-guided pseudo-labeling, but are typically limited to single-task adaptation.The cited works are Papadopoulos et al., Kennerley et al. [61], and Liao et al., respectively.Source: §2 · p. 3
  • Semi-supervised methods commonly use teacher-student frameworks and adaptive pseudo-labeling, but operate within single datasets rather than scaling annotation across datasets.Source: §2 · p. 3
  • Few approaches systematically combine dataset harmonization, semi-supervised annotation, and cross-dataset generalization in one pipeline for cross-task unification and annotation upcycling.Source: §2 · p. 3

3 Data Unification and Recycling

ReLEAP unifies heterogeneous datasets through source abstraction, taxonomy-based label mapping, and configurable aggregation, while deterministic conversions recycle annotations across tasks. Its 15-dataset agriculture collection expands usable data through recycling and learning-based upcycling.

3.1 Framework Design

  • ReLEAP abstracts data sources behind lightweight dataset interfaces, so core functions operate independently of each dataset's structure.Shared modules support hosting platforms and standard annotation-format conversions, reducing dataset-specific code.Source: §3 · p. 3, §3.1 · p. 4
  • A shared hierarchical taxonomy maps source and target labels to synonymic nodes and parent classes, enabling consistent alignment and aggregation at arbitrary detail.The agriculture taxonomy draws on botanical ontologies and maps plant families through individual species.Source: Taxonomy-Based Label Mapping · p. 4, Figure 2 · p. 4
  • Deterministic conversions, such as deriving boxes from instance masks or classification crops from detection data, extend datasets to additional tasks.These conversions are integrated into the aggregation pipeline to maximize usable data for an application.Source: Annotation Recycling · p. 5

3.2 Data Integration and Analysis

  • The 15 integrated agriculture datasets span major crops, weeds, and niche species, with varied viewpoints, environments, annotation schemes, tasks, and image sizes.Table 1 summarizes dataset counts, task availability, species, and image dimensions.Source: §3.2 · p. 5, Table 1 · p. 6
  • Usable samples increase by factors of 1.3 for detection and 27 for classification through rule-based recycling; classification label classes increase by a factor of 4.2.Additional gains come from upcycling to higher-level tasks.Source: §3.2 · pp. 5–6

4 Data Upcycling

This section evaluates instance-segmentation upcycling from unannotated images and with lower-level prior information. It compares cross-dataset methods under varying supervision and examines few-shot adaptation to new domains.

4 Data Upcycling

  • Upcycling generates higher-level annotations unavailable through deterministic conversion, using instance segmentation as the test case and minimizing prior information and manual post-processing.Methods range from multi-class or class-agnostic models trained on unified instance-segmentation data to SAM 3 prompts and separate classification using broader ReLEAP classification data.Source: §4 · p. 6, Unsupervised Methods · p. 7, Weakly Supervised and Semi-Supervised… · p. 8

4.1 Evaluation Protocol

  • Across five evaluation rounds, each of five datasets is held out entirely from training, and generated masks are compared with manual annotations; single-dataset transfer models serve as baseline.The reported metrics are mAP50-95, precision, and recall. Taxonomy matching accommodates coarser target annotations by counting subclasses of an originally annotated class as correct assignments.Source: §4.1 · p. 8, Metrics · p. 8

4.2 Method Comparison

  • Across target datasets, the preferred strategy depends on data characteristics: classification expands class coverage and recall, while SAM prompting can improve results but requires prior information and increases runtime.Domain-specific classifiers can miss classes absent from training; fine-grained classes challenge all strategies, and foundation models incur higher runtime.Source: Table 3 · p. 11, Unsupervised and Weakly Supervised… · pp. 11–12
  • Box-prompted SAM achieves the highest accuracy, but assumes bounding-box annotations; point prompts underperform box-based strategies and only marginally surpass unsupervised methods on most datasets.Using boxes with class-agnostic masks offers no advantage over directly prompting with the same boxes for most datasets.Source: Figure 4 · p. 10, Table 3 · p. 11, Semi-Supervised Approaches · pp. 12–13

4.3 Few-Shot Domain Adaptation

  • Few-shot adaptation helps more on structurally simpler LettuceMOTS, where a missing class can be added with few samples, than on CropAndWeed, where random samples may miss classes absent from training.The experiments fine-tune target-specific models using randomly selected manually annotated images, so gains depend on target dataset characteristics and class distribution.Source: §4.3 · pp. 13–14

5 Conclusion

ReLEAP unifies heterogeneous vision datasets and extends their utility through annotation recycling, with 15 agricultural datasets integrated into a shared foundation. Experiments show that upcycling can improve annotation quality and expand reusable training data, while the framework is envisioned as a growing community resource.

5 Conclusion

  • ReLEAP unifies heterogeneous vision datasets by abstracting data sources, harmonizing labels through a shared taxonomy, and integrating 15 agricultural datasets.Deterministic task conversions further expand training-data availability.Source: §5 · p. 14
  • Aggregating data improves class-agnostic mask quality, while separating mask generation from classification benefits fine-grained class assignment; both outperform single-dataset training on average.Given the same prior information, lightweight domain-specific models perform comparably to much larger foundation models with inference time roughly an order of magnitude lower.Source: §5 · p. 14
  • The authors envision ReLEAP as a community-driven resource that grows with new datasets, taxonomies, and upcycling methods.They hope unified access and reusable annotation pipelines will improve research efficiency, accessibility, and reproducibility in precision agriculture and potentially other heterogeneous vision domains.Source: §5 · p. 14

Annotation (Supplementary Material)

The supplementary document adds dataset statistics, implementation details, and experiment results to the main paper. Its three sections cover dataset/framework comparisons, upcycling methods and label mappings, and cross-dataset experiments.

Annotation (Supplementary Material)

A Data Unification and Recycling

This section describes ReLEAP features in relation to other data-harmonization frameworks and provides detailed statistics for surveyed and integrated datasets.

A Data Unification and Recycling

A.1 Dataset Survey and Selection

An extensive survey of public datasets with plant images and annotations from early growth stages informed ReLEAP’s initial dataset selection. The survey covers task types, recording years, locations, and dataset-specific characteristics; 15 datasets were selected for integration and experiments.

A.1 Dataset Survey and Selection

  • The survey informed selection of 15 datasets for ReLEAP integration and experiments.Datasets were analyzed for recording and annotation characteristics to form a balanced initial selection.Source: §A.1 · p. 23
  • Tables 5–8 list surveyed datasets by highest-level annotation task, with recording and dataset statistics and public-availability information.The tables cover classification, object detection, semantic segmentation, and instance segmentation; selected integration datasets are marked in bold.Source: Table 5 · p. 24, Table 6 · p. 25, Table 7 · p. 26
  • Figure 6 summarizes surveyed datasets by highest-level annotated task, recording year, and location.The task categories shown are classification, object detection, semantic segmentation, and instance segmentation.Source: Figure 6 · p. 27

A.2 Framework Comparison

ReLEAP combines cross-dataset aggregation, taxonomy-based label harmonization, and annotation recycling in one configurable open-source workflow. Its domain-level taxonomy is reused across datasets and target label spaces, rather than defined through pairwise label-remapping rules.

A.2 Framework Comparison

  • A shared taxonomy is curated once per application domain and mapped to each dataset and target label space, rather than using pairwise label-remapping rules.Task-driven recycling selects deterministic annotation conversions automatically from the requested target task; end-to-end aggregation requires all capabilities in one configurable pipeline.Source: §A.2 · p. 28
  • ReLEAP unifies cross-dataset integration, taxonomy-based label harmonization, and task-driven annotation recycling in one open-source framework.Table 9 compares documented workflow capabilities; its marks do not indicate maturity or scope.Source: Table 9 · p. 28

B Upcycling Methods

The methods generate instance masks using image-only, dataset-level label-name, or lower-level annotation inputs. ReLEAP maps and filters labels and instances to prepare shared training label spaces.

B Upcycling Methods

  • Unsupervised methods use only target images, drawing on models trained on other integrated sources or foundation models.ISEGSpec trains a multi-class model on available instance-segmentation data while excluding the target dataset; its label space is limited to classes in those datasets.Source: B Upcycling Methods · p. 29
  • ISEGAgn+CLS uses class-agnostic masks to crop image patches, then assigns classes with a separate classifier trained without target-dataset samples.The segmentation and classification confidence thresholds are 0.4 and 0.1, respectively; unclassified instances are removed.Source: B Upcycling Methods · p. 29
  • SAMAgn+CLS uses the same classification model as above, with agnostic masks generated by prompting SAM 3 [14] with generic superclass names plant, vegetation, crop, and weed; combined results undergo non-maxima suppression at IoU 0.7.Source: B Upcycling Methods · p. 29
  • Weakly supervised methods need dataset-level metadata rather than image-level annotations; SAMLabels prompts SAM 3 with the target dataset’s class names.Generic background labels are excluded, and the approach represents upcycling unannotated images when crop and weed species are known.Source: B Upcycling Methods · pp. 29–30
  • Semi-supervised methods use per-image boxes or stem points; ISEGAgn+Box assigns labels by IoU matching, while SAMBox uses boxes as SAM 3 prompts.ISEGAgn+Box uses a 0.1 IoU threshold and discards unmatched masks; SAMPt instead prompts SAM 3 with points geometrically derived from ground-truth masks.Source: B Upcycling Methods · p. 30

B.1 Label Mapping

  • ReLEAP maps and filters dataset labels and instances before training multi-class instance-segmentation and classification models.Tables 10–12 define the mapped classes; tables 13–14 report mapped labels and included instances after filtering.Source: §B.1 · p. 30, Table 10 · p. 31, Table 11 · p. 32

C Experimental Results

The supplementary evaluation reports per-dataset method statistics, mapped-class accuracy, and representative qualitative outputs. Its tables and figures complement the main paper’s method comparison.

C Experimental Results

  • Tables 16–20 report instance-mask mAP50-95 for mapped target classes across five datasets, whose class counts and granularities vary.The tables cover PhenoBench, WeedsGalore, SorghumWeedSeg, LettuceMOTS, and CropAndWeed; the passages provide no cell values to compare.Source: C Experimental Results · p. 35, Table 16 · p. 38, Table 17 · p. 38
  • Figures 7–11 show representative input images, ground truth, and method outputs, with colors marking correct masks, wrong-class assignments, errors, and missed instances.Green marks correctly detected mask areas, yellow correctly located areas assigned to wrong classes, red incorrectly detected areas, and blue undetected instances.Source: C Experimental Results · p. 35, Figure 7 · p. 36, Figure 8 · p. 36
  • Table 15 gives agnostic-method precision, recall, and average inference time per image for each target dataset, forming the basis for the main paper’s mean values.Its baseline averages and best results from individual models trained on the other four datasets.Source: Table 15 · p. 35, C Experimental Results · p. 35

Related papers

Loading 2610.32289v1…