PaperlayerView exact revision on arXiv

Source-linked AI summary

Three LiDARs, Twenty-Five Datasets: Cross-Source Generalization in Tree-Species Recognition

Jae Joong Lee, Bedrich Benes

arXiv:2610.22606v1cs.CVcs.AIcs.GR

TL;DR

The paper asks whether strong LiDAR tree-species benchmark scores reflect transfer across independently collected sources. It evaluates a multi-resolution, multi-view classifier and finds substantially lower accuracy under source-species holdout, motivating source-aware evaluation.

  • Problem

    The central question is whether FOR-species20K's species-stratified random-split score reflects transfer to unseen sources, since every source dataset appears in training and test.

    Source: §4 · p. 4, §7 · p. 8

  • Method

    The authors evaluate a multi-resolution, multi-view ConvNeXtV2 classifier and use source-species-pair holdouts where each species retains training examples in other sources to test cross-source transfer.

    Source: §1 · p. 1, §4 · p. 4

  • Results

    Under source-species holdout, the unweighted species mean falls from 82% to 36%, and tree-weighted accuracy falls from 95% to 60%; PureForest shows the same directional split effect.

    Source: §1 · p. 2, §6.2 · p. 7, §7 · p. 8

  • Takeaways & Limitations

    The authors recommend balanced accuracy and macro-F1 alongside overall accuracy, plus a source-disjoint leaderboard track and per-source reporting.

    Source: §7 · p. 8

    The main estimate excludes held-out cases with fewer than 50 same-type training trees elsewhere, so it does not measure transfer for near-unseen species.

    Source: §6.1 · p. 7

Abstract

from arXiv · show

FOR-species20K is the largest public LiDAR benchmark for tree-species classification, pooling 25 independently collected terrestrial, mobile, and airborne laser-scanning datasets across 33 species. We introduce a multi-resolution, multi-view ConvNeXtV2 model that improves official hidden-test accuracy from 79.5% to 85.0%, ranking first at the time of submission. We then examine what this score measures. Because the hidden test is species-balanced, its overall accuracy is more comparable to a macro metric than to accuracy under the public set's long-tailed distribution. Moreover, the official random split places every source dataset in both training and evaluation. A logistic probe recovers source identity from out-of-fold model probabilities with 69% accuracy, while geometric features identify the source above 90% within two widely shared species; 18 of the 33 species occur in only one source. Under a source-species-cell holdout, evaluated on 5,566 trees from eight eligible species that retain the same LiDAR-type training support in other sources, the unweighted species mean falls from 82% to 36%, and tree-weighted accuracy falls from 95% to 60%. Count-matched retraining and experiments with two alternative backbones indicate that this gap is not explained by reduced training support alone, while an external PureForest experiment shows a smaller but consistent grouped-split effect. These results establish the 85% leaderboard score as strong in-distribution performance, but not as an estimate of source-disjoint generalization, motivating balanced metrics and source-aware evaluation. The code is available at: https://github.com/JaeLee18/FOR20K-lidar-tree-species

1. Introduction

FOR-species20K supports large-scale LiDAR tree-species recognition, and this paper improves its leaderboard result while asking whether that score reflects transfer across source datasets. The work situates this question within multi-view recognition and source-aware evaluation.

1. Introduction

  • 85.0% hidden-test accuracy improves on DetailView’s 79.5% and ranked first on the official leaderboard at submission.The model uses seven geometric views, a logit-adjusted objective, and averaged predictions from 256- and 512-pixel renders.Source: §1 · p. 1
  • The official species-stratified random split includes every source in training and testing, so source familiarity can substitute for cross-source species recognition.The species-balanced hidden test makes overall accuracy more like a macro metric than accuracy under the public set’s long-tailed distribution.Source: §1 · p. 1
  • A logistic probe identifies source from out-of-fold model probabilities with 69% accuracy, while geometric features exceed 90% within two widely shared species.On 5,566 trees from eight eligible species, source-species holdout lowers the unweighted species mean from 82% to 36% and tree-weighted accuracy from 95% to 60%.Source: §1 · pp. 1–2

2. Related Work

  • Prior work contrasts point-based networks such as PointNet++ [38] and DGCNN [49] with multi-view image methods, which currently lead on FOR-species20K.Research also documents source-specific cues and cross-site performance drops; grouped or spatially blocked cross-validation is an established remedy.Source: §2 · p. 2

3. The FOR-species20K Benchmark

FOR-species20K combines 25 independent LiDAR datasets across 33 species, with substantial source and species imbalance. Its balanced hidden test makes macro metrics more comparable than public-set overall accuracy.

3. The FOR-species20K Benchmark

  • The benchmark contains 17,707 public labeled trees and 2,250 hidden-test trees across 25 independently collected datasets and 33 species.The hidden set samples a similar number of trees per species, unlike the long-tailed public set.Source: §3 · p. 2
  • 0.899 public-set overall accuracy is less comparable to the 0.850 hidden-test score than to the 0.860 public-set macro-F1 of the ensemble.Across five models, accuracy under the true data distribution exceeds macro-F1 in every case, with gaps from +0.04 for ConvNeXtV2 to +0.20 for DGCNN.Source: §3 · pp. 3–4
  • Source datasets vary in size by two orders of magnitude, and 18 of 33 species occur in only one source.Each source uses one LiDAR type, and the official random split places every source in both training and test sets.Source: Figure 1 · p. 3, §3 · p. 4

4. Cross-Source LiDAR Evaluation

The section presents a multi-resolution, multi-view ConvNeXtV2 model and evaluates tree-species recognition when source–species pairs are withheld. It reports improved hidden-test performance and compares model variants on the public labeled set.

4. Cross-Source LiDAR Evaluation

  • The source–species holdout withholds a species from a held-out source while scoring it only if that species remains in other sources.The same source may remain in training through other species that are not scored in that fold.Source: §4 · p. 4
  • Scoring excludes single-source species and requires at least 50 same-LiDAR-type training trees and 20 held-out trees for the main gap analysis.Four sources contain only single-source species and contribute no eligible pairs, leaving 21 sources for scoring.Source: §4 · p. 4

5. Experimental Setup

  • The model uses seven orthographic views at 256 × 256 and 512 × 512 resolutions, processed in three ConvNeXtV2 streams with tree height added before classification.The view channels encode surface depth, log-transformed point density, and absolute height.Source: §5 · pp. 4–5, Figure 4 · p. 5
  • 0.850 overall accuracy beats DetailView’s 0.795 on the hidden test; the model also leads precision, recall, and F1 and scores 0.856 TLS, 0.934 MLS, and 0.792 ULS.The method ranked first on the official leaderboard at submission.Source: §5 · p. 5, Table 1 · p. 5
  • 0.899 overall accuracy and 0.860 macro-F1 are the two-resolution ensemble’s strongest public labeled-set out-of-fold scores.These ablation scores are not directly comparable to the species-balanced hidden-test results.Source: Table 2 · p. 5, §5 · p. 6
  • DINOv2-Base achieves 0.819 overall accuracy, close to DenseNet-201’s 0.818, and the authors attribute gains to the ConvNeXtV2 and long-tail objective combination.The comparison keeps the training and inference pipeline the same apart from the backbone and its native input size.Source: §5 · p. 6

6. Results and Analysis

Random splits retain dataset-specific cues, and accuracy falls under source-grouped evaluation even when species and same-type training examples remain available. The analysis also finds persistent cross-source effects on PureForest and fine-grained errors among co-occurring deciduous species.

6.1. Random Splits Retain Dataset Cues

  • 69% accuracy recovers dataset identity from class probabilities, while geometric features identify source at 91% for Picea abies and 97% for Pinus sylvestris.A dataset with 1,958 trees from one species reaches 99.95% out-of-fold accuracy, illustrating how dataset familiarity can supply the label.Source: §6.1 · p. 6
  • 82% to 36% for the unweighted species mean and 95% to 60% for tree-weighted accuracy on the same held-out trees from eight eligible species.The drop persists across folds and is not explained by reduced sample count alone.Source: §6.1 · pp. 6–7
  • 86% to 46% for DenseNet-201 and 92% to 49% for DINOv2 under random versus grouped evaluation.The cross-source gap persists across alternative backbones.Source: §6.1 · p. 7

6.2. The Split Effect on an Another Benchmark

  • 94% to 81% on PureForest under random patch versus forest-grouped splits, with a support-controlled estimated drop of 9 points.The effect is smaller than FOR-species20K's 35-point tree-weighted drop; the authors interpret this as directional evidence beyond FOR-species20K.Source: §6.2 · pp. 7–8

6.3. Errors in a Dense Mixed Forest

  • 99% of BlueCat a's errors among co-occurring deciduous species cross genus boundaries, often favoring frequent classes.Across the public set, cross-genus confusions account for 88% of errors; collapsing species into genera changes accuracy from 0.886 to 0.900 while decreasing macro-F1 and balanced accuracy.Source: §6.3 · p. 8
  • ConvNeXtV2 improves over DenseNet for every species, with its largest gains on long-tail classes.The remaining errors in BlueCat a primarily involve co-occurring deciduous species with similar crowns.Source: §6.3 · p. 8

7. Conclusion

The conclusion distinguishes the official score as an in-distribution measure from source-disjoint performance and recommends source-aware reporting and evaluation.

7. Conclusion

  • The source-species holdout lowers the species mean by 46 points and tree-weighted score by 35 points, while PureForest shows a 9-point forest-grouped split drop.The smaller PureForest drop may reflect its single acquisition program, unlike FOR-species20K's 25 independent sources.Source: §7 · p. 8
  • The authors recommend balanced accuracy and macro-F1, a source-disjoint leaderboard track, per-source results, and at least two independent sources per species.They recommend using the 60% source-species holdout result as the reference for unseen sites when same-type training data are available.Source: §7 · p. 8

Funding

The work acknowledges research-award, NSF, and USDA-NIFA support, while disclaiming institutional endorsement of its findings.

Funding

  • The work is supported by the 2026 Google TPU Builders Research Award and in part by three U.S. National Science Foundation awards.The listed NSF award numbers are 2412928, 2506783, and 2417510.Source: Funding · p. 8
  • USDA-NIFA support includes grants on automated tree geometry, soybean canopy architecture, and Eastern U.S. forest resilience.The listed grants are 2024-67021-42879, 2023-08563, and 2023-68012-38992.Source: Funding · p. 8
  • The authors state that findings and recommendations do not necessarily reflect NSF or U.S. Government and NRCS policies.The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes.Source: Funding · p. 8

Supplementary Overview

The supplement documents implementation and evaluation, then provides source-recognition, per-fold, per-species, and error analyses alongside grouped-split results.

Supplementary Overview

  • Sec. A1 covers input representation, network architecture, training objective, optimization, multi-resolution inference, and source-grouped split construction.Sec. A2 covers overall accuracy, balanced accuracy, and macro-F1 under the natural public-set distribution.Source: Supplementary Overview · p. 11
  • Sec. A3 reports source recognition, while Secs. A4–A6 provide fold-level performance, per-species F1, and the full confusion matrix.The listed per-species results compare DenseNet-201 and ConvNeXtV2 across all 33 species.Source: Supplementary Overview · p. 11
  • Sec. A7 gives complete source-species holdout fold results, including the random-split reference and two image backbones; Sec. A8 covers PureForest.The PureForest supplement includes the experiment comparing random patch splits with forest-grouped folds.Source: Supplementary Overview · p. 11

A1. Implementation Details

The model combines geometric multi-view inputs with a logit-adjusted training objective and multi-resolution ensemble inference. Natural-distribution accuracy exceeds macro-F1 for every model, with the largest gap for the lower-performing point-cloud baselines.

A1. Implementation Details

  • Seven geometric views encode depth, point density, and height for three ConvNeXtV2-Base backbones, whose pooled features and tree height feed a 33-class prediction head.The views are rendered at 256 × 256; the system has 264.7M parameters.Source: A1. Implementation Details · p. 12
  • Logit-adjusted cross-entropy adds τ log πc to each class logit, using fold-training frequencies to give rare classes an effective margin without resampling or class weights.Inference uses raw, unadjusted logits.Source: A1. Implementation Details · p. 12
  • The submitted ensemble averages 256- and 512-pixel models, five fold models, and twenty test-time augmentation passes per tree.Training uses a three-epoch head-only warmup followed by full-network fine-tuning; checkpoint selection uses validation macro-F1.Source: A1. Implementation Details · p. 12

A2. Natural-Distribution and Macro Metrics

  • Natural-distribution accuracy exceeds macro-F1 for every model, with the OA–macro-F1 gap ranging from 0.043 for ConvNeXtV2 to 0.199 for DGCNN.The comparison uses public development-set predictions.Source: A2. Natural-Distribution and Macro… · p. 12

A3. Source Recognition from Model Outputs

A logistic probe predicts source dataset from out-of-fold model probabilities, revealing substantial source information in the outputs. ConvNeXtV2 also maintains or improves per-species F1 over DenseNet-201, with especially large differences among some rare classes.

A3. Source Recognition from Model Outputs

  • Source classification from pooled out-of-fold probabilities reaches 0.691 accuracy (95% CI [0.684, 0.698]), versus 0.040 uniform chance and 0.194 after source-label permutation.Accuracy is measured with separate stratified five-fold cross-validation.Source: A3. Source Recognition from Model… · p. 12
  • Source-classification accuracy varies by platform: 0.818 for TLS, 0.477 for ULS, and 0.151 for the four MLS datasets.The MLS datasets cover nine species and are dominated by Picea, Pinus, and Betula, limiting direct comparison with TLS and ULS.Source: A3. Source Recognition from Model… · p. 12
  • PointNet++ and DGCNN yield source-classification accuracies of 0.582 and 0.655, respectively, from reproduced out-of-fold probabilities.Their species-classification accuracies are published leaderboard values reported in the main paper.Source: A3. Source Recognition from Model… · p. 13

A4. Per-Fold Results

  • ConvNeXtV2’s standard deviation across five out-of-fold folds is below 0.004.The fold results are internal figures and are not comparable to the balanced hidden test.Source: A4. Per-Fold Results · p. 13

A5. Per-Species F1 (All 33 Species)

  • ConvNeXtV2 matches or exceeds DenseNet-201 F1 for all 33 species, with several of the largest differences among the rarest classes.Per-species F1 is sorted by training support in Figure A1.Source: A5. Per-Species F1 (All 33 Species) · p. 13

A6. Full Confusion Matrix

The pooled five-fold out-of-fold confusion matrix shows a near-diagonal block for conifers and geographically distinctive broadleaves, while errors concentrate among co-occurring deciduous broadleaves.

A7. Full Confusion Matrix

  • Figure A2 presents ConvNeXtV2’s row-normalized confusion matrix across 33 species, pooled over five-fold out-of-fold predictions for 17,707 trees.Rows are true species, columns are predictions, and cells show within-row prediction rates.Source: A6. Full Confusion Matrix · p. 13
  • Conifers and geographically distinctive broadleaves form a near-diagonal block in the confusion matrix.The matrix covers all 33 species.Source: A6. Full Confusion Matrix · p. 13
  • Most off-diagonal mass lies among co-occurring deciduous broadleaves in the upper-left block.Off-diagonal cells at or above 0.05 are annotated.Source: A6. Full Confusion Matrix · p. 13

A7. Complete Leave-Source-Out Results

The leave-source-out results report accuracy for scored species-platform cells, alongside surviving same-platform training support and a random-split baseline. Eligibility thresholds restrict the evaluation to a subset of cross-source cells.

A7. Complete Leave-Source-Out Results

  • Of 67 held-out species-platform cells among multi-source species, 25 meet both thresholds, covering eight species and 5,566 held-out trees.Eligibility requires at least 20 held-out trees and 50 surviving same-platform training trees.Source: A7. Complete Leave-Source-Out Results · pp. 14–15
  • Eighteen of 33 species occur in a single source and are never cross-source evaluable; seven additional multi-source species are not scored because their trees are concentrated in one source.Holding out that source would remove nearly all their training examples and would not isolate cross-source transfer.Source: A7. Complete Leave-Source-Out Results · pp. 14–15
  • The 42 excluded cells comprise 17 with fewer than 20 held-out trees and 25 with fewer than 50 surviving same-platform training trees.These exclusions apply the two stated eligibility thresholds.Source: A7. Complete Leave-Source-Out Results · p. 15
  • Table A6 reports support-controlled leave-source-out accuracy by species-fold cell, including surviving same-platform support, random-split OOF baseline, and two backbones’ LSO accuracies.CNX LSO denotes ConvNeXtV2; DNS LSO denotes DenseNet retrained across all five grouped folds.Source: A7. Complete Leave-Source-Out Results · p. 13, Table A6 · p. 15

A8. Forest-Level Transfer on PureForest

On PureForest, forest-grouped evaluation substantially lowers accuracy relative to random patch-level splits, and the gap persists when grouped-training support is controlled. The experiment measures forest-level transfer within one national acquisition program, not cross-sensor variation.

A8.1. Data and Evaluation Subset

  • The comparison uses 22,993 patches from 137 forests across eight species, selected for five-fold grouped evaluation.The subset criteria excluded three large archives for storage and required at least five forests per species; these criteria were fixed before training.Source: A8.1. Data and Evaluation Subset · p. 19

A8.2. Preprocessing and Training

  • Training examples are subsampled to 2,048 points, normalized, rotated, and noise-perturbed; both split conditions use the same PointNet++ model and optimization settings.Five models are trained per split condition, with checkpoints selected by validation macro-F1.Source: A8.2. Preprocessing and Training · p. 19

A8.3. Random and Forest-Grouped Splits

  • Random patch-level folds can share forests between training and validation, whereas grouped folds assign each forest to exactly one fold and pair predictions on identical patches.Fold 0 shares 114 of 137 forests under the random split; the grouped split shares none.Source: A8.3. Random and Forest-Grouped Splits · p. 19

A8.4. Aggregate Results

  • 0.936 random-split accuracy versus 0.815 forest-grouped accuracy—a 12.1-point patch-weighted drop.The forest-level bootstrap 95% confidence interval for the raw accuracy drop is [7.0, 18.4] points.Source: A8.4. Aggregate Results · p. 19

A8.5. Sensitivity to Remaining Training Support

A8.6. Per-Species Results

  • Grouped accuracy is lower for all eight species, with the largest reductions for Abies alba and Quercus robur, which have the weakest forest-level support.The unweighted species mean gives a 22.8-point drop; for Abies alba, one forest contains 717 of 811 patches.Source: A8.6. Per-Species Results · p. 20, Table A10 · p. 20

A8.7. Relation to FOR-species20K

  • PureForest tests forest-level transfer within one national acquisition program, using fixed-area patches rather than individual trees, and does not include FOR-species20K’s cross-sensor variation.Its random folds are an experimental baseline for forest overlap, not a replacement for the official forest-disjoint split.Source: A8.7. Relation to FOR-species20K · p. 20

Related papers

Loading 2610.22606v1…