PaperlayerView exact revision on arXiv

Source-linked AI summary

SatoyamaCT: A Multi-Axis Night-IR Camera-Trap Benchmark for Monitoring Crop-Damaging Wildlife in Japanese Agroforestry

Keito Inoshita, Kohei Hisayama, Haruto Sugeno, Kota Nojiri

arXiv:2609.30278v1eess.IVq-bio.QM

TL;DR

Camera-trap models are often evaluated under conditions like their training data, despite real deployments transferring across regions and sensors. SatoyamaCT combines expert-verified night-IR-dominant data with multi-axis evaluation, finding persistent regional difficulty and joint Wakayama degradation in classification and open-set detection.

  • Problem

    Models are often evaluated under training-homogeneous conditions, although operational deployment transfers them to different regions, seasons, and sensor types.

    Source: §1 · p. 2

  • Method

    SatoyamaCT provides 12,642 expert-verified crops from three Japanese regions and jointly evaluates domain shifts, open-set detection, and selective prediction with capture-event leakage control.

    Source: §1 · pp. 3–4, §6 · pp. 29–30

  • Results

    A 16 to 27 percentage-point regional accuracy gap persists in Wakayama, where open-set detection also degrades alongside classification.

    Source: §1 · pp. 4–5, §6 · p. 30

  • Takeaways & Limitations

    The evaluation identifies regional shift as the dominant remaining challenge for pest monitoring, while placement-minority recall improves after label correction.

    Source: §1 · p. 4

    Region and modality are confounded because photographs come from Mie and video from Shizuoka and Wakayama, so a pure modality effect cannot be isolated.

    Source: §5.3 · p. 29

Abstract

from arXiv · show

Crop and forest damage from sika deer, wild boar, and Japanese macaque is a serious economic problem in Japanese satoyama, where farmland and forest intermingle. Camera traps enable scalable monitoring, yet existing benchmarks evaluate recognition in-distribution, rarely prioritize night infrared imagery, and do not jointly address regional domain shift, novel-species detection, and uncertainty-based triage on a single dataset. We introduce the Satoyama Camera Trap Dataset (SatoyamaCT), 12,642 expert-verified crops from night-IR-dominant camera traps across three satoyama regions. An iterative annotation protocol combining BioCLIP embeddings with confidence-ordered confirmation reduced expert effort while all labels were verified by domain ecologists; species-level inter-annotator agreement reached Cohen's kappa = 0.900. A multi-axis protocol jointly evaluates domain generalization across region, camera placement, and illumination; open-set novel-species detection; and selective prediction, all under capture-event-based leakage control. Difficulty separates into distinct failure modes: a data-inherent regional gap of 16 to 27 percentage points persists in Wakayama across four backbones and domain-generalization methods including CORAL, DANN, and GroupDRO, and open-set detection reaches AUROC 0.93 to 0.96 overall yet degrades jointly with classification in Wakayama. Selective prediction recovers Wakayama accuracy from 0.593 to 0.815 at 50% coverage, supporting an uncertainty-aware triage workflow for practical pest monitoring in agroforestry.

1. Introduction

Camera-trap models face deployment conditions that differ from training, while existing benchmarks do not jointly evaluate night infrared imagery, distribution shifts, and reliability. SatoyamaCT combines expert-verified data with multi-axis evaluation; results identify regional shift and joint classification/open-set degradation in Wakayama as key challenges.

Motivation

  • Most accuracy estimates use conditions homogeneous with training, although deployments transfer models to different regions, seasons, and sensor types.Existing benchmarks also commonly omit night infrared as a primary axis and do not jointly evaluate placement, reliability, and Asian agricultural landscapes.Source: §1 · pp. 2–3

Dataset and annotation

  • SatoyamaCT contains 12,642 expert-verified crops from three Japanese regions, with 9,583 recorded in night-infrared monochrome.The dataset includes photographs and video, and its long-tailed species distribution includes rare and novel individuals.Source: §1 · pp. 3–4

Benchmark design

  • The benchmark jointly evaluates regional, camera-placement, and illumination shifts, open-set novel-species detection, and selective prediction under capture-event leakage control.Annotation combines BioCLIP-embedding clustering and confidence-ordered confirmation, with final labels verified by domain ecologists.Source: §1 · p. 4

Key findings

  • Wakayama remains the hardest regional split, and open-set detection also degrades there alongside classification.The regional difficulty persists across methods; placement-minority recall improved after label correction, leaving regional shift as the dominant challenge.Source: §1 · pp. 4–5

2. Related Work

Camera-trap research has expanded across datasets and tasks, but existing resources still leave important operational shifts and reliability questions unintegrated. SatoyamaCT brings these axes together, using foundation-model embeddings and model-in-the-loop annotation with independent label-quality assessment.

2.1. Camera Trap Datasets and Benchmarks

  • Existing camera-trap datasets expanded geographic and taxonomic coverage, but largely centered evaluation on species-classification accuracy.Benchmarks still commonly lack night-IR-centered evaluation, mid-scale satoyama coverage, and a framework jointly assessing placement, open-set detection, and uncertainty.Source: §2.1 · pp. 5–6

2.2. Domain Generalization, Modality, Open-Set Recognition, and Uncertainty

  • Inter-location shift, modality differences, and unseen-species detection motivate evaluating domain generalization, open-set detection, calibration, and selective prediction together.The paper integrates AUROC, FPR@95TPR, ECE, and AURC into its multi-axis evaluation.Source: §2.2 · pp. 6–7

2.3. Foundation Model Embeddings and Model-in-the-Loop Annotation

  • SatoyamaCT uses BioCLIP and DINOv2 as reference baselines, and its difficulty is reported as independent of backbone choice.Its annotation approach combines BioCLIP embedding clustering with confidence-ordered active confirmation, while a second annotator labels the video seed set for agreement assessment.Source: §2.3 · pp. 7–8

3. Dataset Construction

SatoyamaCT is built from camera-trap data across three Japanese satoyama regions through detection, crop extraction, model-in-the-loop annotation, and multi-axis benchmark design. Its construction accounts for region–modality confounding and capture-event leakage.

3. Dataset Construction

  • The construction pipeline proceeds from night-IR camera-trap capture to detection and crop extraction, model-in-the-loop annotation, and multi-axis benchmark design.These stages correspond to panels A–D of Figure 1.Source: §3 · p. 8, Figure 1 · p. 9

3.1. Data Sources and Crop Extraction

  • SatoyamaCT draws on three regions and two genuine modalities, with Wakayama deployment points stratified by local placement and vegetation.Region and modality are structurally confounded: Mie contributes photographs, while Wakayama and Shizuoka contribute video.Source: §3.1 · p. 9

3.2. Model-in-the-Loop Semi-Automatic Annotation

  • 54.5% hit rate on 6,906 non-seed crops accompanied model-assisted annotation, with all final labels confirmed by domain experts.Correction rates were 68.0% in the [0, 0.5) confidence band and 1.15% in the [0.9, 1) band.Source: §3.2 · pp. 11–12

3.3. Label Quality Verification: Inter-Annotator Agreement

  • Species-only agreement reached κ = 0.900, while 85.2% of disagreements involved unknown or false-detection labels.Across all 13 categories, agreement was κ = 0.687; pure inter-species confusion accounted for 33 cases.Source: §3.3 · p. 13, Figure 5 · p. 14

3.4. Species Taxonomy and Leakage Control

  • The benchmark labels five sufficiently sampled species as known, rarer real species as novel, uncertain cases separately, and empty detections as junk.To limit leakage, near-duplicate bursts are grouped within 120 seconds, while independent events use a 30-minute gap; one video clip is one group.Source: §3.4 · pp. 14–15

3.5. Benchmark Tasks and Evaluation Metrics

  • Region-LOSO, placement-LOSO, open-set recognition, and an illumination-only sika deer comparison test distinct deployment shifts and reliability tasks.Region-LOSO holds out a region; placement-LOSO holds out a Wakayama deployment point, while the illumination task contrasts daytime color and night IR.Source: §3.5 · p. 15

4. Dataset Statistics and Benchmark Results

SatoyamaCT combines expert-verified night-IR-dominant data with evaluation across regional, placement, illumination, open-set, and uncertainty axes. Its central result is a persistent Wakayama regional gap, alongside distinct placement, open-set, and reliability challenges.

4.1. Dataset Composition

  • 12,642 crops span three regions and two modalities, including 9,583 night-infrared crops; placement-stratified Wakayama video and embedding distributions expose additional dataset structure.Photographs come from Mie and video from Wakayama and Shizuoka, so region and modality are structurally confounded; Figure 7 shows regional-modality differences and overlapping rare-species clusters.Source: §4.1 · p. 16, Table 2 · p. 17, Figure 7 · p. 17

4.2. Reference Baselines and the Regional Gap

  • Linear-probe accuracy was 0.593 in Wakayama, versus 0.839 in Mie and 0.816 in Shizuoka; the gap persisted in video-only evaluation and across placement splits.Wakayama’s prototype accuracy was 0.492, below Mie’s 0.733 and Shizuoka’s 0.610, locating the gap even in embedding geometry.Source: §4.2 · pp. 18–19, Table 4 · p. 20

4.3. The Regional Gap Is Independent of Backbone Choice and Generalization Method

  • Wakayama remained hardest by 16 to 27 percentage points across four backbone configurations, with ∆W = −0.162 to −0.270.The gap persisted across domain-generalization methods and end-to-end fine-tuning, indicating a data-inherent regional shift.Source: §4.3 · pp. 19–20, Table 5 · p. 21
  • CORAL, DANN, and GroupDRO did not remove Wakayama’s lowest macro-F1; GroupDRO narrowed the ERM gap from 0.143 to 0.085, but Wakayama remained hardest.End-to-end fine-tuning likewise left Wakayama hardest across ERM, GroupDRO, and SWAD.Source: §4.3 · p. 21, Table 6 · p. 23, Table 7 · p. 23

4.4. Placement-Level Recall: Improvement after Label Correction

  • After label correction, placement-level macro-F1 rose from 0.38–0.79 to 0.53–0.88, attenuating the minority-species recall difficulty.Broadleaf-lower macro-F1 reached 0.672 averaged over three seeds, up from 0.572 ± 0.062 before correction.Source: §4.4 · p. 23

4.5. Open-Set Novel Species Detection and Illumination Shift

  • Fine-tuned MC-Dropout entropy achieved overall AUROC 0.955 and IR-only AUROC 0.946, while prototype distance fell from AUROC 0.69 to 0.59 in the IR-only setting.For sika deer, daytime-to-night-IR recall was 0.63, compared with 0.75 for night-IR-to-daytime transfer.Source: §4.5 · pp. 24–25, Table 9 · p. 25

4.6. Calibration, Selective Prediction, and Region-Conditional Vulnerability

  • Temperature scaling improved ECE in Mie and Shizuoka but worsened it in Wakayama, from 0.088 to 0.147; selecting the top 50% by confidence raised Wakayama accuracy from 0.593 to 0.815.Wakayama’s AURC was 0.192, about five times Mie’s 0.036 and Shizuoka’s 0.041.Source: §4.6 · pp. 25–26, Table 10 · p. 26
  • Region-conditional open-set performance also degraded in Wakayama, where MSP AUROC was 0.663 versus 0.964 in Mie and 0.844 in Shizuoka.Wakayama was therefore a joint difficulty for classification and novel-species detection, unlike the tractable aggregate open-set result.Source: §4.5 · p. 24, §4.6 · p. 26, Table 11 · p. 26

5. Discussion

The discussion distinguishes a persistent regional failure mode from a placement-level difficulty reduced by label correction. It argues for uncertainty-aware operation and notes important scope limits in the dataset.

5.1. Key Findings

  • The remaining dominant difficulty is the Wakayama regional gap, while label correction attenuated placement-level minority-species recall problems.The discussion attributes these difficulties to different sources: regional distribution shift and community composition within a placement.Source: §5.1 · pp. 26–27

5.2. Applications and Implications

  • In Wakayama, deferring approximately half of low-confidence predictions to expert review raises accuracy from 0.593 to 0.815, supporting selective triage for pest monitoring.The paper also presents SatoyamaCT as an evaluation platform for distribution-aware calibration, domain adaptation, and open-set detection.Source: §5.2 · p. 28

5.3. Limitations

  • Region and modality are confounded because photographs come from Mie while video comes from Shizuoka and Wakayama, so a pure modality effect cannot be isolated.The data also come from one provider network and one satoyama zone, limiting extrapolation to wider geographic or ecological contexts.Source: §5.3 · p. 29

6. Conclusion

SatoyamaCT pairs expert-verified camera-trap crops with a multi-axis evaluation protocol for distribution shifts and uncertainty. Its evaluation identifies Wakayama as a persistent regional challenge, while selective prediction and region-conditional open-set results reveal operationally relevant differences across axes.

6. Conclusion

  • The benchmark combines 12,642 animal crops, expert-confirmed labels, and evaluation of region, camera placement, illumination, open-set detection, calibration, and selective prediction.Evaluation includes capture-event-based leakage control and linear-probe and ViT-B/16 full-fine-tuning baselines.Source: §6 · pp. 29–30
  • Wakayama remained the hardest region across three model configurations, with accuracy 0.593, 0.803, and 0.724 ± 0.004, each 16 to 27 percentage points below other regions.Placement-level macro-F1 improved from 0.38 to 0.79 before correction to 0.53 to 0.88 after correction, while aggregate open-set AUROC reached 0.94–0.96 and degraded jointly with classification in Wakayama.Source: §6 · p. 30
  • The protocol and annotation scheme provide a reproducible way to diagnose where automated wildlife recognition fails in real-world pest-monitoring deployments.Planned work includes denser video sampling, matched-location photographs and video, and uncertainty-centered deployment pipelines.Source: §6 · pp. 30–31

CRediT Author Contribution Statement

The contribution statement assigns conceptualization, methodology, software, analysis, investigation, data curation, visualization, writing, supervision, project administration, and funding acquisition to Keito Inoshita. Kohei Hisayama and Haruto Sugeno contributed investigation, data curation, and writing review; Kota Nojiri contributed data curation and writing review.

CRediT Author Contribution Statement

Data Availability

SatoyamaCT and its baseline code are publicly available through GitHub, with evaluation metadata, splits, images, and embeddings provided under CC BY-NC 4.0.

Data Availability

  • SatoyamaCT and baseline code are publicly available in the project repository under CC BY-NC 4.0.The repository includes JSON metadata and recommended evaluation splits; crop images and pre-computed embeddings are GitHub Release assets.Source: Data Availability · p. 31
  • Precise geographic coordinates are withheld to protect rare species, while region and habitat type are retained.Source: Data Availability · p. 31
Loading 2609.30278v1…