PaperlayerView exact revision on arXiv

Source-linked AI summary

MCLC-NET: Multimodal Continual Learning for Leaf Counting

Ruchi Bhatt, Pratibha Kumari, Shreya Bansal, Vedant Agnihotri, Dwarikanath Mahapatra, Mukesh Saini

arXiv:2609.18129v1cs.CVcs.LG

TL;DR

Leaf counting needs methods that handle RGB limitations and sequentially arriving agricultural data without losing prior knowledge. MCLC combines RGB, depth, and thermal modalities with memory-based continual learning and introduces the MMLC benchmark. Across crop-wise, time-wise, and mixed task orderings, it consistently outperforms compared methods, with average AMSE values of 0.675±0.027, 0.542±0.069, and 0.745±0.057.

  • Problem

    RGB-only leaf counting is vulnerable to occlusion and environmental variation, while multimodal continual leaf counting is underexplored and existing methods assume all training data are available at once.

  • Method

    MCLC uses RGB, depth, and thermal inputs with memory-based rehearsal, retaining informative samples from previous tasks while learning sequentially.

  • Results

    MCLC consistently achieves the best performance across all three task sequences, with average AMSE values of 0.675±0.027, 0.542±0.069, and 0.745±0.057.

  • Takeaways & Limitations

    MMLC and MCLC provide a benchmark and framework for multimodal domain-incremental leaf counting under changing crop and environmental conditions.

  • Takeaways & Limitations

    MCLC assumes aligned RGB, depth, and thermal modalities during training and inference and uses a fixed memory buffer.

Abstract

from arXiv · show

Leaf counting is an important task in plant phenotyping for monitoring plant growth and estimating crop yield. Most existing methods rely on RGB images, but their performance is often affected by occlusion, lighting variations, and other real-world challenges. Additional modalities, such as depth and thermal images, can provide useful complementary information. However, multimodal leaf counting remains underexplored. Also, many existing methods assume that all training data are available simultaneously, which is impractical in real agricultural settings, where data is collected over time from multiple sources. To address these challenges, we propose MCLC-NET, a multimodal continual learning framework for leaf counting. It learns tasks sequentially using a memory-based strategy with a memory buffer to retain important samples from previous tasks. We also introduce MMLC, a real-world multimodal leaf-counting dataset designed for a domain incremental scenario (DIS) in CL. It contains RGB, depth, and thermal images collected across different crop types under varying environmental conditions, arranged in three orderings: crop-wise, time-wise, and mixed. Experimental results, averaged over three random seeds, demonstrate that MCLC-NET consistently outperforms existing methods across all three task orderings, achieving the lowest AMSE of 0.675$\pm$0.027, 0.542$\pm$0.069, and 0.745$\pm$0.057, respectively.

I. INTRODUCTION

Leaf counting needs multimodal inputs and continual learning because RGB-only methods face real-world variability, while agricultural data arrives sequentially across changing domains. MCLC addresses these gaps with memory-based multimodal continual learning and uncertainty- and diversity-aware replay.

  • I. INTRODUCTION: RGB-only leaf counting is challenged by occlusion, variable lighting, and complex backgrounds, motivating complementary depth and thermal modalities.RGB captures color and texture, depth provides relative distance, and thermal imagery reflects temperature-related patterns.
  • I. INTRODUCTION: Multimodal leaf counting remains underexplored, and existing datasets are few and primarily collected in controlled environments rather than variable field conditions.Prior work also largely addresses static datasets or classification and unimodal settings instead of multimodal regression under continual learning.
  • I. INTRODUCTION: Sequential agricultural data creates catastrophic forgetting because existing methods assume all training data are available simultaneously.Domain-incremental learning instead adapts to changing crop types, growth stages, capture times, and environmental conditions while retaining past knowledge.
  • I. INTRODUCTION: Rehearsal-based learning is selected because memory replay is simpler and often more effective than generative replay, while multimodal regression remains challenging.The challenge combines complementary-modality utilization with continual adaptation for a regression task rather than classification.
  • I. INTRODUCTION: MCLC combines RGB, depth, and thermal data with memory-based continual learning to adapt sequentially while retaining knowledge from previous tasks.The framework stores important samples from prior crop, capture-time, and environmental-condition tasks for replay.

III. PROPOSED DATASET

MMLC is a real-world multimodal leaf-counting dataset organized for domain-incremental learning across crop and capture-time shifts. It provides eight tasks and three task orderings for continual-learning benchmarking.

  • III. PROPOSED DATASET: MMLC contains over 6.3K RGB, depth, and thermal images from four vegetable crops captured in real agricultural fields.The crops are capsicum, zucchini, cucumber, and cauliflower, with samples collected during the first four weeks of growth.
  • III. PROPOSED DATASET: Plants were captured twice daily to incorporate natural illumination and temperature variation into the benchmark.The morning and evening acquisitions are used to represent changing environmental conditions.
  • III. PROPOSED DATASET: The dataset defines eight domain-incremental tasks by crossing four crop types with two capture times, morning and evening.These variations create distribution shifts through crop species and lighting or thermal conditions.
  • III. PROPOSED DATASET: MMLC supports three sequence orderings: crop-wise CS1, time-wise CS2, and mixed CS3.CS1 groups tasks by crop, while CS2 groups morning and evening observations across crops; the figure presents the split examples and ordering structure.
  • III. PROPOSED DATASET: The proposed MCLC pipeline combines multimodal feature fusion with continual learning to address non-stationary agricultural data distributions.Algorithm 1 trains sequentially with replay and distillation, then updates a memory buffer before freezing the current model.

A. Problem Formulation

The problem models leaf counting as sequential domain-incremental regression across multimodal tasks, using buffered prior samples so later models preserve earlier knowledge.

  • Continual Setting: The model is trained sequentially because only the current data volume is available at each session.The objective is to preserve performance on previous tasks despite their data no longer being retained directly.
  • Task Definition: Each task corresponds to a domain, such as a plant species and imaging conditions, with multimodal inputs and leaf-count labels.The task sequence is represented as {T1, T2, . . . , Tn}, and each input comprises RGB, thermal, and depth modalities.
  • Objective: The learning objective uses regression loss for each task and evaluates the final model across all task data volumes.The text identifies MSE as the formal regression loss and describes final evaluation over {D1, . . . , Dn}, while the implementation passage specifies Smooth L1 for task training.
  • Replay Formulation: A buffer of prior-task samples is combined with current data to train each subsequent model.The buffer enables each new model to retain knowledge from previously seen tasks.

B. Multimodal Leaf Counting

The multimodal leaf-counting model encodes RGB, depth, and thermal images separately, adaptively weights their embeddings, fuses them through cross-attention, and regresses a single count.

  • Architecture: The architecture comprises modality encoders, modality attention, cross-attention fusion, and a regression head for leaf-count prediction.Each modality is processed separately before the representations are weighted and fused.
  • Modality Encoder Block: Each modality uses a ResNet-50-based encoder and projection head to produce a 512-dimensional embedding.The projection head flattens features and maps the 2048-dimensional representation to 512 dimensions with ReLU and dropout.
  • Modality Attention: A two-layer MLP with Softmax generates normalized weights that dynamically adjust the RGB, depth, and thermal contributions.The re-weighted embeddings are passed to the fusion stage.
  • Cross-Attention Fusion: RGB supplies the query while depth and thermal supply keys and values for multi-head cross-attention.The module produces a fused 512-dimensional feature vector containing integrated multimodal information.
  • Regression Head: The fused feature is passed through a multilayer perceptron to predict a single scalar leaf count.The regression head converts the integrated multimodal representation into the final prediction.

C. Continual Learning Strategy

The continual-learning strategy combines experience replay, uncertainty-diversity buffer selection, and knowledge distillation to preserve information from earlier tasks.

  • Experience Replay: Experience replay trains on current data and stored exemplars through both feature-level and input-level replay.Stored fused features train the regression head directly, while stored raw samples pass through the full model.
  • Uncertainty-Diversity Sampling: Five stochastic dropout passes estimate uncertainty through prediction variability during inference.This uncertainty measure prioritizes samples whose predictions vary under dropout.
  • Uncertainty-Diversity Sampling: The buffer retains a fixed number of samples selected by a unified score combining prediction uncertainty and embedding diversity.Uncertainty comes from Monte Carlo dropout, while diversity uses Minimum Embedding Distance; the highest-scoring B samples update the buffer.
  • Knowledge Distillation: A frozen previous model is used for feature- and output-space distillation during training of the current model.The distillation constraint is intended to stabilize representations and reduce catastrophic forgetting.
  • Combined Objective: The overall objective combines current-task, replay, and distillation losses.The replay implementation weights feature-level replay by λ1 = 0.3 and input-level replay by λ2 = 0.7.

V. EXPERIMENTS AND RESULTS

The experiments compare the proposed model with baseline and state-of-the-art methods using quantitative and qualitative analyses, including computational complexity.

  • Evaluation: The evaluation reports quantitative and qualitative comparisons of the proposed model against baseline and state-of-the-art methods.The section also introduces the experimental setup, evaluation metrics, and compared methods.
  • Evaluation: The experiments include an analysis of computational complexity alongside model-performance evaluation.
  • Evaluation: The section presents the experimental setup and evaluation metrics before analyzing model performance.

A. Experimental Setup

The experiments evaluate multimodal continual leaf counting across MMLC, MSU-PID, and CVPPP-related settings, using sequential tasks and three random seeds. MMLC provides three domain-incremental sequences with RGB, depth, and thermal modalities.

  • A. Experimental Setup: Each continual-learning task contains RGB, depth, and thermal images, with leaf count as the prediction target.
  • A. Experimental Setup: Experiments train methods sequentially with Smooth L1 and MSE losses while selecting buffer samples using a hybrid uncertainty-diversity criterion.
  • A. Experimental Setup: Results are averaged across three random seeds—42, 123, and 999—to provide statistically reliable comparisons.
  • A. Experimental Setup: The evaluation uses MMLC, MSU-PID, and CVPPP, with MMLC evaluated under three sequences and MSU-PID adapted into six sequential splits.

B. Comparable Methods

The study compares rehearsal-based, gradient-based, regularization-based, and non-continual learning methods using task-wise train-test evaluation. Baselines include ER, DER++, GEM, AGEM, and other approaches with specified memory configurations.

  • B. Comparable Methods: Compared continual-learning methods include ER, DER++, GEM, AGEM, EWC, and AVQACL, alongside non-CL cumulative and joint-training references.
  • B. Comparable Methods: ER and DER++ use replay buffers, whereas GEM and AGEM regulate interference through gradient projection using stored samples.
  • B. Comparable Methods: The train-test matrix evaluates performance on every task after sequential training through six tasks.

C. Evaluation metrics

Evaluation measures overall continual-learning performance, forgetting, and knowledge transfer using AMSE, BWT, and FWT. The train-test matrix exposes the trade-off between acquiring new knowledge and retaining earlier performance.

  • C. Evaluation metrics: The train-test matrix records MSE on each test task after training through each successive task.
  • C. Evaluation metrics: This matrix captures the stability-plasticity trade-off between retaining prior knowledge and acquiring new knowledge.
  • C. Evaluation metrics: AMSE is computed across all tasks after completion of the final task to measure overall performance.
  • C. Evaluation metrics: BWT and FWT quantify forgetting and knowledge transfer, respectively, with lower values indicating better performance.

D. Computational Complexity and Scalability

MCLC uses bounded replay memory to support sequential adaptation with modest computational demands. Across MMLC and MSU-PID, it achieves strong continual-learning performance, stable task-wise errors, and reliable predictions despite difficult multimodal inputs.

  • D. Computational Complexity and Scalability: 13.06 minutes per task and 3.75 GB peak GPU memory are required with buffer size 80, while inference takes 10.68 ms per sample.
  • D. Computational Complexity and Scalability: MCLC achieves the lowest AMSE across all three MMLC sequences: 0.675±0.027, 0.542±0.069, and 0.745±0.057.
  • D. Computational Complexity and Scalability: On MSU-PID, MCLC achieves the lowest continual-learning AMSE of 0.664±0.071 and lowest FWT of 0.979±0.104.
  • D. Computational Complexity and Scalability: Task-wise AMSE generally decreases as tasks accumulate, with MCLC maintaining low errors and stable knowledge retention across sequences.
  • D. Computational Complexity and Scalability: Qualitative examples show that RGB, thermal, and depth cues support accurate counts, while small plants, uneven illumination, and low-contrast modalities increase errors.

VI. ABLATION STUDY

The ablation study selects regression for leaf counting, evaluates multimodal fusion and task-order effects, and examines how replay memory and hyperparameters affect continual-learning performance.

  • A. Choice of Leaf Counting Paradigm: Regression achieves the lowest Train MSE of 0.34 and Test MSE of 1.08 on CVPPP, so it is selected as the core leaf-counting paradigm.The comparison is conducted among segmentation, regression, density estimation, and object detection in a non-CL RGB setting.
  • C. Impact of Performance across Task Sequences: Task ordering significantly changes continual-learning performance, with most methods varying across CS1, CS2, and CS3.CS1 groups tasks by crop type and produces gradual domain transitions; CS2 groups them by capture time and introduces greater variation across crops.
  • C. Impact of Performance across Task Sequences: Rehearsal-based methods provide greater stability and lower errors across sequences, whereas EWC is highly sensitive to task order and distributional shifts.In CS1, ER obtains an AMSE of 0.843±0.090, compared with MCLC’s 0.675±0.027 and EWC’s 3.110±1.626.
  • D. Effect of Memory Size and Hyperparameters: Increasing replay memory generally improves performance, but representative sample selection matters as much as buffer size.ER’s CS1 AMSE decreases from 1.001±0.070 with a buffer of 30 to 0.843±0.090 with a buffer of 80; MCLC remains competitive at moderate sizes and improves further at 80.

E. Evaluating Generalization Across Datasets

MCLC consistently outperforms baseline methods across multimodal continual leaf-counting task orders and shows similar trends on the additional MSU-PID dataset. Its performance remains subject to the assumption that aligned RGB, depth, and thermal modalities are available during training and inference.

  • MSU-PID generalization: On MSU-PID, rehearsal methods generally improve as buffer size increases, with GEM and A-GEM reaching 0.869±0.052 and 0.813±0.084 at B=10×8.These results are reported in Table X under varying hyperparameter settings.
  • MSU-PID generalization: On MSU-PID, EWC remains between 0.882±0.074 and 0.896±0.135, DER++ performs significantly worse, and AVQACL does not improve consistently with larger buffers.These patterns indicate differing robustness among continual-learning strategies under changing data distributions.
  • Overall results: 0.675±0.027, 0.542±0.069, and 0.745±0.057 are MCLC's lowest AMSE values across the three task orderings, averaged over three random seeds.MCLC also maintains low BWT and FWT values, indicating knowledge retention and forward transfer across task orders.
  • Overall results: MCLC consistently outperforms EWC, ER, GEM, and A-GEM, particularly with larger buffer sizes.The comparison covers both regularization-based and memory-based continual-learning baselines.
  • Limitations and future work: MCLC assumes aligned RGB, depth, and thermal modalities are available during both training and inference, limiting deployment when sensors are missing, noisy, or unavailable.The paper identifies missing-modality handling and more memory-efficient buffers as future-work directions.
Loading 2609.18129v1…