PaperlayerView exact revision on arXiv

Source-linked AI summary

LettuceVisSim: A Simulator That Generates Lettuce Image Time-series for Vision-Based Reinforcement Learning

Ziye Zhu, Bert van 't Ooster, Congcong Sun, Eldert van Henten, Sjoerd Boersma

arXiv:2609.12505v1cs.CVcs.AIcs.SYeess.SYmath.OC

TL;DR

Vision-based RL in CEA is constrained by scarce labelled crop images, despite the potential of image-based crop-state observation. LettuceVisSim links a lettuce growth model to canopy layouts and rendered images; its reported validations support the simulator as a data generator across dynamic cultivation conditions.

  • Problem

    Biomass-based control assumes dry-weight measurements, which are infrequent and inaccurate, while vision-based RL requires large labelled crop-image datasets that are generally unavailable in agriculture.

    Source: §1 · p. 2

  • Method

    LettuceVisSim combines a process-based lettuce biomass model, a canopy layout algorithm, and a Unity renderer to generate labelled top-view crop-image time series.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    R2 = 0.84 for PBM shoot dry weight prediction on unseen cultivation strategies; PPA mapping reached R2 = 0.94, and canopy layout was validated across 12 unseen strategies.

    Source: §5 · p. 32

  • Takeaways & Limitations

    LettuceVisSim provides a simulator for generating labelled crop-image time series for vision-based RL in CEA.

    Source: §1 · p. 2, §5 · p. 32

    The PBM and PPA mapping were each based on a single lettuce cultivar, so generalization to other cultivars remains unestablished.

    Source: §4.1 · p. 28, §4.2 · p. 29

Abstract

from arXiv · show

Vision-based reinforcement learning holds strong potential for decision-making in controlled environment agriculture (CEA). However, its development is hindered by the scarcity of labelled crop images. To address this gap, LettuceVisSim, a lettuce growth simulator that generates labelled time series of crop images, was developed and validated. The simulator contains a process-based model (PBM) for shoot dry weight dynamics, a canopy layout algorithm for deriving canopy layout representations from shoot dry weight, and a Unity rendering engine for image generation. Five findings support the simulator. First, the PBM reproduced shoot dry weight under dynamic plant-density management with $\mathrm{R}^{2}=0.84$. Second, a piecewise cubic regression mapped shoot dry weight to potential projected area with $\mathrm{R}^{2}=0.94$. Third, the canopy layout representation was validated using 12 experimental datasets each having different dynamic environmental and spacing conditions. It reproduced the ground coverage ratio dynamics observed in measured images, achieving $\mathrm{R}^{2}=0.84$ when driven by measured shoot dry weight and $\mathrm{R}^{2}=0.40$ (0.76 excluding one outlier) when driven by PBM-simulated values. Fourth, the Unity rendering engine converted canopy layout representations into RGB and segmentation images at less than 10~ms. Fifth, a demonstration showed that a lighting-control policy can be learned and applied by observing only crop images that were generated with LettuceVisSim, providing a proof of concept of vision-based reinforcement learning in CEA using LettuceVisSim.

1. Introduction

CEA requires climate-control setpoints to optimize crop growth, while vision-based RL offers a non-destructive alternative to biomass-based control but lacks large labelled crop-image datasets. LettuceVisSim addresses this gap through a biomass-to-canopy-to-image pipeline, evaluated across five research questions.

1. Introduction

  • CEA grows crops under artificially regulated indoor conditions, and optimizing growth requires setting climate-control setpoints.CEA includes greenhouse and plant-factory production systems.Source: §1 · p. 1
  • Biomass-based control commonly assumes measured dry weight, but destructive sampling is infrequent and inaccurate; image-based RL can learn from crop images but needs extensive labelled data.Crop images are non-destructive and can support estimation of a biomass proxy over time.Source: §1 · p. 2
  • Existing crop models do not provide efficient, labelled crop-image time series linked to traits such as shoot dry weight under dynamic conditions.PBMs cannot generate images; FSPMs may be static or computationally expensive, while diffusion models are slow and lack an explicit pixel-to-trait link.Source: §1 · p. 2
  • LettuceVisSim combines a lettuce PBM, a canopy layout algorithm, and a Unity renderer to generate labelled top-view crop images over time.The PBM represents biomass responses to environmental variables and spacing management; the layout maps shoot dry weight to canopy representation, which the renderer converts to images.Source: §1 · pp. 2–3
  • The adopted Van Henten lettuce model was developed and validated at fixed plant density, so its validity under CEA’s dynamic spacing management required examination.The model takes plant density as an input, while CEA routinely reduces density as the canopy expands.Source: §1 · p. 3
  • The canopy algorithm maps shoot dry weight to potential projected area, then combines area and plant density on a uniform grid to approximate canopy coverage.Its layout permits ground coverage ratio calculation in the same way as for measured images.Source: §1 · p. 3
  • The study’s five questions cover dynamic-density biomass prediction, shoot-dry-weight-to-area mapping, canopy coverage fidelity, rendering performance, and image-only RL training.The questions assess the simulator’s component models and whether its generated images can support vision-based RL in principle.Source: §1 · p. 3

2. Material and methods

This section describes LettuceVisSim’s biomass simulation, canopy-layout construction, and Unity image rendering, as well as the datasets and experimental procedures used to develop and validate the simulator. It also covers evaluation of the PPA mapping, canopy-layout representation, rendering engine, and image-based lighting-control demonstration, together with the computational environment.

2.1. Simulator overview

  • LettuceVisSim combines a process-based model (PBM), a canopy layout algorithm, and a Unity rendering engine to generate synthetic crop images from simulated biomass.The canopy layout algorithm and rendering engine are newly proposed; the PBM was previously validated and applied in control studies.Source: §2.1 · pp. 3, 5

2.1.1. Process based model

  • The PBM represents lettuce shoot dry weight with a discrete-time ODE integration using forward Euler and a 5-minute growth time step.Air temperature, incident radiation, CO2 concentration, and plant density are inputs; image sampling is independently configurable but cannot be faster than the growth step.Source: §2.1 · p. 3, §2.1.1 · p. 5

2.1.2. Canopy layout algorithm

  • The canopy layout maps shoot dry weight to projected plant area, then represents the 1 m2 top-view canopy as equal-radius circles positioned on a regular grid.The construction assumes uniform plant distribution, includes partially visible boundary plants, and models canopy closure as geometric overlap without contact-driven deformation or adaptation.Source: §2.1.2 · pp. 5–6

2.1.3. Unity rendering engine

  • Unity reconstructs the canopy from its layout representation and renders top-view RGB images and binary segmentation masks with a virtual camera.Plant appearance uses prefabricated models with three leaves at transplanting, increasing by one leaf daily to a cap of 34; models are scaled by the layout radius, with optional small rotation and scale variation.Source: §2.1.3 · pp. 7–8

2.2.1. Data description

  • Development and validation used two public lettuce datasets collected in Bleiswijk, Netherlands, in 2021 and 2022.The 2021 dataset contains RGB-D images and shoot dry weights for individual plants, including cv. Lugano; the 2022 dataset contains hourly canopy images and synchronized climate, spacing, and weekly dry-weight records across 12 cultivation strategies.Source: §2.2.1 · p. 9

2.2.2. Data selection and processing

  • Processing selected cv. Lugano plants and screened canopy images for visibility before deriving projected area and ground-coverage time series.The 2021 selection yielded 96 shoot-dry-weight/PPA pairs; for 2022, a PAR threshold above 5 µmol m−2 s−1 retained about 40% (4,942) of images, and a DeepLabv3+ model trained on 131 annotated images achieved 93.5% test mIoU.Source: §2.2.2 · pp. 9–11

2.3. Experiment description

  • The five experiments addressed PBM validation, shoot-dry-weight-to-PPA model selection, canopy-layout validation, rendering diversity and efficiency, and vision-based reinforcement learning.Experiment 1 used six-fold leave-strategy-out cross-validation across 12 strategies, calibrating the five most influential PBM parameters on training strategies and testing on held-out strategies.Source: §2.3 · p. 12, §2.3.1 · p. 12

2.3.2. Experiment 2: model selection for the shoot dry weight–PPA mapping

  • Experiment 2 compared one-phase regression models with a proposed two-phase piecewise cubic regression for mapping shoot dry weight to PPA.The piecewise model uses two cubic segments joined at a breakpoint with continuity in the function value and first derivative.Source: §2.3.2 · p. 13
  • Five-fold cross-validation compared PPA mapping models and breakpoint settings, including stage-specific performance, before the best model was refitted on the complete dataset.The two-phase model was compared with one-phase models at a fixed 0.7 g breakpoint, then evaluated with breakpoints from 0.3 to 1.1 g in 0.2 g increments.Source: §2.3.2 · p. 14

2.3.3. Experiment 3: validation of the canopy layout representation

  • The canopy-layout representation was assessed against image-derived ground coverage ratio using both measured and PBM-simulated shoot dry weight inputs.Measured-input comparisons used daily averaged image-derived GCR on destructive-sampling days; PBM-driven time series used leave-strategy-out cross-validation predictions and were compared with image-derived GCR at matched hourly times.Source: §2.3.3 · p. 15

2.3.4. Experiment 4: evaluation of the Unity rendering engine

  • Rendering diversity was tested by varying lighting, image resolution, and randomized plant rotations, while computational efficiency was measured across pipeline stages, sampling intervals, and resolutions.Five independent runs were conducted per full-pipeline configuration; the evaluation used a 30-day scenario with climate and spacing records from strategy cva1.Source: §2.3.4 · pp. 15–16

2.3.5. Experiment 5: proof of concept of vision-based reinforcement learning with LettuceVisSim

  • The control demonstration tasked a vision-based agent with selecting daily light intensity to reach a 15–17 g plant−1 terminal harvest band over 30 days without biomass decreases.The agent's sole observation was an 84 × 84 top-view canopy RGB image; PPO (Schulman et al., 2017) used an image-dependent actor and a critic that additionally received shoot dry weight during training. The reward combined daily biomass gain, penalties for decreases, and terminal-band reward or deviation penalty.Source: §2.3.5 · pp. 16–17

2.3.6. Experiment environment

  • Experiments ran on a ThinkPad P15 Gen 2i with Pop!_OS, using Python-based simulation and analysis and a headless Unity rendering engine.The system had an Intel Xeon W-1885M CPU, 32 GB RAM, and an NVIDIA RTX A5000 laptop GPU; software included Python 3.10, NumPy 1.26, OpenCV 4.9, PyTorch 2.0, SciPy 1.15.3, and Unity 6000.0.48f1.Source: §2.3.6 · p. 17

3. Results

The experiments evaluate LettuceVisSim’s biomass prediction, shoot-weight-to-area mapping, canopy-layout validation, rendering, and image-based lighting-control demonstration. Results show strong agreement in several stages, with errors concentrated in particular strategies and rendering cost increasing at finer sampling and higher resolution.

3.1. Experiment 1

  • The PBM achieved test agreement of R2 = 0.836 ± 0.131 (RRMSE 38.6 ± 17.1%), comparable to train agreement in six-fold validation.Test R2 varied from 0.624 to 0.955; veggie-might1 and koala2 formed the weakest fold, with test R2 = 0.624 and RRMSE = 70.0%.Source: §3.1 · p. 17, Table 2 · p. 18

3.2. Experiment 2

  • Across five-fold cross-validation, the two-phase model achieved the best average overall PPA performance, with R2 = 0.94 and RRMSE = 22%.Its clearest advantage was early-stage performance (R2 = 0.78, RRMSE = 29%); late-stage performance was closer to that of top one-phase models, and wb = 0.7 g was selected.Source: §3.2 · pp. 19–21, Table 3 · p. 20

3.3. Experiment 3

  • With measured shoot dry weight, simulated GCR achieved R2 = 0.84 across 12 strategies, with RRMSE ranging from 4.2% to 36.3%.Excluding veggie-might1 raised overall performance to R2 = 0.93, with mean RRMSE of 9.4%.Source: §3.3 · p. 22, Figure 8 · p. 23
  • With PBM-simulated shoot dry weight, mean GCR agreement was R2 = 0.40 and mean RRMSE was 16.8% across strategies.Excluding veggie-might1 raised average performance to R2 = 0.76 and mean RRMSE = 13.8%, below the measured-weight case’s R2 = 0.93.Source: §3.3 · p. 23, Figure 9 · p. 24

3.4. Experiment 4

  • Lighting and background changes varied rendered appearance without changing canopy geometry, while plant-level randomisation changed geometry and corresponding segmentation masks.Rendered and measured images showed minor spatial-arrangement discrepancies attributed to the layout’s uniform-grid assumption.Source: §3.4 · pp. 24–25, Figure 10 · p. 25, Figure 11 · p. 26
  • Rendering reproduced major temporal image trends, while runtime increased with finer sampling and higher resolution, reaching about 60 ms per frame at the highest tested resolution under hourly sampling.At hourly rather than daily sampling, rendering time per frame increased about 22 times; the PBM contribution remained comparatively small.Source: §3.4 · pp. 24, 26, Figure 10 · p. 25, Figure 12 · p. 27

3.5. Experiment 5

  • From crop images, the trained lighting policy reached a maximum episode return of about 5.09 and produced 16.1 g · plant−1 shoot dry weight within the harvest band.Training converged after around 100k steps; the reported outcome is from a single rollout.Source: §3.5 · pp. 27–28, Figure 13 · p. 28

4. Discussion

The discussion finds that the simulator’s canopy-level agreement depends on both the quality of predicted biomass and the simplified layout representation. Its components support modular development, but the reported validity remains conditional on crop, environment, and imaging assumptions.

4.1. Validation of the process-based model under dynamic plant density

  • Test R2 = 0.836 ± 0.131 versus train R2 = 0.873 ± 0.031 supports using the Van Henten PBM under dynamic density within the examined cultivars and conditions.Agreement varied across strategies, with veggie-might1 showing the weakest test agreement and later-stage deviation from measurements.Source: §4.1 · p. 28

4.2. Modelling choices for the PPA–shoot dry weight mapping

  • The two-phase PPA mapping’s clearest advantage was early growth, where the developmental relationship appears to change around its breakpoint.Its numerical advantage over one-phase models should be interpreted cautiously because the comparison used average cross-validation metrics without statistical testing.Source: §4.2 · pp. 28–29

4.3. Validity of the canopy layout representation

  • The layout’s validity is functional rather than geometric: circular projections and uniform spacing omit plant-level heterogeneity and increasingly diverge from measured masks as cultivation progresses.The representation is most suitable for controlled CEA with regular planting and relatively homogeneous canopy development.Source: §4.3 · pp. 29–30
  • R2 = 0.84 with measured shoot dry weight and R2 = 0.40 with PBM-simulated inputs; excluding veggie-might1 raised these to 0.93 and 0.76, respectively.The downstream agreement therefore depends on the quality of its shoot dry weight input.Source: §4.3 · p. 30
  • The representation generalised to unseen dynamic conditions: PPA parameters were validated on a separate dataset, and PBM-driven trajectories came from leave-strategy-out cross-validation.The evaluation covered both the layout representation alone and its use in the full pipeline.Source: §4.3 · p. 30
  • Before canopy closure, shoot dry weight errors propagate into GCR; after closure, saturation makes idealised geometry and retained canopy gaps more influential.Koala2 had accurate predicted shoot dry weight but markedly reduced downstream GCR agreement, pointing to representation-level effects.Source: §4.3 · p. 30

4.4. Implications and future work

  • LettuceVisSim’s modular components preserve a link between crop state and visual output while enabling error attribution and component replacement.Future work includes broader cultivar and environmental coverage, improved imaging validation, and exploring vision-based RL under more realistic control settings.Source: §4.4 · pp. 31–32

5. Conclusion

LettuceVisSim generated labelled lettuce image time series and was validated across its modelling, canopy-layout, rendering, and image-based control components.

5. Conclusion

  • R^2 = 0.84 for PBM shoot dry weight predictions under dynamic plant-density management on unseen cultivation strategies.The two-phase piecewise cubic regression mapped shoot dry weight to potential projected area with R^2 = 0.94.Source: §5 · p. 32
  • R^2 = 0.84 with measured shoot dry weight and R^2 = 0.40 with PBM-simulated values for canopy-layout validation across 12 unseen strategies.Excluding one outlier strategy raised the simulated-value result to R^2 = 0.76.Source: §5 · p. 32
  • Below 10 ms per frame at low resolution, the Unity rendering engine generated images from canopy-layout representations.Source: §5 · p. 32
  • A lighting-control policy was learned and operated by observing crop images without biomass estimation, providing a proof of concept for vision-based RL in CEA.Source: §5 · p. 32

Declaration of competing interest

The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.

Declaration of competing interest

CRediT authorship contribution statement

The statement assigns conceptual, methodological, supervisory, analytical, software, validation, visualization, and writing contributions across five authors.

CRediT authorship contribution statement

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The authors disclose using ChatGPT to check grammatical errors during manuscript preparation and state that they reviewed and edited the content.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

Appendix A. Process-based model equations

Appendix A collects the PBM equations and parameter information used in the study. It also specifies a temperature-dependent expression for gcar.

Appendix A. Process-based model equations

  • The appendix summarizes the PBM governing equations and directs readers to Table A.1 for parameter definitions, values, and units.The parameter set is denoted p.Source: Appendix A · p. 33, Table A.1 · p. 34
  • The PBM parameter set is denoted p, and its individual definitions, values, and units are listed in Table A.1.Source: Appendix A · p. 33, Table A.1 · p. 34
  • gcar is expressed as a quadratic function of temperature, with coefficients −1.32 × 10^-5, 5.94 × 10^-4, and −2.64 × 10^-3.The equation is labeled (A.10).Source: Appendix A · p. 33

Appendix B. DeepLabv3+ training and validation performance

The segmentation model achieved 93.5% mIoU on the held-out test set. The appendices also show training-performance curves and relate plant density to visible lettuce counts.

Appendix B. DeepLabv3+ training and validation performance

  • 93.5% mIoU was obtained on the held-out test set for the segmentation model.The metric is expressed as a percentage and is defined using the lettuce and background class masks.Source: Appendix B · p. 34
  • Figure B.1 plots training and validation loss and validation mIoU over training epochs.Source: Appendix B · p. 34, Figure B.1 · p. 35

Appendix C. Plant density and annotated image-based lettuce counts

  • Figure C.1 shows temporal plant-density changes alongside visible plant counts in the camera field of view for each of 12 cultivation strategies.The visible counts are derived from annotated images.Source: Appendix C · p. 35, Figure C.1 · p. 36

Appendix D. Evaluation metrics

The study quantifies predictive performance using R2 and RRMSE. The observed and predicted values used in these metrics vary by experiment.

Appendix D. Evaluation metrics

  • Predictive performance is quantified using the coefficient of determination (R2) and relative root mean squared error (RRMSE).Source: Appendix D · p. 36
  • The roles of observed and predicted values differ by experiment, with the specific variables defined in each experiment’s description in Section 2.3.Source: Appendix D · p. 37

Appendix E. Vision-based reinforcement network architecture and training details

The RL demonstration uses an asymmetric actor–critic network that takes crop images as observations. The appendix identifies the PPO hyperparameter table used for training.

Appendix E. Vision-based reinforcement network architecture and training details

  • The actor encodes an 84 × 84 RGB image into a 256-dimensional feature vector and outputs a Gaussian over actions rescaled to 0–360 µmol m−2 s−1.The actor depends on the image alone.Source: Appendix E · p. 37
  • The critic shares the image encoder, additionally receives privileged true shoot dry weight during training, and outputs a scalar state-value estimate.Source: Appendix E · p. 37
  • The full PPO hyperparameter set is listed in Table E.1.Source: Appendix E · p. 37, Table E.1 · p. 37

Appendix F. Per-strategy evaluation metrics

Appendix F lists per-strategy evaluation metrics for the model and canopy-layout evaluations shown in Figures 5, 8, and 9. The tables distinguish measured from PBM-simulated shoot dry weight inputs.

Appendix F. Per-strategy evaluation metrics

  • Table F.1 reports per-trajectory agreement between PBM-simulated and measured shoot dry weight across 12 test trajectories.Each trajectory uses fold-specific parameters from the cross-validation run that held out its strategy; n counts destructive sampling occasions.Source: Table F.1 · p. 38
  • Table F.2 reports per-strategy agreement between simulated and image-derived ground coverage ratio using measured shoot dry weight across 12 cultivation strategies.n counts measurement days for each strategy.Source: Table F.2 · p. 38
  • Table F.3 reports per-strategy agreement between simulated and image-derived ground coverage ratio using PBM shoot dry weight across 12 cultivation strategies.Each strategy uses fold-specific parameters from the cross-validation run that held it out; n counts individual images.Source: Table F.3 · p. 39

Related papers

Loading 2609.12505v1…