A Vision-Language Model (VLM)-based Pipeline for End-to-End Procedural Modeling of Field-Grown Maize from Point Clouds

• A zero-shot VLM annotates leaf midlines in rendered point-cloud views.Source: Highlights · p. 2
• 99.4% of curated reference leaves were recovered at an intersection-over-union of at least 0.5.Source: Highlights · p. 2
• Measured organ traits populate an editable procedural model without manual tuning, and differentiable NURBS refinement fits each leaf to its scan points.Source: Highlights · p. 2
• 5.4 mm was the median whole-plant Chamfer distance across 100 field-grown maize plants.Source: Highlights · p. 2

• Accurate 3D crop models support phenotyping, functional-structural plant modeling, and agricultural digital twins, but constructing them from field sensor data remains challenging.Source: §1 · p. 3
• Scanned or image-reconstructed plant geometry is unstructured and may be noisy, occluded, or unevenly sampled, while trait extraction requires organ-level structure.Source: §1 · p. 4
• Procedural models provide interpretable, editable parameters, but manual descriptor tuning has been reported to take 2 to 6 hours per plant.Source: §1 · p. 4
• Supervised segmentation methods require substantial labels that must be recollected when species, growth stage, or acquisition setup changes.Source: §1 · p. 4
• The render-segment-back-project paradigm had not been applied to plant phenotyping, where overlapping organs and fine instance boundaries complicate metric measurements.Source: §1 · p. 4
• Measured organ parameters initialize NURBS refinement, avoiding the earlier stochastic search while keeping control points tied to organ features.Source: §1 · p. 5
• Existing approaches leave gaps: some return organ labels without parameters, others lack parameter-wise editability, and canopy-scale recovery does not produce a single-plant organ descriptor.Source: §1 · p. 5
• The proposed pipeline reconciles multi-view VLM annotations into 3D leaves, measures organ parameters for procedural generation, and refines leaf surfaces from their scan points.Source: §1 · p. 5

2.1. Dataset

• The evaluation processes 100 field-grown MaizeField3D plants with one fixed parameter set and no per-plant tuning.Ten distinct genotypes spanning plant-height, leaf-angle, and canopy-compactness variation are also reported leaf by leaf.Source: §2.1 · p. 7

2.2. Preprocessing

• Each scan is converted to a centimeter-scale, canonical frame with its estimated stalk base at the origin.Horizontal orientation is set by principal-component analysis, and the base is estimated from the lowest points in a central vertical core.Source: §2.2 · pp. 7–8

2.3. Multi-view orthographic rendering

• The pipeline renders each preprocessed cloud from eight orthographic views, preserving organ scale and reusing each camera’s pixel-to-world mapping for back-projection.The views include front, back, and six side viewpoints.Source: §2.3 · p. 8, Figure 3 · p. 9

2.4. Per-view VLM-based segmentation

• Zero-shot VLM prompts generate per-view leaf midlines and node markers; geometric checks validate outputs, while accepted annotations are reviewed using rendered views alone.Failing images are retried, and overlay extraction separates colored strokes and magenta node dots using HSV thresholds and shape filters.Source: §2.4 · pp. 9–11

2.5. Mask extraction and skeleton-based leaf separation

• Skeletonization and tangent-based regrouping separate crossing leaf strokes and reconnect fragments that a plain connected-component analysis would mishandle.The regrouped arms receive a shared leaf identifier when their outward tangents continue smoothly through a junction.Source: §2.5 · p. 11

2.6. Per-view back-projection and cross-view consensus

• Back-projected point labels are reconciled by mutual-best matches with at least 40 shared points, and leaves supported by fewer than two views are discarded.Consensus also splits supported bridged leaves when node dots form separated height clusters and the points do not continue smoothly across the boundary.Source: §2.6 · pp. 12–13

2.7. Blade-level segmentation refinement and node repair

• Consensus midlines seed full-blade propagation over an orientation-weighted surface graph, whose edge costs discourage transitions between touching leaves.A seed-consistency gate suppresses contradictory crossing seeds before Dijkstra labeling; the stage also corrects supported blade-fragment assignments and determines final nodes.Source: §2.7 · pp. 13–14
• A stalk centerline is traced as a radial-penalized shortest path between back-projected endpoint anchors, then fitted with a cubic B-spline.If endpoint recovery fails, the pipeline estimates a geometric centerline from the full cloud; the result supports blade propagation and node projection.Source: §2.7 · p. 14
• Initially unlabeled points remain excluded from measurements, while hysteresis changes a connected group of at least eight labeled points only when the alternative improves cost by at least 8 cm.This avoids forcing background, tassel, or unsupported material into leaves.Source: §2.7 · p. 15
• Final attachment nodes must lie within 6 cm of their leaf bases and at least 4 cm from neighboring nodes, with inadmissible nodes repaired only when constraints remain satisfied.Missing nodes and nodes on both leaves involved in a split are projected; other nodes are preserved unless a valid replacement can be found without making an admissible neighbor inadmissible.Source: §2.7 · p. 15
• Ellipsoid-chain envelopes test whether whole blade fragments fit their assigned leaves, supplementing pointwise assignment with a leaf-level geometric criterion.Each leaf is represented by 18 equal-arc-length patches; patch widths and thicknesses use trimmed 90th-percentile offsets, and points with envelope score σ ≤1.5 are treated as inside.Source: §2.7 · pp. 15–16
• Fragment moves and splits require geometric support from destination midlines, surface proximity, tangent alignment, and envelope scores, with ambiguous cases left unchanged.Touching-leaf components can be split by moving only points within a 25° cone along a destination endpoint tangent; small unsupported components may remain unassigned.Source: §2.7 · pp. 16–17
• Corrections are verified against the current point set and accepted only if the leaf partition, nodes, and attachment order remain valid.Moves are applied transactionally and rolled back as a set if validation fails; a selected split must pass its own validation before publication.Source: §2.7 · p. 17
• After corrections, each accepted leaf’s midline is rebuilt as a geodesic base-to-tip path on its full-blade cloud, and leaves are renumbered by node position along the stalk.The graph uses k = 12 nearest neighbors, discards edges longer than five times the median nearest-neighbor distance, and samples the spline at a minimum of 200 ordered points.Source: §2.7 · p. 17

2.8. Geometric parameter extraction and procedural model generation

• Plant-specific descriptor parameters come from corrected full-blade clouds and rebuilt midlines; leaves lacking a required full-blade cloud are omitted rather than inferred from sparse midlines.The descriptor records whether each width profile was measured or derived from a scalar summary.Source: §2.8 · pp. 18–19
• A distance-weighted graph and Dijkstra path recover each leaf midline, with adaptive edge caps and minimum bridge edges preventing fold shortcuts while reconnecting scan gaps.If graph construction or path recovery fails, a greedy proximity-and-progress walk is used instead.Source: §2.8 · p. 18
• A path-start repair runs only when the start-to-base gap is at least 2 cm and backtracking is at least 1 cm, and removes no more than 20% of the path.It anchors the first sample at the trusted leaf base and tapers displacement to zero over 5 cm.Source: §2.8 · p. 18
• Leaf width is recorded as an ordered profile along control points, while orientation comes from the proximal midline tangent and stalk diameter is measured per internode.The profile preserves finite measured values rather than replacing them with smoothing or scalar taper; unsigned pitch does not erase rise or descent in the 3D control points.Source: §2.8 · pp. 18–19
• FloraForge generates stalks by sweeping interpolated radii along the stalk centerline and leaves as NURBS surfaces shaped by the measured width profile.Measured widths are interpolated onto 11 blade-length rows; only leaves without a measured profile use analytical taper.Source: §2.8 · p. 19

2.9. Differentiable NURBS refinement from procedural initialization

• Each leaf’s procedural NURBS surface is refined against up to 4,000 corrected blade points for 500 Adam iterations, while its node-side control row remains fixed.Numerical failures trigger a restart from initialization, with at most three attempts; leaves that fail all attempts are not promoted.Source: §2.9 · p. 19
• The refinement objective combines one-sided target-to-surface Chamfer distance with surface and control-net regularizers because self-occluded blade regions lack scan support.The regularizers constrain unsupported regions; the transverse-compactness term limits spreading beyond blade edges.Source: §2.9 · p. 20

2.10. Full-plant assembly and export

• A plant is accepted for cohort results only when every expected leaf succeeds and its fidelity metrics, output files, and checksums pass validation.The report stage checks these requirements for all 100 evaluated plants.Source: §2.10 · p. 22

3.1. Qualitative results

• Visual examples showed that refined models preserved overall plant architecture, leaf arrangement, blade curvature, and attachment order, while view annotations sometimes omitted or merged upper-canopy leaves.The visual assessment showed annotations, segmentation, initial procedural models, and refined models; mask and consensus stages corrected some annotation errors when other views or curve continuity supported them.Source: §3.1 · p. 22, Figure 7 · p. 23

3.2. Organ-level recovery

• Eight-view consensus reconciled 890 per-view components into 117 global leaves, and segmentation recovered one more, for 118 leaves across ten plants.Of 936 possible leaf-view pairs, 835 (89.2%) were annotated; consensus excluded 41 unsupported components and reconciled 14 fragments, while four leaves had only the two-view minimum support.Source: §3.2 · p. 24, Table 2 · p. 24
• The ten-plant segmentation left 8.0% of non-stalk points unlabeled, while four leaves had only two-view support and were among the smallest consensus clusters.The unlabeled fraction was highest on Kui2021 (15.1%) and A679 (14.7%); the two-view leaves occurred at canopy extremes, supporting retention of the two-view floor and conservative correction gates.Source: §3.2 · pp. 24–25
• The constrained node rule preserved annotation-derived nodes on 114 leaves, and 116 of 118 attachments (98.3%) met both placement conditions.Two CML331 leaves were projected after a split; CML69 leaf 14 and CML331 leaf 12 retained reported gap violations, with no manual node correction. The automatic fragment tests rejected 579 candidates and accepted two corrections, both on CML331.Source: §3.2 · pp. 24–25, Table 3 · p. 26

3.3. Geometric fidelity

• Across 118 leaves, the median fitted-surface distance to observed blade points was 5.20 mm, while the eight-point B-spline midline reconstruction RMSE was 1.7 mm.In a random-fifth holdout rerun, mean RMS was 5.64 mm on withheld points and 5.30 mm on fitted points; this internal check used points from the same segmented clouds.Source: §3.3 · pp. 25–26, Table 4 · p. 26
• Exported-surface and segmented-cloud traits agreed closely for blade length, node height, and azimuth, while vertical insertion angle had a larger disagreement tail.Blade length differed by a mean absolute 2.8 cm (R2 = 0.962), node height by 1.3 cm (R2 = 0.999), and angle cosines were 0.9932 for vertical insertion and 0.9995 for azimuth. Independent physical trait measurements were unavailable, so this was an internal consistency assessment.Source: §3.3 · p. 27, Figure 9 · p. 29

3.4. Results across the 100 plants

• 5.4 mm was the median whole-plant symmetric Chamfer distance across 100 plants, with 1,072 leaves recovered and a 4.3 mm median blade-level residual.The whole-plant median had a 95% bootstrap confidence interval of 5.13 to 5.52 mm; the blade-level median interval was 4.13 to 4.47 mm.Source: §3.4 · pp. 27–28, Figure 10 · p. 30

3.5. Behavior of the geometric checks

• Geometric checks removed folds on 14 of 28 folded leaves after local-roll refitting, while 14 of 109 screened leaves retained a fold signal; threshold sweeps found the prealignment margin most consequential.Loosening that margin from 0.08 to 0.04 admitted two rotations that worsened leaf distances to 14.87 mm and 6.68 mm, supporting retention of 0.08; retained fold signals do not guarantee local non-self-intersection.Source: §3.5 · pp. 29–31

3.7. Computational cost

• 82 s versus 2740 ± 180 s for procedural versus PSO initialization yielded a reported 33-fold speedup, while refinement was roughly 5 times faster.The historical and proposed timings came from different hardware and scopes, so these ratios describe reported workloads rather than controlled speedups. The five local stages took 601 ± 55 s per plant; adding the pooled 214 s annotation mean gave an approximate 815 s total.Source: §3.6 · p. 31, Table 6 · p. 32, §3.7 · pp. 32–33

3.6. Comparison with the PSO+NURBS-Diff pipeline

• 5.4 mm versus 9.2 mm median whole-plant Chamfer distance, with the proposed pipeline closer to the scan on 99 of 100 plants.Both model sets were evaluated under one protocol; this is an end-to-end workflow comparison, not a controlled isolation of initialization, and the historical baseline used curated organ labels. The proposed pipeline also outputs an editable descriptor with per-leaf traits and control points.Source: §3.6 · p. 31, Figure 11 · p. 32

3.8. Validation of the organ segmentation

• 15 of 1,073 consensus leaves depended on exactly two views; removing any single view lost 1–7 leaves, while a six-view rerun recovered 115 of 118.Across the 100 plants, 87 lost no leaf under any single-view removal; removing two of eight views in the rerun reduced recovered organs by 2.5%. This analysis does not establish effects on blade extent or fitted-surface fidelity.Source: §3.8 · pp. 33–34
• Repeated front-view annotations had mean pairwise IoU 0.729, while cross-view consensus retains leaves only when at least two views support them.The repeat test covered four plants and annotation variability alone; the complete eight-view workflow was not measured. A second-model probe produced partial curves and misplaced components, but its outputs were not tested downstream.Source: §3.8 · p. 34
• The exported total was 94.4% of the manual count, and the upper canopy was the weakest region for leaf recovery.On ten plants, manual counting found 125 leaves versus 118 exported; common missed-leaf conditions involved small uppermost leaves or sparse or twisting blades. The count reflects rendered scans, not completeness against the plant, and accepting all 41 single-view candidates would lack cross-view corroboration.Source: §3.8 · p. 34, Table 8 · p. 35
• 1,017 of 1,023 curated leaves were recovered at IoU ≥0.5, a 99.4% detection rate, with mean matched-leaf IoU of 0.948.Per-point recall and precision were 97.3% and 96.7%, respectively. These results use reviewed annotations and independent organ labels, not labels used for task-specific training or pipeline input; they do not establish an accuracy advantage over supervised networks, and generalization remains untested.Source: §3.8 · p. 35

3.9. Future directions

• Future work should validate physical traits, evaluate other plant architectures and acquisition conditions, and assess a reviewed plant-specific VLM with calibrated automated checks.The paper proposes co-registered physical measurements, testing field clutter and occlusion, and an execution layer that can rerun stages or route uncertain cases for human review.Source: §3.9 · p. 36

4. Conclusions

• The pipeline recovered 1,072 leaves across 100 plants, with a 5.4 mm median whole-plant Chamfer distance and closer scans on 99 of 100 plants than the earlier PSO-initialized pipeline.It also matched 1,017 of 1,023 reference leaves (99.4%) at IoU ≥0.5 without using curated labels as input or for task-specific training.Source: §4 · p. 36
• Zero-shot VLM annotations become reliable organ-level measurements when downstream geometric stages reconcile, test, and correct them.The pipeline records applicable geometric-test outcomes so users can identify attachments or surfaces needing manual review.Source: §4 · p. 36
• Validation against co-registered physical measurements remains future work, although traits measured on exported surfaces agreed closely with those measured on segmented blade clouds.The reported 118 exported leaves had a 5.2 mm median blade-level residual, under 1% of the 70.2 cm median blade length.Source: §4 · p. 36

CRediT authorship contribution statement

• Mozhgan Hadadi handled data curation, formal analysis, investigation, software, validation, visualization, and the original draft; the other three authors shared conceptualization, methodology, project administration, supervision, and review.The listed coauthors are Talukder Z. Jubery, Adarsh Krishnamurthy, and Baskar Ganapathysubramanian.Source: CRediT authorship contribution statement · p. 37

Declaration of generative AI and AI-assisted technologies in the writing process

• Claude was used to improve the manuscript’s language, grammar, and readability, and the authors reviewed and edited the resulting text.The authors state that they take full responsibility for the publication’s content.Source: Declaration of generative AI and… · p. 37
• The VLM was part of the reported methodology, while Claude and GPT coding assistants were used during implementation and the authors verified and validated the algorithms, settings, and results.The VLM provider, version, and prompts are reported in Section 2.4.Source: Declaration of generative AI and… · p. 37

Appendix A. Additional methodological details

• The appendix provides the exact VLM prompts, standard NURBS basis and differentiation formulas, conditional base-collar repair, and parameter table for the reported pipeline.Source: Appendix A · p. 42

Appendix A.2. NURBS basis and differentiation

• B-spline basis functions are defined recursively over a non-decreasing knot vector, with degree p and highest control-point index n.A curve has n+1 control points and n+p+2 knots.Source: Appendix A.2 · p. 42
• B-spline surfaces extend curves to two dimensions through tensor products of basis functions over two knot vectors and a control-point grid.The grid has dimensions (n+1) × (m+1).Source: Appendix A.2 · p. 42
• NURBS assign a weight to each control point, enabling representation of standard shapes such as circles and ellipses as well as complex free-form surfaces.The weighted surface uses rational basis functions.Source: Appendix A.2 · p. 42
• NURBS-Diff supplies closed-form derivatives with respect to NURBS parameters for an approximate Jacobian, while this pipeline optimizes control-point positions and holds weights and knot vectors fixed.Following Hadadi et al., only control-point positions are optimized.Source: Appendix A.2 · p. 42

Appendix A.3. Conditional base-collar repair

• The repair applies only to leaves whose path-start repair was used, leaving the fixed node-side row attached while downstream rows remain free.The issue arises when the extracted path begins behind the trusted blade base.Source: Appendix A.3 · p. 43
• For rows 1–3, the method first tests a Hermite center path with tangent scale 0.65 and blended transverse offsets, then tests a linear center path if needed.Row 4 and later rows remain unchanged.Source: Appendix A.3 · p. 43
• A candidate is accepted only if center progression, transverse-edge alignment, and sampled collar-cell orientation pass checks; otherwise the original fit is retained.Post-repair Chamfer is saved for auditing but is not an additional acceptance gate.Source: Appendix A.3 · p. 43

Appendix A.4. Pipeline parameters

• Table A.1 distinguishes four parameter provenance sources: run configuration (P), per-plant blade-refinement configuration (L), refinement-stage arguments (A), and source-code module constants (M).The table pairs parameters with the stage that applies them and their value source.Source: Appendix A.4 · p. 43
• The tabulated parameter values were checked against the saved configuration and source code used for the reported runs.Source: Appendix A.4 · p. 43
• Section 3.5 varied three parameters: the prealignment confidence margin and two local-roll acceptance limits.Source: Appendix A.4 · p. 43

Appendix B. Per-plant qualitative results

• Table B.1 shows qualitative pipeline outputs for ten maize plants, with each row following the genotype order of Table 1.Source: Appendix B · p. 43, Table B.1 · p. 43
• The columns show the input scan, consensus midlines, geometrically corrected blade segmentation, initial procedural model, and NURBS-Diff-refined model.Panels within each row use the same camera.Source: Table B.1 · p. 43, Appendix B · pp. 44–45
• Plants are framed separately, so apparent size is not comparable across rows.Source: Table B.1 · p. 43

Appendix C. Per-plant fidelity and check activity

• Table C.1 reports per-plant blade-level residuals, whole-plant symmetric Chamfer distances, and counts of adopted geometric modifications.The blade-level measure compares corrected blade points with fitted leaf surfaces; the whole-plant measure compares the assembled mesh with the complete scan.Source: Appendix C · p. 45
• The fold diagnostic flagged 28 of 109 eligible fits, with 21 fold repairs adopted; rigid prealignment was accepted on 9 leaves.Source: Appendix C · p. 45
• CML69 and M162W accounted for 7 of 9 accepted rotations; Kui2021 had none, and no base-collar repair was selected for the ten plants.Source: Appendix C · p. 45
• A679 had the lowest blade-level residual but one of the higher whole-plant distances; unmodeled tassels and sheaths may contribute, but the metrics do not identify individual causes.Source: Appendix C · p. 45

Appendix D. Repeated annotation of a single view

• Mean (minimum) pairwise intersection over union was 0.792 (0.764) for Plant A, 0.772 (0.739) for Plant B, 0.682 (0.626) for Plant C, and 0.670 (0.635) for Plant D.Each plant was annotated five times from the same rendered image using the same prompt and model.Source: Appendix D · pp. 45, 47
• The narrow annotation curves make the mask-overlap measure sensitive to small lateral displacements.Source: Appendix D · p. 47
• The repeats measured variability in one view, not in the complete eight-view workflow, and did not validate the requirement for support from at least two views.Measuring end-to-end variability would require repeating all eight annotations and downstream stages.Source: Appendix D · p. 47

Appendix E. Probe of a second image-output model

• Foreground-mask recall averaged 78.1% across 20 second-model returns and ranged from 28.7% to 98.5%.This measure includes all thresholded image foreground, including render overlays, and does not establish exact pixel preservation or 3D point registration.Source: Appendix E · p. 47
• All 20 calls returned curves and magenta components in the extractor's encoding, but visual inspection found partial or split midlines and components away from the stalk.Detected magenta-component counts ranged from 10 to 18 per call for Plant A, compared with 9 to 10 for the primary model, and from 15 to 21 for Plant C, compared with 11 to 12.Source: Appendix E · p. 47, Figure E.1 · p. 50
• The second-model outputs were not passed through mask extraction or reconstruction, so observed annotation defects do not establish errors in exported leaves or nodes.Source: Appendix E · p. 47
• The probe characterizes annotation-stage requirements rather than ranking services; a substitute must preserve pixel-to-point correspondence sufficiently for back-projection and produce extractable annotations.The reported reconstructions used the model named in Section 2.4.Source: Appendix E · p. 47