Source-linked AI summary
From Language to Motion: Task-Conditioned Focal-Stack Trajectory Integration for Microscopic Robots
Junjie Xie, Chuxuan He, Junkai Huang, Heng Zhang, Angen Ye, Yujia Song, Yuqing Li, Pengsong Zhang, Dapeng Zhang
TL;DR
Microscopic robot tasks require accurate ordered geometry despite changing parts, instructions, and focus. The paper combines constrained language-to-geometry specifications with reusable perception and path-space focal integration, then connects recovered paths to robot execution. Across nine matched part–illumination conditions, trajectory-space integration reduces RMSE and P95 relative to image-first multi-focus fusion.
Problem
Microscopic robot tasks need ordered geometric paths despite part and request changes, while shallow depth of field can leave different path segments in focus in different images.
Source: §I · p. 1Method
The framework maps language into constrained task specifications, reuses frozen perception, and combines locally reliable focal-plane trajectories through confidence weighting and dynamic programming.
Source: §I · pp. 1–2, B. Confidence-Weighted… · p. 4Results
Across nine matched conditions, trajectory-first integration reduces average RMSE by 56.4% (14.41 to 6.28 pixels) and P95 by 59.5% (20.07 to 8.13 pixels) against image-first fusion.
Source: B. Trajectory-Level Focal-Stack… · p. 6Takeaways & Limitations
The zero-new-label configuration achieves 6.28-pixel RMSE with 15 min of adaptation, compared with 5.47 pixels and 165 min for the most accurate 100-label U-Net.
Source: §V · p. 7
Abstract
from arXiv · showhide
Microscopic robots require accurate task geometry despite changes in language, parts, and focus. We present a semantic-to-physical framework that maps instructions to constrained geometric operators, reuses frozen open-vocabulary perception, and integrates locally reliable focal-plane trajectories by confidence weighting and dynamic programming. Calibrated multi-view geometry connects 2-D paths to physical execution. Prompt, unseen-part, and geometry reconfiguration tests yield 6.30-6.59-pixel RMSE. Relative to part-specific U-Net training with 20-100 labels, the proposed zero-new-label configuration takes 15 rather than 72-165 min. Across nine part-illumination conditions, trajectory-space integration reduces RMSE from 14.41 to 6.28 pixels (56.4%) and P95 error from 20.07 to 8.13 pixels (59.5%) compared with image-first multi-focus fusion. An ablation isolates the roles of confidence and path-wise selection. In representative robot experiments, target-region coverage improves from 83.5% to 92.9%. Dispensing provides a measurable physical trace, not a task-specific limitation of the method.
I. INTRODUCTION
Microscopic robot tasks need ordered geometric paths that remain accurate across changing parts, requested geometry, and focal planes. The paper pairs language-conditioned geometry extraction with focal-stack path integration and calibrated multi-view execution.
I. INTRODUCTION
- Part-specific supervised systems can be accurate, but repeated annotation and training incur costs in high-mix, low-volume manipulation.The introduction positions reusable open-vocabulary perception as an alternative component, not as a novel visual backbone.Source: §I · p. 1
- The framework separates language-specified task geometry from visual measurement, enabling reconfiguration by changing instructions and operator settings instead of retraining a dedicated detector.It evaluates accuracy and adaptation time against supervised baselines.Source: §I · p. 1
- Shallow depth of field can leave parts of a 3-D contour out of focus, so the method integrates locally extracted trajectories directly in ordered path space.Confidence weighting favors locally reliable evidence, while dynamic programming encourages continuity and allows a null state when evidence is inadequate.Source: §I · p. 1
- Calibrated multi-view geometry reconstructs a 3-D task trajectory from separately recovered 2-D paths for physical execution.Approximately orthogonal views are a platform-specific configuration, not a requirement of the algorithm.Source: §I · p. 1
- The framework targets geometric requests supported by available operators and sufficient image evidence, rather than arbitrary instructions executable without configuration.Physical-trace dispensing is used to measure outcomes after geometry recovery, calibration, and robot motion.Source: §I · pp. 1–2
II. RELATED WORK
Language-grounded geometry extraction uses reusable perception components while leaving image-coordinate measurement to visual processing. For focus variation, this work makes the ordered path—not a fused image—the target of integration.
A. Language-Grounded Geometry Extraction
- Language specifies the geometry to measure, while calibrated visual processing determines its image coordinates; the paper does not propose a new visual foundation model.Grounding DINO detects described objects, SAM estimates masks, and Grounded SAM composes them –[6].Source: A. Language-Grounded Geometry Extraction · p. 2
B. Focus Variation and Task-Space Fusion
- Unlike multi-focus image fusion, which synthesizes a composite image, the method integrates focal-plane hypotheses into an ordered geometric path.The path-level optimizer can leave an interval unspecified rather than force an unreliable prediction.Source: B. Focus Variation and Task-Space Fusion · p. 2
- Focus transitions and image blending may displace edges used for downstream measurements,, motivating direct path-space integration rather than a fused-image output.Source: B. Focus Variation and Task-Space Fusion · p. 2
C. Perception-to-Execution Bridge
The method extracts task geometry in 2-D, integrates focal-plane trajectories, then uses an established bridge from image measurements to robot motion. Language yields a constrained measurement specification, while visual processing and task-specific operators produce the geometry.
C. Perception-to-Execution Bridge
- Multi-view reconstruction and vision-based robot control connect image measurements to robot motion, while the evaluation distinguishes image-space trajectory error from downstream physical-trace outcomes.The methodological focus is upstream 2-D geometry, not a new 3-D reconstruction algorithm.Source: C. Perception-to-Execution Bridge · p. 2
III. METHOD
- The method has three stages—language-guided 2-D task-geometry extraction, trajectory-level focal integration, and a standard 2-D-to-3-D execution bridge—with the first two as its methodological focus.The third stage connects recovered geometry to the robot.Source: §III · p. 2, Figure 2 · p. 3
A. Language-Guided 2-D Task Geometry
- A semantic model converts a language request into a constrained specification of desired and excluded targets, region or spatial relation, geometric primitive, and lightweight settings.The semantic model specifies what to measure, not image coordinates, metric coordinates, or motor commands.Source: A. Language-Guided 2-D Task Geometry · p. 3
- Frozen general-purpose perception backbones provide visual inputs, and a task-specific post-processor converts masks into the requested 2-D geometry.The post-processor may include geometric processing or lightweight task-specific learning; the intended reconfiguration path changes the request and specification rather than rebuilding the full training pipeline.Source: A. Language-Guided 2-D Task Geometry · p. 3
B. Confidence-Weighted Trajectory-Level Focal Integration
The method integrates local trajectories from separate focal planes rather than first collapsing the stack into an all-in-focus image. Confidence-weighted hypotheses and global path selection produce a per-view 2-D trajectory while allowing unsupported intervals to remain unspecified.
B. Confidence-Weighted Trajectory-Level Focal Integration
- The method avoids all-in-focus synthesis, extracting local trajectories from focal planes where each is reliable over only part of the path.This preserves focal-plane evidence instead of collapsing it into an image before trajectory extraction.Source: B. Confidence-Weighted… · p. 3
- Each focal-plane candidate is resampled by normalized arc length and scored using local confidence from task processing, perception, focus quality, and crossfocal consistency.Missing or rejected observations carry no weight; confidence compares observations within the same task rather than serving as a universal reliability model.Source: B. Confidence-Weighted… · p. 3
- The integration groups geometrically compatible observations, confidence-weights consistent ones, and keeps clearly separated edges as competing hypotheses rather than averaging them.The final 2-D trajectory preserves focal-source information.Source: B. Confidence-Weighted… · pp. 3–4, Figure 3 · p. 4
- Dynamic programming selects a globally ordered path, including a null state when no observation is sufficiently supported.Its costs account for local evidence, spatial discontinuity, and unsupported focal-source switching; backtracking yields accepted portions of the fused per-view trajectory.Source: B. Confidence-Weighted… · p. 4
C. From 2-D Task Geometry to 3-D Execution
The framework lifts corresponding per-view 2-D trajectories into an executable 3-D task trajectory using calibrated multi-view geometry. The validation platform uses approximately orthogonal views as one configuration, while image-space and physical-trace metrics remain separate.
C. From 2-D Task Geometry to 3-D Execution
- Corresponding 2-D trajectories from calibrated camera views are lifted through standard multi-view geometry into an executable 3-D task trajectory.The reconstruction follows established multi-view formulations rather than introducing a new reconstruction algorithm.Source: C. From 2-D Task Geometry to 3-D… · p. 4
- Approximately orthogonal microscopic views are one practical platform configuration, not a method requirement, and focal-plane indices are not treated as metric depth.Source: C. From 2-D Task Geometry to 3-D… · p. 4
- Evaluation reports manually annotated 2-D path errors separately from target-relative physical-trace outcomes after robot execution.The metric families are kept separate to avoid conflating image-space localization with robot-level accuracy.Source: C. From 2-D Task Geometry to 3-D… · p. 4
IV. EXPERIMENTS
The experiments address adaptation under changes in language, part, or requested geometry; focal-evidence integration and its components; and measurable target-aligned physical traces. The real-system validation depicts the platform, complementary views, robot interaction, and deposited traces.
IV. EXPERIMENTS
- The experiments ask whether language, part, and geometry changes preserve trajectory accuracy at low adaptation cost, whether focal integration improves localization, and whether execution yields measurable target-aligned traces.Source: §IV · pp. 4–5
- Physical validation uses dispensing as a trace-preserving measurement task, with target-relative performance summarized separately in Table V.Figure 4 depicts the platform and auxiliary setup, complementary microscope views, robot–part interaction, and deposited traces with local dimensional annotations.Source: Figure 4 · p. 5
A. Language Generalization and Reconfiguration
Controlled changes in wording, part identity, and requested geometry retain similar image-space errors. Against part-specific U-Net training, the zero-new-label configuration trades a small accuracy gap to the most accurate model for less adaptation time.
A. Language Generalization and Reconfiguration
- RMSE varies 6.30–6.57 pixels across phrasings, 6.39–6.59 across unseen parts, and 6.37–6.56 across geometric requests.The tested requests include canonical, formal, semantic, and colloquial phrasings, previously unseen parts D–F, and three distinct geometric outputs.Source: Table I · p. 5, A. Language Generalization and… · p. 5
- The tested reconfiguration conditions bound the evidence; they do not establish unrestricted open-set performance.Source: A. Language Generalization and… · p. 5
- The zero-new-label configuration takes 15 min and reaches 6.28-pixel RMSE, versus 165 min and 5.47 pixels for the most accurate 100-label U-Net.Against the 20-label U-Net, it is quicker (15 versus 72 min) and more accurate (6.28 versus 6.46 pixels).Source: A. Language Generalization and… · pp. 5–6, Table II · p. 6
B. Trajectory-Level Focal-Stack Integration
Trajectory-first integration combines local path hypotheses across focal planes rather than first synthesizing an all-in-focus image. Across nine matched part–illumination conditions, it improves image-space trajectory error over the image-first baseline, while ablations distinguish confidence weighting from path-wise selection.
B. Trajectory-Level Focal-Stack Integration
- The comparison evaluates the same manually annotated reference path for each method in each of three parts under three illumination settings.The baseline synthesizes an all-in-focus image before extracting a trajectory; the proposed method extracts focal-plane candidates and integrates them.Source: B. Trajectory-Level Focal-Stack… · p. 6
- 56.4% lower average RMSE (14.41 to 6.28 pixels) and 59.5% lower P95 (20.07 to 8.13 pixels) than image-first MFIF across nine matched conditions.The trajectory-first RMSE spans 6.23–6.36 pixels across conditions, versus 14.34–14.47 pixels for MFIF.Source: B. Trajectory-Level Focal-Stack… · p. 6
- Combining confidence with dynamic programming yields the lowest RMSE (6.28 pixels), P95 (8.13 pixels), and source-switch count (9.0) among the tested variants.Confidence-only integration improves RMSE from 7.00 to 6.75 pixels; DP-only yields 6.56 pixels and cuts source switching from 60.6 to 11.2.Source: Table IV · p. 6, B. Trajectory-Level Focal-Stack… · p. 6
C. From 2-D Geometry to Physical Validation
Calibrated multi-view geometry connects per-view 2-D paths to the robot's executed physical trace. The representative dispensing evaluation reports target-relative trace outcomes separately from image-space trajectory errors.
C. From 2-D Geometry to Physical Validation
- Per-view 2-D paths are converted to a 3-D task trajectory through calibrated multi-view geometry and executed on the robot.Dispensing is used because its deposited trace is persistent and measurable, not because the perception method requires a dispensing tool.Source: C. From 2-D Geometry to Physical… · p. 6
- Relative physical trace error falls from 1.000 to 0.519, a 48.1% reduction; target-region coverage rises from 83.5% to 92.9%.Out-of-region deposition falls from 12.5% to 5.4%; these physical-trace measures are distinct from pixel-space RMSE.Source: C. From 2-D Geometry to Physical… · p. 6
D. Scope of the Evaluation
The evaluation spans wording variants, held-out parts, geometry requests, adaptation budgets, nine part–illumination conditions, and one representative physical task. The conclusion reports gains in trajectory integration and physical-trace outcomes while limiting numerical generalization to the evaluated settings.
D. Scope of the Evaluation
- The evaluation covers wording variants, three held-out part identities, three geometry requests, three U-Net adaptation budgets, nine part–illumination conditions, and one representative physical task; broader settings are needed to generalize numerical effect sizes.Source: D. Scope of the Evaluation · pp. 6–7
V. CONCLUSION
- Across nine matched part–illumination conditions, trajectory-space integration reduces RMSE and P95 by 56.4% and 59.5%, respectively, relative to image-first MFIF.The final comparison evaluates both methods against the same manually annotated paths in each condition.Source: §V · p. 7
- The zero-new-label configuration achieves 6.28-pixel RMSE with 15 min of adaptation, versus 5.47 pixels and 165 min for the most accurate 100-label U-Net.Source: §V · p. 7
- The physical trace experiment reports a 48.1% reduction in normalized target-relative error, higher target-region coverage, and reduced out-of-region deposition.Source: §V · p. 7
- The framework decouples semantic requests from metric measurement, reuses frozen open-vocabulary perception, and integrates focal-plane trajectories in ordered path space using confidence weighting and dynamic programming.Source: §V · p. 7