PaperlayerView exact revision on arXiv

Source-linked AI summary

LICOA: Learning Instruction-Conditioned Optical Actions for Mobile Spatial Augmented Reality

Yuchen Deng, Shunkai Zhou, Ruixuan Zhu, Haibin Ling, Bingyao Huang

arXiv:2610.21282v1cs.ROcs.CV

TL;DR

Most spatial augmented reality systems assume a fixed projector pose or predefined target, while instruction-conditioned placement requires choosing how to move and where and at what scale to project. LICOA combines mobile and local optical control with a shared action-model interface and matched simulation training. Across 480 real-world rollouts, adapted π0.5 outperformed the same architecture trained from scratch on success rate and success weighted by path length.

  • Problem

    Instruction-conditioned optical action lacks a corpus pairing spatial instructions, external and onboard images, robot state, and coordinated optical commands, as well as a common evaluation protocol.

    Source: §5 · p. 6

  • Method

    LICOA uses a matched simulation embodiment and corpus of 150 spatial placement tasks and 7,500 rendered episodes to fine-tune action models for real-world evaluation.

    Source: §1 · p. 2, §5.2 · p. 6

  • Results

    57.50% success and 40.64% success weighted by path length for adapted π0.5, versus 8.33% and 5.15% for the same architecture trained from scratch across 480 real-world rollouts.

    Source: §8 · p. 10

  • Takeaways & Limitations

    The results establish a physical platform, shared perception and optical action interface, corpus, and real-world evaluation protocol for treating projected light as an embodied action.

    Source: §8 · p. 10

    The external camera simplifies supervision and evaluation, making the setup more instrumented than a fully self-contained consumer system.

    Source: §7.3 · p. 9

Abstract

from arXiv · show

Spatial augmented reality (SAR) augments physical environments with projected content, but most systems assume that the projector pose, target region, or motion path has been fixed or specified. We instead make projected light an embodied action: given a language instruction and current visual observations, a system must decide how to move and where and at what scale to place the projection. We introduce LICOA to realize this capability. Its differential-drive chassis provides room-scale approach and repositioning, while a steerable mirror with two degrees of freedom (2DoF) and projection scale provide rapid local control of the beam. A shared interface maps synchronized external and onboard RGB cameras with robot control state to physical optical actions across pretrained action models. Because instruction-conditioned optical action lacks dedicated training data and a common evaluation protocol, we construct a matched simulation embodiment and a corpus with 150 tasks and 7,500 rendered episodes, then evaluate models trained only on this corpus in previously unseen real-world environments. Across 480 complete closed-loop real-world execution trials (rollouts) on 12 tasks, the best adapted action model achieves a 57.50% success rate and 40.64% success weighted by path length, compared with 8.33% and 5.15% for the same architecture trained from scratch. A further 150 real-world ablation rollouts support the need for both camera views and coordinated control of the chassis and mirror. These results establish a physical platform, shared perception and optical action interface, corpus, and real-world evaluation protocol for treating projected light as an embodied action.

1 INTRODUCTION

LICOA treats projected light as an instruction-conditioned physical action, requiring a system to identify a target and coordinate movement, beam direction, and projection scale. It combines a mobile optical platform, a shared interface for pretrained action models, and a matched simulation corpus with real-world evaluation.

1 INTRODUCTION

  • The work addresses the absence of a corpus and common evaluation protocol pairing spatial instructions, two camera roles, robot state, and coordinated optical actions.It establishes a matched simulation corpus and evaluates models trained only on that corpus in real-world settings.Source: §1 · p. 2
  • 57.50% success and 40.64% success weighted by path length for adapted π0.5, versus 8.33% and 5.15% for the same architecture trained from scratch.The comparison covers 480 complete closed-loop real-world rollouts; another 150 rollouts support the sensing and actuation ablations.Source: §1 · p. 2
  • LICOA makes projected light an embodied action: the system must decide how to move and where and at what scale to place a projection from an instruction and visual observations.The task includes identifying a referenced region, reaching a feasible projection pose, redirecting the beam, adjusting its scale, and deciding when placement is complete.Source: §1 · p. 2
  • The platform combines room-scale chassis repositioning with rapid local mirror steering and projection-scale control in one action space.The coupled physical action space also includes episode termination.Source: §1 · p. 2
  • A shared interface maps external and onboard RGB images with robot control state to physical actions across pretrained action models.The interface supports models with different action-generation approaches.Source: §1 · p. 2

2 RELATED WORK

Prior spatial augmented reality systems provide perception, content correction, or mobile projection, but commonly rely on fixed setups or predefined targets and paths. LICOA connects language and current observations to learned closed-loop placement actions, and adapts generalist action models through a shared interface.

2.1 Spatial augmented reality and mobile projection

  • Fixed projector-camera systems handle geometry, photometry, or content adaptation, while generally leaving physical placement to a configured projector and target surface.Semantic scene description extends perception, but does not select and execute a projection action.Source: §2.1 · p. 2
  • Mobile and steerable systems expand coverage or local beam control, but prior targets and motion are supplied by planes, markers, paths, scripts, maps, or scoring rules.The comparison distinguishes projector-origin mobility, exposed action components, spatial instruction, learned closed-loop control, and matched simulation.Source: §2.1 · pp. 2–3, Table 1 · p. 3
  • LICOA connects a natural-language spatial goal and current observations to learned placement actions for a room-scale chassis, 2DoF steerable mirror, and projection scale.Its distinction is the learned closed-loop optical action interface and matched embodiment.Source: §2.1 · p. 3

2.2 Generalist action models for optical action

  • The study instantiates continuous flow matching, autoregressive action tokenization, and video-action co-training as three action-generation paradigms.π0.5, π0-FAST, and Fast-WAM are not presented as an exhaustive survey or as directly comparable pretrained architectures.Source: §2.2 · p. 3
  • A shared interface gives the models the same instruction, external and onboard RGB images, and robot control state, then maps predictions to the same physical actions.This tests model transfer and interface compatibility without changing the physical task or embodiment.Source: §2.2 · p. 3

3 LICOA PHYSICAL PLATFORM

LICOA combines chassis motion for broad repositioning with mirror steering and projection-scale control for local placement. Its sensing and control setup integrates onboard and external RGB views with the robot state and routes model action chunks to the physical actuators.

3.1 Mobile optical embodiment

  • The physical platform separates broad chassis repositioning from local beam direction and scale control.The mirror allows pan and tilt without rotating the platform, while mirror steering alone cannot reposition the projector.Source: Table 2 · p. 4, Figure 2 · p. 4, §3.1 · p. 4

3 LICOA PHYSICAL PLATFORM

  • The differential-drive chassis changes the projector origin for broad approach and repositioning, while the 2DoF mirror redirects the beam locally and projection scale changes its footprint.Episode termination marks completion, making placement executable as an embodied action.Source: §3 · p. 4, §3.1 · p. 4

3.1 Mobile optical embodiment

  • The onboard camera changes viewpoint with chassis motion, while the external camera provides a global view of the robot, target region, and projected light.The host synchronizes both camera streams with robot control state, receives an action chunk, and relays its per-step actions to the physical components.Source: §3.1 · p. 4
  • The Jetson dispatches chassis and projection-scale commands, while an STM32 closes the mirror pan-tilt motor loops locally at approximately 260 Hz.This local loop separates high-rate mirror control from action-model inference.Source: §3.1 · p. 4

4 SHARED PERCEPTION AND OPTICAL ACTION INTERFACE

LICOA gives action models a shared interface from synchronized visual observations and robot control state to physical optical actions. Model-specific adapters preserve that interface, while deployment repeatedly replans from fresh observations.

4.1 Observation and action interface

  • The synchronized control state records active chassis velocities and mirror and scale targets, but omits the global chassis pose.The observation combines timestamp-aligned external and onboard images with that state; the instruction conditions the model separately.Source: §4.1 · p. 4
  • Action chunks combine chassis velocity, mirror pan and tilt, projection scale, and a termination indicator, making each output a physical optical action.The shared interface pairs instruction, external and onboard RGB images, and robot control state with these actions.Source: §4 · p. 4, §4.1 · p. 5

4.2 Connecting pretrained action models

  • Input, target, and output adapters connect different pretrained model representations to the same physical action interface while retaining each model family’s native training objective.The objective may include model-specific action or future-visual targets, and adaptation may fine-tune all or selected parameters.Source: §4.2 · pp. 5–6

4.3 Closed-loop action execution

  • At deployment, the controller clears stale action queues, executes eight actions, then replans at 15 Hz from fresh observations.This receding-horizon loop synchronizes the external camera and robot streams and executes a short prefix of each predicted chunk.Source: Figure 3 · p. 5, §4.3 · p. 6

5 SIMULATION CORPUS AND EVALUATION PROTOCOL

LICOA addresses the lack of a corpus and common evaluation protocol for instruction-conditioned optical action with matched simulation training and real-world evaluation. The corpus covers spatial placement tasks, and success is judged from the final physical projection.

5.1 Matched simulation and data collection

  • Simulation reproduces the physical platform’s kinematic chain, camera configuration, action ranges, and projection scale, and demonstrations use the same trajectory format.Teleoperated trajectories vary initial configurations and include both local corrections and chassis repositioning before mirror adjustment.Source: §5.1 · p. 6

5.2 Task collection

  • The corpus contains 150 spatial placement tasks across 10 scenes, with five external viewpoints and three tasks per viewpoint.Each task specifies a scene and viewpoint, initial robot configuration, and instruction describing a target region and projection scale.Source: §5.2 · p. 6, Table 3 · p. 6
  • Each of the 150 tasks yields 50 rendered episodes, for 7,500 total, by rendering 10 source trajectories under five appearance settings.The four randomized variants alter lighting intensity, color temperature, wall materials, and floor materials while preserving geometry and source actions.Source: §5.2 · p. 6

5.3 Real-world evaluation protocol

  • The protocol evaluates final projection placement in previously unseen real-world environments with new instructions, rather than relying on a held-out source trajectory benchmark.Because the corpus lacks geometric target masks and calibrated target measurements, a human operator judges success; a rollout succeeds upon termination or at least three seconds of stable placement.Source: §5 · p. 6, §5.3 · p. 7, §5.2 · p. 7

6 EXPERIMENTS

The experiments evaluate pretrained action-model families through a shared interface, testing simulation-to-real deployment and the contributions of camera views and coupled chassis–mirror control. Adapted π0.5 leads the real-world results, while ablations support complementary sensing and actuation.

6 EXPERIMENTS

  • The experiments assess interface generality, simulation-to-real transfer, pretraining, and the complementary roles of external and onboard views and chassis and mirror control.The camera ablations retrain without one view, whereas action ablations retain the full checkpoint and suppress actuator commands at deployment.Source: §6 · p. 7, §6.3 · p. 8

6.1 Experimental setup

  • The setup compares three pretrained action-model families and a same-architecture π0.5-scratch control using shared observations, control state, action space, and training data.Real-world evaluation covers 12 previously unseen tasks with 10 rollouts per task and reports SR, SPL, and MT; MT is computed over successful trials only.Source: §6.1 · p. 7

6.2 Interface generality and real-world deployment

  • 57.50% SR and 40.64% SPL make adapted π0.5 the strongest overall model, exceeding π0.5-scratch by 49.17 and 35.49 percentage points, respectively.π0.5 is the only model to succeed at least once on all 12 tasks; the evaluation comprises 120 rollouts per model.Source: §6.2 · p. 7, Table 4 · p. 7, Figure 4 · p. 8

6.3 Evidence for sensing and action design

  • The full camera-and-actuation configuration reaches 60.00% SR and 37.09% SPL, while removing either view or restricting control lowers performance.External-camera removal yields 26.67% SR and 13.47% SPL; onboard-camera removal yields 30.00% and 24.61%; mirror-only and chassis-only yield 30.00% and 13.33% SR.Source: §6.3 · pp. 8–9, Table 5 · p. 8

7 DISCUSSION

The discussion reports placement behavior with projection appearances beyond training and proof-of-concept downstream operations after placement. It also bounds the current system by content, sensing, environmental, hardware-generalization, and termination-supervision limitations.

7.1 Applicability to other settings

  • With no additional fine-tuning, the full π0.5 model places circular light, green light, and an image on a wooden surface despite training with uniform white rectangular projection.These are qualitative rollouts providing initial evidence that learned placement can accommodate appearances beyond the fine-tuning corpus.Source: §7.1 · p. 9, Figure 6 · p. 9

7.2 Additional applications

  • After placement, proof-of-concept trials use structured-light patterns for local geometry reconstruction and projector compensation on one textured surface.The compensation demonstration is a single-pose proof of concept, separate from the real-world evaluation protocol.Source: §7.2 · p. 9

7.3 Limitations and future work

  • The current scope evaluates physical placement, not generation, layout, or readability of rich projected content, and relies on an external camera.Results also remain sensitive to surface material, ambient light, camera exposure, and actuator repeatability; hardware and room-scale generalization require additional calibration or adaptation data.Source: §7.3 · p. 9
  • Future work proposes richer content, onboard-only state estimation, improved mirror speed and scale control, larger models, mixed real and simulated data, and self-supervised exploration.The paper also identifies more reliable termination annotation and explicit supervision of task completion over time as areas for further work.Source: §7.3 · p. 9

8 CONCLUSION

The conclusion presents LICOA as a mobile system for instruction-conditioned optical action, combining chassis motion, mirror steering, scale control, and a shared interface. Simulation-trained adapted π0.5 achieves stronger real-world results than its scratch counterpart, and ablations support the sensing and actuation design.

8 CONCLUSION

  • Across 480 real-world rollouts, adapted π0.5 achieves 57.50% SR and 40.64% SPL, versus 8.33% and 5.15% for the same architecture trained from scratch.Both models were trained only on a matched simulation corpus of 150 spatial placement tasks and 7,500 episodes.Source: §8 · p. 10

1 PHYSICAL PLATFORM AND EMBEDDED CONTROL

LICOA combines chassis movement for room-scale repositioning with mirror steering and projection-scale control for local beam placement. Its embedded system routes model actions through the Jetson while the STM32 closes the mirror motor loops locally.

1 PHYSICAL PLATFORM AND EMBEDDED CONTROL

  • The physical platform separates room-scale chassis movement from local mirror steering and footprint adjustment through projection scale.The projector is carried by the differential-drive chassis; the 2DoF mirror redirects the beam without moving the platform.Source: §1 · p. 12

1.1 System interfaces and power

  • The Jetson bridges host actions to the chassis, mirror, and projector, while forwarding synchronized onboard imagery and robot state alongside the external camera stream.The host synchronizes the two camera streams and sends a 16-step action chunk for execution.Source: §1.1 · p. 12

1.2 Custom laser projector

  • The custom projector converts Jetson HDMI output to RGB888 for its projection electronics, and changes projection scale by rendering a corresponding pattern.The projection electronics coordinate image processing, laser control, RGB sources, and a MEMS scanner.Source: §1.2 · p. 12

1.3 2DoF steerable mirror wiring

  • The STM32 receives mirror targets and telemetry over USB-to-UART, while a separate USB connection supplies logic power and a 12 V rail supplies motor drivers.The AS5600 encoders provide feedback for the locally closed motor loops.Source: §1.3 · p. 12, Figure S1 · p. 13

1.4 Jetson Nano command bridge

  • The Jetson command bridge clamps mirror targets to platform limits, dispatches chassis, mirror, and scale commands, and stops chassis motion when termination is signaled.The mirror targets are pan and tilt in semantic joint coordinates; the serial controller maps them to firmware angles.Source: §1.4 · p. 12

1.5 STM32 motor control

  • The STM32 closes both mirror motor loops locally at about 260 Hz, interpolates streamed targets, and drives the motors using SVPWM at about 25 kHz.The deployed vertical-axis path uses continuous feedback rather than the legacy staged open-loop fast approach mode.Source: §1.5 · pp. 12–13

2 MATCHED SIMULATION CORPUS CONSTRUCTION

To address the lack of dedicated training data and common tasks for instruction-conditioned optical action, the authors reproduce the physical embodiment and shared interface in simulation. They construct 150 tasks and collect accepted demonstrations with matched observations and actions.

2 MATCHED SIMULATION CORPUS CONSTRUCTION

  • The simulated corpus matches the physical embodiment, camera roles, robot state, and action interface used for real-world deployment.The matched interface supports demonstrations collected in simulation and used for real-world evaluation.Source: §2 · p. 13
  • The Isaac Sim environment uses 10 large indoor scenes, five fixed external viewpoints per scene, and three placement tasks per viewpoint, yielding 150 tasks.Each task includes the LICOA articulation, an initial robot pose, and static scene geometry for navigation and optical interaction.Source: §2 · p. 13
  • Operators teleoperate the chassis, 2DoF mirror, and projection scale; only demonstrations accepted as satisfying the instruction are retained.Failed and timed-out attempts are discarded, and saved observations align with complete simulator steps.Source: §2 · pp. 13–14
  • Recorded simulator states are replayed to render synchronized external and onboard images while preserving the source action sequence.Each trajectory retains its original appearance plus four deterministic lighting and material variants, with geometry, camera poses, projection light, and actions unchanged.Source: §2 · p. 14

3 ACTION MODEL ADAPTATION AND TRAINING DETAILS

The action models share fixed physical inputs and outputs while retaining model-specific representations and objectives. Their training uses the simulation corpus rather than real-world LICOA training data.

3 ACTION MODEL ADAPTATION AND TRAINING DETAILS

  • The shared interface fixes physical inputs and outputs across models, while the reported runs use no real-world LICOA training data.The models retain their own representations and objectives.Source: §3 · p. 14
  • All four main-comparison configurations train for 30,000 updates with batch size 32 and seed 0, using one GPU.Online image transformations are disabled; camera images are mapped to model-specific feature keys.Source: §3 · p. 14
  • The tested model recipes cover full fine-tuning, autoregressive action tokenization, and video-action co-training, while π0.5-scratch uses the π0.5 architecture without a pretrained checkpoint.Fast-WAM consumes the robot state and action directly; π0-FAST uses the FAST tokenizer and causal cross-entropy.Source: §3 · p. 14
  • The camera ablation removes either the external or onboard image while holding the remaining inputs, action interface, update budget, and optimizer fixed.Both variants reuse the full π0.5 recipe.Source: §3 · p. 14

4 OFFLINE ANALYSIS OF ADAPTATION AND DATA SCALE

The offline analyses compare adaptation scope and simulation-data scale using fixed-validation flow-matching loss, providing optimization and data-efficiency evidence rather than closed-loop task-success evidence. They find that Action expert nearly matches Full fine-tuning on this metric, while more trajectories and broader task coverage improve held-out action prediction.

4 OFFLINE ANALYSIS OF ADAPTATION AND DATA SCALE

  • These offline ablations assess trainable-parameter scope and simulation-data amount and diversity using held-out flow-matching loss, not real-world rollouts or task success.The authors treat them as optimization and data-efficiency evidence rather than a substitute for the main paper’s real-world metrics.Source: §4 · p. 14

4.1 Corpus and offline validation split

  • The corpus contains 150 placement tasks across 10 InteriorAgent scenes, with 10 source trajectories and five appearance variants per task, totaling 7,500 episodes.The appearance variants randomize lighting intensity, color temperature, wall materials, and floor materials.Source: §4.1 · p. 15
  • Validation holds out one source trajectory and all five of its appearance variants per task, yielding 750 episodes across all 150 tasks; all scenes contribute to training.The authors state that this fixed holdout tests offline validation loss, not generalization across scenes or instructions.Source: §4.1 · p. 15

4.2 Training and validation protocol

  • The action models use synchronized onboard and external RGB images with robot control state, and π0.5 predicts six-channel action chunks over a 16-step horizon.The validation loss uses a fixed validation set and common noise/time stream; runs use one seed, which is not a multi-seed uncertainty estimate.Source: §4.1 · p. 15, §4.2 · pp. 15–16

4.3 Parameter adaptation scope

  • 22.3% lower endpoint loss for Action expert than Projection heads (0.3030 versus 0.3901); Full improves by a further 0.0026 but uses more trainable parameters, memory, and runtime.Action expert nearly matches Full with 83.3% fewer trainable parameters and 45.8% lower peak memory; main-paper π0.5 results nevertheless use Full fine-tuning.Source: §4.3 · p. 16, Table S5 · p. 16

4.4 Data scale analysis

  • 65.7% lower validation loss when available episodes per task rise from 10 to 50 (0.8755 to 0.3004); 71.3% lower when training tasks rise from 10 to 150 (1.0454 to 0.3004).All data-scale runs use Full fine-tuning; the two axes vary trajectory density and spatial task diversity, respectively.Source: §4.4 · p. 16, Table S6 · p. 16
  • Smaller training subsets often turn upward earlier, while the complete-split run reaches 0.2589 at 15,000 updates before ending at 0.3004 at the prespecified 30,000-update endpoint.The authors describe the pattern as consistent with overfitting and retain the common endpoint to avoid post-hoc checkpoint selection.Source: §4.4 · p. 16, Figure S2 · p. 17

Related papers

Loading 2610.21282v1…