Source-linked AI summary
Vision Guided Target Conditioned Control for Autonomous Excavation
Shuai Zhao, Ji-An Pan, Junwei Li, Xun Tang, Fansen Xi, Qing Xu, Keqiang Li, Jianqiang Wang
TL;DR
Autonomous excavation needs to translate spatial work intent into coordinated control under contact-rich soil interaction. The paper combines target masks, mask-conditioned ACT action chunks, and paired-condition demonstrations in a physics-based simulation workflow. Paired-condition control achieves the strongest target compliance and sequential pile-clearing results among the tested approaches.
Problem
Autonomous excavation must map a desired work region to coordinated commands despite long-horizon, multi-joint, nonlinear soil interaction.
Method
An image-aligned target mask conditions an ACT policy on multi-view observations and proprioception, while paired demonstrations contrast targets and corresponding action chunks.
Results
Paired-condition mask-conditioned ACT outperforms non-paired and no-condition baselines in diagnostic target success and sequential pile clearing.
Takeaways & Limitations
Target conditioning matters most when repeated scoops must be distributed across a larger work area, and paired demonstrations expose target-dependent action variation.
Abstract
from arXiv · showhide
Autonomous excavation requires an intelligent control system that can convert spatial work intent into coordinated bucket motion under contact-rich soil interaction. This paper presents a target-conditioned intelligent control framework for autonomous excavation in a physics-based deformable-soil simulation workflow. An image-aligned target mask serves as a visual spatial command for the desired digging region, while a mask-conditioned Action Chunking Transformer maps multi-view RGB observations, proprioception, and the target mask to temporally extended joystick commands. To reduce target-ignoring behavior, demonstrations are organized with paired-condition supervision, where the same or closely matched scene is demonstrated with different target masks and corresponding action chunks. The framework is evaluated through both a diagnostic manipulation task and an excavation simulation benchmark with single-scoop and sequential pile-clearing protocols. In manipulation, target success is 4\% for no-condition ACT, 63\% for non-paired mask-conditioned ACT, and 96\% for paired-condition mask-conditioned ACT. In sequential pile clearing, paired-condition mask-conditioned ACT removes 76.8\% of the pile versus 27.4\% and 15.7\% for the two baselines, with 91.0\% human-normalized efficiency. The results show that visual target conditioning, paired demonstration structure, and action-chunk control form a practical cyber-physical simulation pipeline for excavator automation.
I. INTRODUCTION
Autonomous excavation must convert a desired work region into coordinated, long-horizon commands despite nonlinear, spatially varying soil interaction. The paper addresses target-following by combining visual masks, mask-conditioned action chunks, and paired-condition demonstrations.
- Excavation automation must coordinate four work-control joints through a long-horizon dig-load-dump cycle under nonlinear, spatially varying soil interaction.
- Imitation-learning performance depends on policy architecture and demonstration distribution, while action chunking alone may not ensure spatial-target compliance.
- The framework feeds an image-aligned dig-region mask, multi-view RGB observations, and proprioception into a mask-conditioned ACT policy producing joystick-command chunks.
- Paired-condition supervision contrasts different target masks and corresponding action chunks in identical or closely matched scenes.
- Sequential pile clearing removes 76.8% of the pile with paired-condition conditioning, versus 27.4% without paired contrasts and 15.7% without target conditioning.
A. Imitation Learning for Continuous Control
The paper situates excavation control within imitation learning for continuous visuomotor behavior, emphasizing temporally extended actions and spatial target interfaces. Its target mask specifies excavation intent while testing whether continuous actions change with the commanded region.
- Behavior cloning acquires visuomotor behavior from demonstrations, but results depend on policy architecture and demonstration-distribution structure.
- Sequence-level policies predict temporally extended actions to reduce compounding error, with ACT using a transformer policy for action chunks.
- Visual goals, spatial prompts, language, and multimodal task descriptions can condition robot policies, but the policy must learn how each condition modulates continuous actions.
- In this paper, the target mask is an engineering interface for spatial excavation intent and a controlled test of behavior changes across commanded regions.
- Excavation control is long-horizon, contact-rich, and coupled to deformable terrain, motivating a visual-region interface for continuous joystick-style work-control chunks.
III. CLOSED-LOOP CONTROL ARCHITECTURE
The closed-loop architecture receives visual observations, proprioception, and a target condition, then predicts and executes temporally extended joystick commands before evaluating task state.
- At each time t, the system receives multi-view RGB images, proprioceptive state, and an image-aligned binary target mask.
- The closed-loop system alternates sensing, target-conditioned prediction, command execution, and task-state evaluation.
- The policy predicts a future action chunk At = (at, at+1, . . . , at+K−1) from observation ot and target condition c.
- Each excavation action is a four-dimensional joystick-style command for swing, boom, arm, and bucket.
B. Target-Conditioned Action-Chunk Controller
The target-conditioned controller injects the excavation mask into visual processing and predicts temporally extended commands suited to coherent digging phases. It is trained by supervised imitation on demonstrated future action chunks.
- The target mask is concatenated with the aligned RGB image as a fourth channel for the main task view.Auxiliary views receive zero-mask channels.
- The mask-conditioned ACT policy encodes RGB-mask visual tokens and proprioceptive tokens before decoding a temporally extended command chunk.
- Action chunks match excavation phases including approach, digging, curling, lifting, and dumping.
- The policy is trained by supervised imitation using the observation, target condition, and demonstrated future action chunk.
- The implementation sets λKL = 10 in the training objective.
C. Paired Demonstration Design
Paired-condition supervision presents matched scenes with different target masks and corresponding action chunks, creating direct contrasts between visual context and commanded behavior. This discourages scene-identity shortcuts and supports selecting scoops by work region.
- Paired-condition supervision collects demonstrations from the same or closely matched initial context with multiple target masks and corresponding action chunks.
- Within-scene contrasts discourage scene-identity shortcuts because changing the mask changes the correct control sequence.
- The design addresses excavation settings where visually similar piles support several locally valid scoops selected by the commanded work region.
IV. SIMULATION ENVIRONMENT AND DATA WORKFLOW
The simulation workflow combines deformable-soil physics, joystick demonstrations, aligned target masks, synchronized rollout data, and closed-loop action-chunk execution. It supports controlled evaluation of target-conditioned excavation across single-scoop and sequential protocols.
- The excavation environment models cohesionless sand in a physics-based deformable-soil simulation with fixed-base excavator control and shared terrain settings.The terrain uses density 1474 kg/m3, friction angle 39.0°, and a 0.02 s dynamics time step.
- Joystick demonstrations execute dig-load-dump cycles for commanded regions, with paired data varying targets across the same or closely matched pile state.This keeps visual context largely fixed while changing the target mask and correct joystick sequence.
- Single-scoop trials test one mechanically effective cycle, while sequential pile-clearing trials compose repeated local commands over a larger pile.
- At each control step, the policy consumes visual observations, proprioception, and the target mask, predicts an action chunk, and executes joystick-style commands through the simulator.
- Target masks remain aligned with the main RGB frame through matching crop and resize operations, with platform-specific mask generation procedures.
B. Engineering Metrics
The evaluation protocol measures both target compliance and excavation utility in controlled simulation. Its engineering metrics cover local digging performance, material removal, retained deposited mass, and efficiency relative to joystick demonstrations.
- The protocol evaluates target compliance and excavation utility through single-scoop and sequential pile-clearing trials.
- Fig. 2 uses commanded dig regions and material-removal metrics for controlled deformable-soil evaluation, while quantitative excavation comparisons are reported in simulation.
V. EXPERIMENTAL VALIDATION
A diagnostic manipulation study tests whether policies use image-aligned target masks to select the specified block. Paired-condition mask-conditioned ACT substantially outperforms unconditioned and non-paired alternatives, while corrupted masks sharply reduce original-target success.
- Diagnostic Manipulation Study: The manipulation task uses four blocks, an image-aligned binary target mask, RGB+mask observations, proprioception, and action chunks of length 100 at 50 Hz.
- Diagnostic Manipulation Study: The strict success criterion requires grasping the block specified by the target mask and placing it into a side tray.
- Diagnostic Manipulation Study: 4% target success is achieved by no-condition ACT, compared with 63% for non-paired mask-conditioned ACT and 96% for paired-condition mask-conditioned ACT.
- Diagnostic Manipulation Study: 0–2% original-target success under zero, random, and mismatched masks indicates dependence on the provided target condition.
B. Excavation Simulation Results
The excavation benchmark separates single-scoop execution from sequential allocation across a pile. Paired-condition mask-conditioned ACT is strongest for repeated target-directed clearing, while the study remains simulation-based rather than full-size autonomy.
- Sequential pile clearing: 76.8% of the pile was removed by paired-condition mask-conditioned ACT, versus 27.4% for non-paired mask-conditioned ACT and 15.7% for no-condition ACT.Sequential pile clearing tests whether repeated scoops are allocated across the pile.
- Single-scoop trials: 46/50 target successes were achieved by paired-data mask-conditioned ACT in single-scoop trials.Single-scoop trials evaluate one dig-load-dump primitive, whereas pile clearing evaluates repeated scoop allocation.
- Interpretation: Target conditioning matters most when local excavation actions must be distributed across a larger work area.No-condition ACT can retain a useful repeated scoop but lacks an input channel for selecting the next dig region.
- Interpretation: Paired-condition demonstrations improve sequential behavior by exposing target-dependent action variation during training.The framework tests target following through manipulation, single-scoop execution through one primitive, and repeated coverage through pile clearing.
- Scope: The framework validates control in physics-based simulation rather than claiming full-size construction-site autonomy.The scaled testbed does not verify calibrated material-removal performance or direct deployment to full-size machinery.
VII. CONCLUSION
The paper presents target-conditioned action chunking for autonomous excavation and evaluates it in physics-based deformable-soil simulation. Paired-condition supervision helps the policy use commanded regions rather than default behavior, improving sequential pile clearing over the baselines.
- The framework connects target masks, joystick demonstrations, action-chunk policy execution, and physics-based deformable-soil simulation.
- Paired-condition supervision provides target-action contrasts that help the policy use the commanded region rather than execute a default behavior.
- Paired-condition mask-conditioned ACT improves sequential pile clearing over no-condition and non-paired baselines.The findings support target-conditioned action chunking as a practical intelligent-control implementation for simulation-based excavator automation.