PaperlayerView exact revision on arXiv

Source-linked AI summary

Size Doesn't Matter: Material-State Reinforcement Learning for Excavator Transferable Soil Manipulation

Lennart Werner, Pol Eyschen, Sean Costello, Pierluigi Micarelli, Andrei Cramariuc, Marco Hutter

arXiv:2609.12677v1cs.RO

TL;DR

Contact-rich earthmoving remains difficult for autonomous systems because common methods have limited access to soil mechanics and terrain state. This paper trains local soil-manipulation policies in a GPU-parallelized particle simulation and transfers them through calibrated normalized interfaces. On a full-size excavator, the controller built a 42 m embankment in 45 minutes at a speed comparable to an expert operator.

  • Problem

    Existing earthmoving methods often use heuristic soil models or limited terrain-state representations, constraining approaches to contact-rich material redistribution tasks.

    Source: §II · p. 2, B. Learning-Based Excavation and… · p. 2

  • Method

    The paper trains local soil-manipulation reinforcement-learning policies in a GPU-parallelized MPM particle simulation and separates them from machine-specific control through a calibrated normalized interface.

    Source: A. Soil Simulation · p. 5, §VI · p. 14

  • Results

    Across 201 rollouts, the M445 built a 42 m, 2.1 m high embankment in 45 min without intervention, matching expert progression speed while producing a higher embankment with lower height variation in the reported trial.

    Source: 4) Conclusion · p. 8

  • Takeaways & Limitations

    The results support particle-based reinforcement learning as a local skill layer for contact-rich soil manipulation, with a calibrated normalized interface for reuse across machines.

    Source: §VI · p. 15

    The policies omit history and use no time-aware network, preventing them from learning multistroke motions such as repositioning or pushing from above during compaction.

    Source: C. Limitations · pp. 13–14

Abstract

from arXiv · show

Earthmoving tasks such as excavation, backfilling, or embankment construction require deliberate repositioning of deformable soil. For these tasks, human operators use all shovel faces, while autonomous systems so far are limited to excavation and dumping. Current methods often rely on heuristic models but do not incorporate soil mechanics. We address this shortcoming by using Reinforcement Learning in a GPU-parallelized Material Point Method particle simulation. Our controllers are conditioned on material state such as shape and compactness, enabling skills that use multiple contact faces of the tool and displace material both inside and outside of the shovel. To use the same learned weights across machines, our policies operate in a normalized end-effector space and are deployed through a calibrated machine interface. We evaluate this calibrated transfer on an 11.5t hydraulic excavator and a 500g tabletop robot. We validate performance through autonomous construction of a 42m long, 2.1m high embankment in 45min, executing 201 individual policy strokes without failure, retry, or operator intervention. In a direct comparison, the autonomous controller matches an expert operator's progression speed and produces a higher, more consistent embankment. Additional qualitative backfilling and compaction experiments demonstrate the material-state awareness and calibrated transfer across machines.

I. INTRODUCTION

Earthmoving requires both accurate machine control and soil-mechanics-aware manipulation, especially for contact-rich reshaping beyond dig-and-dump. The paper combines particle-simulation reinforcement learning with a normalized local skill and reviews gaps in existing planning, learning, and simulation approaches.

I. INTRODUCTION

  • The paper frames automation as a response to hazardous, physically demanding work and a shortage of experienced construction-equipment operators.Its evaluation spans field embankment formation and qualitative backfilling and compaction demonstrations on a full-size excavator and tabletop robot.Source: §I · pp. 1–2
  • The learned local controller uses simulated material state to manipulate soil for embankment formation, backfilling, and compaction, while separating soil interaction from machine-specific motion control.The paper describes a custom parallelized 2D MPM simulation and rewards that can use state such as surface hardness, which is weakly observable from height alone.Source: §I · p. 1

B. Learning-Based Excavation and Earthmoving

  • Existing earthmoving approaches can handle bucket filling and dumping, but contact-rich tasks such as compaction, piling, and backfilling require more advanced tool-soil interaction understanding.The review identifies limited terrain-state representation as a central constraint: heuristic simulations are use-case-specific and expose little terrain mechanics beyond height fields.Source: §II · p. 2, B. Learning-Based Excavation and… · p. 2

A. Model-Based Excavation Planning and Soil-Tool Interaction

  • Trajectory-planning methods can model geometric, kinematic, hydraulic, and soil-tool constraints, but soil mechanics are usually absent from planning techniques, leaving interaction-heavy behavior beyond bucket filling to engineered motions.The passage cites – for planning approaches and their modeling capabilities.Source: A. Model-Based Excavation Planning and… · p. 2

C. Particle and Granular Simulation for Robotic Soil Interaction

  • Particle-based simulators model soil mechanics and material state, but prior approaches cited here generally lacked real-time policy training with material-state rewards for earthmoving.This work aims to expose internal states such as compaction as rewards while providing throughput for model-free reinforcement learning.Source: C. Particle and Granular Simulation… · p. 3

D. Transfer, Embodiment Dependence, and Field Deployment

The method isolates a local soil-interaction skill from machine-specific motion control, then reuses the policy through calibrated interfaces. The controller acts in a single aligned workspace and uses normalized observations and shovel-target actions.

III. METHOD AND DEPLOYMENT ARCHITECTURE

  • The workspace planner selects aligned local workspaces and attack points, while a calibrated interface rescales policy variables and a machine-calibrated controller tracks Cartesian shovel targets.The presented planner is feed-forward in coordinate space, and the local policy does not reposition the excavator or actuate its cabin rotation joint.Source: §III · p. 3, Figure 2 · p. 4

D. Transfer, Embodiment Dependence, and Field Deployment

  • A normalized task-space policy and conventional machine-specific controllers separate transferable soil manipulation from embodiment-dependent actuation, allowing checkpoint reuse after interface calibration.The stack also uses conventional workspace planning and state estimation; the planner selects local workspaces and attack points, while the low-level controller tracks shovel targets.Source: §III · p. 3, D. Transfer, Embodiment Dependence,… · p. 3, Figure 2 · p. 4

A. Single-Workspace Task-Space Policy

  • The policy controls 2D shovel motion in one aligned vertical workspace, using the in-plane boom, stick, and bucket joints while the planner aligns the workspace with the cabin joint.This arrangement matches the workflow of expert operators on heavy hydraulic machines.Source: A. Single-Workspace Task-Space Policy · p. 3

1) Actions:

  • Each action updates the Cartesian shovel target in the arm plane and its angle of attack; a leading target pose is updated before it is reached to mitigate jerky end-effector motion.The low-level machine controller tracks the command, removing hydraulic joint control and inverse kinematics from the learning objective.Source: 1) Actions · pp. 3–4

2) Observations:

  • Observations normalize workspace positions by local dimensions, orientation by 2π, and contact force by machine force capability, while including shovel pose, target coordinate, force utilization, and terrain profile.Terrain samples update only when the shovel is above a threshold; force magnitude indicates proximity to stalling, and training adds pose and force noise.Source: 2) Observations · pp. 4–5

3) Training:

The policy learns soil-manipulation strokes from simulated material state, while calibrated interfaces handle machine-specific control. The simulator and worksite planner support training and deployment across machine setups.

3) Training:

  • Material-state rewards let the simulated shovel learn motions for physical outcomes without embedding machine embodiment in the policy.Training uses a local-workspace simulation with soil and a free-floating shovel controlled through a simplified command-following interface.Source: 3) Training · p. 5

4) Deployment:

  • The deployment interface transforms terrain and rescales shovel targets and force utilization; a stiff MPC-based controller tracks targets on hydraulic machines.The learned checkpoint applies to machines with comparable bucket geometry and compatible workspace aspect ratios.Source: 4) Deployment · p. 5

B. Worksite Supervisor

  • A high-level state machine decomposes work into shovel-width workspaces, positions the machine, and runs the stroke policy to task-specific terminal conditions.The same supervisor structure is used across tasks, with only completion criteria changed.Source: B. Worksite Supervisor · p. 5

A. Soil Simulation

  • A 2D explicit MPM solver models elastoplastic soil, shovel contact, and persistent plastic compaction memory.The memory records mostly hydrostatic plastic compression, hardens the material, and provides a state signal for compaction tasks.Source: A. Soil Simulation · p. 5

B. Solver Additions and Throughput

  • Compaction memory hardens soil and shrinks particle reference volume, while a compressive yield cap bounds stored deviatoric strain.The cap limits frictional strength at high confinement; compaction memory also relaxes rebound while a clamp prevents artificial tensile suction.Source: B. Solver Additions and Throughput · p. 5
  • Around 2000 times real time is reached with four-GPU training; Figure 4 compares control-step throughput across environment counts and three GPUs.The implementation parallelizes over environments and replays captured physics substeps to reduce launch overhead.Source: B. Solver Additions and Throughput · p. 6, Figure 4 · p. 6

C. Learning Soil Interaction

  • Reset randomization varies terrain, shovel pose, soil parameters, compaction, force capability, speed limits, and observation noise.Training across different force limits teaches the policy to adjust its stroke to the capability provided by the deployment interface.Source: C. Learning Soil Interaction · p. 6

D. Reward Functions

The reward functions score material-state change for compaction and target-profile progress for embankment formation and backfilling. Termination rules constrain strokes to task and machine limits.

D. Reward Functions

  • All tasks share an action penalty, and a two-stage curriculum first anchors the control point before expanding reset and observation randomization.The second stage removes the reference-pose term and opens the randomization.Source: D. Reward Functions · p. 6
  • Embankment reward measures attention-weighted distance to a sampled target profile and proved more stable than added-material or peak-height rewards.Progress is measured against the best distance so far, with a dense bonus on the raised target region and a penalty for forces exceeding sampled capability.Source: D. Reward Functions · pp. 6–7
  • Backfilling rewards profile progress toward a uniform level, penalizes movement away from the target, and adds a transport term for source-to-target movement.A Gaussian bonus favors samples already at the target level; the transport term is gated by shovel proximity to the source and soil contact.Source: D. Reward Functions · p. 7
  • Compaction reward uses attention-zone mean plastic compaction, rewarding improvement over the episode best with a dense term.The particle count in the zone penalizes material pushed out, while angle and time terms discourage unhelpful orientations and stalling.Source: D. Reward Functions · p. 7

E. Termination and Training

  • Termination checks enforce time, workspace, force, and task-specific conditions; compaction rules prevent success through pulling or cutting instead of pressure.Backfilling succeeds when profile distance falls below a tolerance, while embankment success follows the shovel exiting the upper workspace.Source: E. Termination and Training · p. 7

V. EVALUATION

The evaluation tests embankment formation on a full-size excavator and calibrated transfer to a tabletop robot, with backfilling and compaction shown qualitatively. In the field trial, the controller built a 42 m embankment in 45 minutes and matched an expert’s progression speed while producing a higher, more consistent embankment in the direct comparison.

2) Per-Stroke Statistics:

  • 201 policy executions displaced 110 m3 (0.55 m3 per execution), with repeatable stroke envelopes despite rocks or hard soil.The shovel penetrates at a high angle, flattens while filling, then re-opens while pulling material out.Source: 2) Per-Stroke Statistics · pp. 7–8, Figure 7 · p. 8

1) Field Deployment:

3) Expert Operator:

  • 2.11 m versus 1.46 m: the autonomous 9 m embankment was higher than the expert’s, with 9 cm versus 11 cm height variation; the expert progressed at 1.03 m/min.The comparison was a single trial at one site and soil condition.Source: Figure 8 · p. 8, 3) Expert Operator · p. 8

B. Validation

  • The same checkpoint transferred through calibrated interfaces to LeExcavator; backfilling used the shovel underside, while compaction pressed soil without visible pile collapse.Backfilling and compaction were qualitative demonstrations on both platforms.Source: B. Validation · pp. 9–10
  • The policy excavated soft soil, scraped hard soil, and changed its trajectory as terrain rose, cutting progressively deeper on later passes.Median penetration increased from about 0.76 m on the first pass to about 0.96 m on the third; median height gain fell from about 0.25 m to 0.13 m.Source: B. Validation · pp. 10, 12, 13, Figure 25 · p. 13
  • 0.158 versus 0.017: only the policy trained with compaction reward improved compaction over baseline across 500 simulated rollouts.Both policies began from the same settled baseline.Source: B. Validation · p. 10, Figure 16 · p. 11
  • A 100 kN versus 50 kN force normalization produced about 15 cm deeper penetration; force-aware policies concentrated usage near Fmax, unlike ablated policies with broader usage and over-limit rollouts.The stronger normalization setting applied about 60 kN, versus about 40 kN with the weaker setting.Source: B. Validation · p. 12, Figure 21 · p. 12
  • Replay with hand-picked training-distribution soil parameters yielded similar simulated and measured force profiles and terrain trends, rather than inferred material parameters.The two-dimensional simulator’s force was scaled by one constant calibrated on a different motion; residual differences were attributed mainly to parameter mismatch and homogeneous simulated soil.Source: B. Validation · p. 13

C. Limitations

The learned skills remain single-stroke, so long-horizon work needs higher-level planning. Policy reuse across machines is conditional, and the simulator omits three-dimensional and heterogeneous-material effects.

C. Limitations

  • Omitting history prevents the policy from learning multistroke motions, so longer-horizon tasks require a higher-level controller.This limitation is visible in compaction: single-stroke compaction succeeds, but repositioning and pushing from above are not learned.Source: C. Limitations · p. 14
  • Reusing the same checkpoint requires compatible workspace aspect ratio, perception, force normalization, and low-level tracking; incompatible setups require additional training.The experiments use compatible shovel geometries, and the simulator does not train over arbitrary bucket shapes.Source: C. Limitations · p. 14
  • The 2D simulator omits heterogeneous materials and three-dimensional effects during training, with task repetition and overlapping workspaces compensating for these gaps.The authors also characterize the simulation replay as qualitative support because its soil parameters were selected manually.Source: C. Limitations · p. 14
  • The replay check provides qualitative support only: soil parameters were manually selected, and force agreement is reported for one tested motion.The measured shovel pose was replayed from a laser-scan terrain initialization; the compared force magnitudes and temporal shapes agreed for that motion.Source: C. Limitations · p. 14, Figure 27 · p. 14, Figure 26 · p. 14

VI. CONCLUSION

The particle-simulation controller learns shovel motions from simulated material state and separates transferable manipulation from machine-specific actuation. Field construction matched an expert’s progression speed while producing a higher, less variable embankment; transfer is currently limited to compatible bucket geometries.

VI. CONCLUSION

  • The controller learns task-space shovel motions from rewards on terrain shape, material displacement, contact forces, and compaction memory.Source: §VI · p. 14
  • The embankment policy changes its trajectory in response to soil softness or hardness, terrain profile, and indicated machine force capability.Source: §VI · p. 14
  • A 42 m field embankment was built at a progression speed comparable to an expert operator, with a higher profile and lower height variation.Source: §VI · p. 15

VII. FUTURE WORK

  • Transfer is demonstrated for compatible bucket geometries because bucket shape is fixed in simulation; future work proposes broader tool-geometry training.Source: §VII · p. 15

APPENDIX A SOIL SIMULATOR IMPLEMENTATION

The simulator extends a two-dimensional explicit Material Point Method solver with compaction memory, shovel coupling, and GPU-parallel execution. Its particle and grid updates model soil deformation and contact, while accumulated contact force supplies a policy observation and reward signal.

APPENDIX A SOIL SIMULATOR IMPLEMENTATION

  • The two-dimensional explicit MPM solver uses NVIDIA Warp GPU kernels and adds compaction memory, shovel coupling, compaction-dependent hardening, and gradual reference-volume shrinkage.Source: Appendix A · p. 17
  • The solver uses an 80 × 48 grid over a 5.0 m × 3.0 m domain, with 50 physics updates per 10 Hz control update, and batches simulations across environments.Source: Appendix A · pp. 17, 19
  • Each soil particle stores position, velocity, deformation gradient, affine velocity, reference volume, and scalar compaction memory; the grid stores separate soil and shovel mass and momentum.Source: Appendix A · p. 17
  • Quadratic B-spline interpolation uses a 3×3 stencil; P2G uses APIC momentum, and after grid operations G2P reconstructs velocity vn+1.Source: Appendix A · p. 17
  • A corotated elastic predictor and Drucker-Prager plasticity are augmented with persistent volumetric compaction memory that supports hardening and compaction-related rewards.Source: Appendix A · pp. 17–18
  • Compaction hardening scales stiffness and cohesion with the common factor 1 + khνp, while friction angle increases additively and is capped.Source: Appendix A · p. 18
  • The shovel is represented by rigidly transformed particles, with grid contact resolving normal collision and Coulomb-like tangential friction.Source: Appendix A · pp. 18–19
  • The accumulated contact force includes normal and frictional resistance and serves as a force observation and reward penalty for commanded shovel motion.Source: Appendix A · p. 19

APPENDIX B LEARNING FORMULATION

Appendix B details the environment setup used in the learning formulation. It covers initialization, randomization, rewards, and termination conditions summarized in Sections IV-C–IV-E.

APPENDIX B LEARNING FORMULATION

  • The appendix details environment initialization, randomization, rewards, and termination conditions summarized in Sections IV-C–IV-E.Source: Appendix B · p. 19

A. Environment Initialization and Randomization

The environment initializes terrain from smoothed random positive and negative features, then samples particles under the resulting profile. It also randomizes shovel and machine limits and soil properties, with soil randomization hand-tuned for task suitability.

A. Environment Initialization and Randomization

B. Reward Functions

The reward design uses staged training and shared control terms alongside task-specific objectives for embankment formation, backfilling, and compaction. These objectives measure terrain-profile change or compaction while penalizing unsafe forces and poorly tracked commands.

  • Training first shapes commanded control points near a reference pose, then removes that shaping and adds terrain-height offsets and greater shovel-position spawn jitter.The switch occurs after mean return reaches a task-specific threshold.Source: B. Reward Functions · p. 20
  • Shared terms penalize large action increments and poor pose tracking, with initialization shaping active only in the first curriculum stage.Reward weights and thresholds are task-specific, so shared symbols can take different values across tasks.Source: B. Reward Functions · p. 20
  • Embankment rewards improve an attention-weighted profile distance to a target whose height exceeds what one stroke can reach, plus a dense bonus over the target feature.The policy observes the target coordinate, but not the target height or attention mask.Source: B. Reward Functions · p. 20
  • Backfilling rewards movement toward a uniform profile across the pile and depression, with consecutive-step progress scoring that penalizes regressions.Attention weights focus the objective on the two features to equalize.Source: B. Reward Functions · pp. 20–21
  • A transport bonus discourages leveling only the pile by rewarding commanded motion from surplus toward deficit when both regions exist and the shovel is at the source, working height, and in soil contact.Transport motion in free space is worth at most βw of its value when loaded.Source: B. Reward Functions · p. 21
  • Compaction rewards progress and continuous compaction while penalizing force-limit violations, particle loss, unsuitable shovel orientation, and stalling.Progress is measured against the best compaction observed so far.Source: B. Reward Functions · p. 21

C. Episode Termination

Episodes end on timeout or workspace-bound violations, with task-specific success and safety conditions. Terminal rewards distinguish successful embankment strokes, completed backfilling, and unsafe compaction motions.

  • Timeouts and workspace-bound violations are shared termination checks, with timeout reward set to zero and workspace violations receiving a small penalty.Episode duration and workspace safety margins define the timeout and bounds.Source: C. Episode Termination · p. 21
  • An embankment episode succeeds when the shovel exits through the upper workspace region at a finishing-stroke angle, receiving accumulated profile-improvement reward.Excessively negative normalized vertical force also terminates the episode with zero reward.Source: C. Episode Termination · p. 21
  • Backfilling terminates successfully when attention-weighted profile distance falls below a tolerance, and also has workspace and shovel-angle bounds.The supplied passage also identifies an excessively negative vertical shovel force as a termination condition.Source: C. Episode Termination · p. 21
  • Compaction terminates unsafe motions, including upward pulling, excessive tangential force, or force dominated by its tangential component rather than downward compaction.These checks prevent satisfying the objective through pulling or cutting.Source: C. Episode Termination · pp. 21–22

APPENDIX C FIELD DEPLOYMENT SYSTEM

The field-deployment system combines worksite planning and supervisory execution with a sequence of locally planned embankment strokes. It supports driving over uneven terrain, LiDAR scans, and repeated policy execution as soil geometry changes.

A. Worksite Decomposition and Supervisor

  • The planner divides work into shovel-width local workspaces, selects the next workspace, and assigns base poses, aligned workspaces, and attack points to cover the target path.A supervisor executes the plan through driving, alignment, scanning, and learned-policy states.Source: A. Worksite Decomposition and Supervisor · p. 22

B. Embankment Mission Execution

  • The target path is discretized into overlapping shovel-width targets, while force-based leg control maintains base height, horizontal alignment, and wheel contact on uneven terrain.A curvature-based controller tracks an offset base path that places targets within arm reach.Source: B. Embankment Mission Execution · p. 22, Figure 30 · p. 22
  • A Reeds-Shepp-based planner routes the machine to the first workspace, and yaw goals allow reverse driving when it gives a shorter path.The target yaw follows the offset base path, keeping legs and wheels outside the arm workspace.Source: B. Embankment Mission Execution · p. 22
  • At each base pose, the shovel moves through an intermediate point before scanning so it does not obstruct the LiDAR view of the workspace.The intermediate point is selected to preserve an unobstructed scan.Source: B. Embankment Mission Execution · p. 22, Figure 31 · p. 22
  • Each scoop targets the far end of the workspace; after three policy strokes per target in loose soil, the system rescans and updates the attack point before continuing.After completing a target, the cabin slews to the next target, and the supervisor advances to the next workspace when its targets are complete.Source: B. Embankment Mission Execution · p. 22
Loading 2609.12677v1…