LAW: Unifying Language, Action, and World Modeling for Agentic Physical Intelligence

1. Introduction

• A one-time language plan cannot verify intermediate completion, while observation-conditioned action generation may miss which changes matter to the larger goal.Errors in grasp assessment, stage recognition, or objective-guided action selection can compound over time.Source: §1 · p. 1
• Semantic subtask progress provides the temporal interface for updating plans, judging actions against objectives, and retaining salient observations.State assessment runs continuously without blocking control, action evaluation occurs when the WAM proposes a chunk, and planning operates at the task scale.Source: §1 · pp. 1–2
• LAW plans remaining semantic subtasks, ranks and fuses WAM action chunks, and asynchronously assesses completion to continue execution or advance the plan.Planning, action selection, and execution monitoring operate at different frequencies.Source: §1 · p. 2
• MoTE-WAM combines shared and modality-specific experts; dual-stream hierarchical memory preserves context, and subtask-local value learning retains successful-prefix supervision from failed episodes.Value returns terminate at semantic boundaries, preserving positive supervision from completed subtasks rather than assigning zero value to every action in a failed episode.Source: §1 · pp. 2–3
• 94.21% average success across 50 RoboTwin tasks; OpenWAM-α raises the average to 95.56%, and LAW averages 65% across three real-robot tasks.The OpenWAM-α result establishes a new open-source state of the art on RoboTwin.Source: §1 · p. 3

2.1. Vision-Language-Action Models for Robot Control

• VLA methods connect semantic representations to control, while CoT-VLA predicts visual subgoals and VEGA aligns visual and geometry-aware features [45].SayCan grounds language plans in affordances, whereas VoxPoser translates language objectives and constraints into spatial maps for trajectory optimization.Source: §2.1 · p. 3

2.2. World Action Models and Video-Based Policy

• World-action models jointly connect future visual modeling with executable actions [27] [30] [32] [51] [53], with differing use of predictive information at deployment.LingBot-VA uses causal video–action modeling with observation feedback and asynchronous execution [27], whereas Fast-WAM omits test-time future prediction [51].Source: §2.2 · p. 3
• LAW coordinates semantic planning, asynchronous state assessment, and value-guided WAM selection, while MoTE uses shared and modality-specific experts and separate memory streams.Unlike trajectory-level returns used by MV-WAM [10], LAW terminates value targets at semantic subtask boundaries, retaining supervision from completed prefixes of unsuccessful episodes.Source: §2.2 · p. 4, Figure 2 · p. 4

3.1. LAW Framework Overview

• The current subtask links a remaining-plan prediction to WAM action chunks, value-based top-K fusion, and execution feedback that triggers replanning at subtask transitions.The planner runs at semantic decision events, while the value expert scores candidate chunks and the state expert assesses completion asynchronously.Source: §3.1 · pp. 4–5

3.2. Hybrid-Expert VLM for Progress-Aware Control

• Three role-specific experts share a frozen VLM backbone but use task-specific LoRA adapters; asynchronous completion assessment supports replanning and improves inference throughput by 15%.The state expert also receives auxiliary failure-prediction supervision.Source: §3.2 · pp. 5–6

3.3. Joint Video–Action Modeling with MoTE

• MoTE adds sparse, modality-routed expert transformations after mixed attention, using private experts to preserve video and action statistics and shared experts to model common dynamics.The default configuration has two private experts per modality and one shared expert.Source: §3.3 · p. 7

3.4. Dual-Stream Hierarchical Memory

• Two memory streams encode shared observation history in VLM and WAM token spaces, retaining full-resolution recent and boundary observations while spatially compressing long-term history.The VLM stream supports planning, state estimation, and candidate evaluation; the WAM stream conditions video prediction and action generation.Source: §3.4 · p. 8

3.5. Subtask-Local Value Learning

• Subtask-local supervision preserves positive targets for completed stages in failed episodes, excludes unobserved subtasks, and withholds positive reward from the failed final subtask.Chunks stop at subtask boundaries, and bootstrapping occurs only when a valid successor chunk remains within the same subtask.Source: §3.5 · pp. 8–9
• At execution, the value expert weights top-K candidate chunks by their scores, averages positions and gripper commands, and combines quaternions with the weighted Markley mean.The robot executes an aggregate chunk prefix before generating new proposals from the next observation.Source: §3.5 · p. 9

4.1. Training Details

• Training uses two stages: planner and state experts learn from semantic annotations while the WAM learns from video-action sequences, then the frozen components support value-expert training on successful and annotated failure trajectories.The real-world training set contains 300 teleoperated trajectories per task across Puzzle, Domino, and Organize.Source: §4.1 · p. 10

4.2. Main Results

• Across 50 RoboTwin tasks, LAW averages 94.21% success (94.22% clean; 94.20% randomized), outperforming selected baselines in both settings; without robot-data pretraining, it beats Fast-WAM (91.83%) by 2.38 percentage points.Source: §4.2 · p. 10
• LAW leads the listed methods across all five RoboDojo dimensions without additional robot-domain or embodied-interaction pretraining, averaging 6.47 / 3.82%.On Long-Horizon it reaches 11.16 / 6.67%, and on Memory 7.46 / 5.44%.Source: §4.2 · pp. 10–11
• 65% average success across three real-robot tasks, versus 28.33% for π0.5 and 25% for Fast-WAM, with tasks spanning three, five, and nine semantic subtasks.On Organize, LAW achieves 65% success and a score of 85.54, exceeding Fast-WAM by 25 percentage points and 15.57 points, respectively.Source: §4.2 · p. 11

4.3. Component Ablations

• Hybrid experts, failure-trajectory supervision, and hierarchical memory improve planning and state estimation; completion accuracy rises to 99.02% and weighted F1 to 99.02%.The full planner reaches 99.64% exact match, compared with 70.10% for a text-only planning baseline without images.Source: §4.3 · p. 11
• Value-guided action selection achieves the best cumulative RoboTwin result at 94.21%, after subtask conditioning and MoTE reach 93.16% and hierarchical memory raises it to 93.80%.The MoTE-only and subtask-only variants provide comparable gains when each is added independently to Fast-WAM.Source: §4.3 · p. 11

5.1. WAM Backbone Extensibility

• 95.56% average success with OpenWAM-α establishes a new open-source state of the art on RoboTwin; LAW also raises Fast-WAM from 91.83% to 94.21%.LAW raises OpenWAM-α from 93.60% to 95.56%, using the same LAW configuration with only the WAM backbone replaced.Source: Table 6 · p. 12, §5.1 · p. 12

5.2. Effect of VLM Choice

• 93.80% average success for the default 2B hybrid-expert VLM exceeds Embodied-R1.5 at 88.8% and GPT-5.5 High at 88.7%.The default VLM also performs better in both evaluation settings; the comparison disables value-guided action fusion.Source: Table 7 · p. 13, §5.2 · p. 13

5.3. MoE Placement and Shared Experts

• 92.67% average success with full MoTE exceeds 91.93% without shared experts; generic Attn-MoE and FFN-MoE remain close to the no-MoE baseline.MoTE transforms representations after mixed attention has exchanged video and action information; modality-specific experts preserve distinct statistics while shared experts capture common transformations.Source: Table 8 · p. 14, §5.3 · p. 14

5.4. Subtask Granularity

• 95.8% ten-task average success with semantic-level subtasks exceeds 92.4% without subtasks and 89.1% with low-level subtasks.Semantic stages provide consistent units for planning, asynchronous completion decisions, and hierarchical memory; the decomposition does not improve every saturated task.Source: §5.4 · pp. 14–15, Table 9 · p. 15

5.5. Subtask-Local Value Learning Ablation

• 90.80% average success on five multi-subtask tasks with subtask-local supervision exceeds 88.20% without value guidance and 87.90% with episode-level supervision.The gains hold in clean and randomized settings; completed stages of failed trajectories retain positive supervision for evaluating actions against the current objective.Source: Table 10 · p. 15, §5.5 · p. 15

5.6. Memory Component Ablation

• 7.46 RoboDojo Memory score and 5.44% success rate with subtask-boundary memory exceed the preceding short- and long-term memory settings.Short-term memory raises score and success rate from 4.06 and 3.44% to 5.57 and 4.00%; compressed long-term memory raises them further to 6.48 and 4.89%.Source: Table 11 · p. 15, §5.6 · pp. 15–16

• LAW uses semantic subtask progress to unify language reasoning, action generation, and world modeling.The framework coordinates specialized planning, completion assessment, and value-guided action fusion at complementary frequencies.Source: §6 · p. 16
• MoTE combines shared and modality-specific experts for video–action modeling, while dual-stream hierarchical memory preserves recent context and semantic-boundary observations.Subtask-local value learning retains supervision from completed stages of failed trajectories.Source: §6 · p. 16
• 95.56% RoboTwin success with OpenWAM-α establishes a new open-source state of the art and demonstrates extensibility to stronger pretrained WAMs.The paper identifies transfer across robot embodiments and adaptive subtask decomposition for longer, more diverse tasks as future work.Source: §6 · p. 16

• 95.56% overall average success with LAW + OpenWAM-α combines 95.60% clean-setting and 95.52% randomized-setting success.The OpenWAM variant uses 100 episodes per task and setting.Source: A. Per-Task RoboTwin Results · p. 21
• 94.22% clean-setting and 94.20% randomized-setting success yield a 94.21% overall average for LAW + Fast-WAM.Means are computed with equal weight per task and, for the overall score, per evaluation setting.Source: A. Per-Task RoboTwin Results · p. 21

• Two six-degree-of-freedom PiPER arms with parallel-jaw grippers operate on the AgileX Cobot Magic platform.Source: §B.1 · p. 22
• Three RGB views comprise a shared-workspace scene camera and one wrist camera on each arm.The scene view provides task-level context, while wrist views resolve local gripper, object, and receptacle interactions.Source: §B.1 · p. 22
• Robot placement and camera mounting geometry remain fixed during data collection in a tabletop workspace.Source: §B.1 · p. 22

• Teleoperated demonstrations vary object positions and in-plane orientations within each task’s reachable workspace, changing required grasps and placements while preserving the task objective.Each trajectory pairs multiview observations with bimanual states, action sequences, a language instruction, and an episode outcome.Source: §B.2 · p. 23
• The collection retains successful, partially completed, and failed executions, with failures annotated by stage and principal cause to preserve completed prefixes.Data validity is assessed separately from task success; episodes with missing data, unusable alignment, or severe manipulation occlusion are excluded.Source: §B.2 · p. 23
• Trajectories use LeRobot format, 30-fps video, and aligned semantic boundaries; bimanual actions are 16-dimensional vectors of position, unit quaternion, and gripper command per arm.Source: §B.2 · p. 23

• Domino requires four domino placements followed by toppling, with success depending on stable placement and alignment for the intended sequence.Source: §B.3 · p. 23
• Puzzle comprises three semantic stages, one per colored piece, and succeeds only when all pieces are matched, aligned, and seated in their corresponding recesses.Source: §B.3 · p. 23
• Organize sorts three colored blocks into matching drawers through nine stages of opening, placing, and closing, requiring correct assignments and closure of every drawer.Source: §B.3 · p. 23

• Complete-task success is recorded per episode as a binary indicator of whether the full task objective is satisfied.Source: §B.4 · p. 23
• Progress gives equal weight to semantic stages, with five, three, and nine stages for Domino, Puzzle, and Organize, respectively.Source: §B.4 · p. 23
• A partially completed execution can earn nonzero progress while remaining unsuccessful under the complete-task criterion.Source: §B.4 · p. 23

C. Motion-Based Progress Labels

• Motion-based labels derive offline progress within each annotated subtask interval from cumulative robot-state motion.Position, rotation, and gripper increments are normalized by their own total subtask motion; components below stated motion thresholds are excluded, and included components receive equal weights.Source: C. Motion-Based Progress Labels · pp. 23–24
• For 14-dimensional joint-state trajectories, progress uses joint-motion proxies rather than Cartesian translation and geodesic rotation, with the same component normalization and thresholds.Source: C. Motion-Based Progress Labels · p. 24
• Completion marks the contiguous suffix with progress at least 0.995, or retains the annotated endpoint if no frame meets that threshold.The threshold gives a narrow endpoint tolerance; motion locates the completion region, while semantic meaning comes from the subtask annotation.Source: C. Motion-Based Progress Labels · p. 24

D. Observation and Action Conventions

• The implementation uses scene and wrist RGB views, separate VLM and WAM visual-memory streams, and absolute 16-dimensional end-effector targets for both arms.WAM inputs include 16-dimensional end-effector state; the VLM state expert receives 14 joint and gripper values.Source: D. Observation and Action Conventions · p. 24

E. Semantic Subtask Decomposition

• Semantic subtasks pair an objective with a trajectory interval; grasp, transport, and release may form one placement stage, while handovers or confirmation actions may be separate stages.Repeated motions for one objective may remain within a single subtask.Source: E. Semantic Subtask Decomposition · p. 24

E.1. Real-World Subtask Sequences

• Real-world sequences specify five, three, and nine stages for Domino, Puzzle, and Organize, respectively.Source: §E.1 · p. 26, Table 15 · p. 26

E.2. Segmented Trajectory Examples

• Simulation examples segment trajectories in execution order by changes in semantic objective rather than uniform temporal sampling.Examples include distinct covering and uncovering stages, transfer and confirmation stages, and repeated screw-fastening operations.Source: §E.2 · p. 27, Figure 14 · p. 28, Figure 15 · p. 28