X-Planner: Event-Structured Task Planning for Embodied Intelligence

• The listed project code repository is https://github.com/X-Square-Robot/Xplanner, dated September 22, 2026.Source: X Square Robot Team · p. 1

• The data gap is that task-level labels can conceal event boundaries, retries, failed contacts, and pose corrections needed to supervise ongoing planning.Source: §1 · p. 1
• The granularity gap is that fixed semantic levels or visual targets can split one behavior or merge several, unlike action-grounded semantic events.Source: §1 · p. 1
• The latency gap is that token-by-token CoT can repeatedly process overlapping context, while serial latent steps and long traces retain delay or error-propagation concerns.Source: §1 · p. 2
• X-Planner is a planning front-end that turns high-level instructions and multi-view observations into event-structured representations conditioning a downstream world-action model.Source: §1 · p. 2
• Its Ego, UMI, and teleoperation supervision uses a shared hierarchy with source-dependent depth, takeover-time labels, and human-designed failures.Source: §1 · p. 2
• The shared VLM offers explicit event states and continuous CoT states relayed across staggered Transformer depths; frozen latent-to-text training anchors their semantics.Source: §1 · p. 2
• X-Planner ranks above Qwen and Doubao but below kimi3 on both offline planning metrics, while its coupled system leads in average Task Progress on the reported suites.Source: §1 · p. 2
• Controlled comparisons between the discrete and latent plan forms are left for future work.Source: §1 · p. 2

2.1 Reasoning and Planning in VLA Models

• VLA systems often lack explicit task structure; linguistic and visual CoT approaches trade semantic readability against spatial specificity and generation cost.Source: §2.1 · p. 2

2.2 Explicit, Latent, and Parallel Reasoning

• Latent reasoning reduces textual serialization, but many methods retain serial dependencies; Staircase Decoding relays plan states across staggered Transformer depths.Source: §2.2 · p. 2

2.3 Data and Annotation for Embodied Planning

• Episode-level task strings conceal sub-goal structure, motivating finer supervision; X-Planner combines Ego, UMI, and teleoperation with takeover-time and designed-failure annotations.Source: §2.3 · p. 3

3.1 Sources and Hierarchical Annotation

• Teleoperation annotations span L0–L3, while Ego and UMI retain L1–L3 because their faster motions make L0 boundaries difficult to annotate reliably.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2 Failure-Aware Ongoing Supervision

• Takeover times supervise error recognition during execution, and human-designed failure demonstrations broaden coverage beyond the collecting policy’s failure distribution.Source: §3.2 · p. 3

3.3 Coverage and Selection

• The deterministic subset retains 1,500 of 1,654 candidate episodes, removing redundant frequent combinations while preserving the observed tail; fixed-seed tie-breaking makes selection repeatable.Source: §3.3 · pp. 3–4
• The analysis subset’s temporal distributions have medians of five subtasks and 37.1 seconds, with 90th percentiles of 15 subtasks and 113.1 seconds.Source: Figure 2A · p. 4
• The action and task coverage counts are multi-label episode shares; 41 atomic-action labels were inferred from two ground-truth subtask captions per episode.Source: Figure 3a · p. 4
• Source-defined action- and subtask-complexity bins are not calibrated difficulty scores, and deterministic selection trims dominant modes, including rigid-object share by 3.7 percentage points.Source: §3.3 · p. 4, Figure 3D · p. 4

C Complexity bins (%)

• Panels A–C show episode shares and complexity-bin composition, while D–E report retained-minus-candidate shifts in percentage points, with multi-labels where applicable.All panels use the same subset and denominator.Source: Figure 3 · p. 5

4 Method

• Given a high-level instruction, multi-view observations, and optional history, X-Planner produces event-structured representations that condition a downstream world-action model.Source: §4 · p. 5

4.1 Two Plan Forms

• The discrete form exposes inspectable event states, while the latent form generates compact continuous CoT states in parallel for downstream cross-attention.Source: §4.1 · p. 5

4.2 Text Conditioning Interface

• A shared text-conditioning interface projects selected text-token states into downstream space, where they join image tokens in world-action cross-attention.Source: §4.2 · p. 6

4.3 Staircase Latent CoT

• Staircase Decoding shares lower-layer visual-language context through a relay representation, then updates latent reasoning states in parallel across upper blocks.This reduces architectural serial computation but does not by itself establish lower end-to-end control latency.Source: §4.3 · pp. 6–7

4.4 Frozen Latent-to-Text Reconstruction

• A frozen language model reconstructs textual CoT from a soft prefix projected from latent states, encouraging retained high-level semantics without matching token-level hidden trajectories.Source: §4.4 · p. 7

4.5 Inference: Coupling X-Planner to a World-Action Model

• Event mode updates structured states as observations arrive; unified mode uses Kc continuous CoT states to condition fixed-length action chunks through the same cross-attention interface.Source: §4.5 · p. 7

5.1 Structured Plan-State Supervision

• Ego and UMI provide L1–L3 labels, while teleoperation adds L0; takeover times and human-designed failures supervise ongoing error recognition.The full hierarchy is L3 Task, L2 Subtask, L1 Action, and L0 Segment.Source: §5.1 · p. 7, §5 · p. 7
• The structured response jointly supervises captions, progress, continuation state, and annotated errors, with missing annotation levels excluded rather than treated as negative labels.Training targets the complete compact JSON response rather than only the next-event caption or an independent progress scalar.Source: §5.1 · pp. 7–8

5.2 Feature Alignment for Text Conditioning

• Feature alignment regularizes the planner conditioning path toward the original text encoder’s feature space expected by the fixed downstream interface.It is a separate model-side objective, not an alternative assistant target.Source: §5.2 · p. 8

5.3 Staircase Distillation

• The staircase branch produces latent plans for downstream cross-attention, while a frozen language model reconstructs annotation-derived textual CoT traces from a soft prefix.Latent reconstruction updates the staircase branch and prefix projector; it is distinct from compact-JSON SFT and uses different targets and readouts.Source: §5.3 · p. 8
• The objectives have separate roles, and the system-level experiments do not isolate their individual causal contributions.Structured-response SFT defines the target, feature alignment regularizes the downstream interface, and latent reconstruction supervises the staircase representation.Source: §5.3 · p. 8

6.1 Offline Two-Step Planning Evaluation

• X-Planner ranks second among four models on both offline metrics: BERTScore-F1 is 0.9011 and Overall is 1.411.The scores exceed Qwen and Doubao and remain below kimi3 on both metrics.Source: §6.1 · p. 9, Table 2 · p. 9

6 Experiments; 6.2 Real-Robot Evaluation

• The evaluation pairs two-step text comparisons with tabletop bimanual-robot trials, using 0–100 Task Progress to credit partial completion.The real-robot system runs in event mode, where the planner’s next-event description conditions the world-action backbone; U-Scratch is a system-level reference, not a component-isolating intervention.Source: §6 · p. 8, §6.2 · p. 9

6.2 Real-Robot Evaluation

• Robot suite means are descriptive comparisons only: trial-level scores and uncertainty estimates are unavailable, and no error bars are shown.The figure should not be read as a variance or significance analysis.Source: §6.2 · p. 9, Figure 5 · p. 10

6.2.1 Reasoning Manipulation

• Reasoning Manipulation tests category grounding, relational matching, ordered execution, and instruction-conditioned selection across five tasks.Task-level results vary: U-Scratch leads on shape classification, π0.5 on item pairing, and X-Planner and U-Scratch tie on headphone sorting at the same rounded score.Source: §6.2.1 · p. 9, Table 3 · p. 10

6.2.1 Reasoning Manipulation; 6.2.2 Generalization

• 71.60 Task Progress for X-Planner versus 59.50 for U-Scratch in Reasoning Manipulation, and 53.75 versus 28.50 for DreamZero in Generalization.These are the highest reported suite means in the respective comparisons; Figure 5 presents the suite-level results.Source: §6.2.1 · p. 9, Figure 5 · p. 10, §6.2.2 · p. 10

6.2.2 Generalization

• Generalization measures instruction-conditioned transfer in a shared, cluttered tabletop scene; LingBot-VA has no result available for this suite.The reported comparison includes X-Planner, DreamZero, π0.5, and U-Scratch.Source: §6.2.2 · p. 10, Figure 5 · p. 10

6.3 Interpreting the Results

• The evaluations characterize planning-text quality and downstream execution, but do not isolate component contributions or establish an end-to-end latency advantage.The reported results support event-grounded planning as an interface between high-level instructions and action generation, while leaving a gap to kimi3 in offline text comparison.Source: §6.3 · p. 10

7 Discussion and Conclusion

• Ego, UMI, and teleoperation data share a hierarchy with source-dependent annotation depth, while takeover times and designed failures supervise ongoing error recognition.Designed failures are intended to reduce dependence on a single policy’s error distribution.Source: §7 · pp. 10–11
• BERTScore-F1 of 0.9011 without baseline rescaling and a Doubao Judge Overall rating of 1.411 place X-Planner above Qwen and Doubao and below kimi3; event-mode also leads average Task Progress on both robot suites.The reported robot comparison uses the complete event-mode system.Source: §7 · p. 11
• Discrete event descriptions offer an interpretable plan for inspection or revision, while the downstream model remains responsible for physical realization.Source: §7 · p. 11
• The latent interface uses textual reconstruction as a semantic anchor, and Staircase Decoding shortens the planner’s serial decoding path; end-to-end latency still depends on execution and scheduling.Source: §7 · p. 11
• Evaluation is limited to offline planning text and tabletop bimanual execution; offline aggregate scores do not establish physical success or statistical significance.Training histories and optional initial plans are annotation-derived, unlike deployment’s rolling model outputs; rollout-aligned supervision is needed to narrow this oracle-context gap.Source: §7 · p. 11

Contributors and Acknowledgments

• The listed contributors include Howard Lu, Shalfun Li, Porter Pan, Cris, Lumen, Cyril, Eric Hu, and the other named authors.Source: Contributors and Acknowledgments · p. 11
• Howard Lu is marked as a core contributor, Shalfun Li as project lead, and Hao Wang as corresponding author.Source: Contributors and Acknowledgments · p. 11