Source-linked AI summary
LY-GWM-RoboCasa-Human300: Prediction-Guided Action Selection with a Graph Dynamics Model
TianLiang Ma, Ting Ma, ZhongYuan Ma
TL;DR
The report addresses how to select among plausible actions by assessing their consequences. It combines a Human300-fine-tuned GR00T policy with action-conditioned graph prediction and learned reward scoring, then reports a positive historical local difference and a separate official re-evaluation.
Problem
A policy can produce several plausible actions for one observation, so selecting among them requires assessing their consequences.
Source: §1 · p. 1Method
The system samples four GR00T action chunks, predicts each candidate's future features and robot state, and executes the candidate with the highest learned reward logit.
Source: §1 · p. 1, §3.1 · p. 4, §3.3 · p. 4Results
The local joint-policy evaluation recorded 207/2,500 successes, a positive historical difference of 26 over S31; separate official re-evaluation recorded 354/1,500 successes (23.6%).
Source: §10 · p. 10Takeaways & Limitations
The report documents an active prediction-and-selection system and a positive historical difference in the evaluated RoboCasa setting.
Source: §10 · p. 10The official re-evaluation is separate from the local experiment, and its difference from 8.28% cannot be attributed to a model update or isolated graph-dynamics contribution.
Source: §5.4.1 · p. 7
Abstract
from arXiv · showhide
We present LY-GWM-RoboCasa-Human300, a policy that combines a Human300-fine-tuned GR00T-N1.5 action generator, an action-conditioned graph dynamics model, and a learned binary state-reward scorer. At inference, the policy samples four action chunks, predicts the future latent features and robot state for each, and executes the candidate with the highest predicted-state reward logit. Predictions therefore influence the actions sent to the environment. The evaluated system uses a 16-step prediction horizon and keeps all three components frozen during evaluation. In the local evaluation on 50 RoboCasa 1.0.1 tasks in the pretrain kitchen split, with 50 episodes per task, the system records 207/2,500 successes (8.28%), compared with 181/2,500 (7.24%) in the historical self-trained GR00T-N1.5 baseline. The observed difference is 26 successes, or 1.04 percentage points. The atomic group improves from 141/900 to 150/900. Separately, the RoboCasa maintainers re-evaluated the submission and reported 354/1,500 successes (23.6%) across 50 tasks, with leaderboard rates of 48.3%, 16.3%, and 3.1% for Atomic-Seen, Composite-Seen, and Composite-Unseen. The local and official results are reported separately; their difference is not attributed to a model improvement. Recorded local audits identify 212,700 changed environment-input chunks, including 28,223 in atomic tasks. We describe the actual training stages, implemented inference mechanism, evaluation provenance, and public reproduction entrypoints. These results document an active prediction-and-selection system and a positive historical difference.
1. Introduction
The report documents an inference-time composition that uses candidate-specific predictions and learned reward scoring to choose among actions proposed by GR00T. Its contribution is the concrete RoboCasa implementation and evaluated prediction-and-selection pipeline.
1. Introduction
- The system separates action proposal, future-state prediction, and candidate scoring so predicted consequences inform which action is executed.GR00T proposes actions, LY-GWM predicts candidate-dependent outcomes, and a learned scorer selects an action.Source: §1 · p. 1
- The report documents the concrete RoboCasa implementation and its evaluated prediction-and-selection pipeline.Source: §1 · p. 1
- One shared GR00T backbone evaluation supports four action-head samples, which a trained latent graph model and reward head evaluate.The report documents training, interfaces, results, action-selection evidence, and limitations for the joint-policy evaluation.Source: §1 · p. 1
2. Background and scope
The report situates its RoboCasa implementation among established work on generative policies, world models, and graph dynamics, while defining the contribution as a documented composition. Internal identifiers distinguish the policy, dynamics, scorer, and evaluation stages; the report limits claims about broader superiority and generalization.
2. Background and scope
2.1 Component names and experiment identifiers
- S31 evaluates the fine-tuned policy alone at 181/2,500 successes, whereas S42 evaluates the policy, dynamics, and scorer together at 207/2,500.The report also identifies R5 as policy fine-tuning, W5 as graph-dynamics training, and S38 as reward-scorer training; these are separate stages, not successive model versions.Source: §2.1 · p. 2
- S30 is a separate local evaluation of the public GR00T-N1.5 checkpoint on the target setting.Source: §2.1 · p. 2
2.2 Related work and positioning
- Unlike iterative trajectory optimization in Visual Foresight [11] and TD-MPC2 [12], this system predicts one endpoint at t + 16 for each of four proposed chunks and selects the highest state-reward logit.The report says it does not use a terminal value function or claim those systems' planning capabilities.Source: §2.2 · p. 3
- The system uses separately trained, frozen components rather than online reinforcement learning or actor-critic updates on imagined trajectories.The report connects this design to DreamerV3 [13] and DayDreamer [14] through the use of learned action consequences for control.Source: §2.2 · p. 3
- Its 16 feature nodes are not supervised object identities, and learned graph edges are not established physical or causal relations.The report therefore limits interpretation of the graph as an object-level physical simulator and describes its contribution as a documented RoboCasa Human300 composition, not evidence of superiority or unrestricted generalization.Source: §2.2 · p. 3
3. Implemented method
The implemented policy samples four action chunks from a shared GR00T backbone evaluation, predicts each candidate's future latent features and robot state with a graph dynamics model, and selects by learned reward logit. It executes the selected chunk while all model weights remain frozen.
3.1 Inputs and candidate generation
- Four action-head samples share one cached backbone evaluation; three additional calls use separately seeded random streams, and replay checks differ from the original output by at most 1e-6.The backbone is not run four times, and candidate seeding is deterministic from recorded task, request context, and candidate identity.Source: §3.1 · p. 4
3.2 Action-conditioned graph dynamics
- The graph predictor uses 16 feature nodes and one robot-state node, action-conditioned message passing, and residual output heads to predict state at t + 16.The complete candidate chunk conditions predictions of future features and robot state.Source: §3.2 · p. 4
3.3 Predicted-state scoring and selection
- For each candidate, the scorer combines mean-pooled predicted features with predicted state and selects the candidate with the maximum reward logit.The scorer has no explicit goal input and is trained on recorded same-frame binary reward, not discounted return or calibrated eventual rollout success.Source: §3.3 · p. 4
- Each request encodes observations once, predicts and scores four candidates, then executes up to 16 actions from the selected padded output before observing again.All model weights remain frozen; environment rewards and observed-future diagnostic probes do not enter selection.Source: §3.3 · p. 4
4. Training and data
Training separates Human300 policy fine-tuning, graph-dynamics fitting, and reward-scorer training. The graph and reward stages do not update GR00T, and no model is updated during evaluation.
4. Training and data
- Three separately trained stages produce the policy, graph dynamics, and reward scorer; the graph and reward stages do not update GR00T.No model is updated during S42 evaluation.Source: §4 · p. 5
4.1 GR00T-N1.5 policy fine-tuning (R5)
- R5 fine-tunes the GR00T-N1.5 projector and action model on 30,000 Human300 demonstrations while freezing the language and visual backbones.The final checkpoint is step 120,000.Source: §4.1 · p. 5, §4 · p. 5
4.2 LY-GWM graph dynamics training (W5)
- W5 fits future feature and state prediction on 108,000 training and 12,000 validation 16-step pairs, held out from W5 and S38 but not R5.Its validation data are demonstration-action pairs.Source: §4.2 · p. 5, §4 · p. 5
4.3 State-reward scorer training (S38)
- S38 trains the binary reward scorer on observed current and future states labeled with recorded rewards; W5 validation predictions do not train or select it.Checkpoint selection uses observed-validation BCE.Source: §4.3 · p. 5, §4 · p. 5
5. Local evaluation protocol and results
The local S42 evaluation records a positive historical difference over S31, while offline dynamics validation and the independent official re-evaluation are reported separately. The official result is not a replacement for the local comparison.
5.1 Benchmark setting
- The local benchmark covers 50 RoboCasa 1.0.1 pretrain-kitchen tasks, with 50 episodes per task and recorded task horizons.Success is the worker's logical OR of chunk-end success over an episode.Source: §5.1 · p. 6
5.2 Comparison with the self-trained baseline
- 207/2,500 (8.28%) for S42 versus 181/2,500 (7.24%) for S31, a +1.04 percentage-point local difference.Table 1 includes all 2,500 episodes in each arm.Source: Table 1 · p. 6, §5.4.1 · p. 7
5.3 Offline dynamics validation
- Table 2 evaluates future-feature and robot-state prediction errors against persistence on 12,000 held-out W5 pairs of demonstrated actions.Validation is used for W5 checkpoint selection.Source: Table 2 · p. 6, §5.3 · p. 6
5.4 Official independent re-evaluation
- 354/1,500 (23.6%) is the official independent RoboCasa re-evaluation result.The maintainers published the re-evaluation in leaderboard PR #19.Source: Table 1a · p. 7, §5.4 · p. 7
5.4.1 Relationship to the local experiments
- The official 23.6% is a separate re-evaluation, not a replacement denominator for the local result or evidence of a model-update improvement.The public re-evaluation comment does not specify every runtime setting.Source: §5.4.1 · p. 7
5.4.2 Official task-level results and source
- Appendix A.1 reproduces all 50 task rates from the maintainer's comment, while Appendix A retains the local counts and recorded horizons.The official source and leaderboard are cited in.Source: §5.4.2 · p. 7
6. Outcome and action-selection analyses
The outcome analyses describe uncertainty in the historical local comparison, document how often selection changed environment inputs, and report out-of-bound action values retained in the evaluation.
6.1 Exploratory paired outcome accounting
- The task-level counts show 20 increases, 13 decreases, and 17 unchanged; matching constructor seeds does not ensure identical later states after different actions.The aggregate difference is 102 gains minus 76 losses; the analysis was added after the rollout and was not a prespecified test.Source: Table 3 · p. 8, §6.1 · p. 8
- The exploratory task-level bootstrap interval is [0.00, 2.12] percentage points, while the paired exact binomial calculation gives p = 0.06065 under its stated null.The bootstrap resamples task mix, not policy seeds; the paired calculation is descriptive given task clustering and the historical design.Source: §6.1 · p. 8
6.2 Evidence that selection affects environment inputs
- 74.947% of selections were non-first candidates, and the audit marks prediction and score divergence on all 283,800 chunks.Changed inputs are measured against candidate zero in the same request and refer to chunks passed to the environment, not post-controller motor commands.Source: §6.2 · p. 8, Table 4 · p. 8
6.3 Result aggregation and action-input diagnostics
- The acceptance rule was revised after rollout to treat action-bound counters as diagnostics, without changing recorded actions, seeds, rewards, or outcomes.The revised collection rule retained episode, report, and evidence checks.Source: §6.3 · p. 8
- 116,968 selected-chunk scalars and 116,292 consumed-prefix scalars were out of bounds, with the score retaining all 2,500 episodes.The worker applies the original inverse transforms without extra clipping or denormalization.Source: §6.3 · p. 8
7. Public-checkpoint reference results
The public-checkpoint result is presented as a separate historical reference, distinct from the pretrain-split evaluations and the official leaderboard report.
7.1 Evaluation setting and recorded results
- The public GR00T-N1.5 checkpoint recorded 212/2,500 successes (8.48%) in the historical S30 local evaluation on the target split.S31 and S42 used the pretrain split, so this is a separate reference rather than a same-protocol comparison.Source: §7.1 · p. 9
- Table 5 is a separate historical reference, and its task-group labels do not remove the kitchen-split difference from S42.The author audit identifies the S30 protocol digest in Appendix B.Source: Table 5 · p. 9
- As checked on 25 September 2026, the official leaderboard listed GR00T N1.5 at 23.9% overall, with 50.7%, 14.8%, and 2.7% by group.The leaderboard noted reevaluation with a 1.5x longer horizon relative to the paper's results.Source: §7.1 · p. 9
8. Public artifacts and reproduction resources
The public materials connect the described method to fixed implementation and result records, while distinguishing inference and evaluation entrypoints from full retraining.
8. Public artifacts and reproduction resources
- The release provides inference implementations, reviewer entrypoints, training materials, weights, episode records, audit summaries, and trace hashes.These materials connect the method description to fixed implementation and result records.Source: §8 · p. 9
- Training documentation covers policy fine-tuning, graph-dynamics training, and reward-scorer training, while released execution entrypoints focus on inference and evaluation.Full retraining additionally uses custom data preparation and orchestration, caches, and reward labels.Source: §8 · p. 9
- Historical verification uses exported episode records and audit summaries; a fresh evaluation generates its own complete selection and execution traces.A new evaluation also requires compatible local environments and simulator assets.Source: §8 · p. 9
9. Reproduction procedure
The reproduction procedure pins code and model revisions, verifies exported historical records, and provides steps to prepare and run a new evaluation.
9. Reproduction procedure
- Use the specified fixed code and model revisions rather than assuming a mutable main branch reproduces the evaluated system.The report lists immutable revisions for evaluation code, inference release, upstream training source, and base model.Source: §9 · p. 10
- The historical verification script checks exported file hashes, all 2,500 episode records, 207 successes, receipt digests, and recorded action-evidence totals.This is a CPU record-integrity check; new GPU evaluation follows the preparation and execution procedure.Source: §9 · p. 10
- Prepare a new run with the fixed model snapshot, policy and simulation environments, source trees, and simulator assets.The preparation script checks source, configuration, and weight hashes, and writes to a new output directory outside the repository.Source: §9 · p. 10
- A full new evaluation runs task indices 0 through 49 on allocated GPUs before collecting results.The scripts do not allocate or submit GPU jobs; the collector checks complete coverage and newly generated selection and execution traces.Source: §9 · p. 10
- Partial-task continuation and mixing episodes from different jobs are unsupported, and a fresh run may obtain a different success count.The archived cluster job scripts contain historical paths and are not the reviewer entrypoint.Source: §9 · p. 10
10. Conclusion
The report documents an active prediction-guided action-selection system and a positive historical local difference, while keeping local and official evaluations separate. The appendix identifies the local task-result table’s success-count and horizon conventions.
10. Conclusion
- 207/2,500 local successes versus 181/2,500 for the historical S31 baseline is a positive difference of 26 successes; official re-evaluation is reported separately.The official result is 354/1,500 successes (23.6%).Source: §10 · p. 10
- S42 implements prediction-guided selection on a Human300-fine-tuned GR00T-N1.5 policy, and the evaluated components remain frozen during evaluation.The report describes the system as a joint-policy evaluation.Source: §10 · p. 10
Appendix A. Local task results and horizons
- The local task table records successes rather than percentages, with 50 episodes per policy in every row; H denotes the recorded maximum environment-step horizon.S31 is baseline-policy evaluation, S42 is joint-policy evaluation, and Delta is joint minus baseline.Source: Appendix A · p. 11
Appendix A.1. Official re-evaluation task results
The official re-evaluation reports task-level rates across Atomic-Seen, Composite-Seen, and Composite-Unseen tasks, with 30 episodes per task. The accompanying source map distinguishes evaluation records, archival evidence, and public reproduction materials.
Appendix A.1. Official re-evaluation task results
- The official group totals use 30 episodes per task, and the report explicitly keeps these measurements separate from local horizons and success counts.The rates are transcribed from the RoboCasa maintainer's re-evaluation.Source: Appendix A.1 · p. 12
Appendix B. Provenance and source map
- The source map points to the revised S42 result and receipt, the historical baseline episode outcomes, and reviewer scripts for verification and new runs.The supplement adds analysis of exported outcomes but no new GPU experiment.Source: Appendix B · p. 13, Data and code availability · p. 14
- The W5 and S38 training workers were author-archived and byte-hashed; the pinned inference repository does not contain the full workers.The archival evidence corroborates specified training details, while the supplement exposes relevant non-private facts and recorded digests.Source: Appendix B · p. 13
Data and code availability
- Fixed code and weights are available at, with a reviewer guide in; the conceptual preprint and implementation report serve different purposes.The report recommends linking the two when describing the submission.Source: Data and code availability · p. 14