RynnUnify: Towards Unified Embodied Brain

1 Introduction

• Existing unified multimodal models face challenges sustaining long-horizon interaction because scene changes and prediction errors accumulate, and static predictions can miss intermediate dynamics.The paper motivates interleaved reasoning to connect embodied understanding with temporally coherent video prediction.Source: §1 · p. 2
• The 16B model combines an autoregressive reasoner and diffusion generator, while an RL-post-trained agent selectively invokes the frozen model to interpret predicted videos.The full model is also extended to a World-Action Model through latent planning and joint video-action training.Source: §1 · p. 2
• Rynn-VP evaluates long-horizon plan consistency, while Rynn-GEU tests reasoning about unobserved consequences and whether generated videos improve answers to the same questions.Rynn-GEU covers action consequences, physical changes, and spatial relationships after motion or viewpoint changes.Source: §1 · p. 2
• RynnUnify achieves state-of-the-art embodied understanding among evaluated UMMs, while its WAM reaches 51.5% success on 20 selected composite RoboCasa365 tasks.Astribot S1 experiments also demonstrate effective multi-step manipulation involving body reorientation, precise object handling, and history-dependent execution.Source: §1 · p. 2

2 Model Architecture

• The autoregressive reasoner and rectified-flow generator use separate Transformer parameters and interact through joint attention; the generator attends to reasoner representations.The generator adds cross-chunk scene memory and first-and-last-frame-conditioned generation while retaining the reasoner architecture unchanged.Source: §2 · p. 3

2.1 Reasoner

• For visual planning, the reasoner uses the task goal, current and initial observations, and prior subtask history to predict the next executable subtask, anticipated visual evolution, and duration.The generated clip’s final frame becomes the updated observation for subsequent reasoning.Source: §2.1 · p. 3

2.2 Visual tokenization and input organization.

• Scene-memory frames are independently encoded token groups placed before the main-video tokens, rather than inserted as extra frames on the video’s temporal axis.The generator processes memory and video tokens through existing attention layers, without a separate memory encoder or additional cross-attention modules.Source: §2.2 · p. 4

2.3 Cross-chunk scene memory.

• Four reference frames retain a persistent initial observation, selected earlier-history keyframes, and a recent-step reference; a continuous prefix carries recent motion across chunks.The references anchor scene appearance and layout, while the prefix preserves recent motion and interaction context.Source: §2.3 · p. 4

2.4 Generation modes and fixed conditions.

• I2V fixes the first latent frame, FL2V fixes the first and final frames, and continuation fixes a prefix from the preceding video’s tail.Scene-memory and conditioning latents remain fixed during denoising; only positions to be predicted contribute to generation loss.Source: §2.4 · p. 4

2.5 Textual and temporal conditioning.

• Structured conditions specify the subtask, anticipated visual evolution, and duration, while variable-length videos use the VAE-compatible form 4n + 1 frames.For continuation, the requested duration covers the new operation and excludes the historical prefix.Source: §2.5 · p. 5

3 Data; 3.1 General Understanding Data

• General understanding uses 10% of the RynnBrain pretraining corpus and combines broad image, video, temporal, embodied, spatial-grounding, and manipulation-planning supervision.The data are intended to retain broad understanding capabilities alongside interleaved video–language reasoning.Source: §3.1 · p. 5

3.2 Visual Planning Data

• Visual-planning demonstrations are organized into temporally ordered subtask instructions and clips; interleaved examples connect task decomposition, visual prompts, and continuous video prediction.Pretraining first fine-tunes the reasoner, then freezes it while training the generator with self-forcing and ground-truth future-video supervision.Source: §3.2 · pp. 5–6, Figure 3 · p. 10

3.3 Generation Enhanced Understanding (GEU) Data; 3.3.1 RynnUnify-Phys

• GEU examples interleave an observation and question with a world-modeling instruction, video context, and reasoning; RynnUnify-Phys questions use object trajectories and events in future continuations.The Phys data include 10,028 supervised fine-tuning examples and 1,553 numerical questions for reinforcement learning.Source: §3.3 · p. 7, §3.3.1 · p. 7

3.3.2 RynnUnify-Mess

• RynnUnify-Mess models how controlled interventions in cluttered tabletop scenes affect other objects, using simulated dynamics, generated videos, and questions about resulting changes.Its examples connect an intervention question and video-generation prompt with a rollout and answer.Source: §3.3.2 · pp. 7–8

3.3.3 RynnUnify-Spatial

• RynnUnify-Spatial covers position, direction, distance, and size; its position subset contains 100,000 examples, and paired-frame video context supports prompt formulation and spatial reasoning.Position questions span object–camera, camera–camera, camera-motion, and object–object relations.Source: §3.3.3 · p. 8

3.3.4 RynnUnify-ActionReasoning

• RynnUnify-ActionReasoning covers seven planning and predictive-world-modeling capabilities, retaining 14,000 examples split into 13,300 training and 700 test examples.The seven categories include action-primitive selection, translation and reach, intent, push direction, rotation and orientation, metric action, and affordance and feasibility.Source: §3.3.4 · pp. 8–9

3.3.5 RynnUnify-Affordance and Area

• RynnUnify-Affordance and RynnUnify-Area pair task-conditioned interaction-point or target-region questions with execution clips and world-modeling descriptions, yielding 33,813 training examples.The total comprises 20,039 affordance samples and 13,774 area samples.Source: §3.3.5 · p. 9

3.3.6 RynnUnify-Trajectory

• RynnUnify-Trajectory predicts an ordered sequence of up to ten image-space points with coordinates normalized to 0–1000; its dataset contains 9,892 examples.Examples with projected paths shorter than 20 pixels are excluded, including motions primarily in depth with little image-plane displacement.Source: §3.3.6 · p. 9

4.1.1 Training Objectives

• Reasoner fine-tuning targets spatiotemporal understanding, physical grounding, subtask planning, and video-prompt formulation to support later reasoning over predicted visual evidence.Prompt supervision describes the current scene and anticipated visual evolution.Source: §4.1.1 · p. 10

4.1.2 Training Format

• Training interleaves observations, questions, generation instructions, continuation videos, and reasoning so the model learns to formulate predictions and interpret their visual evidence.For planning, the reasoner predicts an executable subplan, visual prompt, and operation duration from the goal, observations, and subtask history.Source: §4.1.2 · pp. 10–11

4.1.3 Optimization

• The reasoner uses autoregressive loss on assistant response tokens, supervising prompt- and answer-generation turns while excluding system and user turns and their visual inputs.Only the reasoner is optimized in this stage.Source: §4.1.3 · p. 11

4.2 Stage 2: Generator Fine-tuning

• Generator fine-tuning combines I2V, FL2V, and cross-chunk continuation with scene references and recent video prefixes, applying flow-matching supervision only to predicted main-video positions.The training objective excludes fixed conditioning positions and independently encoded scene-memory items.Source: §4.2 · p. 11, §4.2.1 · p. 12, §4.2.2 · p. 12

4.2.3 Self-forcing Training

• Self-forcing exposes the generator to degraded starting frames and cached generated endpoints or histories, while retaining the ground-truth initial observation as a scene reference and the ground-truth continuation as target.Joint history and endpoint replacement uses the same generated predecessor for historical context and the current starting observation.Source: §4.2.3 · pp. 12–13

4.3 Stage 3: Agentic Reinforcement Learning

• BCPO trains the agent to improve reasoning along direct and generation-assisted paths and to select between them, while keeping the video generator fixed.For each question, it samples four trajectories per path and autonomously selects a branch at inference.Source: §4.3 · pp. 13–14

4.3.1 Optimization Objective

• Within-branch reward comparisons improve reasoning on each path, while soft routing targets favor generation when its reward is higher and combine protocol validity with task-answer quality.Task-specific scoring uses measures such as relative error for numerical and spatial questions and geometric distances for localization and trajectories.Source: §4.3.1 · p. 14, §4.3.2 · p. 15

4.3.3 Training Data and Recipe

• The agent is initialized from RynnUnify’s fine-tuned 8B reasoner and trained on 22,007 questions spanning embodied perception, physical prediction, spatial reasoning, interaction outcomes, and localization.Each update uses 64 questions and eight rollouts per question; the generator and reference policy remain fixed.Source: §4.3.3 · p. 15

5 Post-training for the World-Action Model (WAM)

• The WAM extension adapts RynnUnify for robot control in three training stages.Robot adaptation uses about 1,600 hours of atomic-task and 177 hours of composite-task trajectories from RoboCasa365.Source: §5 · p. 15, §5.2 · p. 16

5.1 Model Architecture

• The reasoner turns task instructions and visual context into a subtask latent that conditions future-video generation, whose features drive action-chunk prediction.An auxiliary camera-pose head provides geometric supervision during training.Source: §5.1 · pp. 15–16, §5.2 · p. 16, Figure 5 · p. 16

5.2 Training

• Stage I trains the reasoner to identify subtasks; Stage II trains the generator with the reasoner frozen; Stage III jointly trains the generator and action head while keeping the reasoner frozen.Stage III also freezes the generator’s original action output layer; the auxiliary camera-pose head predicts future poses relative to the current observation.Source: §5.2 · p. 16
• At inference, the action head uses 16 Euler steps to generate a 48-step action chunk, but only the first 16 actions are executed.Video features are extracted once and reused during action sampling; the camera-pose head is training-only.Source: §5.2 · p. 17

6.1 Rynn-VP: Long-Horizon Visual Planning

• Rynn-VP evaluates 200 manually collected egocentric task scenarios using generated videos or ordered step images and a simulator-free protocol.The protocol assesses task progress, operation dependencies, scene continuity, and physical plausibility against held-out reference demonstrations and subtask annotations.Source: §6.1 · p. 17
• Rynn-VP scores Planning Intelligence, Visual Realization, and Visual Quality on a 0–100 scale.The dimensions cover goal achievement and plan validity, visual and physical consistency, and perceptual quality and task-relevant evidence clarity.Source: §6.1 · p. 18

6.2 Rynn-GEU: Embodied Understanding with Generated Visual Context

• Rynn-GEU evaluates whether video prediction helps reason about unobserved states affected by actions, physical interactions, or viewpoint changes.Its ten subsets cover spatial relations, physical prediction, chain reactions, action reasoning, affordance and area grounding, and trajectory prediction.Source: §6.2 · pp. 18–19
• Rynn-GEU examples are filtered to emphasize questions answered better with future visual evidence, while retaining some examples answerable without it.Training and evaluation scenes are separated to limit scene memorization; the future-video comparison is a selection heuristic, not a definitive label of whether generation is necessary.Source: §6.2 · p. 19
• Rynn-GEU uses task-specific metrics on a common 0–100 scale, including choice accuracy, exact matching, normalized distance errors, and spatial agreement between point sets.Trajectory scoring evaluates path geometry and point ordering using discrete Fréchet distance; Mess scores predicted effects on non-manipulated objects.Source: §6.2 · p. 19

7.1.1 Foundational Embodied Understanding

• 57.9 average score makes RynnUnify the highest-scoring evaluated UMM, with leads on six of ten embodied-understanding metrics.It scores 83.0 on EmbSpatial and 60.3 on RefSpatial-Bench, and improves ShareRobot Affordance from 28.0 to 41.8.Source: §7.1.1 · p. 20

7.1.2 Embodied Reasoning with Agentic Visual Generation

• 50.0 average on Rynn-GEU makes the agentic extension the strongest compared method, up from RynnUnify’s direct-answering score of 44.0.It improves seven of ten tasks over direct answering and reaches a 44.6 average across public reasoning benchmarks.Source: §7.1.2 · pp. 20–21, Table 3 · p. 21

7.1.3 Case Study

• Qualitative examples show direct answers from observations and generation-assisted reasoning using task-specific prompts, sampled video frames, and textual or spatial outputs.Examples include object-motion reasoning, camera-view transitions, placement points, and gripper trajectories grounded in the input image.Source: §7.1.3 · pp. 22–23, Figure 7 · p. 22

7.2.2 Main Results

• 54.80 overall, RynnUnify leads the evaluated UMMs on Rynn-VP, improving over Cosmos 3 Nano’s 45.05; PI and VR rise while VQ is slightly lower.The overall score is driven by planning and visual realization.Source: Table 4 · p. 23, §7.2.2 · pp. 23–24

7.2.1 Evaluation Settings

• Rynn-VP evaluation interleaves subtask reasoning and visual generation over 200 egocentric scenarios, with rollouts capped at 15 steps.The comparison includes UMMs and a composed GPT-5.6 Terra with MiniMax-H3 system.Source: §7.2.1 · p. 23

7.2.3 Visualization

• On Rynn-VP, interleaved reasoning and video generation illustrate task progression in toy storage and successive fabric manipulations in pillowcase folding.The examples contrast coherent progression with baseline sequences that can repeat interactions or abruptly change object appearance, scale, or layout.Source: Figure 8 · p. 23, §7.2.3 · p. 24

7.3.1 Evaluation Settings on Simulation Benchmarks

• 51.5% success on Selected-20 is the highest among compared methods; RynnUnify also ranks first on Composite-Unseen at 50.0%.Selected-20 covers 20 composite mobile-manipulation tasks.Source: §7.3.1 · p. 25, Table 5 · p. 25

7.3.2 Evaluation Settings on Real Robot

• 46.7% average complete-task success on Astribot S1 is higher than GR00T N1.7’s 26.7% and π0.5’s 25.0%.RynnUnify records 55.0% on Turn-and-Grasp, 35.0% on Block Swapping, and 50.0% on Bowl ×3 then Spoon.Source: Table 6 · p. 25, §7.3.2 · p. 26

8 Conclusion and Future Work

• RynnUnify combines embodied supervision, scene memory, and self-forcing for visual planning, with agentic and world-action extensions for reasoning and control.The paper introduces Rynn-VP and Rynn-GEU and reports improvements on both benchmarks.Source: §8 · p. 27
• Generated videos can distort fine-grained geometry, and physical execution needs broader validation across tasks and environments.Future work also targets reliable evidence selection and lower computational cost for closed-loop control.Source: §8 · p. 27

B Qualitative Examples

• Additional qualitative examples cover embodied cognition, localization, agentic rollout, and long-horizon visual planning.Source: B Qualitative Examples · p. 32

• Six MindCube and ERQA examples cover viewpoint reasoning, camera motion, interaction grounding, and anticipated object motion.Source: §B.1 · p. 32

B.2 Embodied Localization

• Affordance questions produce interaction points, target-area questions produce placement-region points, and trajectory questions produce ordered image-space coordinates.Source: §B.2 · p. 32

B.3 Agentic Rollout

• Four agentic-rollout examples connect each input question to a task-specific video prompt, intermediate frames, and a final answer.The examples address camera motion, object relationships, and action outcomes.Source: §B.3 · p. 32

• Additional Rynn-VP cases provide qualitative examples of long-horizon visual planning, complementing examples in the main paper.Source: §B.4 · p. 32, Figure 14 · p. 36
• Six MindCube and ERQA cases illustrate viewpoint reasoning, camera motion, interaction grounding, and anticipated object motion through visual observations, questions, and answers.Source: Figure 11 · p. 33
• Six localization cases cover affordance, target-area, and trajectory questions, with answers grounded in the input observation as interaction points, region points, or ordered trajectories.Source: Figure 12 · p. 34
• Four agentic rollout cases show input observations, video-generation prompts, sampled frames, and answers for camera motion, object relationships, button pressing, and slicing.Source: Figure 13 · p. 35
• The illustrated planning instructions include gathering and stacking sticky notes and packing items into schoolbags or backpacks.Source: §B.4 · p. 36

C.1 Evaluation and Inference Prompts

• Table 7 collects benchmark prompts and planning and agentic interaction formats, including the visual-input question template and world-modeling generation request.Source: §C.1 · pp. 37–38, Table 7 · p. 37
• The illustrated interaction format returns generated video frames as visual context before the model provides reasoning and a final answer.Source: §C.1 · p. 38

C.2 Evaluation Hyperparameters

• Text generation is generally greedy; ERQA, RoboSpatial, and ShareRobot use temperature 0.2, top_p 0.95, and top_k 50, while videos are sampled at 2 fps up to 512 frames.Source: §C.2 · p. 38
• RefSpatial-Bench and ShareRobot use a higher minimum image pixel budget, and video inputs have per-frame minimum and total-video pixel caps.Source: §C.2 · p. 38
• Rynn-VP generation uses 480p, 24 fps, 50 diffusion steps, and rollouts terminate at task completion or 15 steps; GEU agentic videos use 35 diffusion steps.Source: §C.2 · p. 38

C.3 SFT Training Hyperparameters

• Table 8 summarizes optimization settings for reasoner fine-tuning, generator fine-tuning, and the three WAM post-training stages.Source: §C.3 · p. 38, Table 8 · p. 38

• Rynn-VP evaluates 200 task instances across six indoor scene categories, assessing whether predicted videos or step images support executable plans, task-relevant actions and effects, and sufficient visual quality.Source: D Detailed Evaluation Protocol for… · pp. 38–39
• A shared task specification records entities, goal states, action dependencies, protected relationships, and allowed variations; alternative valid strategies and operation orders are accepted.Source: D Detailed Evaluation Protocol for… · p. 39
• The evaluator uses five review stages, including entity-continuity and hallucination auditing, with at most 50 source frames and 30 candidate panels organized into five storyboards.Source: D Detailed Evaluation Protocol for… · p. 39
• Planning Intelligence and Visual Realization are weighted averages of evidence-calibrated criteria, while Visual Quality is a holistic judgment informed by seven diagnostic dimensions.Source: D Detailed Evaluation Protocol for… · p. 39, Table 9 · p. 40
• Hallucination review distinguishes physical execution, entity lifecycle, and scene consistency violations; sparse or occluded evidence without confirmation is recorded as an evidence gap.Source: D Detailed Evaluation Protocol for… · p. 39
• Complete execution requires support for all necessary goals, valid subgoals receive partial credit, and actions that undo required terminal states reduce goal achievement.Source: D Detailed Evaluation Protocol for… · p. 39
• Ten participants reviewed 300 participant–case assignments across the 200 tasks, guiding evaluator refinement; reports provide mean PI, VR, VQ, and Overall scores and flag unevaluable cases.Source: D Detailed Evaluation Protocol for… · pp. 39–40

• The agent trains for one epoch on 22,007 questions, initialized from RynnUnify’s 8B reasoner, while the visual-generation model, reference policy, and semantic judge remain frozen.Source: E Detailed Training Protocol of… · p. 40
• Each direct trajectory answers from the observations; each generation-assisted trajectory requests one video of up to 10 seconds before producing reasoning and an answer.Source: E Detailed Training Protocol of… · p. 41
• For routable questions, both branches share a task-specific answer score gated by protocol validity, so their reward difference measures generation benefit under the same criterion.Source: E Detailed Training Protocol of… · p. 41
• Numerical and geometric scoring preserves signs and units, scores point sets in image coordinates, and resamples trajectories to 15 points for Fréchet-distance comparison.Source: E Detailed Training Protocol of… · p. 41, Table 10 · p. 41
• A frozen Qwen3.8-27B-FP8 judge scores semantic correctness and completeness, while malformed judgments receive up to three attempts and generator failures count as missing feedback.Source: E Detailed Training Protocol of… · p. 41