PaperlayerView exact revision on arXiv

Source-linked AI summary

PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policies

Arhan Jain, Mingtong Zhang, Kanav Arora, William Chen, Marcel Torne, Muhammad Zubair Irshad, Sergey Zakharov, Yue Wang, Sergey Levine, Chelsea Finn, Wei-Chiu Ma, Dhruv Shah, Abhishek Gupta, Karl Pertsch

arXiv:2512.16881v2cs.ROcs.LG

TL;DR

Real-world evaluation of generalist robot policies is costly and difficult to scale, while existing simulation benchmarks can poorly reflect real performance. PolaRiS reconstructs short video scans into interactive simulations and uses lightweight simulation co-training to bridge visual gaps. Its evaluations achieve strong real-to-sim correlation, including an average Pearson correlation of 𝑟= 0.9 across diverse DROID environments, but cover only a subset of capabilities measured in comprehensive real-world benchmarks.

  • Problem

    Generalist policies require broad evaluation, but real-world rollouts are costly and existing simulations have visual and physical gaps that limit their reliability.

    Source: §1 · p. 1

  • Method

    PolaRiS reconstructs short real-world videos into interactive high-fidelity simulations and co-trains policies with a small amount of simulation data for unseen-environment evaluation.

    Source: §1 · p. 2, §4.1 · p. 6, §4.2 · p. 8

  • Results

    Across diverse DROID environments, PolaRiS matches real-world policy performance with an average Pearson correlation of 𝑟= 0.9.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PolaRiS provides a scalable tool for rapidly estimating generalist policy performance through customized, shareable simulated environments.

    Source: §1 · p. 2, §6 · p. 13

    The evaluations cover only a small subset of RoboArena capabilities, particularly non-rigid, soft-body, and complex-contact behaviors, so they do not replace real-world evaluation.

    Source: §5.2 · p. 11

Abstract

from arXiv · show

A significant challenge for robot learning research is our ability to accurately measure and compare the performance of robot policies. Benchmarking in robotics is historically challenging due to the stochasticity, reproducibility, and time-consuming nature of real-world rollouts. This challenge is exacerbated for recent generalist policies, which has to be evaluated across a wide variety of scenes and tasks. Evaluation in simulation offers a scalable complement to real world evaluations, but the visual and physical domain gap between existing simulation benchmarks and the real world has made them an unreliable signal for policy improvement. Furthermore, building realistic and diverse simulated environments has traditionally required significant human effort and expertise. To bridge the gap, we introduce Policy Evaluation and Environment Reconstruction in Simulation (PolaRiS), a scalable real-to-sim framework for high-fidelity simulated robot evaluation. PolaRiS utilizes neural reconstruction methods to turn short video scans of real-world scenes into interactive simulation environments. Additionally, we develop a simple simulation data co-training recipe that bridges remaining real-to-sim gaps and enables zero-shot evaluation in unseen simulation environments. Through extensive paired evaluations between simulation and the real world, we demonstrate that PolaRiS evaluations provide a much stronger correlation to real world generalist policy performance than existing simulated benchmarks. Its simplicity also enables rapid creation of diverse simulated environments. As such, this work takes a step towards distributed and democratized evaluation for the next generation of robotic foundation models.

1 Introduction

PolaRiS addresses the difficulty of scalable, realistic evaluation for generalist robot policies by reconstructing real scenes into simulation and aligning policies with lightweight simulation co-training. Its evaluations correlate strongly with real-world performance while enabling rapidly created, shared environments.

1 Introduction

  • Existing simulation benchmarks sacrifice visual, dynamic, and geometric realism, producing poor correlation with real-world performance and requiring substantial manual effort for targeted scenes.Real-world evaluation remains costly, tedious, and difficult to control systematically, especially across the many tasks and environments needed for generalist policies.Source: §1 · pp. 1–2
  • PolaRiS converts real-world scenes into high-fidelity simulated evaluation environments through neural reconstruction, targeting scalable assessment of generalist policies.The framework combines environment-generation tools, simulation data, off-the-shelf environments, and a sharing hub.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • Lightweight co-finetuning with simulation data bridges remaining real-to-sim gaps without teaching new tasks, enabling zero-shot evaluation in unseen PolaRiS environments.The simulation data aligns visual representations and need not be collected in the target environment or for the target task.Source: §1 · pp. 2–3
  • An average Pearson correlation of 𝑟= 0.9 shows that PolaRiS matches real-world performance for state-of-the-art generalist policies across diverse DROID environments.The evaluation is described as operating out of the box on the evaluated policies.Source: §1 · p. 2
  • PolaRiS releases tools and shared resources intended to support community-driven development of realistic, scalable simulation evaluation environments.The released components include environment-creation tools, an open-source simulation dataset, off-the-shelf environments, and a sharing hub.Source: Figure 1 · p. 2, §1 · p. 3

2 Related Work

Prior simulated benchmarks provide scalable evaluation but often fail to reflect real-world performance for generalist robot policies because of visual, physical, and interaction gaps. PolaRiS instead reconstructs realistic scenes for evaluation across unseen environments and supports moving-camera policies.

2 Related Work

  • Existing simulated benchmarks are valuable for scalable, repeatable evaluation but struggle with generalist policies because of large visual and physical domain gaps.Real-robot evaluation remains comprehensive but costly, laborious, and difficult to reproduce at scale.Source: §2 · p. 3
  • PolaRiS uses high-fidelity Gaussian-splatting environments with physics simulation to preserve realistic appearance, dynamics, and policy interactions during evaluation.This design contrasts with action-conditioned video models, which the paper says struggle to model physical interactions reliably.Source: §2 · pp. 3–4
  • Earlier real-to-sim methods address navigation or driving, while manipulation methods have not reliably evaluated modern generalist policies across many tasks and unseen environments.Green-screening approaches also do not support moving cameras, excluding many generalist policies with wrist cameras.Source: §2 · pp. 3–4
  • The paper positions PolaRiS as the first pipeline for evaluating real-robot policies in unseen Gaussian-splatting simulation environments with strong real-to-sim correlation.Related Gaussian-splatting evaluation work focused on narrow, single-task policies in environments seen during training.Source: §2 · p. 4

3 Preliminaries

The paper formalizes faithful simulation evaluation as preserving real-world policy performance and rankings, then describes neural scene reconstruction and metrics for measuring this fidelity. Its generalist policies are language-conditioned vision-language-action models.

3 Preliminaries

  • The evaluation goal is to construct simulated environments whose policy rankings and relative performance approximate those measured in the real-world environment distribution.Real and simulated policy performances are represented as expected episodic performance over their respective environment distributions.Source: §3 · p. 4
  • Pearson correlation measures linear agreement between real and simulated policy performance, with the ideal value approaching 1.The means of real and simulated scores are used in the correlation computation.Source: §3 · pp. 4–5
  • Mean Maximum Rank Violation measures the average worst-case real-world performance gap caused by policy rank inversions under simulation.A faithful simulator should have low MMRV, approaching 0.Source: §3 · pp. 4–5
  • PolaRiS reconstructs scene geometry and appearance with 2D Gaussian splats, then extracts meshes for physics-based simulation while retaining splats for photorealistic rendering.The representation is optimized for differentiable rendering and supports mesh extraction for robotic interaction.Source: §3 · p. 5
  • The paper defines generalist policies as language-conditioned multi-task vision-language-action models mapping visual observations and language instructions to robot actions.These policies are built on transformer-based vision-language models fine-tuned to produce actions.Source: §3 · p. 5

4 Scalable Simulation Evaluations with Real-World Video Scans

PolaRiS converts short real-world video scans into interactive, high-fidelity simulation environments and uses simulation co-training to improve real-to-sim evaluation correlation. Its reconstruction, articulation, scene composition, and policy refinement pipeline supports zero-shot evaluation in unseen environments.

4 Scalable Simulation Evaluations with Real-World Video Scans

  • PolaRiS combines neural scene reconstruction, automatic robot articulation, and simulation co-training to create scalable, correlated evaluations for generalist policies.The system transforms real-world video into interactive simulation and addresses remaining visual domain gaps with policy-agnostic co-training.Source: §4 · p. 6, §4.2 · p. 8

4.1 Environment Generation with PolaRiS

  • 2–5 minutes of monocular video plus separate background and robot recordings provide the inputs for reconstructing composable scene and robot assets.A ChArUco board assists global orientation and metric scale, while 5–10 minutes of capture typically suffices for a scene.Source: §4.1 · p. 6
  • TSDF-derived meshes provide geometry and contact dynamics in IsaacSim, while Gaussian splats provide high-fidelity photometric rendering.The automated environment-creation framework typically completes within 30 minutes on an NVIDIA RTX 4090.Source: §4.1 · p. 6
  • Gaussian primitives anchored to kinematic links produce interactive, photorealistic articulated robots whose splats follow the underlying mesh model.The scanned robot can be reused across simulation environments, making articulation a one-time process per robot system.Source: §4.1 · pp. 6–7
  • The Scene Composition GUI assembles environment, robot, and object assets into USD scenes, with new environments typically requiring less than 20 minutes of human effort.The total wall time, including Gaussian-splat training, is typically less than one hour; the GUI procedure shown in Figure 3 typically takes less than 5 minutes.Source: §4.1 · p. 7, Figure 3 · p. 7
  • Full 3D reconstruction enables novel-view synthesis, third-person viewpoints, and wrist-camera inputs unavailable to prior 2D background-copying approaches.This expands simulation evaluations to the richer sensor inputs required by generalist policies.Source: §4.1 · p. 8

4.2 Enabling High Correlation Policy Evaluation with PolaRiS using Co-Training

  • PolaRiS co-training targets visual domain gaps such as lighting variation and missing inter-object shadows, which hardware-in-the-loop experiments identify as the dominant error source.The method mixes simulation data with original pretraining data using a behavior-cloning objective controlled by λ.Source: §4.2 · p. 8
  • A few hundred simulated episodes and fine-tuning steps improve robustness to subtle visual gaps, and co-training environments can differ from unseen evaluation environments.The co-training data is typically completed in < 25 mins and does not need to cover the target tasks or environments, enabling zero-shot evaluation.Source: §4.2 · pp. 8–9

5 Experiments

PolaRiS evaluates generalist robot policies through paired real and simulated environments, testing correlation, baseline comparisons, and design choices affecting zero-shot real-to-sim evaluation.

5.1 Experimental Setup

  • The experiments use six paired real-world and simulated environments spanning diverse visual scenes and rigid manipulation tasks, plus 15 separate co-training environments.The co-training environments and evaluation environments share no scenes or objects.Source: §5.1 · pp. 9–10
  • Policies are fine-tuned for 1k steps on approximately 350 simulation demonstrations, then evaluated with 50 simulated rollouts per task and automatically scored on a [0 . . . 1] progress scale.Real-world rankings use 20 evaluation rollouts per policy, with Pearson correlation and MMRV reported between real and simulated performance.Source: §5.1 · p. 10
  • The study compares PolaRiS against MSE, Libero-Score, and human-scored Ctrl-World video evaluation, using DROID policies including π0, π0-FAST, PaliGemma-binning, and π0.5.Libero-Score uses substantially more finetuning and evaluation compute, while existing real-to-sim approaches do not support wrist cameras.Source: §5.1 · pp. 9–10

5.2 Real-to-Sim Correlation in Unseen Environments

  • Existing evaluation methods perform poorly: MSE and Libero-Score correlate weakly with real-world performance, while Ctrl-World produces hallucinations and policy mis-rankings with MMRV: 0.22.Libero policies cluster between 90 and 95% success despite spanning the full real-world performance range.Source: §5.1 · p. 10, §5.2 · p. 11
  • PolaRiS achieves strong real-to-sim correlation across environments, accurately ranking policies with a worst-case Pearson correlation of r= 0.81 across six environments.The pipeline estimates both average policy performance and performance in specific deployment scenarios.Source: §5.2 · p. 11
  • PolaRiS scores strongly correlate with RoboArena [22] real-world rankings, although the simulated tasks cover only a small subset of RoboArena capabilities.Non-rigid dynamics, soft-body interactions, and complex contact behaviors remain challenging for current simulators.Source: §5.2 · p. 11, Figure 8 · p. 11

5.3 Ablation Studies and Analysis

  • Gaussian-splatting rendering produces the strongest correlation among visual-fidelity variants, while 1k co-finetuning steps reliably perform well and excessive finetuning worsens correlation through overfitting.A few hundred steps can already significantly improve correlation.Source: §5.3 · pp. 11–12, Figure 9 · p. 12, Figure 11 · p. 12
  • Simulation co-training is essential: OOD data from held-out environments yields approximately comparable correlation to in-domain data, enabling evaluation in unseen scenes without targeted data collection.Near-domain data performs slightly better than target-task demonstrations, possibly because target demonstrations cause quick overfitting.Source: §5.3 · p. 12, Figure 10 · p. 12

6 Discussion

PolaRiS combines neural reconstruction and simulation assets to create high-fidelity evaluation environments, while identifying remaining opportunities to improve rendering, dynamics, and scene complexity.

6 Discussion

  • PolaRiS supports personalized simulation environments reconstructed from real-world videos and can benefit from advances in 3D reconstruction and generation.The discussion frames these environments as tools for scalable generalist-policy evaluation, with future methods expected to improve fidelity and ease of use.Source: §6 · p. 13
  • The framework’s main improvement opportunities are richer diffusion-based rendering, stronger system identification, and hybrid simulation for complex objects and scenes.These directions target visual gaps, simplistic dynamics modeling, and scalability beyond current physics-based scene configurations.Source: §6 · p. 13

A.1 2DGS

  • 2DGS produces high-fidelity novel views and extractable geometry for policy simulation, with geometry regularization reducing thickness and multi-view inconsistency.The method uses depth and normal regularizers, then extracts geometry for simulation; the paper reports better zero-shot evaluation performance for real-world-trained policies.Source: §A.1 · p. 18

B.1 Reconstruction

  • PolaRiS standardizes reconstructed scenes in a ChArUco-based world frame and converts reconstructed assets into USD for composition and simulation.COLMAP poses are aligned to the ChArUco canonical frame before assets are assembled through the scene-composition GUI.Source: §B.1 · p. 18, §B.2 · p. 18

B.2 Scene Composition

  • Figure 12 compares absolute joint error over time during open-loop replay for joint velocity and joint position control modes.The supplied caption identifies the comparison but does not state its outcome.Source: Figure 12 · p. 19

C.1 Tasks

The evaluation covers six tasks in distinct environments, using dense, normalized progress scores that are graded consistently in simulation and the real world.

C.1 Tasks

  • Six tasks are evaluated in distinct environments with task-specific partial-progress criteria, providing denser measurements than binary success.Simulation grades all criteria autonomously using privileged state information, while humans apply the same rubric to real-world rollouts.Source: §C.1 · p. 19
  • Scores are normalized between 0 and 1 because tasks have different score ranges, enabling comparisons across environments.The normalization is applied after task-specific criteria are defined.Source: §C.1 · p. 19

C.2 Cotraining

Cotraining uses simulation data collected on a DROID-platform twin and mixes it with real DROID data to support policy transfer between real and simulated environments.

C.2 Cotraining

  • Cotraining combines 90% real DROID data with 10% simulated DROID data using matching VR collection, RLDS formatting, and openpi training procedures.Simulation data is collected with an Oculus Quest 2 and processed to match the original DROID RLDS structure.Source: §C.2 · p. 19
  • Joint-position policies transfer better from real to simulation, which the authors associate with avoiding compounding error; Figure 12 compares them with joint-velocity policies.The study also uses independent seeds and 50 rollouts per checkpoint for finetuning analysis.Source: §C.2 · p. 19
  • Task scores use normalized rubrics so simulation and real-world evaluations can be compared under the same criteria.The supplied task description specifies autonomous privileged-state grading in simulation and human grading in the real world.Source: Table 1 · p. 20

C.3 Datasets

The experiments vary cotraining data distribution and visual fidelity to test how closely training conditions should match the evaluation setting.

C.3 Datasets

  • Cotraining ablations compare OOD, near-domain, and in-domain datasets to vary the closeness of training data to the test-time distribution.The dataset examples and compositions are documented in Table 4.Source: §C.3 · p. 20
  • Visual-fidelity ablations compare splat, raytraced, untextured-environment, and untextured-everything datasets, using OOD data with rendering as the only difference.Examples of the four conditions are provided in Table 5.Source: §C.3 · p. 20

D.1 Hardware-in-the-loop Experiments

Hardware-in-the-loop experiments isolate visual and dynamics gaps by selectively combining real and simulated rollout components. Across three tasks, simulated dynamics with real visuals caused no performance drop, whereas simulated visuals with real dynamics caused about a 25% drop.

D.1 Hardware-in-the-loop Experiments

  • The experiments isolated dynamics and visual gaps by selectively replacing rollout components between real and simulation.Dynamics testing used real observations with achieved simulation joint positions, while visual testing used simulation renders with real-robot actions and achieved real joint positions sent to simulation.Source: §D.1 · pp. 20–21
  • About a 25% performance drop across 3 tasks occurred when simulation visuals were paired with real dynamics, while simulated dynamics with real visuals caused no drop.The comparison used the π0 FAST DROID Joint Position policy and hardware-in-the-loop rollouts.Source: §D.1 · p. 21
  • Hardware-in-the-loop evaluations ran at a 15 Hz control frequency for DROID using blocking control and real-to-simulation state exchange.All hardware-in-the-loop experiments used the π0 FAST DROID Joint Position policy.Source: §D.1 · p. 21
  • PolaRiS also reports correlation metrics for individual environments to assess targeted policy evaluation in addition to general performance.Figure 13 presents the per-environment correlation metrics.Source: Figure 13 · p. 22
Loading 2512.16881v2…