PaperlayerView exact revision on arXiv

Source-linked AI summary

Splat2Act: From Captured Scenes to Personalized VLA Manipulation

Xiaomeng Ye, Krishna Jaganathan, Gyubeum Lim, Gunjan Chhablani, Muhammad Zubair Irshad, Zsolt Kira

arXiv:2610.13945v1cs.ROcs.CV

TL;DR

Generalist manipulation policies can fail in captured deployment scenes, and collecting demonstrations for each user environment is costly. Splat2Act builds a simulation-ready digital twin from scene and object captures, generates oracle demonstrations, and personalizes a VLA. Same-scene broad adaptation followed by personalization improves aggregate pickup success over direct personalization, with real-world performance close to simulation on the reported tasks.

  • Problem

    Pretrained π0.5 achieves 0.0% pickup success in two target scenes, while collecting real demonstrations for each new user, receptacle, object, or configuration requires recurring human effort.

    Source: §I · pp. 1–2

  • Method

    Splat2Act turns scene and object captures into simulation-ready environments and oracle-generated pickup demonstrations, then uses broad adaptation followed by personalization to the deployment scene and user objects.

    Source: §I · p. 1, D. Policy Adaptation · p. 4

  • Results

    17.5 percentage points: same-scene broad adaptation followed by personalization improves aggregate pickup success over direct personalization, with 63.9% real-world SR and 66.7% simulated SR on the reported tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    The pipeline scales to 101 scanned indoor environments and produces 68,021 successful pickup demonstrations across 852 automatically discovered receptacles.

    Source: §I · p. 2

Abstract

from arXiv · show

A user who brings a generalist manipulation policy into their home or office wants it to pick up their objects from their shelves, counters, and tables. Yet even a strong generalist like $π_{0.5}$ achieves 0% pickup success in our captured deployment scenes without adaptation, and collecting real demonstrations in every user's environment is prohibitively expensive. We present Splat2Act, an automated real-to-sim-to-real pipeline that personalizes a pretrained vision-language-action (VLA) policy to a specific scene and object set using only a handheld phone scan and images of the user's objects. Unlike prior real-to-sim work that reconstructs individual tabletops, Splat2Act operates on building-scale indoor environments: it builds a photorealistic digital twin with 3D Gaussian Splatting and collision geometry, automatically discovers tabletop and non-tabletop receptacles, constructs feasible pickup tasks, and generates successful demonstrations with a privileged simulation oracle, with only scene and object capture required from a human. Applied to 101 scanned campus environments with a median navigable area of 104.87 m$^2$ and 7 receptacles, the pipeline yields 852 receptacles and over 68,000 pickup demonstrations. We fine-tune $π_{0.5}$ in two stages: Stage 1 adapts to hundreds of public object assets, drawn either from the deployment scene alone or from all 101 scenes; Stage 2 personalizes to the user's own objects on the deployment scene's receptacles. Our strongest personalized policy reaches 62.5% aggregate pickup success in the target scene. Deployed on a real Stretch 3, the same policy achieves 63.9% success on a 12-task subset, closely tracking its 66.7% simulated success on those tasks. These results suggest that automated real-to-sim-to-real training can personalize VLA manipulation policies using only simulation-oracle data.

I. INTRODUCTION

Generalist VLAs can fail to pick up objects in captured deployment scenes, while collecting demonstrations anew for each environment is costly. Splat2Act automates scene-grounded task and demonstration generation from scene and object captures, then adapts a VLA for the captured environment.

Motivation

  • Pretrained π0.5 achieves 0.0% pickup success in two target scenes, while collecting real demonstrations requires repeated human effort for new users, receptacles, objects, or configurations.The introduction frames this recurring collection cost as a barrier to personalizing generalist manipulation policies.Source: §I · pp. 1–2

Approach

  • Splat2Act builds a simulation-ready scene and object representation, discovers valid receptacles, and uses a privileged-state geometric oracle to generate successful pickup demonstrations.Human input is a 20–30 minute handheld scene scan plus one image and dimensions for each target object; no robot or operator expertise is needed.Source: §I · p. 1

Results

  • 17.5 percentage points: same-scene broad adaptation followed by personalization improves aggregate pickup success over direct personalization, with 63.9% real-world SR versus 66.7% simulated SR on the reported tasks.The paper also reports that a policy adapted on diverse multi-scene data retains robustness to unseen receptacles.Source: §I · p. 2

Scale

  • The pipeline scales to 101 scanned indoor environments, generating 68,021 successful pickup demonstrations across 852 automatically discovered receptacles.The scanned environments have a median navigable area of 104.87 m2.Source: §I · p. 2

Evaluation

  • The study evaluates broad simulation adaptation and personalization through controlled simulation and real-world experiments.Its stated contribution includes a dataset of 101 building-scale indoor scenes with 852 diverse receptacles, task metadata, and demonstrations.Source: §I · p. 2

II. RELATED WORK

Prior work studies VLA adaptation, automated data generation, and real-to-sim manipulation, but Splat2Act targets deployment-specific personalization in building-scale captured environments. Its distinction is autonomous receptacle discovery and analysis of how scene-specific and multi-scene data affect personalization and transfer.

A. Vision-Language-Action Models and Adaptation

  • Prior VLA work improves fine-tuning efficiency, performance, or user-specific object grounding, while Splat2Act couples object assets with adaptation to captured-scene appearance, receptacles, and feasible pickup configurations.The cited approaches include action-representation and training-objective studies, and visual prompting for user-specific objects, [14].Source: A. Vision-Language-Action Models and… · p. 2

B. Automated and Deployment-Specific Data Generation

  • Automated data-generation methods synthesize tasks, scenes, or demonstrations, but deployment personalization still requires grounding generated experience in the environment where the policy will operate.The passage distinguishes this need from methods that amplify limited human demonstrations or generate simulation supervision more generally.Source: B. Automated and Deployment-Specific… · p. 2

C. Real-to-Sim-to-Real Manipulation

  • Splat2Act autonomously discovers manipulation opportunities across building-scale captures for deployment-scene personalization, processing median 104.87 m2 navigable area and finding median 7 usable receptacles per scene, including non-tabletop surfaces.Source: C. Real-to-Sim-to-Real Manipulation · p. 3
  • Splat2Act processes building-scale captures with a median of 104.87 m2 navigable area and discovers a median of 7 usable receptacles per scene, including non-tabletop surfaces.The paper contrasts this with tabletop-scale pipelines and room-scale systems that require task objects, targets, or support surfaces to be specified or annotated.Source: C. Real-to-Sim-to-Real Manipulation · p. 3

III. METHOD

Splat2Act targets receptacle-local pickup with a mobile manipulator, converting scene and object captures into executable tasks and demonstrations for adaptation. The described pipeline requires only scene and object capture from a human.

III. METHOD

  • Splat2Act constructs executable pickup tasks from a captured scene and separately chosen object inventory, then uses an oracle to generate demonstrations for adaptation to that deployment scene.The paper identifies the robot as a Hello Robot Stretch 3 and states that only scene and object capture are required from a human.Source: §III · p. 3
  • The method is framed as receptacle-local pickup with a mobile manipulator, using a standard-phone video for scene capture and a separately selected object inventory.The cited comparison table is titled as a comparison with capture-conditioned manipulation pipelines; its caption alone does not provide specific comparison outcomes.Source: §III · p. 3, Table I · p. 3
  • The method’s stated human-input boundary is scene and object capture, rather than task-by-task teleoperation.This is the pipeline-level description; no specific comparison-table values are given in the supplied passage.Source: §III · p. 3

A. Scene and Object Representations

The scene representation combines photorealistic Gaussian rendering with mesh geometry, while object assets come from public models or images of user objects. Receptacle discovery and reachability checks turn scene geometry into feasible pickup configurations.

A. Scene and Object Representations

  • A hybrid scene representation uses 3D Gaussian Splatting for photorealistic egocentric observations and an extracted mesh for collision, receptacle discovery, and navigation analysis.The mesh is extracted from the trained Gaussian representation, with depth and normal regularization used for large-scene geometric consistency.Source: A. Scene and Object Representations · p. 3
  • Object assets for broad adaptation come from filtered public 3D assets, while personalization assets are generated from single images of real objects and scaled to user-specified dimensions.The objects are inserted as movable assets without assuming exact reconstruction.Source: A. Scene and Object Representations · p. 3

B. Receptacle Discovery and Task Assembly

  • Receptacle discovery filters upward-facing mesh regions by height and area, clusters candidate surfaces, then applies geometric extent, stability, and height filters.The pipeline overview also describes receptacle discovery as part of constructing feasible pickup tasks from the reconstructed scene.Source: B. Receptacle Discovery and Task… · p. 4, Figure 2 · p. 4
  • Candidate placement points are checked against the robot’s navigable region and reach, with further settlement, grasp, and oracle-feasibility tests before configurations are assembled.A retained stand point requires at least 6 placement samples within the robot’s 0.70m horizontal reach radius.Source: B. Receptacle Discovery and Task… · p. 4

C. Demonstration Generation

A privileged-state simulation oracle generates pickup demonstrations through grasp selection, planning, retries, and recording. The policy is then adapted with two-stage supervised fine-tuning, first broadly and then to the deployment scene and user objects.

C. Demonstration Generation

  • 60 successful pickup demonstrations per A40 GPU-hour are generated on average across 64 parallel environments.The oracle uses privileged object poses, geometry, and robot state to select grasp candidates, plan and retry motions, and record successful demonstrations.Source: C. Demonstration Generation · p. 4

D. Policy Adaptation

  • Two-stage supervised fine-tuning first adapts across public object assets and receptacles, then personalizes the policy to the user's objects on deployment-scene receptacles.Stage 1 data comes either from the deployment scene alone or from all reconstructed scenes; direct Stage-2 fine-tuning from pretrained π0.5 is a comparison.Source: D. Policy Adaptation · p. 4
  • Conditional flow matching trains the policy from successful oracle demonstrations by mapping noisy action chunks toward oracle action chunks.Training interpolates ground-truth action chunks with Gaussian noise at sampled times.Source: D. Policy Adaptation · p. 4
  • The adaptation fine-tunes the PaliGemma language backbone and action expert while freezing SigLIP to preserve its pretrained visual-semantic prior.The action model can specialize to the reconstructed scene, robot embodiment, receptacle distribution, and task family.Source: D. Policy Adaptation · p. 4

E. Real-World Deployment

The study builds a large-scale scene and demonstration collection, evaluates two captured deployment scenes under easy and distractor-rich conditions, and uses personalized object assets across their receptacles.

E. Real-World Deployment

  • Deployment streams head- and wrist-camera images, proprioception, and a user prompt to a remote policy server, which returns action chunks for synchronous position- and velocity-controlled execution.The server runs on a single A40 GPU and communicates through bidirectional gRPC.Source: E. Real-World Deployment · pp. 4–5

IV. DATASETS AND TASKS

  • Evaluation uses Office and Conference deployment scenes, with each task tested both with the target alone and with distractors, and success pooled across conditions.The other 99 scenes are used only as broad multi-scene adaptation data.Source: §IV · p. 5

A. Scene and Demonstrations

  • 101 campus scenes contain 11,909.78 m^2 of navigable area, with a median of 104.87 m^2 and 7 receptacles per scene.The retained broad training subset includes 852 final training receptacles.Source: A. Scene and Demonstrations · p. 5
  • 68,021 successful pickup trajectories were generated across 852 receptacles and 556 compatible object assets.The broad campaigns recorded 273.46 control hours; the demonstrations are a subsampled fraction of the potential task space due to storage constraints.Source: A. Scene and Demonstrations · p. 5
  • Seven real objects are spawned across seven Office receptacles and six Conference receptacles for personalized tasks.The objects include a pill bottle, Spam, tomato soup, cardboard box, Rubik’s cube, spray can, and water bottle.Source: A. Scene and Demonstrations · p. 5

B. Broad and Personalized Datasets

Broad adaptation datasets vary in scene coverage, while personalization data pairs selected user objects with designated receptacles. Evaluation uses disjoint robot-target configurations and an explicit pickup-success protocol.

B. Broad and Personalized Datasets

  • Office1k and Multi1k each contain 1,024 demonstrations, spanning 550 identities and 7 Office supports versus 556 identities, 101 scenes, and 852 receptacles.The first subset draws from Office; the second draws from all reconstructed scenes.Source: B. Broad and Personalized Datasets · p. 5
  • Personalization uses pill bottle, Spam, and tomato soup as targets, with the other six objects eligible as distractors.Source: B. Broad and Personalized Datasets · p. 5
  • The personalized datasets cover 21 Office and 18 Conference object-receptacle pairs, with 504 Office and 500 Conference Stage 2 demonstrations.Each Office pair has 24 variations and each Conference pair around 28 augmentations.Source: B. Broad and Personalized Datasets · p. 5

C. Evaluation

  • Evaluation banks contain 40 Office and 35 Conference configurations, covering all 39 object-receptacle pairs with easy and hard tasks.Included configurations have successful states verified by the demonstration-generation module and robot-target configurations disjoint from training.Source: C. Evaluation · p. 5
  • Success is the fraction of successful evaluated episodes across easy and hard conditions, with unsuccessful policy termination counted as failure.Episodes run for up to 700 steps, and success is assessed at policy stop or the horizon before reset.Source: C. Evaluation · p. 5

V. EXPERIMENTS

The experiments examine broad adaptation, scene personalization, real-world transfer, and generalization to a similar unseen scene. The setup standardizes the pretrained policy and training conditions, while Table III concerns scaling receptacle count and augmentation depth.

V. EXPERIMENTS

  • The experiments ask how broad adaptation choices and scene personalization affect pickup performance, and whether policies transfer to real deployment and similar unseen scenes.Source: §V · p. 5

A. Experimental Setup

  • Policies start from the same pretrained π0.5 checkpoint and map head and wrist RGB, proprioceptive state, and task text to 10-action chunks in a 10-dimensional Stretch action space.The action vector includes arm, lift, wrist, gripper, termination, and base controls.Source: A. Experimental Setup · p. 5
  • Each run trains for one epoch at batch size 64, while optimizer-step counts vary by condition because trajectory lengths differ.Office1k and Multi1k use 39,489 and 45,705 steps; Office and Conference Stage 2 use 15,963 and 17,390 steps.Source: A. Experimental Setup · pp. 5–6
  • Table III reports in-distribution pickup success as receptacle count and augmentation depth scale, using four rendered 3D training objects.Source: Table III · p. 6

B. Scaling Receptacles, Objects, and Augmentations

Personalized success depends on the broad-adaptation source and deployment scene, while scaling Stage 2 data requires more augmentation as receptacles and objects diversify.

B. Scaling Receptacles, Objects, and Augmentations

  • At 24 tasks per object, success was 75.0%, 87.5%, and 75.0% with one, two, and four receptacles, respectively, as the corpus grew from 96 to 384 trajectories.At eight tasks per object, success remained 21.9–43.8%; the results motivated using 24+ augmentations per pair in Stage 2.Source: B. Scaling Receptacles, Objects, and… · p. 6

C. Broad Adaptation Sources

  • 62.5% Office SR followed Office1k Stage 1, while 51.4% Conference SR followed Multi1k Stage 1 among the highest personalized results.Fig. 3 compares pretrained, Stage 1, and scene-matched Stage 2 policies on both task banks.Source: Figure 3 · p. 6, C. Broad Adaptation Sources · p. 6
  • Broad adaptation produced the highest observed SR in each scene: Office1k Stage 2 improved 17.5 percentage points over π0.5-base Stage 2 on Office, while Multi1k improved 8.6 points on Conference.Distractors cost Stage 2 policies 10–26 percentage points, with the smallest penalty after broad adaptation; the strongest policy reached 71.4% easy and 52.6% hard Office SR.Source: C. Broad Adaptation Sources · p. 6
  • Before personalization, Office1k Stage 1 reached 40.0% SR on Office but 17.1% on Conference, while Multi1k reached 37.5% and 40.0%, respectively.The pretrained π0.5 base reached 0.0% on both banks, attributed to no exposure to the Stretch embodiment or action space.Source: C. Broad Adaptation Sources · p. 7

D. Scene Personalization

Scene-matched personalization substantially outperformed cross-scene personalization, while Office policies transferred to the real robot with similar aggregate success to simulation.

D. Scene Personalization

  • Matching the personalization scene improved SR in all six comparisons by 14.3%–47.5%; Office1k personalization reached 62.5% on Office versus 15.0% after Conference personalization.Conference personalization of Office1k Stage 1 also fell below its 40.0% Office Stage 1 SR.Source: D. Scene Personalization · p. 7

E. Sim-to-Real Transfer in Office

  • 66.7% simulation SR and 63.9% real SR were achieved by the Office1k Stage 1 personalized policy across the 12 evaluated tasks.The evaluation used 12 simulated episodes and 36 real attempts over the same task cells; the Multi1k-based policy achieved 50.0% and 58.3%, respectively.Source: E. Sim-to-Real Transfer in Office · p. 7
  • Both policy aggregates transferred within 8 percentage points of simulated SR and preserved their ordering.Real contact was more forgiving: gripper compliance and rubber-fingertip friction allowed some imperfect grasps to succeed.Source: E. Sim-to-Real Transfer in Office · p. 7

F. Generalization to the unseen

In an unseen lab, personalized policies achieved lower success than in their personalization scene, with results varying across the two tables. The paper concludes that scene-matched broad adaptation followed by personalization can support robust behavior in simulation and real-world deployment scenarios.

F. Generalization to the unseen

  • 33.3% and 22.2% SR were achieved on the black table by Office1k- and Multi1k-based policies, respectively; on the yellow table, they achieved 22.2% and 27.8%.Results average easy and hard trials; each table used three targets, two difficulty conditions, and three attempts per configuration.Source: F. Generalization to the unseen · p. 7, Figure 6 · p. 8
  • The ordering reversed between tables, so the results establish no consistent winner; both policies dropped substantially from their 62.5% and 47.5% personalization-scene SRs.The conclusion states that same-scene broad adaptation followed by personalization can lead to robust behaviors in simulation and real-world deployment scenarios.Source: F. Generalization to the unseen · p. 7, §VI · p. 8
Loading 2610.13945v1…