Source-linked AI summary
BURGER: Benchmarking and Understanding Robotic Generalization via Embodied Drawing
Xiangyu Xu, Bangshuai Peng, Ziyi Liu, Mengjie Zhao, Chenhao Lu, Zhengyang Fan, Huazhe Xu, Kaizhe Hu
TL;DR
VLA models are increasingly trained on larger, more diverse datasets, but studying their language-conditioned behavior requires scalable robot data, while physical collection remains costly. BURGER introduces prompt-driven robotic drawing trajectories and evaluates model scaling, grounding, and multimodal behavior. Results show scaling benefits for some models, generalization to unseen compositions of familiar objects, and reactive real-robot drawing under human intervention.
Problem
Studying whether VLA models exploit larger, more diverse language-conditioned robot datasets requires controllable scale and diversity, but collecting varied physical-robot demonstrations remains costly.
Source: §1 · p. 1Method
Results
Some flow-matching models generally improve with more demonstrations or training tasks, while evaluated policies compose familiar objects in unseen spatial arrangements but struggle with unseen object categories.
Source: §1 · p. 2Takeaways & Limitations
Real-robot policies can draw with visual feedback, respond to human-added strokes, and produce different valid trajectories for the same task.
Source: §4.5 · p. 9, §5 · p. 9BURGER focuses on sketch-level strokes rather than contact-rich media such as brush painting, and its automatic stroke-order heuristics may differ from human drawing behavior.
Source: §5 · p. 9
Abstract
from arXiv · showhide
How effectively can vision-language-action (VLA) models learn diverse language-conditioned behaviors as training data scales? Investigating this question requires large and varied robot datasets, whose collection on physical robots remains costly. We introduce BURGER, a language-conditioned robotic drawing benchmark with a scalable, prompt-driven pipeline for generating continuous drawing trajectories, both in simulation and the real world. In BURGER, an agent starts from the same blank canvas and controls a robot arm to draw the object or spatial composition specified by language, while each instruction allows an open-ended variety of valid drawings and stroke orders. Using 28,200 simulated demonstrations spanning 282 drawing tasks, we evaluate five representative VLA models across data scales, language grounding, and multimodal behavior. BURGER supports fine-grained analysis of language grounding, spatial composition, and action generation, revealing distinct failure patterns across VLA models. Our experiments suggest that some VLA models can benefit from greater task diversity and demonstration scale, capture multimodal drawing behaviors, and generalize to unseen compositions of familiar objects. Finally, we deploy a trained policy on a Franka Panda arm, demonstrating the ability to draw reactively under human intervention. Project homepage and code can be found at https://siyingzhao.github.io/BURGER/.
1 Introduction
BURGER uses robotic drawing to study how VLA models learn language-conditioned behaviors from scalable, diverse trajectories. Its benchmark supports analysis of grounding, spatial composition, and action generation, including how models respond to data scale and generalize.
1 Introduction
- Drawing starts from a blank canvas, preventing inference of the requested drawing from the initial observation alone, while valid stroke orders and trajectories are multimodal.Inspecting canvases and trajectories makes semantic, spatial, and execution failures directly diagnosable.Source: §1 · pp. 1–2
- BURGER generates robot end-effector trajectories from natural-language prompts and includes 132 single-object tasks and 150 compositional tasks, each with 100 demonstrations.The dataset spans diverse objects, spatial compositions, and trajectory realizations.Source: §1 · p. 2
- Some flow-matching models generally improve with more demonstrations per task or more training tasks, and π0 retains multiple trajectory modes.Evaluated policies also generate unseen spatial compositions of familiar objects but struggle with object categories absent from training.Source: §1 · p. 2
- Drawing preserves language grounding and long-horizon continuous action demands relevant to manipulation, while simplifying contact-rich interaction and object dynamics.The paper positions it as a lens on VLA readiness for larger and more diverse training datasets.Source: §1 · p. 2
2 Related Work
Related work covers VLA action-generation paradigms, diagnostic benchmarks, data-scaling studies, and robotic drawing systems. BURGER differs by using language-conditioned sketching to make varied valid trajectories and their failures inspectable.
2 Related Work
- VLA systems span autoregressive action prediction, continuous flow-matching or diffusion generation, and hybrid approaches; BURGER evaluates these paradigms on multimodal, long-horizon drawing data.Examples include OpenVLA [2], π0 and π0.5, RDT-1B and GR00T N1 [18] [19], and DiffusionVLA.Source: §2 · p. 3
- Diagnostic VLA benchmarks assess capabilities beyond binary success, while drawing exposes distinct valid stroke orders, geometries, and trajectories through canvases and rollouts.Prior benchmarks cover long-horizon reasoning, grounding, robustness, behavioral metrics, and memory-dependent manipulation [6] [7] [21] [22] [23].Source: §2 · p. 3
- Prior scaling studies examine how data quantity and composition affect manipulation-policy generalization, and BURGER studies a related question in controlled language-conditioned drawing.The cited factors include object and environment diversity, camera viewpoints, spatial arrangements, and dataset fragmentation [24] [25] [10].Source: §2 · p. 3
- Earlier robotic drawing and painting work targets stroke planning, motor control, or real-world execution, whereas BURGER studies sketching as a language-conditioned VLA benchmark.Examples include hierarchical reinforcement learning for sketching [26], FRIDA [27], and oil-painting reproduction with learned pixel dynamics [28].Source: §2 · pp. 3–4
3 Method
BURGER formulates drawing as sequential language-conditioned control, generates demonstrations from prompts, and evaluates both trajectory diversity and drawing quality. The dataset supports varied single-object and compositional tasks, with a sketch-specific metric validated against human judgments.
3.1 Task Formulation
- At each time step, the policy uses language, the current canvas, and robot proprioception to produce drawing actions without receiving a target sketch at inference.Canvas observations change as strokes accumulate, and pen-lifting transitions separate disconnected strokes.Source: §3.1 · p. 4
3.2 Prompt-to-Trajectory Data Generation
- The prompt-to-trajectory pipeline turns generated sketch candidates into ordered SVG strokes, replays them in simulation, and records observation-action pairs.It uses an end-effector pose controller and is implemented in SAPIEN, MuJoCo, and Isaac Sim.Source: §3.2 · p. 4, Figure 4 · p. 4
3.3 The BURGER Dataset
- 7.13 effective modes per instruction for BURGER versus 1.09 for LIBERO, a 6.6× increase under the reported clustering setting.The analysis clusters planar end-effector trajectories with a shared codebook and uses k = 24; higher multimodality persists across tested k choices.Source: §3.3 · p. 5
3.4 Quantifying Robot Trajectory Quality
- A frozen DINOv3 backbone underlies a sketch-specific evaluator that scores target-class similarity and separately assesses object identities and spatial relations in compositions.In a 10-class user study, the metric agrees more closely with human judgments than CLIPScore across Spearman correlation, Kendall’s τb, and pairwise agreement.Source: §3.4 · p. 5
4 Experimental Results
The experiments use BURGER to compare VLA scaling, language and spatial grounding, multimodal behavior, failure patterns, and real-robot execution. Results vary by model, with controlled analyses and physical trials revealing both capabilities and limitations.
4.1 Scaling Behavior of VLA Models
- π0 and π0.5 generally improve as training classes or trajectories per class increase, whereas π0-FAST scales inconsistently.The evaluated ranges are 10–132 classes and 10–100 trajectories per class.Source: §4.1 · p. 6, Figure 6 · p. 6
4.2 Language-conditioned Generalization
- In-distribution object instructions produce distinct requested objects, but π-family models fail to generate the corresponding object for unseen categories.The findings indicate effective grounding in trained semantic categories without direct generalization to unseen object concepts.Source: §4.2 · p. 6
- Policies generate valid left-right and over-under compositions for object pairs absent from training and distinguish the corresponding spatial instructions.For example, a policy draws a cup left of an apple despite not seeing that specific pair in training.Source: §4.2 · p. 7
4.3 VLA Models Capture Multimodal Drawing Behaviors
- Across eight representative tasks, training demonstrations exhibited 4.0–7.7 effective trajectory modes per instruction, while π0 rollouts exhibited 2.1–6.7 modes, showing that π0 preserved rollout multimodality after BURGER training.Source: §4.3 · p. 7
4.4 BURGER Reveals Fine-Grained VLA Failure Modes
- Failures include initial-position sensitivity, failure to terminate, generic strokes for unseen objects, and degraded object quality as compositional relations expand.Correct spatial relations can coexist with degraded object geometry, especially when tasks add vertical to horizontal relations.Source: §4.4 · p. 8
- In circle drawing, OpenVLA can stagnate in local circles, while OpenVLA-OFT with chunk length 8 improves contact and trajectory coherence over chunk length 1.OpenVLA-OFT and π0-FAST also concentrate on subsets of starts or directions and may overdraw beyond one revolution.Source: Figure 9 · p. 8, §4.4 · p. 8
4.5 Real Robot Results
- Fine-tuned π0.5 executes real-world drawing tasks on a Franka Panda and remains moderately robust to camera-viewpoint changes.Figure 10 depicts viewpoint changes alongside real-robot drawing outcomes.Source: §4.5 · p. 9, Figure 10 · p. 9
- The real-robot policy uses visual feedback to add a missing cup handle, skip a human-drawn bicycle wheel, and continue the remaining structure.It also produces different valid trajectories for the same task on the physical robot.Source: §4.5 · p. 9, Figure 10 · p. 9
5 Conclusion and Limitations
BURGER provides a scalable benchmark for language-conditioned drawing and fine-grained VLA evaluation, while its scope and generated data impose boundaries. Its sketch-focused setup omits contact-rich painting and may not represent natural human drawing styles.
5 Conclusion and Limitations
- BURGER supports scalable generation of diverse robotic drawing trajectories and evaluation of grounding, spatial reasoning, multimodal behavior, and action generation.The experiments examine responses to data scale and task diversity, distinct failure patterns, and real-world drawing with visual feedback.Source: §5 · p. 9
- The sketch-level drawing-tool setup excludes brush dynamics and limits coverage of contact-rich artistic media such as brush painting.The authors note that this simplification makes the benchmark scalable.Source: §5 · p. 9
A.1 Prompting and Sketch Generation
- Sketch images are generated by substituting object names into a fixed prompt template, with optional diversity instructions and 4 × 4 candidate grids split into PNGs.The generation pipeline uses GPT-Image 2.Source: §A.1 · p. 14
A.2 Data-Generation Algorithms
The data-generation algorithms convert PNG sketches into ordered stroke representations that can be replayed as robot trajectories. They skeletonize images, extract paths, and encode those paths as SVG commands.
A.2 Data-Generation Algorithms
- PNG-to-skeleton conversion binarizes images, detects contours, suppresses approximately circular filled regions, and skeletonizes the cleaned foreground.The algorithm saves the resulting binary skeleton images for the next conversion stage.Source: §A.2 · pp. 14–15
- The ordered stroke paths become SVG commands, preserving stroke structure and ordering for replay in the robot environment to generate observation–action trajectories.The pipeline first decomposes the skeleton into ordered paths, then converts each path into SVG commands.Source: §A.2 · pp. 14–15
- Skeleton-to-SVG conversion builds a graph from 8-connected foreground pixels, extracts paths, samples them at spacing ∆, and smooths approximately circular paths.The paths are ordered by drawing heuristics: large outlines first, then smaller details from top to bottom using local nearest-neighbor ordering.Source: §A.2 · p. 15
A.3 Dataset Composition and Statistics
BURGER’s dataset combines single-object and compositional drawing tasks, with demonstrations showing variation in geometry and stroke order. It also includes real-robot examples collected by replaying generated SVG trajectories.
A.3 Dataset Composition and Statistics
- The dataset contains 132 single-object and 150 compositional tasks, with 100 demonstrations each, totaling 28,200 simulated demonstrations.The compositional set includes 100 horizontal tasks and 50 using vertical relations such as “over” and “under.”Source: §A.3 · p. 16
- Demonstrations vary in stroke order and object geometry, and single-object sketches combine into compositions with language-specified spatial relations.Figure 12 visualizes 10 representative tasks.Source: §A.3 · p. 16
- Real-robot data are collected by replaying generated SVG trajectories without teleoperation or closed-loop feedback, recording instructions, camera observations, proprioception, and expert actions.Either the front or rear camera view can serve as the visual observation input.Source: §A.3 · p. 16
A.4 Comparison with Other Sketch Sources and LIBERO
BURGER’s prompt-driven sketch pipeline produces drawings with a favorable balance of recognizability and stroke simplicity, and its trajectories show greater instruction-conditioned diversity than LIBERO under the tested clustering settings.
A.4 Comparison with Other Sketch Sources and LIBERO
- The compositional-task table lists 100 horizontal A-left-of-B tasks and notes that BURGER also contains 50 tasks with vertical relations.Each horizontal task pairs a left-side object A with a right-side object B.Source: Table 3 · p. 17
- BURGER sketches balance visual recognizability and stroke simplicity with moderate line counts, smooth contours, and clear object structure.The paper describes these properties as suitable for conversion into robot-executable trajectories.Source: §A.4 · pp. 16–17
- The comparison uses 40 instructions with approximately 42 episodes each, resampling trajectories to 32 points before clustering pooled descriptors with a shared k-means codebook.The effective mode count uses cluster frequencies; p_j denotes the fraction assigned to cluster j.Source: §A.4 · p. 17
- Examples illustrate variation in object shapes and drawing sequences, with color encoding stroke order from dark blue through light blue and light red to dark red.Figure 12 visualizes stroke-order and geometry diversity.Source: Figure 12 · p. 18
- At k = 24, BURGER has 6.57-fold more effective trajectory modes than LIBERO, and it consistently exceeds LIBERO across ten initializations and tested clustering resolutions.BURGER rises from 3.63 to 9.22 effective modes, while LIBERO remains at 1.05–1.10.Source: §A.4 · p. 18
B.1 VLA Post-Training Protocol
The post-training protocol adapts each policy using its model-specific action objective, then evaluates it across shared initial-state and episode-length conditions without task-specific adaptation.
B.1 VLA Post-Training Protocol
- Training samples pair a language instruction and simulator canvas observation with robot proprioception and the expert action recorded during SVG replay.Prompts include single-object and compositional instructions.Source: §B.1 · pp. 18–19
- π0 and π0.5 are fine-tuned with their original flow-matching action objective, while π0-FAST uses autoregressive prediction of FAST action tokens.All models use batch size 32 and are evaluated without additional task-specific adaptation.Source: §B.1 · p. 19
- Evaluation uses the same initial-state distribution and maximum episode length across models, with ten episodes per task and average scores reported.Episodes end when the end-effector rises above a predefined height threshold, so policies must learn both drawing and termination behavior.Source: §B.1 · p. 19
B.2 Rollout Evaluation
The rollout evaluation compares a sketch-specific learned metric with human judgments and assesses policy drawings. The accompanying sections describe real-robot interaction, qualitative scaling effects, and multimodal rollouts.
B.2 Rollout Evaluation
- The sketch metric combines frozen DINOv3 image features into a normalized 256-dimensional embedding, then trains a projection head and cosine classifier with cross-entropy and supervised contrastive learning.The contrastive term clusters same-class drawings and separates different-class drawings.Source: §B.2 · pp. 19–20
- For compositional tasks, scoring separates foreground strokes into two object regions, compares each region with its class center, averages the similarities, and penalizes incorrect spatial ordering.Single-object scoring compares the final canvas with the prompted object’s class center.Source: §B.2 · p. 20
- 0.834 Spearman correlation, 0.655 Kendall’s τb, and 0.875 pairwise agreement are achieved by the latent-based metric, versus 0.592, 0.433, and 0.803 for CLIPScore, respectively.The three measures compare aggregate human rankings and individual pairwise preferences.Source: §B.2 · pp. 20–21, Table 4 · p. 21
B.3 Real-Robot Implementation
- Real-robot inference merges stroke observations from two depth-camera views, and human hand entry pauses the policy until the user modifies the canvas and leaves.The policy then receives the updated canvas and resumes drawing with a Franka Panda arm.Source: §B.3 · pp. 21–22
C.1 Additional Scaling Results
- Increasing either demonstrations per class or training classes improves π0.5’s qualitative drawing quality, with demonstration count having a particularly strong effect.The authors suggest sufficient within-class coverage is important for reliable stroke-level action generation.Source: §C.1 · p. 22
C.2 Additional Multimodal Rollouts
- Training examples vary in stroke order and geometry, while trained VLA rollouts reproduce multiple valid drawing modes for the same object.Representative model comparisons select the highest-scoring rollout for each model and task.Source: §C.2 · p. 22, §C.1 · p. 22
C.3 Additional Spatial-Composition Results
The reported spatial-composition results show VLA models applying learned spatial relations to object pairs not seen during training, alongside in-distribution compositional drawing.
C.3 Additional Spatial-Composition Results
- VLA models apply learned spatial relations to unseen object pairs as well as generating compositional drawings for trained object pairs.Figure 20 contrasts in-distribution spatial-relation grounding with generalization to unseen object compositions.Source: §C.3 · p. 22, Figure 20 · p. 26
- The accompanying qualitative figures depict multiple valid drawing realizations and show drawing quality across demonstration and training-class scales.Figure 16 varies demonstrations per class and training-class count; Figures 18–19 show multimodal examples.Source: Figure 16 · p. 23, Figure 18 · p. 24, Figure 19 · p. 25
- Figure 20 presents rollouts on trained object compositions and unseen object pairs, making transfer of spatial relations across object combinations the key comparison.The figure caption states that models can apply learned relations to pairs not observed during training.Source: §C.3 · p. 22, Figure 20 · p. 26