Source-linked AI summary
CoBrush: A Hierarchical Planning Framework for Human-Robot Co-Painting
Dantong Qin, Yike Guo, Qinlin Liu, Alessandro Bozzon, Pan Wang
TL;DR
Open-ended co-painting requires robots to update a shared physical canvas as human intent evolves, but existing systems are commonly reactive sketch assistants or single-shot painters. CoBrush coordinates semantic planning, spatial grounding, and stroke execution in a multi-round framework. Its evaluations report stronger semantic and spatial results and higher perceived plausibility than single-shot baselines.
Problem
Existing reactive sketch assistants and reference-driven robotic painters provide limited structured support for coherent multi-round collaboration on a real canvas.
Source: §I · p. 1Method
CoBrush uses a VLM to select a semantic target and coarse spatial prior, then refines the region and executes it as brush strokes.
Source: §I · p. 1, A. CoBrush: Semantic planning · p. 2, 2) Coarse spatial prior · p. 2Results
CoBrush reports stronger semantic alignment, more accurate spatial localization, and higher perceived plausibility than single-shot approaches; its PSA is 0.675 and mean P-IoU is 0.376.
Source: §I · p. 2, A. Quantitative Results · pp. 5–6Takeaways & Limitations
The authors present hierarchical organization as a practical basis for stable and flexible human–robot collaboration on real physical canvases.
Source: §VI · pp. 7–8
Abstract
from arXiv · showhide
Embodied co-painting requires a robot to repeatedly update a shared physical canvas while human intent evolves over interaction. Existing reference-driven painters or reactive assistants are typically optimized for single-shot rendering or sketch completion, limiting their ability to sustain coherent multi-round collaboration or to construct complex, content-rich scenes over time. We present CoBrush, a hierarchical framework that formulates multi-round co-painting as a coordinated semantic, spatial, and execution process. By separating high-level intent inference from spatial grounding and stroke-level control, the system supports progressive scene development on real acrylic canvases. We evaluate the framework through real human-robot painting sessions, stress tests, and user studies. Compared to single-turn baselines, our approach achieves stronger semantic alignment, more stable spatial progression, and higher perceived plausibility of robot actions. These results demonstrate that structured multi-stage reasoning improves the coherence and robustness of interactive painting and supports the progressive development of content-rich physical artworks.
I. INTRODUCTION
The paper frames co-painting as an open-ended, multi-round physical collaboration in which intent and visual structure evolve through interaction. CoBrush separates semantic planning, spatial grounding, and physical execution, and its real-session evaluation reports improvements over single-shot approaches.
I. INTRODUCTION
- The framework addresses open-ended painting by repeatedly deciding what to add, where to place it, and how to realize it on a shared canvas.Goals and visual structures emerge gradually through interaction rather than being fixed in advance.Source: §I · p. 1
- Existing digital co-drawing systems emphasize reactive sketch completion or refinement, while physical painting systems are largely single-shot and reference-driven.The paper identifies a gap in structured planning for coherent multi-round collaboration on a real canvas.Source: §I · p. 1
- CoBrush uses a VLM to infer a semantic target and coarse spatial prior, refines the location, then translates the region into executable brush strokes.The robotic arm realizes the updates on an acrylic canvas within the interaction round.Source: §I · pp. 1–2
- Real multi-round sessions, stress tests, and user studies report stronger semantic alignment, more accurate localization, and higher perceived plausibility than single-shot approaches.The evaluation compares representative single-shot approaches under the same interaction settings.Source: §I · p. 2
II. RELATED WORK
Prior work includes interactive digital sketching and physical robotic painting, but the paper identifies limited compositional planning for evolving guidance. CoBrush assigns high-level reasoning to a VLM and delegates localization and stroke execution to separate modules.
A. Human-AI collaborative drawing system
- Digital collaborative drawing spans multi-turn stroke interaction, object-level completion, and multimodal or diffusion-based refinement, but remains largely reactive to user input.These systems mainly complete, refine, or embellish input rather than reason structurally about an evolving composition.Source: A. Human-AI collaborative drawing system · p. 2
B. Vision-language models
- CoBrush uses a VLM for high-level semantic planning while dedicated modules handle localization and stroke execution.This contrasts with SketchAgent, which couples VLM semantic reasoning directly to grid-based stroke coordinates in a digital sketch environment [24].Source: B. Vision-language models · p. 2
C. Robotic painting
- Most robotic painting systems use reference-driven or predefined single-shot objectives; interactive systems still lack structured planning for evolving guidance.CoBrush instead treats each round as a decision-making process for intent-to-stroke mapping and continuous collaboration.Source: C. Robotic painting · p. 2
III. METHOD
CoBrush organizes embodied co-painting into semantic planning, spatial grounding and synthesis, and stroke-based execution. Its planner predicts what to add and where, using semantic abstractions and a discretized grid prior for spatial grounding.
III. METHOD
- CoBrush combines a semantic planner, a grounding-and-synthesis module, and a stroke-based executor in an iterative human–robot collaboration loop.Together, these modules connect intent inference and spatial localization to physical rendering.Source: §III · p. 2
A. CoBrush: Semantic planning
- At each round, a frozen VLM maps the evolving canvas and human-provided theme into a semantic target and coarse spatial prior.The structured output is (yt, bt), describing what content to add and approximately where it should go.Source: A. CoBrush: Semantic planning · p. 2
1) Semantic abstraction:
- The VLM summarizes the current layout and dominant elements, predicts a plausible next development under the theme, and derives a target with a brief justification.An example target is “eyes and nose.”Source: 1) Semantic abstraction · p. 2
2) Coarse spatial prior:
- A 10 × 10 grid turns region prediction into discrete selection: the prior specifies a top-left cell and region extent in grid units.The passage motivates discretization as a way to reduce ambiguous coordinate regression to low-cardinality classification and improve stability across canvas resolutions.Source: 2) Coarse spatial prior · p. 2
3) Self-critique refinement:
CoBrush uses self-critique to revise rejected spatial predictions, while visual synthesis and coarse-to-fine localization turn an abstract target into a painting region. The synthesized reference is intended to align with the target content and local canvas style.
3) Self-critique refinement:
- The VLM checks whether a predicted region matches the intended composition and revises rejected predictions, typically in one to two iterations.The check uses a visible grid overlay and a binary Yes/No status with a short rationale; the loop helps reduce misalignment.Source: 3) Self-critique refinement · pp. 2–3
B. CoBrush: Visual synthesis and Coarse-to-fine localization
- The visual synthesis and localization module converts a semantic target and coarse prior into a visual reference and a precise painting region.That region is intended for rendering, bridging high-level intent and low-level execution.Source: B. CoBrush: Visual synthesis and… · p. 3
1) Visual synthesis:
- GPT-Image-1 generates an exemplar image from the visual summary and target phrase to show how missing content should appear on the current canvas.The reference is prompted to align with the intended content and local painting style.Source: 1) Visual synthesis · p. 3
2) Weakly-supervised region adaptation:
CoBrush refines a planner’s coarse box using weakly supervised local visual features, then combines semantic and geometric cues to obtain a painting region for stroke planning.
2) Weakly-supervised region adaptation:
- DINOv2 patch features inside the coarse box form weak positives, while a surrounding ring supplies weak negatives for a Ridge regression probe.The features are augmented with a bias before fitting.Source: 2) Weakly-supervised region adaptation · p. 3
- The classifier scores each patch, and the selected rectangle maximizes accumulated relevance while penalizing excessive area.This refines the coarse box into the final rectangular region for instance-level localization.Source: 2) Weakly-supervised region adaptation · p. 3
- For stroke planning, LLaVA-1.6 cross-attention heatmaps are combined with Grounded-SAM instance masks to form a target-consistent, geometrically complete region.The heatmap highlights areas relevant to the target phrase; the masks provide precise geometric boundaries.Source: 3) Instance-level localization · p. 3
3) Instance-level localization:
CoBrush converts a refined painting region and target-conditioned reference into executable strokes and colors, then adapts those outputs to the robot’s physical paint palette and workflow.
C. CoBrush: Physical execution
- Given the refined region and target-conditioned reference image, CoBrush translates the desired update into physically executable strokes and colors.Source: C. CoBrush: Physical execution · p. 3
1) Stroke rendering:
- The stroke renderer converts the reference image into cubic Bézier strokes, samples them into canvas waypoints, and maps them to 3D end-effector poses.The robot tracks the resulting trajectories with Cartesian velocity control.Source: 1) Stroke rendering · p. 4
2) Palette-aware robot execution:
- Stroke colors are quantized into 21 palette entries, and the robot uses predefined dipping and cleaning poses when changing colors or replenishing paint.The palette has seven color bins and three value levels.Source: 2) Palette-aware robot execution · p. 4
D. Human–Robot Collaboration
CoBrush operates in a turn-based loop in which human and system updates alternate on a shared canvas; the reported study used individual participant sessions and evaluated 11 completed artworks.
D. Human–Robot Collaboration
- A fixed shared theme and overhead-camera image condition each system update, after which control returns to the human for another round.The human can react to, modify, or extend the system’s result.Source: D. Human–Robot Collaboration · p. 4
A. Implementing details
- The system runs on a 7-DoF Franka arm, and pruning same-color strokes with over 80% spatial overlap reduced execution time by more than 5× despite some geometric loss.A sponge pad is used for brush cleaning between color switches.Source: A. Implementing details · p. 4
B. Participants
- Eleven completed artworks were used for evaluation after two participants could not finish and one session was excluded due to system failure.Participants completed the study individually; typical sessions lasted 4–5 hours.Source: B. Participants · p. 4
C. Compared Methods
The comparison uses single-shot baselines and an ablated CoBrush variant, while evaluation combines semantic and spatial measures with user ratings of plausibility.
C. Compared Methods
- CLIPDraw, FRIDA, and a CoBrush variant without multi-turn reasoning are evaluated as single-shot updates from the participant’s first-round canvas.The ablation also removes semantic planning and region prediction.Source: C. Compared Methods · p. 4
D. Metrics
- Semantic evaluation includes per-round CLIP similarity between predicted and user-provided targets, plus final-artwork ratings of whether targets appear in user-specified boxes.The latter uses GPT-4.1 and LLaVA-OneVision ratings; SEC adds a yes/no confidence measure.Source: D. Metrics · pp. 4–5
- Spatial evaluation measures per-round IoU against participant-specified boxes and asks users to rate the plausibility of content within those regions.The study uses a hybrid of automatic and user-centric metrics, with Table I comparing methods on alignment or plausibility measures.Source: D. Metrics · pp. 4–5, Table I · p. 5
V. RESULTS
CoBrush reports stronger semantic and spatial results, with higher perceived plausibility of its decisions. Qualitative comparisons show how iterative updates differ from single-turn output on a shared canvas.
A. Quantitative Results
- CoBrush achieves a PSA of 0.675, the best OSA and highest SEC values, while Posterior Rationality ratings are higher for both target and bounding box.Together, the reported results indicate alignment with user intent and more collaborative, plausible decisions.Source: A. Quantitative Results · pp. 5–6
B. Qualitative Results
- Figure 4 shows human and agent contributions alternating across themes, with each agent update labeled by its predicted next-step target.The examples proceed from left to right.Source: Figure 4 · p. 6
- On the same initial canvas, the full multi-turn model confines updates to targeted regions and preserves earlier strokes, unlike the single-turn variant.The single-turn variant reflects only the initial state without incorporating intermediate feedback.Source: B. Qualitative Results · p. 6, Figure 5 · p. 6
C. Region Refinement Effectiveness
Region refinement adjusts coarse spatial predictions toward visually coherent target areas. In the goal-drift stress test, spatial decisions remained stable and rationality ratings high despite ambiguous or conflicting input.
C. Region Refinement Effectiveness
- Refinement shifts or expands insufficient coarse regions to cover more of the visually coherent area associated with the target concept.Features inside the coarse box serve as positives and those in a surrounding ring as negatives; a rectangular search selects regions following high-score areas.Source: C. Region Refinement Effectiveness · pp. 6–7, Figure 6 · p. 7
D. Goal Drift Stress Test
- Under increasing perturbation, P-IoU remains stable and Rationality scores stay high, while overall performance drops only marginally.The test covers mild off-theme decoration, dominant theme-inconsistent content, and strongly conflicting or destructive input; semantic metrics are excluded for the destructive condition.Source: Table II · p. 7, D. Goal Drift Stress Test · p. 7
E. Runtime Breakdown
Runtime is dominated by physical painting, while participants reported favorable, low-effort collaboration with moderate intent alignment. The conclusion frames hierarchical decision-making as a practical basis for stable, flexible co-painting and identifies broader themes and personalization as future directions.
E. Runtime Breakdown
- Physical stroke rendering and robot execution take 22–63 min per round, making execution the dominant runtime component.The arm averages about 3 min per 10 strokes, and a typical 4–5-round session lasts 4–5 hours; semantic planning takes approximately 71–96 s per round.Source: E. Runtime Breakdown · p. 7
F. User Experience
- Satisfaction averaged 7.11 (SD 1.17), while intent alignment averaged 6.33 (SD 1.41), pointing to fine-grained semantic grounding as an improvement focus.Participants also rated perceived workload at 2.33 (SD 1.22), turn-taking flow at 5.56 (SD 2.35), and creative support at 5.56 (SD 2.07).Source: F. User Experience · p. 7
VI. CONCLUSION AND FUTURE WORK
- The conclusion presents hierarchical organization as a practical way to support coherent collaboration with stability and flexibility on a real physical canvas.It proposes adaptive component-invocation policies and modeling user preferences and painting rhythms as future directions.Source: §VI · pp. 7–8