Source-linked AI summary
ACE-Brain-0.6: Scaling Spatial Grounding for Physical Foundation Models
Yinhe Wang, Tao Tao, Zexin He, Shuang Luo, Wei Mao, Yao Mu, Feng Pan, Hanming Deng, Lewei Lu, Zhi Hou, Xingang Pan, Dahua Lin, Dacheng Tao, Xiaogang Wang
TL;DR
Grounding errors can misdirect physical action, motivating the question of whether broader supervision can strengthen an open model’s spatial perception. The paper builds a pipeline that converts diverse visual data into verifiable grounding supervision and fine-tunes Qwen3.5-9B. ACE-Brain-0.6 exceeds GPT-6 Astra on four of seven public benchmarks and achieves a higher seven-benchmark average; the authors identify remaining gaps in complete-instance enumeration and spatially constrained placement.
Problem
Grounding must localize all regions matching a description and recognize absent targets, while clutter and occlusion make this correspondence difficult and errors can misdirect action.
Source: §1 · p. 3Method
The pipeline combines robot recordings with complementary visual sources, automatically annotates robot data, and converts sources into verifiable question–answer supervision for fine-tuning Qwen3.5-9B.
Source: §1 · p. 3, §4.2 · p. 8Results
ACE-Brain-0.6 exceeds GPT-6 Astra on four of seven public grounding benchmarks and scores 75.41 versus 73.68 on their seven-benchmark average.
Source: §1 · p. 4Takeaways & Limitations
Across fixed-recipe training runs, grounding performance rises with data scale, and expanding supervision coverage contributes substantially more than increasing existing-source volume.
Source: §1 · p. 4, §5.4 · p. 11ACE-Brain-0.6 trails Astra on complete-instance enumeration and placement under spatial constraints, where two-dimensional free-space supervision does not capture three-dimensional feasibility.
Source: §6 · p. 12
Abstract
from arXiv · showhide
General-purpose multimodal models, including GPT-6 Astra and Claude Opus 5.5, demonstrate broad capabilities in embodied manipulation. Our analysis indicates that this generality is accompanied by strong spatial perception, as Astra can determine where to interact with task-relevant entities amid visual distractors. Because spatial perception connects high-level reasoning to the physical scene, we investigate scaling visual grounding to strengthen the perceptual foundation of general embodied operation. We develop a unified data pipeline that combines robot manipulation recordings with complementary visual data from diverse sources. After automatically annotating the robot recordings, the pipeline converts the collected data into verifiable grounding supervision. Fine-tuning Qwen3.5-9B on approximately 1.5 million samples produced by this pipeline yields ACE-Brain-0.6. The model outperforms GPT-6 Astra on four of seven public grounding benchmarks, with an overall average score of 75.41 compared with Astra's 73.68. Under a fixed Qwen3.5-9B fine-tuning setup, aggregate grounding performance improves steadily as successive data releases expand the training collection to cover a broader range of grounding scenarios. These results show that scaling verifiable grounding supervision from diverse sources can strengthen the spatial perception of an open 9B model for general embodied operation.
1 Introduction
Spatial grounding links high-level reasoning to visual referents and matters because localization errors can misdirect physical action. The paper tests whether broader, verifiable grounding supervision can improve an open 9B model, reporting gains over Astra on four public benchmarks and in their seven-benchmark average.
1 Introduction
- Grounding links language descriptions to image referents, and errors in cluttered or occluded scenes can misdirect action despite sound high-level reasoning.The paper frames this connection as a reason to study grounding for embodied operation.Source: §1 · p. 3
- A unified pipeline combines robot recordings with complementary visual data, automatically annotates the recordings, and converts sources into verifiable question–answer supervision for Qwen3.5-9B.Fine-tuning on approximately 1.5 million samples for one epoch produces ACE-Brain-0.6.Source: §1 · pp. 3–4
- 75.41 versus 73.68 on the seven-benchmark average, with ACE-Brain-0.6 exceeding GPT-6 Astra on four of seven public grounding benchmarks.The model remains behind Astra on PixMo-Count, RefSpatial, and PointArena.Source: §1 · p. 4
- With the base checkpoint and training recipe fixed, the seven-benchmark average rises approximately linearly with log sample count, while a same-size control adding only existing sources gains less than one point.The control suggests broader supervision coverage contributes to the gains; releases expand both sample count and coverage.Source: §1 · p. 4
2 Related Work
Related work spans visual grounding methods embedded in broader embodied models and varied sources of grounding supervision. This paper focuses on grounding itself and combines heterogeneous supervision to study the effects of its scale and composition.
2.1 Visual Grounding and Embodied Spatial Understanding
- Detector-based and generative grounding methods connect spatial prediction with increasingly broad embodied capabilities, while this work holds the base model and training recipe fixed to study training-data effects.Examples include GLIP and Grounding DINO, Pix2Seq, Florence-2, Rex-Omni, PaLM-E, Gemini Robotics-ER, RynnBrain, and Embodied-R1.5.Source: §2.1 · p. 4
2.2 Grounding Data and Supervision
- Prior supervision uses human annotation, automatic labeling, or geometric derivation; this work combines robot, egocentric, object-centric, and 3D data in a common verifiable question–answer format.Checks address instance completeness, spatial constraints, and target absence.Source: §2.2 · p. 4
3 Analysis of GPT-6 Astra
The analysis connects grounding to geometric manipulation and examines Astra’s breadth and precision across grounding tasks. Case studies show how visual references supply action geometry, while examples reveal localization errors at fine boundaries and under spatial constraints.
3 Analysis of GPT-6 Astra
- Task-relevant grounding identifies objects, interaction regions, and spatial references that analytic geometry uses to determine positions, displacements, and motion parameters.The section uses two manipulation cases before examining Astra’s grounding capabilities.Source: §3 · p. 5
3.1 Case Studies: From Grounding to Geometric Robot Manipulation
- Basket placement uses points on the basket and destination, while T-shirt folding uses garment extent; in both cases, visual references determine the geometric quantities used to construct the action.The basket case estimates relative configuration from calibrated RGB-D observations; the folding case estimates garment dimensions from a contour.Source: §3.1 · p. 5
3.2 A Closer Look at Astra’s Grounding Capabilities
- Astra handles ordinal and object referring, complete instance sets, free-space relations, and functional or interaction targets, but examples also show errors outside a mug-part mask and a requested region.The failures illustrate sensitivity to local geometric boundaries and spatial relations to reference objects.Source: §3.2 · pp. 6–7
4 Method
ACE-Brain-0.6 casts grounding as conditional text generation and trains Qwen3.5-9B on heterogeneous supervision made verifiable against annotations. Its method combines automatic robot-data annotation, controlled question generation, quality checks, and a fixed training recipe.
4.1 Model and Problem Formulation
- Grounding tasks use a shared conditional text-generation interface that outputs points, point lists, boxes, counts, or categorical tokens from images and a natural-language query.Coordinates are integers normalized to [0, 1000], and the interface covers referring, counting, spatial relations, interaction grounding, and two-view robot targets.Source: §4.1 · p. 7
4.3 Training
- The model starts from Qwen3.5-9B, fine-tunes all language-model parameters with a frozen vision encoder, and trains for one epoch using a fixed learning-rate recipe across releases.The objective applies token-averaged cross-entropy to answer tokens only; the collection contains about 1.54 million samples.Source: §4.1 · p. 7, §4.3 · pp. 8–9, §4.2 · p. 8
4.2 Data Pipeline and Data Scale
- Training data combines robot manipulation episodes, egocentric human–object interactions, and general pointing and grounding data, with a small amount of visual question answering data.The sources contribute complementary supervision such as contact regions, exhaustive instance sets, part masks, and 3D spatial annotations.Source: §4.2 · p. 7
- Robot recordings are converted into grounding supervision by selecting keyframes, building scene- and camera-specific object vocabularies, and relabeling detected boxes with recognition concepts.Semantic relabeling corrects labels without moving or adding boxes; question–answer generation enforces verifiable answers, complete instance sets for counting, and balanced formats.Source: §4.2 · p. 8, Figure 4 · p. 8
5 Experiments
ACE-Brain-0.6 is evaluated on seven public grounding benchmarks and a held-out robot-scene benchmark, with analyses of where its gains come from and how performance changes across data releases. It exceeds GPT-6 Astra on four public benchmarks and has a higher Avg-7, while performance rises as releases broaden supervision coverage.
5.1 Setup
- Evaluation spans seven public benchmarks and RobotGrounding, with shared prompts and scoring protocols; closed models are queried through their APIs.The public benchmarks cover spatial reasoning, multi-instance perception, spatial relations and free-space localization, and functional and interaction grounding; RobotGrounding contains 2,591 questions across 14 types.Source: §5.1 · p. 9
5.2 Main Results
- Fine-tuning improves the base model on six benchmarks and raises Avg-7 by almost ten points, with the largest gains on spatial relations and constrained references.Counting shows no gain and costs a few points; ACE-Brain-0.6 remains behind GPT-6 Astra on RefSpatial (71.8 vs. 80.1).Source: §5.2 · p. 9
- ACE-Brain-0.6 exceeds GPT-6 Astra on four benchmarks and records Avg-7 75.4 vs. 73.7, while leading all compared models on RobotGrounding.The table covers seven public benchmarks and RobotGrounding, which is excluded from Avg-7.Source: §5.2 · pp. 9–10, Table 1 · p. 10
5.3 Where the Gains Come From
- For spatially constrained placement, Astra leads on RefSpatial location; adding several thousand placement questions derived from 2D masks leaves placement unchanged.The 2D free-space targets mark unoccupied regions but do not assert 3D feasibility; the authors expect the gap to require targets verified in 3D.Source: §5.3 · pp. 10–11
- ACE-Brain-0.6 leads Astra on part and free-space tasks, while Astra remains slightly ahead on relation judgments; the largest ShareRobot gain coincides with adding action-selected part masks.On RobotGrounding, gains over the base model center on absent-object and absent-attribute recognition and cross-view matching; Astra leads when every instance must be enumerated or a relation resolved.Source: §5.3 · p. 11
5.4 Data Scaling
- Across releases, Avg-7 rises from 69.3 to 75.4 as training data grows from 0.26M to 1.55M samples, about +2.4 points per doubling, with no saturation observed in the tested range.Each point is a full Qwen3.5-9B training run with the same recipe; only the data changes.Source: §5.4 · p. 11, Figure 6 · p. 12
- The main data series expands sample count and supervision coverage together, so their effects cannot be fully separated; a control adding existing sources gains less than one point.Adding more many-point answers also did not improve PointArena questions with many instances.Source: §5.4 · pp. 11–12
6 Conclusions
The conclusion asks how far expanded grounding data can carry an open 9B model toward GPT-6 Astra’s capabilities. ACE-Brain-0.6 improves across public benchmarks as supervision grows, while gaps remain in complete instance enumeration and spatially feasible placement.
6 Conclusions
- ACE-Brain-0.6 is ahead of Astra on four of seven public grounding benchmarks, on par on two, and scores 93.4 versus 91.1 on the robot-scene benchmark.Its seven-benchmark average is also above Astra’s.Source: §6 · p. 12
- Across data releases, grounding performance rises log-linearly with sample count, and the observed rise is driven by supervision coverage.The controlled runs vary data releases while using the same open 9B model setting.Source: §6 · p. 12
- ACE-Brain-0.6 still trails Astra on enumerating every instance and on placement under spatial constraints, where 2D free-space supervision misses 3D feasibility.The proposed next additions are complete multi-instance targets and feasibility-aware placement data.Source: §6 · p. 12
A.2 The RobotGrounding Benchmark
RobotGrounding evaluates grounding in robot-policy scenes using held-out manipulation frames and diverse question types. Its scoring combines answer-specific correctness criteria with an unweighted mean over 14 groups.
A.2 The RobotGrounding Benchmark
- The benchmark contains 2,591 questions on held-out robot-manipulation frames, with hand-annotated boxes checked question by question.Evaluation frames’ episodes and all other evaluation benchmarks are excluded from training.Source: §A.2 · p. 16
- Its questions cover referring, small objects, every-instance pointing, counting, absent-target and spatial judgments, and cross-view correspondence.Positive and negative questions are balanced within each type so that always answering none does not score.Source: §A.2 · p. 16
- The overall score is the unweighted mean over 14 scoring groups, preventing numerous referring questions from dominating fewer, harder groups.Point answers must fall within target boxes, box answers require IoU thresholds, and counts or categorical answers must match exactly.Source: §A.2 · p. 16
- Figure 7 shows one example of each question type and the expected answer form; Figure 8 reports type counts, target sizes, and camera views.Twenty-one percent of targets cover less than 1% of the image, and wrist-camera questions make up 15%.Source: Figure 7 · p. 17, Figure 8 · p. 18