Source-linked AI summary
Look, Reason, Draw a Region: Recursive Refinement for Spatial Context Prediction
Wajeeha Tahir, Anna Tegon, Lionnus Kesting, Gamze İslamoğlu, Kristóf Karacs, Luca Benini, Thorir Mar Ingolfsson
TL;DR
Spatial-context prediction requires grounding an anchor object, interpreting a spatial relation in the appropriate reference frame, and identifying vacant regions. Spatial-TRM uses frozen vision-language features with recursive refinement and a convex region decoder. On RoboSpatial-Home, it improves context accuracy over a parameter-matched single-step model, while its point-derived region targets limit what evaluation establishes about full-region validity.
Problem
Spatial-context prediction requires identifying an anchor, interpreting the requested direction in its reference frame, and distinguishing vacant from occupied regions.
Source: §1 · p. 1Method
Spatial-TRM combines frozen open-vocabulary image and text features with a weight-tied reasoner that refines a latent region representation and a convex-mask decoder.
Source: §1 · p. 1, §4.3 · p. 5Results
Takeaways & Limitations
Repeated refinement improves spatial-context prediction from frozen features at a fixed trainable parameter count on this benchmark.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Spatial context prediction requires visual grounding and spatial reasoning to identify vacant regions relative to objects in a scene. Spatial relationships and physical constraints remain challenging for large vision-language models (VLMs). Recursive reasoning has demonstrated strong performance on discrete and abstract reasoning tasks, motivating its use for improving spatial context prediction in natural images. Weight sharing enables higher accuracy with a small trainable parameter count. We introduce Spatial-TRM, a compact recursive model with frozen vision-language features, a convex region decoder, and just 11.2M trainable parameters. This weight-tied reasoner supports six spatial relationships, three reference frames, and open-vocabulary anchor objects. On the RoboSpatial-Home benchmark, Spatial-TRM achieves 29.51% context accuracy, a gain of 9.02 percentage points (44.0% relative) over a separately trained single-step model with the same parameter count. Including the frozen encoders, the model has 877.8M total parameters, approximately $15\times$ fewer than the reported 13B size of RoboPoint. These results show that repeated refinement improves spatial context prediction from pretrained vision-language features at a fixed trainable parameter count on this benchmark.
1 INTRODUCTION
The introduction asks whether repeated refinement can improve spatial-context prediction from frozen features at a fixed parameter count. Spatial-TRM applies this approach and reports higher RoboSpatial-Home context accuracy than a matched single-step model.
1 INTRODUCTION
- Spatial-context prediction requires identifying an anchor, interpreting direction in its reference frame, and distinguishing vacant from occupied space.The introduction asks whether recursive refinement can improve this task from frozen features at a fixed parameter count.Source: §1 · p. 1
- Spatial-TRM combines frozen open-vocabulary image and text features with a weight-tied reasoner that repeatedly refines a latent region representation.It uses 11.2M trainable parameters, a convex-mask decoder, six spatial relationships, and three reference frames.Source: §1 · pp. 1–2
- On 122 RoboSpatial-Home context questions, Spatial-TRM achieves 29.51% context accuracy with 877.8M total parameters, approximately 15× fewer than RoboPoint’s reported 13B size.The model is trained on RoboSpatial’s spatial-context subset and evaluated on RoboSpatial-Home.Source: §1 · p. 2
- Context accuracy rises from 20.49% to 29.51% over a separately trained single-step model with the same parameter count and frozen image and text features.The paper reports this as a 44.0% relative increase without adding parameters.Source: §1 · p. 2
- The CLIP-backbone model uses 438M parameters and achieves 26.23% accuracy, half the size of the DINOv3 variant at a cost of 3.28 percentage points.The model combines frozen image and text encoders, recursive refinement, and a convex decoder.Source: §1 · p. 2
- Increasing the representative-point budget from one to ten raises accuracy from 29.51% to 44.26% for the same predicted masks.Point accuracy and mask overlap do not establish whether an entire predicted region is physically valid.Source: §1 · p. 2
2 RELATED WORK
Related work spans spatial reasoning and referring, pretrained visual-language representations, recursive refinement, and convex region modeling. Spatial-TRM combines these threads for region prediction from frozen RGB and text features.
2 RELATED WORK
- SpatialVLM, RoboPoint, RoboSpatial, RoboRefer, and RoboBrain 2.0 address different spatial reasoning and referring tasks; this work focuses narrowly on region prediction from frozen RGB and text features.Figure 1 illustrates a bed-anchored query for vacant space to its left in the object’s reference frame, with a mask derived from annotated target points.Source: §2 · p. 2, Figure 1 · p. 3
- DINOv2 Meets Text and DINOv3 provide aligned or transferable visual-language features, while Spatial-TRM trains its grounding, refinement, and region-prediction modules for the spatial task.Freezing the encoders reduces task-specific trainable parameters but retains their cost in the complete system.Source: §2 · p. 2
- Universal Transformers and Tiny Recursive Models reuse shared transformations across processing steps; Spatial-TRM adapts latent refinement to natural-image spatial-context prediction.Weight sharing permits additional refinement without a separate parameter set for each step, while prediction quality and computational cost remain separate evaluation questions.Source: §2 · p. 2
- CvxNet represents three-dimensional shapes with half-space constraints, while this decoder uses a related construction to produce a single two-dimensional convex region.The paper’s contribution is combining this geometric representation with recursive refinement for spatial-context prediction.Source: §2 · p. 2
3 PROBLEM FORMULATION
The paper formulates spatial-context prediction as binary segmentation of a candidate region relative to an anchor and reference frame, rather than as coordinate pointing. Its region targets are convex hulls derived from sparse annotated points and may not fully represent vacant space.
3 PROBLEM FORMULATION
- Spatial-context prediction changes from RoboSpatial’s pointing task, which outputs valid-location coordinates, to binary segmentation of a candidate region.A mask describes both the location and extent of the candidate area, which can be refined using spatial features from the entire image.Source: §3 · p. 3
- The query specifies a region relative to an anchor object and a reference frame.The target mask is the two-dimensional convex hull of annotated points, so it may omit valid locations or span occupied areas between points.Source: §3 · p. 3
4 SPATIAL-TRM
Spatial-TRM combines frozen vision-language features, anchor grounding, shared recursive refinement, and a convex region decoder. Its training objectives supervise mask fidelity, spatial-relation discrimination, and adaptive halting.
4 SPATIAL-TRM
- The model combines frozen encoders, anchor grounding, a shared reasoning block, and a convex decoder, with 11.2M trainable parameters.The full model has 877.8M parameters, of which 11.2M are trainable.Source: §4 · p. 3
4.1 INPUT REPRESENTATION
- Anchor-object text grounds image patches through cross-attention, and the refined patch features join the full-prompt embedding as the reasoner’s input.The two prompt embeddings come from the anchor name and the full prompt, respectively.Source: §4.1 · p. 3
4.2 LATENT RECURSIVE REFINEMENT
- The reasoner repeatedly updates reasoning and answer states with the same two-layer transformer, reusing weights across refinements.Two-dimensional rotary encodings are applied to vision tokens and one-dimensional encodings to text tokens.Source: §4.2 · p. 4
- Adaptive halting uses a learned head during training, but inference uses the full deep-supervision budget.Its intermediate correctness criterion combines a centroid with nearby support points and measures spatial agreement bidirectionally.Source: §4.2 · p. 4, §4.4.1 · p. 5
4.3 FROM LATENTS TO MASKS
- An interior point and fixed-direction half-plane offsets define a connected, hole-free convex polygon; minimum-margin logits thresholded at zero yield its binary mask.With K = 32, the decoder predicts 34 geometric values for a 224 × 224 mask.Source: §4.3 · pp. 4–5, Figure 2 · p. 4
4.4 TRAINING OBJECTIVES
- Mask fidelity uses class-weighted binary cross-entropy, while a relational contrastive loss compares predictions for correct and altered spatial prompts.The negative-relation loss enforces relative alignment but has no valid-region target for the negative prompt.Source: §4.4 · p. 5
5 EVALUATION PROTOCOL
The protocol evaluates whether predicted regions identify vacant space with the requested direction and reference frame. It reports point-in-mask context accuracy and mask-overlap metrics, with annotations that may cover only part of the valid space.
5 EVALUATION PROTOCOL
- A valid prediction must match the requested perspective and direction relative to the anchor while identifying nearby vacant space without intersecting objects.These are the stated conceptual conditions for a valid predicted mask.Source: §5 · p. 6
- Context accuracy counts whether the decoder’s single predicted interior point lies within the ground-truth mask, not whether the full region is valid or unoccupied.The score also depends on representative-point selection.Source: §5 · p. 6
- IoU and Dice measure overlap, but valid vacant space may extend beyond the annotated target, so boundary mismatch can lower scores despite a correct spatial relationship.Low scores can reflect either annotation mismatch or prediction error.Source: §5 · p. 6
6 OUT-OF-DISTRIBUTION GENERALIZATION
On RoboSpatial-Home, recursive refinement improves context accuracy over a same-size single-step model. Broader comparisons are bounded by the point-based evaluation and the benchmark’s scope.
6 OUT-OF-DISTRIBUTION GENERALIZATION
- Evaluation covers 122 RoboSpatial-Home context questions using 40,000 training samples and one predicted interior point; zero-shot generalization to unseen relations is outside scope.The benchmark omits perspective information and has more cluttered scenes than the training set.Source: §6 · p. 6
- 29.51% context accuracy versus 20.49% for the single-step model is a 9.02-percentage-point, 44.0% relative gain at the same 11.2M trainable parameters.Both variants use the same frozen encoder.Source: Table 1 · p. 6, §6 · p. 6
- 29.51% context accuracy exceeds reported VILA, SpaceLLaVA, and LLaVA-NeXT scores but is 1.59 percentage points below RoboPoint fine-tuned on RoboSpatial.Other reported comparisons include Gemini Robotics-ER 1.5 and RoboBrain 2.0, SpaceTools, and RoboRefer; baseline scores were not re-evaluated under a common protocol, and prompting, point selection, and model versions may differ.Source: §6 · pp. 7–8
- 877.8M total parameters, including frozen encoders, is approximately 15× fewer than RoboPoint’s reported 13B.Spatial-TRM has 11.2M trainable parameters; the size comparison does not measure inference speed or computational cost.Source: §6 · p. 8
- CLIP ViT-L/14 achieves 26.23% context accuracy with 438M total parameters, a 3.28-percentage-point accuracy reduction while halving size relative to DINOv3.SigLIP-SO400M/14 achieves 16.39% context accuracy with 888M parameters.Source: Table 3 · p. 8, §6 · p. 8
- 44.26% context accuracy at ten points versus 29.51% at one uses unchanged predicted masks; point hits do not establish full-region validity.The ten-point criterion is more permissive and raises accuracy by 14.75 percentage points.Source: §6 · p. 8
- 6.7% for above (1/15) contrasts with 35.7% for below (5/14); vertical queries are 2.36% of training data but 23.8% (29/122) of benchmark questions.Other reported context accuracies are 34.0% for front, 30.0% for left, 35.3% for right, and 22.2% for behind.Source: §6 · pp. 8–9
- Grounding can fail for small or distant objects; perspective ambiguity and missing occupancy supervision also leave masks vulnerable to intersecting other objects.The objective does not explicitly supervise occupancy or collision avoidance.Source: §6 · p. 9
7 DISCUSSION
Spatial-TRM improves RoboSpatial-Home context accuracy over a matched single-step model, while its compact region representation and point-based supervision limit what the predictions establish. The discussion also distinguishes model size from inference cost.
7 DISCUSSION
- Reducing model size by replacing DINOv3 with CLIP does not establish lower computation, since frozen encoders and repeated refinement still contribute to inference cost.The CLIP variant halves full model size at a 3.28-percentage-point accuracy reduction.Source: §7 · p. 9
- A single convex polygon cannot represent holes or disconnected valid regions, and its shape alone does not establish obstacle-free space, physical support, or object fit.Robot actions would require additional geometric and task-specific checks.Source: §7 · p. 9
- Point annotations yield derived masks that may omit valid locations or include occupied areas, so point accuracy and mask overlap do not establish whole-region physical validity.The authors identify independent occupancy annotations and more flexible region representations as ways to separate annotation mismatch from prediction error.Source: §7 · p. 9
- 9.02 percentage points: recursive refinement raises RoboSpatial-Home context accuracy from 20.49% to 29.51% over a separately trained single-step model with the same 11.2M trainable parameters.The full model has 877.8M parameters, approximately 15× fewer than RoboPoint’s reported 13B size.Source: §7 · p. 9
A.1 TRAINING DATASET
The appendix describes dataset construction, feature encoding and fusion, recursive reasoning, convex decoding, training losses, and validation-based checkpoint selection. It also reports checkpoint sensitivity on the small RoboSpatial-Home evaluation set.
A.1 TRAINING DATASET
- The training data extends RoboSpatial generation to above and below relations, then uses 40,000 samples because gains diminish beyond that scale.The resulting dataset has 36,496 unique training images and 995 validation images; removing shared-image training samples left reported validation metrics unchanged at the stated precision.Source: §A.1 · p. 13
A.2.1 INPUT ENCODING
- For 224 × 224 images, DINOv3 yields 196 patch features of dimension 1024; the text encoder provides 77-token embeddings aligned to that dimension.The CLS and register tokens are discarded, and the text representation retains the second 1024-dimensional half.Source: §A.2.1 · p. 13
A.2.2 FEATURE FUSION
- Projected anchor-object text features cross-attend with image patches, and the result is concatenated with full-prompt embeddings to form the reasoning input.The input latent has shape [B, 273, 512]; no explicit grounding supervision is used.Source: §A.2.2 · pp. 13–14
A.2.3 REASONING BLOCK
- The reasoning module repeatedly injects the input latent into a reasoning state that updates the embedded answer, using shared transformer weights across recursive steps.The default schedule uses four latent updates, three deep recursion cycles, and four deep-supervision repetitions; evaluation executes the full budget.Source: §A.2.3 · p. 14
A.2.4 DECODER
- The decoder combines answer-state and image features with patch coordinates, then predicts an interior point and half-plane offsets defining a convex mask.It uses up to 32 half-planes; normalized margins are converted to pixel units and smoothed across a 10-pixel transition.Source: §A.2.4 · p. 14
A.2.5 OBJECTIVE FUNCTIONS
- Training minimizes binary cross-entropy, relational contrastive loss, and halting loss, with class weighting for foreground imbalance and a 0.25 relation margin.Predictions are pooled to a 28 × 28 grid for spatial alignment assessment.Source: §A.2.5 · p. 15
A.3 CHECKPOINT SELECTION
- The validation-selected checkpoint scores 29.51% context accuracy on RoboSpatial-Home; the 31.97% peak at epoch 600 is 2.46 percentage points higher, equivalent to three correctly answered questions.The benchmark peak illustrates checkpoint sensitivity and is not used to select the final checkpoint.Source: §A.3 · p. 15
A.4 TRAINING SETUP
The recursion-budget experiments find that removing recurrence substantially lowers context accuracy, while modestly increasing the budget yields no gains. Training uses a fixed 1000-epoch setup with periodic evaluation.
A.4 TRAINING SETUP
- Training runs for 1000 epochs on four NVIDIA GH200 GPUs with batch size 1536 and takes approximately five hours.The learning rate warms to 1e−4 over 2000 steps, then remains constant; evaluation occurs every 25 epochs.Source: §A.4 · p. 16
B SENSITIVITY TO RECURSION BUDGET
- 9.02%: setting latent recursion, deep recursion, and deep supervision to one lowers context accuracy from 29.51%; one latent step alone yields 14.75%.Reducing either deep recursion or deep supervision alone to one preserves performance at 30.33% and 28.69%, respectively; reducing both together yields 22.13%. Modestly increasing the budget beyond defaults brings no gains.Source: B SENSITIVITY TO RECURSION BUDGET · p. 16
C INTERPRETING REGION EVALUATION
Figure 5 presents cases where representative points fall outside annotated target masks, cautioning that the examples do not establish whole-region validity.
C INTERPRETING REGION EVALUATION
- Figure 5 shows ground truth in green, predicted masks in red, and representative points as stars, illustrating cases where points fall outside target masks.These examples do not establish that an entire predicted region is vacant or spatially valid.Source: Figure 5 · p. 16, C INTERPRETING REGION EVALUATION · p. 16
C.1 IMPLICATIONS FOR TRAINING
Under-constrained target masks complicate training: standard segmentation losses reward agreement with annotations, while direction-only objectives omit vacancy extent and context.
C.1 IMPLICATIONS FOR TRAINING
- Standard segmentation losses penalize deviations from annotation geometry; overlap-based losses target region agreement, while binary cross-entropy supervises each pixel without explicitly optimizing global overlap.The masks are under-constrained, making annotation agreement challenging during optimization.Source: §C.1 · p. 16
- An objective that penalizes only predicted-region direction and perspective does not supervise vacant-space extent or context.Source: §C.1 · p. 16
- A future objective could supervise direction and vacancy while allowing multiple valid regions; developing one remains future work.Source: §C.1 · p. 16
- Figure 6 shows examples of predicted regions with limited overlap with annotation-derived targets.Ground-truth masks are green and predicted masks are red.Source: Figure 6 · p. 17