PaperlayerView exact revision on arXiv

Source-linked AI summary

ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Scene Rearrangement

Mary Lynn Martin, Yifei Zhang, Martha Palmer, Maria Leonor Pacheco

arXiv:2608.16011v1cs.CLcs.CV

TL;DR

3D vision-language benchmarks have largely evaluated fixed-scene understanding, leaving continuous, verifiable scene changes underexplored. ReRef-3D benchmarks language-guided placement and finds the task learnable but unsolved: the strongest model achieves valid placements for roughly two thirds of instructions, while all models satisfy relations more often than they produce physically valid scenes.

  • Problem

    Most 3D vision-language benchmarks assess fixed-scene understanding, leaving conversion of spatial instructions into continuous, verifiable scene changes underexplored.

  • Method

    ReRef-3D requires models to interpret relational anchors and constraints, then predict a new 3D location evaluated through post-move relation and physical-validity checks.

  • Results

    The strongest model produces valid placements for roughly two thirds of instructions, while the other two remain below one third; relation satisfaction exceeds physical validity across models.

  • Takeaways & Limitations

    Placement is learnable but unsolved, and wording has minimal effect while physically valid scene generation remains harder than satisfying requested relations.

  • Takeaways & Limitations

    The benchmark uses synthetic scenes from a single camera, which isolates spatial reasoning but may not capture the complexity of real scans.

Abstract

from arXiv · show

We introduce ReRef-3D, a benchmark for language-guided placement in 3D scenes. It contains 33,826 instructions across 998 CLEVR-derived scenes, spanning 16 placement families and direct, one-hop, and two-hop references. Each instruction must be resolved into a valid new placement position. Given that an instruction defines a region of acceptable placements rather than one coordinate, our evaluation inserts a prediction into the scene, recomputes relations, and tests relation satisfaction and physical validity. Each instruction also includes a verified naturalized rewrite. After fine-tuning, LLaVA-3D, 3D-LLM, and PlaceIt3D produce valid placements for 68.3%, 31.6%, and 22.4% of instructions, respectively. Across models, relation satisfaction surpasses physical validity, relations such as nearest and between are the most difficult, and phrasing has minimal effect on performance.

1 Introduction

ReRef-3D introduces scene-referential 3D placement, requiring models to resolve self-contained spatial instructions and predict valid new locations. The benchmark evaluates rearrangement by testing the resulting scene because instructions specify acceptable placement regions rather than single coordinates.

  • Task and novelty: ReRef-3D requires interpreting spatial anchors and destination constraints before predicting a new 3D placement location.Unlike grounding, the task changes the scene rather than identifying content already present.
  • Task and novelty: Each instruction may require a two-hop reference chain to identify an anchor, followed by placing an object adjacent to it on the table without collisions.The example demonstrates coupled reference resolution and physical placement constraints.
  • Evaluation challenge: Valid placements form regions rather than single coordinates, so distance to one annotated point is an incomplete success measure.The benchmark instead evaluates the scene after moving the object.
  • Benchmark design: Procedural generation and verification provide structured scenes, symbolic spatial constraints, and verified goal locations, while known geometry enables relation, clearance, and visibility recomputation.The verified goal location is only one valid solution.
  • Benchmark design: The benchmark combines direct, one-hop, two-hop, and ordinal references with placement families covering directional, between, distance-based, and table-region constraints.This design controls both linguistic and geometric complexity.

2 Background and Related Work

Prior work spans spatial referring and grounding datasets, spatial reasoning benchmarks and 3D vision-language models, and language-guided placement or rearrangement systems. These efforts use synthetic compositional scenes, metric and multi-step reasoning, pose or affordance prediction, and constraint-based planning.

  • Spatial referring expressions and 3D grounding: Spatial grounding benchmarks range from natural-image datasets to CLEVR-Ref+ synthetic scenes with functional programs and compositional expressions.In 3D, ReferIt3D and ScanRefer focus on single referents, while ScanEnts3D and Multi3DRefer address additional or multiple referenced objects.
  • Spatial reasoning and 3D VLMs: Spatial reasoning benchmarks cover qualitative relations, metric properties, and multi-step inference across natural images, controlled images, and compositional settings.VSR, What’sUp, SpartQA, and StepGame isolate different aspects of spatial reasoning, while SpatialVLM, SpatialRGPT, RoboSpatial, and SpaRE use spatial supervision.
  • Language-guided object placement and rearrangement: Language-conditioned manipulation systems map instructions to object poses, actions, or spatial affordances using pick-and-place actions, discretized poses, multimodal prompts, and spatial value maps.Examples include CLIPort, PerAct, VIMA, and VoxPoser.
  • Language-guided object placement and rearrangement: Object rearrangement methods generate poses, predict generative structure, or perform constraint-based planning.StructFormer, StructDiffusion, and LGMCTS represent these three approaches, respectively.
  • Language-guided object placement and rearrangement: Other approaches model language-specified goals as target-space distributions or affordance points, while PlaceIt3D predicts valid placement regions for an object.LINGO-SPACE grounds compositional expressions as distributions, RoboPoint predicts image-space affordance points, and RoboSpatial uses paired 2D and 3D supervision for spatial affordances and manipulation.

3 Dataset Generation

ReRef-3D generates language-guided rearrangement examples from CLEVR-derived scenes by combining controlled target-anchor expressions with varied placement geometries and reference complexities. It verifies that instructions are nontrivial, objects resolve uniquely, and proposed placements satisfy the dataset’s spatial and physical constraints.

  • Scene representation: Object motion is represented on the tabletop ground plane, while the moved object’s vertical coordinate is set from its radius so it remains supported.Small objects use ri = 0.35 and large objects use ri = 0.70, with zt = rt.
  • Dataset construction: ReRef-3D builds on CLEVR-derived scenes, vocabulary, geometry, camera-relative relations, and program execution while adding rearrangement templates, placement constraints, goal generation, and post-move verification.Expressions identify a target and specify its desired post-move configuration.
  • Dataset construction: Targets are generated over scene-object pairs with controlled expression counts and placement-type frequencies to reduce target imbalance and prevent any placement family from dominating.Each object is considered as a potential target during construction.
  • Constraint design: The benchmark combines placement families with direct, one-hop, two-hop, and ordinal anchor descriptions, varying placement geometry and referential complexity independently.The families include directional, aligned, between, nearest, and farthest relations, among others.
  • Validation: Before goal generation, symbolic execution confirms intended object resolution and rejects no-op instructions whose requested condition already holds in the source scene.Accepted examples require Ce(S, te) = 0, ensuring the target must be relocated.
  • Rewrite validation: 91.5% of 400 candidate rewrites received agreement from Gemini and both annotators, while human annotators selected the same best rewrite for 40 of 100 expressions.Disagreements concentrated in distance-based relations, and multiple rewrites could be semantically valid.

4 Evaluated Models

The evaluation compares two general 3D vision-language models with a placement-specific model under a shared language-guided placement task. Despite different scene representations and output formats, all predictions are converted into a single 3D placement before scoring.

  • Model selection: All three models consume a 3D scene, condition on an imperative instruction, and produce an output convertible to a target placement.The comparison separates spatial-understanding limitations from limitations caused by representation or output design.
  • Model design: PlaceIt3D predicts a placement region over scene superpoints, testing whether placement-specific design outperforms general models.The model is purpose-built for placement, unlike the two general 3D vision-language models.
  • Scene representation: 3D-LLM uses a precomputed voxelized point-feature cloud, while LLaVA-3D lifts posed RGB-D observations into 3D and generates the goal as text.LLaVA-3D requires no precomputed feature cloud, testing whether view-based 3D grounding is sufficient for placement.
  • Scoring protocol: Generated text is parsed as a world coordinate, whereas PlaceIt3D’s predicted region is reduced to a representative point before scoring.This conversion makes the models’ different output types comparable as single 3D placements.

5 Evaluation

Section 5 evaluates separately fine-tuned models under matched and cross-type instruction conditions, using placement accuracy and its relation-satisfaction and physical-validity components as primary measures. It also reports distance-to-goal metrics and evaluates relational thresholds at both relaxed and original settings.

  • Evaluation setup: Models are fine-tuned separately on templated and naturalized instructions, then evaluated on both instruction sets in matched and cross-type conditions.The conditions use the same scenes.
  • Metrics: Placement accuracy requires both relation satisfaction and physical validity after inserting the prediction and recomputing the scene graph.The two components are reported separately.
  • Metrics: Mean distance measures average Euclidean error to the annotated goal q⋆, while Acc@1.0 measures predictions within one scene unit of q⋆.The placement workspace spans six units.
  • Relation tolerance: 1.5× relational thresholds are used for the main results, while 1.0× original thresholds provide strict evaluation.Relaxation widens all relational criteria except between and does not affect physical validity.

6 Results and Analysis

LLaVA-3D achieves the strongest localization and placement performance, while relation satisfaction exceeds physical validity across models. Performance is robust to phrasing for LLaVA-3D and PlaceIt3D, but nearest and between relations remain difficult and physical validity limits success.

  • Matched Results: LLaVA-3D leads localization, with mean distance ≈1.1 and Acc@1.0 ≈61%, while relation satisfaction exceeds placement accuracy for all models.3D-LLM and PlaceIt3D have mean distance ≈3.2, although 3D-LLM satisfies relations more often.
  • Naturalization: +4.7 points is PlaceIt3D’s placement-accuracy change with naturalized instructions, compared with +1.6 for LLaVA-3D and −0.7 for 3D-LLM.Naturalization is neutral or beneficial overall across the three models.
  • Robustness to phrasing: Off-diagonal placement-accuracy drops are ≤1.4 points for LLaVA-3D, indicating minimal sensitivity to instruction phrasing.PlaceIt3D leaves every metric unchanged when the entire instruction wording is replaced, but its predictions correlate only weakly with the annotated goal (r ≈0.14).
  • Sources of failure: Fewer than 0.5% of 3D-LLM cases yield parseable coordinates with format guidance, making placement accuracy near zero for all models.Zero-shot results are omitted because the base checkpoints share no placement interface.
  • Relation difficulty: Nearest and between are the most difficult relations, while farthest is easiest because it permits a relatively large valid region.LLaVA-3D achieves 32–62% on nearest and between, whereas 3D-LLM and PlaceIt3D remain below 13%; 3D-LLM’s between performance drops sharply after naturalization.
  • Sources of failure: 25.8 points is LLaVA-3D’s gap between relation satisfaction and placement accuracy, showing that physical validity limits task success.Coordinate supervision does not reveal whether nearby positions are occupied; PlaceIt3D’s unchanged performance after scene-encoder unfreezing implicates its mask objective or conditioning scheme.

7 Conclusion and Future Directions

ReRef-3D is learnable but remains unsolved: the strongest model succeeds on roughly two thirds of instructions, while the other two remain below one third. Future work should disentangle grounding from placement, broaden evaluation, and model feasible placement regions rather than single coordinates.

  • Conclusion: The strongest model produces valid placements for roughly two thirds of instructions, while the other two produce valid placements for below one third.The task is learnable but unsolved.
  • Conclusion: All three models satisfy requested relations more often than they produce physically valid scenes, and wording has minimal effect.This suggests placement is not a direct consequence of grounding ability.
  • Conclusion: All code and data will be released to the community.
  • Future Directions: Separating object identification from placement would isolate reference resolution from placement generation and reveal how grounding errors propagate.
  • Future Directions: Evaluation could expand to more 3D models, depth-lifted image-space predictors, proprietary models with structured outputs, and spatial value-map policies.
  • Future Directions: Releasing feasible regions instead of single verified coordinates would support region-based losses and uncertainty-aware outputs.

Limitations

The benchmark uses synthetic, single-camera scenes and LLM-generated naturalized instructions, limiting realism and rewrite agreement. Evaluation also targets arbitrary feasible placements, while compute constraints prevent robust comparison across tuning settings.

  • Evaluation and comparison: One fine-tuning run per condition, without hyperparameter search, means small differences may reflect seed variance or unequal adaptation budgets.Compute limits constrained the comparison protocol.
  • Evaluation and comparison: Coordinate regression targets an arbitrary point in the feasible set because evaluation accepts any valid placement, making distance only a diagnostic.Each item still provides a verified goal despite this non-unique target.
  • Scene and instruction realism: Synthetic scenes with one camera isolate spatial reasoning but may not capture the complexity of real scans.This limits how directly benchmark performance transfers to real-scene settings.
  • Scene and instruction realism: 40 of 100 examples received annotator agreement on the best rewrite (κ = 0.19), although feedback found several candidates often valid and natural.The limited agreement therefore partly reflects ambiguity among acceptable rewrites.

A Model and Implementation Details … C Human Annotation Details

The appendix details a controlled comparison of three placement models and a two-stage pipeline for generating, verifying, and selecting naturalized spatial instructions. All models share data and goals, while their scene representations, output targets, and adaptation strategies differ.

  • A Model and Implementation Details: All three models train on identical instructions, goals, and splits, differing only in scene representation, supervision target, and adaptation strategy.This establishes a controlled comparison across the candidate systems.
  • A.1 Instruction Format; LLaVA-3D; PlaceIt3D: The models receive instructions verbatim without anchor coordinates or answer-format suffixes; general models emit coordinate text, whereas PlaceIt3D predicts a placement region.LLaVA-3D and the other general model use coordinate outputs, while PlaceIt3D uses a placement mask, rotation bucket, and anchor mask.
  • B Expression Naturalization Details; B.1 Candidate Generation Settings; Generation User Prompt: Qwen3-8B generates four rewrites per call with temperature 0, disabled thinking mode, a 512-token limit, and JSON-only output.The generator is instructed to preserve symbolic meaning while producing concise, fluent, human-like placement instructions.
  • B.1 Candidate Generation Settings: 87 of 95 remaining records were recovered through retries, including 16 at temperature 1.2, while eight records remained unresolved.The penultimate retry recovered 71 records, leaving 24 before the final retry.
  • B.1 Candidate Generation Settings; Generation User Prompt: The rewriting prompt treats symbolic relations as ground truth, preserves objects, attributes, directions, relation roles, and chains, and forbids inferred or additional relations.It also requires exactly one placement action and exact preservation of the input pair_id.
  • B.2 Candidate Verification and Selection Settings; Verification + Selection System Prompt: Candidate selection uses Gemini 3.5 Flash Batch API to retain complete verification results and a compact JSONL file containing each pair_id and selected expression.The verification process receives the original expression, gold relations, and four candidate rewrites.
  • Verification + Selection System Prompt: Semantic verification excludes candidates that alter targets, objects, relations, roles, attributes, chains, or instructional intent, using gold symbolic relations and the original expression as ground truth.Invalid candidates receive semantic error labels and are excluded from naturalness ranking.

C.1 Annotators … D.1 Tolerance

The evaluation used two informed adult English-speaking annotators to assess candidate rewrites for semantic preservation, naturalness, and clarity before selecting among valid candidates. A tolerance ablation found that relaxed relation thresholds improve placement accuracy by about 2 points without changing model rankings, while between remains separately tested.

  • C.1 Annotators: Two adult English-speaking evaluators based in the United States conducted the human evaluation.One evaluator was an author, while the other was an uncompensated external volunteer with a computer science degree who had not helped construct the dataset.
  • C.2 Annotation Instructions: Each candidate rewrite was independently evaluated against the original template expression and gold symbolic relations.Annotators were shown the original expression, gold relations, and four candidate rewrites.
  • Stage 1: Semantic Validity: Semantic validity required preserving the placement target, reference objects, spatial relations, directions, roles, and placement-instruction intent.Candidates also had to preserve indexed-object distinctions, relation chains, identifying attributes, and avoid unsupported information.
  • Stage 1: Semantic Validity: Annotators treated gold symbolic relations as ground truth and did not infer additional relations from common sense or geometry.Invalidity reasons included missing or changed objects, target/anchor role errors, modifier-attachment errors, changed attributes, and changed instruction intent.
  • Stage 1: Semantic Validity: For semantically valid candidates, annotators scored fluency and natural wording, along with interpretability, on 1-to-3 scales.The fluency scale ranged from awkward or unnatural to fluent and natural; interpretability ranged from difficult or ambiguous to immediately clear.
  • Final Selection: Annotators ranked only semantically valid candidates from most natural to least natural.They then considered the overall rewrite selection, including the option that none were semantically valid.
  • Which candidate is the best overall rewrite?: The overall rewrite decision included a “None are semantically valid” choice.This option was listed among the available choices for selecting the best overall rewrite.

D.2 Per-Family Results for Naturalized Expressions … E.1 Representative Predictions

Naturalized expressions preserve the original difficulty ordering and model ranking while shifting performance unevenly across relation families. Cross-language evaluation shows wording independence for PlaceIt3D, whereas qualitative examples expose strict relational failure modes.

  • D.2 Per-Family Results for Naturalized Expressions: The naturalized-instruction difficulty ordering and model ranking remain unchanged from the original evaluation.Figure 4 attributes aggregate shifts to family-specific changes rather than reordered difficulty or models.
  • D.2 Per-Family Results for Naturalized Expressions: 34.0 →41.9 and 22.1 →30.5 mark PlaceIt3D’s largest gains on farthest and directional relations, respectively.The gains are broad rather than concentrated in a single relation family.
  • D.2 Per-Family Results for Naturalized Expressions: 31.9 →35.1 and 57.6 →62.2 show LLaVA-3D’s smaller gains on the two hardest groups, nearest and between.These changes account for the model’s improvement under naturalized expressions.
  • D.3 All Conditions and Metrics: Within 0.2 points in every metric, PlaceIt3D’s cross-language cells match its matched-language counterparts.This indicates that PlaceIt3D’s predictions are effectively independent of instruction wording.
  • D.3 All Conditions and Metrics: 91.2–92.5% relation satisfaction remains nearly constant for LLaVA-3D across four cells, while physical validity varies with training language.The contrast separates stable relational performance from language-sensitive physical validity.
  • E.1 Representative Predictions: Table 9’s six validation examples illustrate observed failure modes but are not representative of average performance.Examples 1–2 show LLaVA-3D near the goal while the other models answer near a table edge.
  • E.1 Representative Predictions: 0.05 units from the goal still fails the strict between test because the recomputed relation depends on both anchors, not proximity.Example 3 demonstrates that closeness to the target alone does not satisfy between.

E.2 Where the Errors Come From … F.1 Artifact Licenses

The error analysis separates relation failures from physical-invalidity failures and shows distinct prediction-distribution behaviors across models. Reproducibility uses shared data, splits, and evaluation, while reporting compute and licensing details.

  • E.2 Where the Errors Come From: 25.8% of LLaVA-3D items satisfy the relation but are physically invalid, concentrating its remaining errors in collision avoidance.This bucket equals the gap between relation satisfaction and placement accuracy.
  • E.2 Where the Errors Come From: Near-misses are rare at ≤2.3%, while 8–19% of items are correct despite lying more than 2 units from the annotated goal.The results support treating the annotated point as one valid solution among many.
  • E.3 Prediction Distributions: LLaVA-3D predictions correlate strongly with goals on both horizontal axes (r ≈0.87) and span 157 half-unit cells, indicating genuine location regression.The distribution statistics distinguish its behavior from the weaker models.
  • E.3 Prediction Distributions: 34.7% of 3D-LLM predictions occupy one unit cell, while its most frequent output string accounts for 11.1%; goal correlations are rx = 0.50 and ry = 0.31.These outputs concentrate at table edges and corners, consistent with coarse-region prediction.
  • F Reproducibility Details: All models use the same rearrangement corpus and scene-level split assignment, with naturalized instructions merged by pair identifier.Thus each phrasing shares the same scene, target, anchors, relation, and goal.
  • F Reproducibility Details: A single shared evaluator scores every model, and strict and relaxed metrics come from the same predictions evaluated at both relation tolerances.This enables comparison across native prediction formats without changing runs.
  • F Reproducibility Details: The six reported fine-tuning runs total about 40 GPU-hours on a single NVIDIA H100, with full-card and 3g.40gb MIG evaluation configurations.Wall-clock device-hours count a MIG partition as one device.
  • F.1 Artifact Licenses: CLEVR data use a Creative Commons Attribution 4.0 license, while the dataset-generation code uses a BSD license permitting the reported adaptations and distribution subject to attribution and notices.The work builds on CLEVR’s dataset and generation framework.
Loading 2608.16011v1…