Spatial Latent Reasoning for Embodied Reference Understanding

1 INTRODUCTION

• Pointing direction alone does not determine an object’s full extent, while recognizing objects alone may not identify the intended instance.The introduction frames the challenge as connecting gesture geometry with visual evidence of target identity and spatial extent.Source: §1 · p. 1
• Figure 1 contrasts crop-and-re-encode tool reasoning, LVR region-feature reconstruction, and SLR’s combination of a spatial ray state with four parity-pooled visual states.The figure illustrates the reasoning operations and supervision targets on an EgoPoint-Ground example.Source: Figure 1 · p. 2
• SLR orders one hand-geometry-supervised spatial ray state before four states supervised by parity-pooled target features.The spatial labels use fingertip position and finger-base-to-fingertip direction; the visual targets summarize interleaved region positions.Source: §1 · p. 2
• SLR improves EgoPoint-Ground mIoU over same-backbone SFT by 2.8, 17.5, and 21.1 percentage points on Qwen3.5-4B, Qwen2.5-VL-7B, and Qwen3-VL-8B, respectively.The introduction also reports gains on both hard subsets and 77.6% precision at IoU 0.5 on YouRefIt.Source: §1 · p. 2

2.1 PROBLEM FORMULATION

• Given an image and grounding query, the model generates a target bounding box; finger annotations and target-region features provide auxiliary losses, not inputs to generated states.The box is serialized as integer coordinates on a 1000 × 1000 grid.Source: §2.1 · p. 2

2.2 RECURRENT CONTINUOUS LATENT STATES

• The causal language backbone recurrently generates K = 5 continuous states, each retaining dimension d (2560 for Qwen3.5-4B), by feeding each preceding position’s final hidden vector into the next slot before decoding the answer.In order, the states are hsp, h00, h01, h10, and h11. The visual encoder and projector are frozen.Source: §2.1 · p. 3, §2.2 · p. 3

2.3 SPATIAL RAY-STATE LATENT

• A spatial readout supervises hsp with fingertip position and finger-base-to-fingertip direction, while the geometry labels do not use target boxes or candidate objects.Direction is scaled to the post-merge visual grid to account for nonsquare layouts, then normalized to preserve orientation.Source: §2.3 · pp. 3–4

2.4 PARITY-POOLED VISUAL TARGETS

• Parity pooling partitions target-region tokens by global row and column parity and averages each nonempty phase into a visual target.The four interleaved phase means supervise h00, h01, h10, and h11 in fixed order; empty phases are omitted from the auxiliary loss.Source: §2.4 · pp. 4–5

2.5 VISUAL ALIGNMENT AND JOINT TRAINING

• Stopped-gradient cosine alignment trains the recurrent states against their phase-mean targets, alongside answer-token cross-entropy and spatial supervision.Gradients from answer and target alignment propagate through earlier states; the visual encoder and merger/projector remain frozen.Source: §2.5 · p. 5

2.6 INFERENCE

• At inference, the model generates five states and greedily decodes the box; it does not use the spatial readout or perform ray–object intersection.Auxiliary labels are not required at inference.Source: §2.6 · p. 5

3.1 DATASETS AND EVALUATION

• EgoPoint-Ground provides egocentric hand–target annotations and reports mIoU and thresholded localization accuracy; Hard-Similar and Hard-Complex test distinct grounding conditions.The comparison includes zero-shot and SFT prediction, Text CoT and PointVG-R reasoning, and continuous visual reasoning, with methods grouped by backbone.Source: §3.1 · pp. 5–6, §3.2 · p. 6

3.3 IMPLEMENTATION AND CHECKPOINT SELECTION

• For Qwen3.5-4B, the visual encoder and merger are frozen while the language model and spatial readout are trained; checkpoints are selected by validation mIoU.The selected checkpoint is fixed across the parent test set and both hard subsets, avoiding subset-specific checkpoint selection.Source: §3.3 · p. 6

3.4 MAIN RESULTS

• SLR improves mIoU over same-backbone SFT in all nine backbone–evaluation combinations, with standard-set gains of 2.8–21.1 points.On Qwen2.5-VL-7B and Qwen3-VL-8B, it also exceeds the strongest reported reasoning baseline by 4.3 points in standard-set mIoU for each.Source: §3.4 · p. 6, Table 1 · p. 6
• Both hard subsets retain mIoU gains over SFT across all three backbones, but comparisons vary by condition and overlap threshold.For Qwen3.5-4B, the gain is 6.0 points on Hard-Similar versus 1.5 on Hard-Complex; on Qwen2.5-VL-7B Hard-Complex, PointVG-R leads at P@0.3 while SLR leads at P@0.7 and mIoU.Source: §3.4 · p. 7, Table 2 · p. 7
• On YouRefIt, SLR gains 8.4, 8.0, and 8.1 points over same-backbone zero-shot at IoU thresholds 0.25, 0.5, and 0.75.The comparison is against the pretrained model rather than task-specific SFT, and published literature scores retain their original protocols.Source: §3.4 · p. 7, Table 3 · p. 7

3.5 ABLATION STUDIES

• The full model achieves the best mIoU on the standard and Hard-Similar sets, where combining spatial and target-state supervision outperforms either cue alone.On Hard-Similar, Spatial-only scores 0.286 versus 0.361 for SFT, while joint supervision reaches 0.421; on Hard-Complex, Target-only leads the full model, 0.597 to 0.551.Source: §3.5 · pp. 7–8, Table 4 · p. 8
• With four target states and no spatial state, parity pooling exceeds Quadrant, Random, and Max by 3.6, 2.9, and 1.8 mIoU points on the standard set.Four states lead the budget sweep, but one is close (0.764 versus 0.761), and larger budgets bring no improvement.Source: §3.5 · pp. 8–9, Table 5 · p. 9
• Attention patterns show the spatial state attending predominantly to context, while later visual states combine context, image tokens, and previous latent states.Image attention peaks at the first visual state and attention to earlier latent states rises through the sequence; these patterns illustrate roles but do not establish a causal reasoning mechanism.Source: §3.5 · p. 9, Figure 5 · p. 9

• SLR organizes pointing geometry and target appearance into recurrent continuous states.Source: §4 · p. 9
• The framework improves over same-backbone SFT across three backbones and both EgoPoint-Ground hard subsets.Source: §4 · p. 9
• Positive YouRefIt results also support task-structured intermediate supervision for embodied reference understanding.Source: §4 · p. 9

5 SCOPE AND FUTURE WORK

• The current work uses single-image, two-dimensional grounding; depth, temporal supervision, weaker annotations, and scene-adaptive states are future directions.Source: §5 · p. 9
• Egocentric imagery can expose private environments, and predicted regions should not be treated as definitive accounts of a person’s intention.Human evaluation requires documented participant and consent procedures.Source: ETHICS STATEMENT · p. 10
• Reproduction requires split manifests, checkpoints, and run configurations beyond the available templates, checks, and evaluation-record merger.The source package does not include model weights, dataset records, or final per-example results.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 REFERRING EXPRESSIONS AND EMBODIED POINTING

• Pointing adds gesture as a reference cue: the work supervises a two-dimensional ray but predicts the box without prescribing a ray–object intersection rule.Related work includes embodied reference benchmarks and methods for geometric reasoning traces (Li et al., 2026d).Source: §A.1 · p. 14

A.2 GROUNDED MULTIMODAL MODELS AND SPATIAL KNOWLEDGE

• Grounded multimodal models represent spatial features or knowledge, while this work’s fingertip-and-direction supervision is narrower and does not reconstruct a three-dimensional scene.The paper combines that geometry objective with target-region visual supervision because the cues supply different information for grounding.Source: §A.2 · p. 14

A.3 EXPLICIT VISUAL REASONING AND THINKING WITH IMAGES

• Explicit visual reasoning uses tools or grounded traces; the paper instead studies recurrent hidden-state feedback without textual rationales or visual-tool calls.The cited comparison cautions that shorter output alone does not establish an accuracy or latency advantage because recurrent computation costs remain.Source: §A.3 · pp. 14–15

A.4 CONTINUOUS AND LATENT VISUAL REASONING

• Continuous latent methods differ in how states are generated and supervised; SLR assigns one recurrent state to pointing geometry and four to parity-pooled target features.RIS (Cui et al., 2026) is a spatial-semantic precedent, while SLR focuses on task-specific factorization in freely generated recurrent states.Source: §A.4 · p. 15
• Decoding geometry from a recurrent state shows representational content, but establishing that box prediction depends on geometry requires interventions or controlled ablations.Source: §A.4 · p. 15

A.5 REGION POOLING AND POLYPHASE REPRESENTATIONS

• Parity pooling averages target-region tokens separately across four interleaved, full-image-grid phase classes rather than contiguous quadrants.The averaging is lossy and does not imply invertibility, anti-aliasing, or translation invariance.Source: §A.5 · pp. 15–16

A.6 AUTOREGRESSIVE FORMULATION AND LVR SUPERVISION

• Answer generation is conditioned on the multimodal prompt and model-generated states, while LVR trains states to reconstruct teacher-forced visual tokens selected by an annotated ROI.At inference, LVR feeds predicted hidden vectors back between latent boundary markers.Source: §A.6 · p. 16

B.1 REFERENCE IMPLEMENTATION AND TRAINING

• The Qwen3.5-4B reference configuration freezes the visual encoder and merger, trains language-side parameters and the spatial head, and retains generic latent markers.It uses a four-epoch training budget; the source checks establish implementation contracts, not reproduced GPU training or benchmark performance.Source: Table 6 · p. 16, §B.1 · pp. 16–17

B.2 BOUNDING-BOX DECODING PROTOCOLS

• The shared implementation offers tokenizer-aware constrained decoding, but the exported Exp3.12 worker uses free greedy decoding.A matched comparison should hold the checkpoint and latent recurrence fixed and report each decoding protocol and caching settings separately.Source: §B.2 · p. 17

B.3 METRICS, RECORD INTEGRITY, AND TIMING

• All examples with annotated target boxes remain in the denominator, while parse failures, end-of-sequence termination, and output-limit hits are logged separately.This separates answer-format outcomes from localization quality.Source: §B.3 · p. 17
• The merger checks complete shard coverage, shared checkpoint identity, hashes, and unchanged parameters, then aggregates metrics over examples rather than averaging shard means.Per-example token IDs and elapsed times are retained in JSONL; the summary CSV omits mean token count and latency.Source: §B.3 · p. 17
• Recorded elapsed time includes collation, transfer, hashing, instrumented generation, decoding, and scoring, so it is not pure decoding latency.A dedicated latency comparison needs consistent hardware, batch, caching, warm-up, synchronization, timing boundaries, and diagnostic overhead; fewer latents need not imply faster inference.Source: §B.3 · p. 17

B.4 REFERENCE POOLING CONTROLS

• Parity, quadrant, and Max controls use the same ROI tokens and produce four detached targets, isolating spatial partition or aggregation choices.Parity averages global row/column parity groups; quadrant averages four box-midpoint regions, while Max takes elementwise maxima within parity groups.Source: §B.4 · p. 18
• The Random control deterministically partitions ROI tokens into four groups matching parity-group sizes, but the configuration templates do not establish whether the reported run used this implementation.Its partition is repeatable for a fixed key and ROI and does not resample during training.Source: §B.4 · p. 18

B.5 TARGET-LATENT BUDGET ANALYSIS

• Four target states attain the highest reported mIoU (0.764), closely followed by one state (0.761); nine or sixteen states do not improve mIoU.One state has higher P@0.7 (0.794 versus 0.783); changing target grouping alongside state count means the sweep compares complete supervision configurations, not a unique optimum.Source: §B.5 · p. 18, Table 7 · p. 18, Figure 6 · p. 18

C PARITY-POOLING PROPERTIES

• Each selected visual token contributes to exactly one descriptor, and all four parity phases are populated when the support spans at least two rows and columns.Thin or unresolved ROIs can leave phases empty; those supervision terms are omitted without duplicating tokens, while all four states remain.Source: C PARITY-POOLING PROPERTIES · p. 19
• Parity phase means retain the information needed to recover the ordinary ROI mean when their counts are known, while exposing differences between interleaved subsets.Phase-wise averaging remains lossy: it cannot reconstruct the feature grid or guarantee distinct descriptors, and fixed phase order does not imply translation invariance.Source: C PARITY-POOLING PROPERTIES · p. 19

D FULL RESULTS ON THE HARD SUBSETS

• The Qwen2.5-VL-7B and Qwen3-VL-8B Standard SFT rows use run 02.Source: D FULL RESULTS ON THE HARD SUBSETS · p. 19
• Table 8 reports Hard-Similar and Hard-Complex results separately, with SLR shading and best grounding scores marked within each backbone and subset.Dashes indicate unreported values and N/A indicates inapplicable entries.Source: Table 8 · p. 19, D FULL RESULTS ON THE HARD SUBSETS · p. 19
• The reported comparison fields include P@0.3, P@0.5, P@0.7, mIoU, token counts, and elapsed time.The table description notes that Appendix B.3 gives conventions for interpreting token counts and elapsed times.Source: D FULL RESULTS ON THE HARD SUBSETS · pp. 19–20

E SUPPLEMENTARY COMPUTATION RECORDS

• Reported zero counts for direct prediction and fixed counts for latent methods are not total answer lengths, and the records do not establish a common counting convention across method families.Continuous states, boundary markers, and generated text require separate accounting.Source: E SUPPLEMENTARY COMPUTATION RECORDS · p. 21
• Elapsed-time values accompany the comparisons, but their timing scope is defined in Appendix B.3 and does not support an efficiency ranking.The main-text localization results do not rely on such a ranking.Source: E SUPPLEMENTARY COMPUTATION RECORDS · p. 21, Table 9 · p. 21
• Table 9 retains token counts and elapsed times for the main comparisons and ablations without converting their units.Standard denotes EgoPoint-Ground.Source: Table 9 · p. 21, E SUPPLEMENTARY COMPUTATION RECORDS · p. 21