PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Should I Join? Robot Group Joining via Language-Guided Goal Prediction

Zilin Fang, Zishuo Wang, Gim Hee Lee, David Hsu

arXiv:2609.28467v1cs.ROcs.AI

TL;DR

Robot group joining must infer a goal from a language-described interaction rather than navigate to a specified target. The paper ranks structured group hypotheses and predicts multimodal joining poses; it reports competitive grounding and stronger goal-prediction results than evaluated baselines, with real-robot demonstrations.

  • Problem

    Social navigation typically assumes a specified goal, while group detection does not select a task-specific target; language-grounded group joining requires identifying the described group and inferring where to join.

    Source: §I · p. 1

  • Method

    The method ranks structured candidate group subsets against language, then predicts a multimodal joining-pose distribution from the grounded group and human-formation priors.

    Source: §I · p. 1

  • Results

    The pipeline outperforms evaluated heuristic and VLM baselines in goal prediction, achieving 89.76% pose validity at approximately 1.1 seconds in offline evaluation.

    Source: §I · p. 1, B. Experiment Results · p. 6

  • Takeaways & Limitations

    Real-robot experiments demonstrate group joining in static and dynamic interactions, including collision-free navigation and efficient goal updates.

    Source: §I · p. 2

    The current formulation treats the environment mainly as a feasibility constraint rather than a direct source of joining affordances.

    Source: §VII · p. 8

Abstract

from arXiv · show

Social navigation typically assumes a specified goal and focuses on reaching it while respecting social conventions, whereas robot group joining requires predicting where to join based on the group's real-time activity and formation. This is a highly semantic task, yet an important capability for applications such as robotic guide dogs and autonomous mobility scooters. We formulate language-grounded robot group joining: given an observation and a natural-language description of a target group, the robot identifies the relevant group members and predicts socially compliant joining poses. For grounding, we generate structured candidate subsets through recursive spectral partitioning and rank them with a language-conditioned image--geometry model. Given the grounded group, a goal predictor leverages human-formation priors to produce a multimodal energy--orientation map over feasible robot poses. Experiments on conversations, queues, and audiences across varying group sizes, crowd densities, and visual ambiguities show that our method achieves competitive grounding accuracy with sub-second inference and outperforms all baselines in joining-pose prediction. Real-robot experiments further demonstrate group joining in both static and dynamically changing interactions.

I. INTRODUCTION

Robot group joining requires inferring a target group and socially compliant, feasible joining pose from language and a cluttered scene. The paper combines language-guided group grounding with goal prediction and evaluates the pipeline in offline and real-robot settings.

  • Unlike typical social navigation, group joining requires inferring the goal from a group’s real-time activity, membership, location, and orientation.The instruction identifies the target interaction, but its appropriate joining pose must be predicted from the scene.Source: §I · p. 1
  • The two-stage method ranks structured candidate groups against language, then predicts a multimodal joining-pose distribution using human-formation priors.The predictor represents sequential, mutual, and directed shared attention topologies.Source: §I · p. 1
  • The pipeline outperforms evaluated heuristic and VLM baselines in goal prediction, while real-robot experiments demonstrate joining in static and dynamic groups.The system joins with collision-free navigation and efficient goal updates; the evaluation spans conversations, queues, and audiences.Source: §I · pp. 1–2

II. RELATED WORK

Prior work addresses goal-directed social navigation, group detection, formation-based joining, or language-guided navigation, but not the combined task of grounding a described group and predicting its joining pose.

A. Social Navigation and Group Detection

  • Social navigation generally assumes a specified goal, while group detection identifies who belongs together without selecting a task-specific navigation target.SANG incorporates group structure into rewards but still navigates to a predefined position; queue-joining work also assumes a predefined coordinate.Source: A. Social Navigation and Group Detection · p. 2

B. F-formation Detection and Group Joining

  • F-formation and group-joining methods mainly address conversations or assume the interaction is already identified, leaving diverse functional groups and formation-preserving joining poses unaddressed.The paper instead trains across attention topologies to predict joining poses for varied group types.Source: B. F-formation Detection and Group… · p. 2

C. Language grounding for navigation

  • Language-guided navigation grounds instructions to landmarks, objects, terrain, or individuals, but does not ground socially defined groups and predict valid joining poses.The paper bridges language grounding to in-group individuals with prediction of geometrically and socially valid poses.Source: C. Language grounding for navigation · p. 2

III. TASK FORMULATION

The task is to infer a distribution of socially compliant, geometrically valid robot poses from one RGB-D observation and an instruction specifying a human group. It separates identifying that group from predicting poses conditioned on its members.

  • The desired output is a multimodal distribution over robot position and heading, rather than a single waypoint, because socially acceptable joining behavior can have multiple valid poses.The input is a single RGB-D scene observation paired with a natural-language instruction.Source: §III · p. 2
  • Group grounding identifies a task-relevant subset of detected people; exhaustive recovery is unnecessary when a coherent subset preserves the interaction context needed for joining.The surrounding environment can provide semantic context, but remains in the original observation without explicit structural extraction.Source: §III · p. 2
  • Goal prediction conditions the pose distribution on grounded members and treats environmental elements as obstacles that exclude infeasible regions, not as sources of prescribed target positions.For people conversing at a table, adjacent seating, across-table seating, or standing nearby may all be socially valid.Source: Figure 3 · p. 2, §III · p. 3

IV. GROUP GROUNDING

Group grounding generates candidate subsets of detected people and ranks them against the language instruction. Recursive spectral partitioning keeps the hypothesis set structured and compact while representing different group sizes and boundaries.

IV. GROUP GROUNDING

  • Candidate group hypotheses are scored against the instruction, and the highest-scoring subset is selected as the target group.The method assumes each group has a predominant identifiable scene-level activity despite possible local sub-interactions.Source: §IV · p. 3

A. Hypothesis Generation

  • Exhaustive subset enumeration grows rapidly with scene size, so the method generates a compact hypothesis set using graph partitioning based on spatial proximity.The exhaustive set can contain 2^N − 1 − N subsets, excluding singletons and the empty set.Source: A. Hypothesis Generation · p. 3
  • Sweeping six evenly spaced thresholds between the 10th and 90th percentiles of the Fiedler-vector distribution creates multiple candidate partitions per split.The threshold diversity addresses ambiguous group boundaries and is motivated as improving robustness to depth and camera-calibration noise.Source: A. Hypothesis Generation · p. 3
  • Top-down recursive partitioning continues until singleton subsets are reached, collecting subgraphs from every level to cover varied group sizes and boundary granularities.Source: A. Hypothesis Generation · p. 3

B. Hypothesis Ranking

The ranking model scores candidate groups by combining visual semantics with their spatial relationships to the instruction. It is trained on ranked group hypotheses and selects the highest-scoring subset at inference.

B. Hypothesis Ranking

  • Candidate subsets are scored for compatibility with the instruction using masked images and a fused image–geometry representation.Person–person and person–language attention supplies geometric features that modulate the image embeddings before comparison with the text embedding.Source: B. Hypothesis Ranking · p. 3
  • Training pairs use text and image negatives, including mismatched descriptions and groups altered by removing, adding, swapping, or randomly sampling members.Pairs are retained when their group-membership F1 scores differ by more than 0.05.Source: B. Hypothesis Ranking · p. 4
  • The filtered training data contains 10,624 training pairs and 1,000 validation pairs, and inference selects the highest-scoring candidate subset.About 8,000 overrepresented Café samples and posed close-ups are removed before training.Source: B. Hypothesis Ranking · pp. 3–4

V. GOAL PREDICTION

The goal predictor uses grounded-group and human-formation information to produce a distribution over feasible joining poses. Its output encodes position energy and robot heading, trained using simulated attention topologies.

V. GOAL PREDICTION

  • The predictor uses human F-formation patterns to learn a distribution over joining poses, with lower positional energy indicating higher likelihood.Its output contains a spatial energy map and cosine and sine maps for robot heading.Source: §V · p. 4
  • Top-down person features encode positions, orientations, group membership, nearby bystanders, and global scale, with language conditioning applied through a FiLM layer.The person features feed a Pose Decoder, while an occupancy map constrains predictions to free space; minimum energy selects position and the corresponding orientation values determine heading.Source: §V · p. 4
  • Training simulations abstract 29 F-formation transitions into sequential, mutual-attention, and directed-shared-attention topologies.These topologies represent following a shared direction, orienting around an o-space, or attending jointly to an external focal entity.Source: §V · p. 4
  • The simulated dataset contains 10,000 training and 2,000 validation samples, with perturbed people, bystanders, obstacles, and topology-specific language instructions.The simulated formations vary member placement, spacing, and orientation according to their attention topology.Source: §V · p. 4

A. Experimental Setup

The experiments assess group grounding, goal prediction with known groups, and the full pipeline on offline and real-robot scenarios. They compare against vision-language, geometric, regression, and robot-navigation baselines using interaction-specific validity and navigation metrics.

A. Experimental Setup

  • The evaluation separates group grounding, goal prediction given the ground-truth group, and end-to-end prediction using grounded groups.Offline scenarios cover queues, conversations, and audiences; the 42-scene benchmark includes 15 conversations, 15 queues, and 12 audiences.Source: A. Experimental Setup · pp. 4–5
  • Zero-shot and few-shot VLM baselines predict group members and joining poses from the same image, depth, and instruction inputs.Image-space predictions are projected into the shared coordinate frame, with infeasible body-position predictions snapped to free ground positions when possible.Source: A. Experimental Setup · p. 5
  • Oracle-group goal prediction is compared with handcrafted F-formation rules and regression models trained on the same simulated data.The regression models are separated by F-formation size; these baselines produce one prediction per scenario.Source: A. Experimental Setup · p. 5
  • Real-robot comparisons include NaVILA, OmniVLA, and a sampling-and-selection baseline, with five trials per method and scenario.The real-robot scenes include seated and standing conversations and static and dynamic queues.Source: A. Experimental Setup · pp. 5–6, Figure 6 · p. 5
  • Offline pose validity requires both an interaction-specific position and heading, while real-robot evaluation reports success, collisions, navigation time, and personal-space violations.Offline position and orientation validity use specified distance and angular thresholds; personal-space violations measure time within 0.25 m of a person.Source: A. Experimental Setup · p. 6

B. Experiment Results

The full pipeline achieves high pose validity with roughly one-second inference, while grounding remains fast and goal prediction performs well even with oracle groups. Real-robot deployment also handles changing interactions, though sampling-and-selection is slower and less reliable in dynamic queues.

B. Experiment Results

  • Agreement-weighted grounding F1 is 0.8125 in 0.936 seconds, while the strongest few-shot VLM is slightly more accurate but substantially slower.The evaluation averages scenes containing 4 to more than 20 people.Source: B. Experiment Results · p. 6
  • With oracle groups, the method achieves high pose validity across interaction types; handcrafted rules are sensitive to noisy poses and irregular formations, while regression struggles with multimodality.These results indicate that goal generation remains non-trivial even when group membership is known.Source: B. Experiment Results · p. 6
  • 89.76% pose validity at approximately 1.1 seconds per query; few-shot VLM pose validity improves but remains below the proposed method.The end-to-end evaluation conditions goal prediction on groups produced by the grounding stage.Source: B. Experiment Results · p. 6
  • Across VLM baselines, orientation validity is consistently higher than position validity, which requires metric 3D reasoning about feasible ground-plane locations.The paper attributes the difference to visible body-pose cues for orientation and frequent VLM difficulty mapping image-space predictions to valid ground-plane coordinates.Source: B. Experiment Results · p. 6
  • The deployed system runs full inference in under 0.7 seconds in tested medium-density scenes and updates goals when group membership changes persistently.The reported robot experiments include navigation with appearing bystanders and newly joining group members.Source: B. Experiment Results · p. 6
  • The sampling-and-selection baseline is slower and less reliable in dynamic queues, with chance-dependent successes that can cause collisions.Its initial goal sometimes fell behind a newly joined person only because it was more than 1.5 m behind the queue end.Source: Figure 8 · p. 7

C. Ablation and Pilot Human Study

The ablation exposes an accuracy–latency trade-off in hypothesis generation, while the pilot human study finds that the predicted joining distribution best matches participants’ judgments.

C. Ablation and Pilot Human Study

  • Increasing threshold-sweeping intervals improves grounding accuracy but increases inference time, with performance saturating near six intervals.Six intervals provide the accuracy–efficiency trade-off used in the main experiments.Source: C. Ablation and Pilot Human Study · p. 7
  • For each RGB image and instruction, 15 participants clicked their most suitable group-joining position; human distributions were compared with model predictions using KL divergence over 500 image bins.Source: C. Ablation and Pilot Human Study · p. 7
  • Our method obtains the lowest KL divergence among evaluated models when compared with human joining-position judgments.The study compared model predictions with clicks from 15 participants.Source: C. Ablation and Pilot Human Study · p. 7

VII. CONCLUSION AND LIMITATION

The approach grounds language-specified groups and predicts valid joining poses across varied settings, with real-world experiments demonstrating effective group joining. Failure analysis identifies grounding errors and overly conservative joining distances, and the formulation treats the environment mainly as a feasibility constraint.

VII. CONCLUSION AND LIMITATION

  • Across varied group sizes, crowd densities, and visual ambiguities, the method grounds groups near-real-time and predicts valid poses across group types and spatial arrangements.Real-world experiments further demonstrate effective group joining.Source: §VII · p. 7
  • The pipeline ranks structured candidate groups, then predicts a multimodal joining-pose distribution using formation-derived attention priors.Source: §VII · p. 7
  • Failure analysis identifies grounding errors, including correct groups missing from distance-based candidates, and overly conservative joining distances, mainly in audience scenes.Source: §VII · pp. 7–8
  • The current formulation treats the environment mainly as a feasibility constraint rather than a direct source of joining affordances.Source: §VII · p. 8
Loading 2609.28467v1…