PaperlayerView exact revision on arXiv

Source-linked AI summary

USA-Net: Unified Semantic and Affordance Representations for Robot Memory

Benjamin Bolte, Austin Wang, Jimmy Yang, Mustafa Mukadam, Mrinal Kalakrishnan, Chris Paxton

arXiv:2304.12164v2cs.ROcs.AI

TL;DR

Robots need both semantic and spatial-affordance information to follow open-ended instructions, while discrete planners can struggle with soft goals and fine scene details. USA-Net combines these representations in a differentiable map and uses a gradient-based planner; the authors report higher semantic similarity and shorter trajectories than prior discrete planners.

  • Problem

    Semantic goal navigation must balance reaching a soft distribution of possible goal locations with avoiding obstacles, while coarse grids may miss fine scene details.

    Source: §I · pp. 1–2

  • Method

    USA-Net combines differentiable semantic and affordance representations in one implicit map and uses a gradient-based planner to optimize semantic goals and trajectory constraints jointly.

    Source: §I · pp. 1–2

  • Results

    Up to 91.2% higher semantic similarity and 39.2% shorter trajectories than prior discrete planners are reported for USA-Net’s gradient-based planner.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors conclude that USA-Net supports autonomous-robot motion planning from implicit scene memory that captures both affordances and semantic information.

    Source: §V · p. 7

    Noisy depth estimates can make the global SDF underestimate true distances, producing predicted SDFs that are too small and an overly cautious planner.

    Source: A. Affordance Representation · p. 3

Abstract

from arXiv · show

In order for robots to follow open-ended instructions like "go open the brown cabinet over the sink", they require an understanding of both the scene geometry and the semantics of their environment. Robotic systems often handle these through separate pipelines, sometimes using very different representation spaces, which can be suboptimal when the two objectives conflict. In this work, we present USA-Net, a simple method for constructing a world representation that encodes both the semantics and spatial affordances of a scene in a differentiable map. This allows us to build a gradient-based planner which can navigate to locations in the scene specified using open-ended vocabulary. We use this planner to consistently generate trajectories which are both shorter 5-10% shorter and 10-30% closer to our goal query in CLIP embedding space than paths from comparable grid-based planners which don't leverage gradient information. To our knowledge, this is the first end-to-end differentiable planner optimizes for both semantics and affordance in a single implicit map. Code and visuals are available at our website: https://usa.bolte.cc/

I. INTRODUCTION

Robots need semantic and spatial-affordance information to follow open-ended goals, but choosing a collision-free route to a soft semantic goal remains challenging. USA-Net combines both kinds of information in a differentiable map and uses a gradient-based planner to optimize semantic goals and trajectory constraints jointly.

I. INTRODUCTION

  • The paper frames the central problem as building a world representation for end-to-end differentiable planning toward semantic goals in cluttered environments.It asks how RGB and depth data can support both semantic goal navigation and motion planning.Source: §I · pp. 1–2
  • Learned semantic goals form a distribution of potential locations, so a semantically ideal target may be blocked or unreachable, while coarse grids can miss fine scene details.The paper illustrates the detail issue with navigating around obstacles to reach a soda can.Source: §I · p. 1
  • USA-Net combines differentiable affordance and semantic representations, jointly supervising empty-space semantics using batchwise nearest-neighbor CLIP embeddings alongside ground-truth SDF values.Its gradient-based planner optimizes semantic goals and trajectory constraints at once.Source: §I · p. 2
  • Up to 91.2% higher semantic similarity and 39.2% shorter trajectories than prior discrete planners are reported for USA-Net’s gradient-based planner.The two gains correspond, respectively, to semantic similarity and trajectory length.Source: §I · p. 2

A. Vision-language navigation

Prior vision-language navigation work builds semantic maps, while motion-planning methods often handle scene semantics and path choice separately. USA-Net instead learns a unified semantic-and-avoidance representation from RGB-D observations and camera information.

A. Vision-language navigation

  • USA-Net jointly learns a representation encoding semantic and avoidance information, unlike prior maps that can require expensive computation or complex downstream systems to translate semantic targets into motion plans.Earlier approaches include language-grounded semantic maps and multimodal maps; some use language models to produce sub-instructions.Source: A. Vision-language navigation · p. 2

B. Motion planning

  • Occupancy-map and sampling-based planners address path search, but are often separate from semantic maps and lack an obvious interface to natural-language goals.This separation can be suboptimal when path choice trades off against the semantic goal.Source: B. Motion planning · p. 2

III. METHODOLOGY

  • The method takes RGB images, per-pixel depths, camera poses, and intrinsics, then samples 3D points and trains their feature vectors to encode semantics and affordances.In the real-world system, Intel RealSense depth and Hector SLAM poses were sufficient despite being noisier than offline methods.Source: §III · pp. 2–3

A. Affordance Representation

USA-Net supervises its affordance representation to predict signed distance values, using batchwise nearest-neighbor sampling to estimate the ground-truth SDF. Noisy robot depth can bias those estimates downward, so the method models and corrects that bias for planning.

A. Affordance Representation

  • Batchwise nearest-neighbor sampling estimates ground-truth SDF values and was simpler to construct than querying the global nearest neighbor in a full-scene point cloud.The batchwise estimates closely matched globally computed SDFs.Source: A. Affordance Representation · p. 3
  • Noisy RealSense depth caused global SDF estimates to significantly underestimate true distances, degrading affordance representations relative to an occupancy-grid baseline.The authors report that accurate simulated ReplicaCAD depth estimates worked well out of the box.Source: A. Affordance Representation · p. 3
  • Batchwise sampling improved estimated SDFs, but robot-trained models still underestimated true SDF and made the planner overly cautious around obstacles.The paper models this bias to correct it in the final planner.Source: A. Affordance Representation · p. 3
  • The real-world SDF received a constant 10 to 20 centimeter adjustment to account for modeled bias.The simulation shows that sampling more points systematically increases final SDF bias for points centered 1 meter from the reference point, with σC = 1 cm.Source: Figure 3 · p. 3, Estimating the SDF Bias · p. 4
  • The occupancy-map baseline trades smaller grid cells for greater mapper and planner computation; its 10-centimeter cell size was chosen for the robot and may vary by agent.The baseline projects point-cloud points into grid cells and thresholds moving-average occupancy values.Source: Baseline Affordance Representations · p. 4

B. Semantic Representation

USA-Net aims for a continuous semantic representation that supports joint optimization of target location and path. It supervises semantics in empty space using the nearest surface point's CLIP embedding, with SDF-weighted contrastive training.

B. Semantic Representation

  • A continuous semantic representation lets the downstream planner jointly optimize the target location and path.The paper identifies determining the semantics of empty space as a challenge for continuous scene representations.Source: B. Semantic Representation · p. 4
  • The model supervises semantic meaning in empty space using the CLIP embedding of the nearest surface point, which has a more defined semantic meaning.Per-sample loss is weighted by SDF, penalizing points less when they are farther from that surface.Source: B. Semantic Representation · p. 4
  • Semantic supervision uses the original CLIP contrastive loss to preserve meaningful dot products between learned embeddings and arbitrary linguistic-query embeddings.The combined training objective joins the semantic loss with the affordance loss.Source: B. Semantic Representation · p. 4
  • The semantic and affordance loss terms are weighted equally because the authors found the choice of loss weight was not particularly important.The weighting constants are λr and λs.Source: B. Semantic Representation · p. 4

C. Grid Motion Planner

The grid planner turns the shared representation into an occupancy grid, then searches that grid for a path. For semantic queries, it selects the grid cell most similar to the query as the target.

C. Grid Motion Planner

  • The baseline instead builds an occupancy map from the raw point cloud and uses breadth-first search with line-of-sight waypoint pruning.The representation-based grid planner can be regenerated at different cell sizes without retaining the full point cloud; its occupancy grids were also smoother, which the authors attribute to network regularization.Source: Occupancy Grid from World Representation · pp. 4–5
  • The grid planner discretizes affordance values into an occupancy grid and uses grid search to find a path.The described search may use A* or RTT.Source: Occupancy Grid from World Representation · pp. 4–5
  • For a semantic query, the planner chooses the grid cell with the highest similarity to the query vector as its target.A cell is considered occupied if any corner's SDF is below the robot radius plus a buffer.Source: Occupancy Grid from World Representation · pp. 4–5

D. Gradient Motion Planner

The gradient planner optimizes waypoints against obstacle avoidance, spacing, semantic goal alignment, and path length. It uses gradient descent on the differentiable representation, with the grid planner's path as its experimental initialization.

D. Gradient Motion Planner

  • The planner optimizes waypoints by gradient descent over differentiable losses for obstacle avoidance, waypoint spacing, query alignment, and path length.The paper says these objectives were chosen because they produced reasonable paths in practice.Source: • Make the path shorter · p. 5, D. Gradient Motion Planner · p. 5
  • Obstacle avoidance uses the representation's SDF and minimum obstacle distance, with the SDF loss clamped for points already sufficiently far from objects.The planner's objective weights are λo = 1, λn = 1, λs = 15, and λd = 25; the authors report that Adam at learning rate 1e-2 works reasonably well.Source: • Make the path shorter · p. 5, Conflicting Loss Objectives · p. 6
  • A waypoint-spacing loss encourages roughly even spacing, reducing the risk that adjacent waypoints lie on opposite sides of an obstacle and form an invalid path.This loss component does not require querying the network.Source: • Make the path shorter · p. 5
  • The semantic loss moves the terminal waypoint toward a higher CLIP score for the target query, while the distance loss is the total path length.The planner samples an initial target point with maximum query similarity, then iteratively updates the path until convergence.Source: • Make the path shorter · pp. 5–6
  • The gradient planner uses the grid planner's output as its initial path because semantic-map gradients are only locally useful and distant initial goals can prevent convergence.The authors note that an unreachable semantic optimum can make the semantic and distance objectives pull in opposite directions.Source: Choosing the Initial Path · p. 6, Conflicting Loss Objectives · p. 6

IV. EXPERIMENTS

The experiments evaluate the motion planner in six scenes drawn from physical-robot recordings, iPhone captures, and simulation.

IV. EXPERIMENTS

  • The evaluation covers six scenes: three recorded with a Stretch robot, two recorded with the Record3D iPhone application, and one from ReplicaCAD simulation.The Stretch scenes are a lab, a nearby kitchen, and a pair of chess boards.Source: §IV · p. 6

A. Navigating to a Target Point

The target-point experiment compares planners using normalized path lengths over random start-and-endpoint pairs. The supplied passages describe the metric and comparison setup but provide no table values.

A. Navigating to a Target Point

B. Navigating to a Semantic Location

The gradient planner uses semantic embedding gradients to improve the final location on a fixed map. The section also reports it was monotonically better than other planners for test queries in a scene.

B. Navigating to a Semantic Location

  • The gradient planner significantly improves the final location’s CLIP embedding relative to other planners on the same fixed map.The comparison uses CLIP correspondence between the linguistic query and the final coordinate; the authors note this is somewhat self-referential because the score is derived from the model.Source: B. Navigating to a Semantic Location · p. 6, Table II · p. 7
  • Empirical comparisons found that CLIP scores correspond well to the linguistic targets.The authors compared the final locations against those targets.Source: B. Navigating to a Semantic Location · p. 6
  • Higher average relative CLIP scores indicate final positions closer in CLIP embedding space to the target query.Table II reports scores for final locations over semantic queries and scenes.Source: Table II · p. 7

PLANNER WAS MONOTONICALLY BETTER THAN ALL OTHER PLANNERS

  • The subsection reports that the planner was monotonically better than all other planners for each test query in a given scene.The associated website provides planned-path visualizations.Source: PLANNER WAS MONOTONICALLY BETTER THAN… · p. 7

V. CONCLUSION

USA-Net supports end-to-end differentiable planning with an implicit scene memory containing affordance and semantic information. The authors identify richer planning objectives and constructing the representation during exploration as future directions.

V. CONCLUSION

  • USA-Net’s implicit scene memory captures affordances and semantics for end-to-end differentiable motion planning.The gradient planner is reported to better optimize arrival at semantic goals and generate shorter paths.Source: §V · p. 7
  • More complex environments may require additional planner objectives, such as optimizing for smoother paths.The authors describe this as a potential future direction for improving performance and efficiency.Source: §V · p. 7
  • The study plans on representations that are already fully constructed; building USA-Net during exploration is future work.The authors suggest this might enable language-driven exploration.Source: §V · p. 7

Related papers

Loading 2304.12164v2…