PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundation-Model-Guided Topology-Aware Semantic Risk Fields for Manipulation

Giung Lee, Weihang Guo, Lydia E. Kavraki

arXiv:2609.36640v1cs.RO

TL;DR

Collision-free motion can still expose manipulated objects to context-dependent semantic risks, and existing risk representations may not account for scene topology. This paper combines foundation-model priors with 3D scene geometry in a topology-aware risk field; its evaluation reports reduced semantic exposure under the same geometric constraints.

  • Problem

    Collision-free motion can remain semantically unsafe, motivating risk representations that account for context and scene topology beyond collision geometry alone.

    Source: §I · p. 1

  • Method

    The method combines six-direction pairwise foundation-model risk priors and spatial decay scales with voxelized geometry, using geodesic distance for topology-aware shielding.

    Source: §I · p. 1, §IV · p. 2

  • Results

    The evaluation reports that the field responds to scene topology and reduces semantic exposure in planning under the same geometric constraints.

    Source: 3) Prior Retrieval and Fallback Runtime · p. 8

  • Takeaways & Limitations

    The results support a practical topology-aware semantic cost representation for manipulation planning beyond collision avoidance.

    Source: §VI · p. 8

    The evaluation is limited to static, near-field scenes and depends on perception, calibration, fixed-frame directional priors, and foundation-model prior quality.

    Source: §VI · p. 8

Abstract

from arXiv · show

Robot motion planning in everyday environments must satisfy hard geometric constraints while accounting for context-dependent semantic risk. We present a foundation-model-guided, topology-aware semantic risk field that extends manipulation safety beyond collision avoidance. For each manipulated-object/scene-object pair, a foundation model provides six directional risk weights and a pair-specific spatial decay scale. The method combines these priors with voxelized 3D scene geometry using topology-aware shielding and geodesic spatial decay. A GPU-parallel backend batches object-level distance and risk computations to construct a dense 3D field that serves as a modular cost for downstream motion planning. We evaluate the field's shielding behavior under full and partial barriers and compare its 3D workspace representation with a pixel-wise semantic-prior baseline. Across three household simulation scenarios, trajectories optimized with the proposed field have lower semantic exposure than collision-only trajectories under the same geometric constraints. We also evaluate the computational practicality and reliability of the supporting pipeline. Together, these results support the proposed field as a practical topology-aware semantic cost representation for manipulation planning beyond collision avoidance.

I. INTRODUCTION · II. RELATED WORK · A. Semantic Safety with Foundation Models

The paper argues that manipulation safety must account for context-dependent semantic risk and scene topology, then presents a foundation-model-guided 3D risk field for downstream planning. Related work motivates this field by identifying limitations in existing semantic-safety representations.

I. INTRODUCTION

  • Collision avoidance is necessary but insufficient: a collision-free path carrying water above a laptop could still cause a damaging spill.Classical PRM, CHOMP, and TrajOpt plan collision-free paths, while semantic manipulation risk depends on the objects and their spatial relationships.Source: §I · p. 1
  • An intervening shelf can shield a laptop from potential consequences, making a trajectory above the shelf lower-risk than one passing directly overhead.This motivates a semantic risk representation that is context-dependent and topology-aware, not based on collision geometry alone.Source: §I · p. 1, Figure 1 · p. 1

A. Semantic Safety with Foundation Models

  • Semantic-Metric Bayesian Risk Fields [9] produce query-conditioned pixel-dense risk images but do not directly provide a planner-ready 3D workspace field or encode connectivity and shielding.Locations close in image or Euclidean space may be separated by an obstacle.Source: §I · p. 1, A. Semantic Safety with Foundation… · p. 2

I. INTRODUCTION

  • In household simulation trajectory optimization, the proposed representation reduces semantic exposure relative to collision-only planning and captures topology-aware changes in risk propagation.The paper presents this as a case study of its modular representation.Source: §I · p. 1
  • A foundation model assigns six directional risk scores and a pair-specific spatial scale σ to manipulated-object/scene-object pairs, stored in a reusable database.The six workspace directions are ±x, ±y, and ±z; σ controls spatial decay.Source: §I · p. 1
  • The topology-aware voxel field combines pairwise priors with RGB-D-derived scene geometry and propagates risk by obstacle-aware geodesic distance, attenuating it along longer free-space detours.The representation provides a modular cost for downstream robotic systems.Source: §I · p. 1

A. Semantic Safety with Foundation Models

  • Prior semantic-safety work includes foundation-model reasoning with 3D semantic maps, 2D safety constraints in navigation [7],, and temporal-logic constraints for longer-horizon behavior.Brunke et al. rely on predefined spatial relations, while CORE [7] and Santos et al. ground constraints in 2D representations; semantic safety also considers object attributes and context.Source: A. Semantic Safety with Foundation… · p. 2

B. Topology-Aware Risk Representations · III. PROBLEM FORMULATION

Geometric and geodesic fields capture distance and free-space connectivity but not object-dependent semantic risk. The formulation adds semantic risk as a soft planning cost while retaining geometric feasibility as a hard constraint.

B. Topology-Aware Risk Representations

  • Prior distance-field methods capture geometric proximity or obstacle-aware connectivity, but not object-dependent semantic risk.Euclidean proximity can understate the collision-free detour to an object; geodesic methods address this topology but focus on geometry or dynamics.Source: B. Topology-Aware Risk Representations · p. 2
  • Semantic grounding identifies risky object interactions, while geodesic fields determine how risk spreads through the workspace.Source: B. Topology-Aware Risk Representations · p. 2

III. PROBLEM FORMULATION

  • The problem defines manipulated object m, scene-object set O, bounded workspace Ω, occupied geometry E, and its collision-free subset Ωfree(E).Source: §III · p. 2
  • The field defines semantic risk over geometrically feasible workspace positions.Source: §III · p. 2
  • For each scene object o, Vo(m, p | E) ≥ 0 represents its semantic risk when m is placed at query position p; occupied regions remain subject to separate collision constraints.Source: §III · p. 2
  • A risk-aware planner adds the field’s accumulated cost along a task-dependent workspace reference path, weighted by λ, to a conventional motion objective.The trajectory remains subject to start, goal, joint, and collision constraints, so geometric feasibility is hard while semantic exposure is a soft cost.Source: §III · p. 2

IV. TOPOLOGY-AWARE SEMANTIC RISK FIELD

The semantic risk field combines pair-specific directional priors with 3D geometry and occupancy to construct object-level risk fields, which are summed into a scene field. Its independent object-pair computations can be batched and parallelized on a GPU.

IV. TOPOLOGY-AWARE SEMANTIC RISK FIELD

  • Object-level fields combine six directional weights and a pair-specific decay scale with scene geometry and occupancy, then sum across scene objects into the final field.The manipulated object is task-specified; scene objects may be supplied by perception or directly in controlled evaluations.Source: §IV · p. 2
  • Because object-level distance and risk computations are independent before summation, they can be batched and parallelized on a GPU.Source: §IV · p. 2

A. Directional Semantic Priors

  • For each manipulated-object/scene-object pair, directional semantic priors specify six risk weights and how far risk should extend.The decay scale is pair-specific.Source: §IV · pp. 2–3

B. Directional Field Construction

  • Directional field construction combines semantic risk with topology-aware shielding and Gaussian distance decay.The section contrasts Euclidean and geodesic distance.Source: §IV · p. 3

D. Pairwise Risk Construction and Scene-Level Composition · A. Directional Semantic Priors

The pipeline combines pair-specific semantic priors with observed 3D scene geometry to construct pairwise risk fields and aggregate them into a scene-level field. Each ordered object pair is represented by six directional risk weights and a spatial decay scale.

D. Pairwise Risk Construction and Scene-Level Composition

  • Pairwise semantic priors and observed 3D scene geometry form pairwise risk fields, which are aggregated into a scene-level field.Source: Figure 2

A. Directional Semantic Priors

  • Six directional weights represent semantic risk for an ordered object pair, capturing direction-dependent severity that scalar pair values and predefined spatial relations do not encode.Source: A. Directional Semantic Priors · p. 3
  • Each pair also has a spatial decay scale σ_m,o; for nonzero-risk pairs, larger scales spread risk over broader free space, while smaller scales localize it closer to object o.The Gaussian decay scale is measured in meters.Source: A. Directional Semantic Priors · p. 3
  • A directional weight is a severity score, not a probability; zero means no additional semantic risk in that direction.Source: A. Directional Semantic Priors · p. 3
  • If all six directional weights are zero, the pairwise risk field is zero and σ_m,o is not used.Source: A. Directional Semantic Priors · p. 3
  • Nonzero weights are calibrated as low (0, 0.30), moderate [0.30, 0.70), high [0.70, 0.99), and operational near-no-go [0.99, 1); typical tabletop σ_m,o is 0.03–0.20 m.Source: A. Directional Semantic Priors · p. 3

B. Directional Field Construction · C. Topology-Aware Shielding

The method builds directional priors from voxel-boundary source sets and blends them according to query direction. It then uses environment-conditioned geodesic distance to attenuate risk when obstacles reduce direct accessibility.

B. Directional Field Construction

  • Each object uses adjacent free-space voxels as a shared source set for Euclidean and collision-free geodesic distance fields.The nearest source voxel defines the local offset from the object boundary.Source: B. Directional Field Construction · p. 3
  • The signed components of the local offset determine the contributions of the six workspace directions at each query position.Source: B. Directional Field Construction · p. 3
  • Linear interpolation blends the directional priors while preserving their [0, 1) scale; diagonal queries blend signed-axis weights, and ε = 10^-6 prevents division by zero.A query directly above the object inherits its +z weight.Source: B. Directional Field Construction · p. 3

C. Topology-Aware Shielding

  • Euclidean distance measures straight-line proximity, whereas geodesic distance measures the shortest path through collision-free workspace around the scene geometry.Source: C. Topology-Aware Shielding · p. 3
  • If no collision-free path connects a query to the object’s source set, its geodesic distance is set to infinity.Source: C. Topology-Aware Shielding · p. 3
  • Topology attenuation is approximately 1 for direct access, lies between 0 and 1 when obstacles force a detour, and equals 0 for unreachable queries.Attenuation varies continuously with lost direct accessibility, so finite-detour regions are not set exactly to zero.Source: C. Topology-Aware Shielding · p. 3

D. Pairwise Risk Construction and Scene-Level Composition

Pairwise semantic risk is constructed by topology-attenuating effective severity before applying a finite smooth mapping, with geodesic distance shaping the field. Summing these pairwise fields yields the scene-level semantic cost queried along a workspace path for planning.

D. Pairwise Risk Construction and Scene-Level Composition

  • A finite smooth mapping preserves graded ordinary risk while making near-one weights strongly discouraging rather than infinite exclusion constraints.Its parameters set ordinary-risk scale, near-one amplification, and finite upper scale; within the bounded evaluated workspace, the upper calibration band retains high cost.Source: D. Pairwise Risk Construction and… · p. 4
  • Topology attenuation is applied before nonlinear amplification, while geodesic distance captures both relative detour and absolute free-space decay.The complete pairwise field uses the same mapping for every effective weight.Source: D. Pairwise Risk Construction and… · p. 4
  • The scene-level field sums pairwise semantic risks and supplies the semantic-cost term queried along the workspace path for planning.The topology-attenuation and geodesic-decay effects are ablated separately in Sec. V-B.Source: D. Pairwise Risk Construction and… · p. 4

V. EXPERIMENTS

The experiments assess whether the field represents semantic risk and scene topology, guides lower-exposure trajectories under matched geometric constraints, and supports reliable, efficient manipulation workloads.

V. EXPERIMENTS

  • The study compares the proposed 3D representation with a pixel-wise semantic-prior baseline and evaluates topology-aware shielding under full and partial barriers.Source: §V · p. 4
  • The study compares risk-aware and collision-only trajectories to test whether the field guides planners toward lower semantic exposure under the same geometric constraints.The supplied passage states the comparison’s purpose, not its outcome.Source: §V · p. 4
  • The study evaluates foundation-model prior consistency across model families and the runtime of risk-field construction, perception, and prior retrieval.Source: §V · p. 4
  • Experiments use an Intel i7-12700K CPU and an NVIDIA RTX 4090 GPU with 24 GB of memory, with α = 50, Λ = 106, and κ = 200 unless stated otherwise.α sets ordinary-risk scale, κ controls amplification near the upper end of [0, 1), and Λ specifies the finite limiting magnitude.Source: §V · p. 4

A. Semantic Risk Representation · B. Topology-Aware Shielding Validation

The paper represents semantic risk in world-aligned 3D space and evaluates whether geodesic distance and topology attenuation shield risk behind scene geometry. The full formulation strongly suppresses risk at a matched laptop query, while the field also represents risk around objects and in free space beyond visible surfaces.

A. Semantic Risk Representation

  • Three OOPSIEVERSE scenarios test fragile-object proximity, spill exposure near a laptop, and proximity to a hot stove using human reference fields hidden from the method and planner.The reference fields are respectively a vertically elongated half-ellipsoid, a rectangle above the laptop, and a distance-decaying region around the stove.Source: A. Semantic Risk Representation · p. 4, Figure 4 · p. 4
  • With identical object identities and risk weights, the pixel-wise Semantic-Metric Bayesian Risk Fields prior [9] shows risky visible surfaces but not risk extending through the surrounding 3D workspace.The comparison isolates spatial representation from perception and risk-prior generation.Source: A. Semantic Risk Representation · pp. 4–5, Figure 4 · p. 4
  • The proposed field assigns risk to world-aligned voxels, representing interaction-specific and directional risk in nearby free space, including spill exposure above a laptop and heat around a stovetop.It can be queried at arbitrary 3D positions.Source: A. Semantic Risk Representation · p. 5

B. Topology-Aware Shielding Validation

  • The evaluation fixes source, query, and voxel resolution while removing only shielding geometry, so deuc stays fixed and dgeo changes with the available collision-free paths.The topology-attenuation ratio A = deuc/dgeo equals 1 without an intervening obstacle and falls below 1 when a barrier forces a detour.Source: B. Topology-Aware Shielding Validation · p. 5
  • Full shielding attenuates risk behind a table, wall, and shelf; partial shielding leaves risk attenuated behind barriers but allows it around uncovered portions.This spatial selectivity distinguishes the method from an all-or-nothing visibility test.Source: B. Topology-Aware Shielding Validation · p. 5, Figure 5 · p. 5
  • With barriers removed, A = 1 in all three scenes; with shielding geometry present, dgeo increases in every case, with the shelf producing the strongest relative detour.A = deuc/dgeo, so smaller values indicate stronger topology-aware shielding; the straight-line distance is fixed at deuc = 0.230 m.Source: B. Topology-Aware Shielding Validation · p. 5, Table II · p. 5

C. Risk-Aware Planning

The section compares collision-only and risk-aware CHOMP-style manipulation planning under shared geometric constraints, evaluating trajectories on human reference risk fields. Risk-aware trajectories detour around semantic hazards that remain collision-free.

C. Risk-Aware Planning

  • Both planners use the same starts, goals, initial trajectories, occupancy information, and collision constraints; the risk-aware objective additionally penalizes semantic-field exposure.The semantic field uses a 1 cm voxel grid, and all risk-aware experiments use a fixed risk weight λ = 1000.Source: C. Risk-Aware Planning · p. 6
  • Human reference-field peak risks are 0.85 for fragile-cluster, 0.999 for laptop, and 0.80 for hot-stove scenarios.Trajectories are evaluated using mean and maximum reference-field risk along sampled trajectory points.Source: C. Risk-Aware Planning · p. 6
  • Risk-aware planning detours around the laptop’s spill-exposure region and avoids other semantic hazards, while collision-only planning passes directly above the laptop.These hazardous regions need not be physical obstacles; the laptop detour still incurs exposure because the reference field extends broadly around it.Source: Table IV · p. 6, C. Risk-Aware Planning · p. 6

D. Foundation-Model Prior Consistency … 2) Perception Frontend Evaluation:

The section reports cross-model consistency of semantic priors and evaluates GPU risk-field construction and a modular perception frontend, including latency–mask-quality tradeoffs.

D. Foundation-Model Prior Consistency

  • All three models produced schema-valid JSON for each of 135 ordered object pairs, with six finite directional weights in [0, 1) and a valid spatial scale σm,o.The models were Claude Sonnet 4.6, GPT-5.4-mini, and Qwen3-235B-A22B, each queried once with the same finite-only prompt.Source: D. Foundation-Model Prior Consistency · p. 6
  • Across 30 reference pairs, Claude Sonnet 4.6, GPT-5.4-mini, and Qwen3-235B-A22B matched 27, 28, and 29 pairs, and matched 162, 165, and 171 of 180 directional decisions.Most differences involved very low-risk assignments; all three models misclassified one reversed ordered pair.Source: D. Foundation-Model Prior Consistency · p. 6

1) Risk-Field Construction Runtime:

  • Runtime tests varied 1–100 YCB scene objects and 8 mm–4 cm voxels in a fixed 1 m × 1 m × 1 m MuJoCo tabletop workspace, repeating each setting 100 times after warm-up.The runtime measurement spans voxel-grid construction through dense scene-level field generation, excluding perception, visualization, and file-saving overhead.Source: 1) Risk-Field Construction Runtime · p. 7
  • At 2 cm resolution, the GPU backend builds a 20-object scene in 56.9 ms (~18 Hz), with runtime scaling approximately linearly in object count and rising at finer resolutions.Table V reports the runtime measurements; the prose identifies the 20-object, 2 cm result.Source: Table V · p. 7, 1) Risk-Field Construction Runtime · p. 7
  • The GPU backend batches obstacle-aware geodesic distances and per-object risk computations on-device, then superposes fields so overlapping object risks increase the scene-level risk.Only the final dense field is transferred to the host.Source: 1) Risk-Field Construction Runtime · p. 7

2) Perception Frontend Evaluation:

  • The perception frontend generates instance masks from RGB observations, labels masked regions with SigLIP 2, and benchmarks SAM 2.1, MobileSAMv2, and FastSAM-s.Each object-count setting used one fixed 640 × 480 observation and 100 recorded runs after warm-up.Source: 2) Perception Frontend Evaluation · p. 7
  • Across six scenes containing 31 objects, SAM 2.1, MobileSAMv2, and FastSAM-s produced valid masks for 29, 30, and 31 objects, with spurious masks in 0, 1, and 6 scenes.Faster mask generation corresponded with spurious detections in more scenes; perception quality itself was outside the work’s scope.Source: 2) Perception Frontend Evaluation · p. 7

3) Prior Retrieval and Fallback Runtime: · VI. CONCLUSION AND LIMITATIONS

The paper presents a foundation-model-guided, topology-aware semantic risk field that combines directional semantic priors with voxelized 3D geometry. Its evaluation reports reduced semantic exposure under matched geometric constraints and practical runtime, while identifying limits of the current static, near-field evaluation.

3) Prior Retrieval and Fallback Runtime:

  • Across 10 trials, the cold LLM fallback took a median of 2.11 s (interquartile range 2.01–2.20 s), compared with less than 0.02 ms to retrieve all eight precomputed priors.The experiment withheld the prior for (hot soldering iron, power drill) and retained the other seven, using Claude Sonnet 4.6 through the Anthropic API.Source: 3) Prior Retrieval and Fallback Runtime · p. 7
  • After caching the generated prior, retrieving all eight priors took less than 0.01 ms, so the initial remote LLM query dominated latency and cached retrieval added negligible overhead to subsequent field construction.The cold fallback condition and the repeated-retrieval-after-caching condition were compared in the eight-object environment.Source: 3) Prior Retrieval and Fallback Runtime · pp. 7–8
  • At 2 cm resolution, the GPU backend constructs a 20-object field in 56.9 ms (approximately 18 Hz), while planning results show reduced semantic exposure under the same geometric constraints.The representation and shielding results also show semantic structure and topology-aware shielding; prior-consistency results show broad agreement across model families and the human-rated prior reference.Source: 3) Prior Retrieval and Fallback Runtime · p. 8

VI. CONCLUSION AND LIMITATIONS

  • The proposed field combines directional semantic priors with voxelized 3D geometry to reduce semantic exposure while capturing topology-aware shielding.The authors frame semantic and context-dependent risk as a complement to collision avoidance.Source: §VI · p. 8
  • The current evaluation is limited to static, near-field scenes and depends on perception, calibration, fixed-frame directional priors, and foundation-model prior quality.Source: §VI · p. 8
  • Future work will explore object-relative priors, dynamic environments, and mobile manipulation and navigation.Source: §VI · p. 8

Related papers

Loading 2609.36640v1…