PaperlayerView exact revision on arXiv

Source-linked AI summary

Affordance-Conditioned Decision Making: Bridging the Semantic-Spatial Gap in Zero-Shot Cross-Floor Vision-and-Language Navigation

Xuekang Yang, Lu Chen, Shuang Luo, Jialing Zhu, Qi Zhang, Yue Gao, Xiang Zhang

arXiv:2609.32292v2cs.ROcs.AI

TL;DR

Correct semantic intent does not ensure reliable execution at constrained cross-floor transitions. PACE grounds transition semantics in traversable affordance poses, conditions short-horizon actions on them, and refines behavior with rollout-derived preferences. Across six zero-shot VLN navigators, it improves average SR on both cross-floor benchmarks and is deployed in unseen real-world environments.

  • Problem

    Correct semantic intent does not ensure reliable execution at constrained transitions, and neither waypoint-based nor end-to-end approaches jointly provide task-conditioned executable grounding and robust closed-loop execution beyond expert states.

    Source: §1 · pp. 1–2

  • Method

    PACE is a supervised local execution module that grounds transition semantics in long-horizon traversable affordance poses, conditions short-horizon actions on them, and uses rollout-derived preference pairs to favor normal or recovery behavior.

    Source: §1 · p. 2

  • Results

    PACE improves average SR by 11.30 and 7.30 percentage points on R2R-CE-CF and RxR-CE-CF, respectively, across six open-source zero-shot VLN navigators, with successful deployment in unseen real-world environments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments across six zero-shot VLN navigators and real-world deployment demonstrate PACE's effectiveness and applicability for cross-floor VLN.

    Source: §5 · p. 8

    PACE exhibits local control instability during stair traversal, including left-right oscillations and reduced railing clearance.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Vision-and-language navigation increasingly relies on general-purpose semantic planners, yet translating correct high-level intent into reliable physical execution remains difficult in spatially constrained transitions. Reaching a staircase, doorway, or narrow passage does not ensure traversal; the agent must identify an executable affordance pose and recover from accumulated action errors. We propose PACE (Preference-refined Affordance-Conditioned Execution), a supervised local execution module that augments frozen zero-shot semantic planners for reliable cross-floor navigation. PACE grounds transition-related semantics into a long-horizon, agent-centric traversable affordance pose and conditions short-horizon action generation on this spatial target, thereby aligning semantic goals with physical execution. We further post-train PACE through failure-aware preference refinement using rollout-derived pairs that contrast normal or recovery behaviors with deviation-amplifying behaviors, thereby improving closed-loop correction. We integrate PACE into six open-source zero-shot VLN navigators and demonstrate consistent improvements on the cross-floor subsets of R2R-CE and RxR-CE, increasing the average success rate from 16.35% to 27.65% and from 4.76% to 12.06%, respectively. Real-world experiments further demonstrate PACE's applicability in unseen environments, highlighting the potential of traversable affordances to bridge semantic intent and reliable embodied behavior.

1 Introduction

Cross-floor navigation exposes a gap between understanding an instruction and executing constrained transitions: locating a staircase does not ensure traversal. PACE addresses this by grounding transition semantics in executable affordance poses, conditioning local actions on them, and refining behavior with rollout-derived preferences.

1 Introduction

  • Correct semantic intent and geometric localization do not guarantee reliable traversal at constrained transitions.Stair traversal remains a major failure source, and reaching a staircase does not ensure successful traversal.Source: §1 · p. 1
  • PACE grounds stair-transition semantics in long-horizon traversable affordance poses that condition short-horizon action generation, augmenting frozen zero-shot semantic planners.The framework is presented as a plug-and-play supervised local execution module.Source: §1 · p. 2, Figure 1 · p. 2
  • Rollout-derived preference pairs favor normal or recovery behaviors over deviation-amplifying alternatives to improve closed-loop correction beyond expert demonstrations.This refinement targets recovery from policy-induced deviations that expert-state-dominated supervision leaves difficult.Source: §1 · p. 2
  • Across six open-source zero-shot VLN navigators, PACE improves average SR by 11.30 and 7.30 percentage points on R2R-CE-CF and RxR-CE-CF, respectively, and is successfully deployed in unseen real-world environments.The gains are reported for the cross-floor subsets of the two benchmarks.Source: §1 · p. 2

2 Related Work

Prior semantic-spatial grounding often localizes environment entities without modeling whether an agent can execute a transition from a particular pose. PACE instead predicts agent-centric traversable affordance poses and conditions local actions on them.

2.1 Semantic-Spatial Grounding

  • Environment-centric semantic representations localize entities, while affordance-aware approaches may depend on predefined skills or external intermediate representations; PACE predicts agent-centric traversable poses from RGB-D and semantic guidance without explicit 3D scene construction.This addresses the distinction between locating a structure and representing its executability for the agent.Source: §2.1 · pp. 2–3

2.2 Semantic-Aware Navigation

  • End-to-end and zero-shot methods map observations to decisions, while hierarchical subgoals often encode geometric accessibility rather than instruction-conditioned affordances; PACE conditions local actions on an inferred affordance pose.The contrast is between accessible locations and executable spatial decisions grounded in task semantics.Source: §2.2 · p. 3

3 Methods

PACE augments frozen semantic planners with local execution that grounds stair-transition guidance in an agent-centric affordance pose and conditions short-horizon actions on it. It combines supervised imitation with rollout-derived preference refinement and receding-horizon inference.

3.2 Dual-Horizon Affordance-Conditioned Execution

  • PACE couples long-horizon affordance estimation with short-horizon action generation, grounding semantic guidance in an executable spatial target.The target is an agent-centric traversable affordance pose, with entry pose during Approach and exit pose during Entry or Traverse; Exit is terminal.Source: §3.2 · pp. 3–4
  • The multimodal encoder fuses RGB, depth, action history, and semantic guidance; an affordance query estimates pose and phase, and a causal decoder generates conditioned actions.Candidate action sequences are reranked for geometric consistency with the predicted pose.Source: §3.2 · p. 4

3.3 Failure-Aware Preference Refinement

  • Rollout-derived preference pairs favor normal or recovery snippets over deviation-amplifying behavior to address imitation learning's limited exposure to policy-induced deviations.Deviation and failure are identified using positional or heading-error thresholds; recovery trajectories are planned back to the expert route.Source: §3.3 · pp. 4–5

3.4 Training, Inference, and Plug-and-Play Integration

  • Training begins with supervised initialization, then iteratively refines PACE using labeled on-policy states and rollout-derived preference pairs; inference executes one action before replanning.The objective mixes expert and on-policy dual-horizon losses with a preference loss.Source: §3.4 · p. 5, §3.3 · p. 5
  • PACE is evaluated as a plug-in local controller: it uses unified RGB-D, semantic-guidance, and action interfaces without accessing navigator internals or requiring navigator retraining.It returns control after Exit or a safety/budget limit, and may reactivate if traversal is incomplete.Source: §3.4 · p. 5

4 Experiments

PACE is evaluated as a local execution module across two cross-floor benchmarks, six zero-shot navigators, standalone prediction and traversal, and an unseen real-world stairwell. Results cover benchmark performance, rollout prediction, real-world behavior, and ablations of affordance conditioning and preference refinement.

4 Experiments

  • The experiments assess affordance and action prediction, baseline improvements, real-world cross-floor navigation, and ablations of affordance conditioning and preference refinement.The shared high-level VLM is locally deployed; real-world trials use a Unitree Go2 Edu with onboard RGB-D and lidar sensors.Source: §4 · p. 5, §4.1 · p. 6

4.1 Experimental Setup

  • Evaluation uses cross-floor subsets of R2R-CE and RxR-CE, trains PACE on their corresponding training splits, and compares six open-source zero-shot navigators under a unified protocol.The subsets retain episodes with start-to-goal vertical displacement greater than 1 m; all baselines are evaluated on the full val-unseen splits.Source: §4.1 · pp. 5–6

4.2 Main Results

  • On R2R-CE-CF, PACE improves all six methods across every metric, with average gains of 11.30 points in SR, 19.03 in OSR, 7.01 in SPL, and 0.082 in NDTW; on RxR-CE-CF, SR and OSR improve for all methods.HSGM+PACE achieves the highest SR, OSR, and SPL on both benchmarks.Source: Table 1 · p. 6, §4.2 · p. 7
  • PACE’s rollout predictions have 0.329 m mean positional error, 9.9° yaw MAE, 96.94% phase accuracy, and action prefix accuracy declining from 74.98% to 72.67% over ten actions.Figure 3 encodes positional-error magnitude and direction by radius and angle, and heading error by arrow deviation.Source: §4.2 · pp. 6–7, Figure 3 · p. 6
  • Across actual-takeover episodes, SR gains range from 9.68 to 44.12 points across navigator–benchmark pairs, with paired-bootstrap 95% confidence intervals above zero.Across full aligned results, 40/48 bootstrap comparisons show robust improvements; the takeover analysis is associative, not causal.Source: Table 2 · p. 6, §4.2 · p. 7
  • In an unseen two-floor stairwell with about 7 m elevation gain, PACE negotiates stair corners and can recover after an early STOP when the VLM reactivates it.Local oscillations and reduced railing clearance remain, with repeated corrections likely related to viewpoint discrepancies and contact perturbations.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 Ablation Studies

  • Compared with waypoint conditioning, PACE gains 4.3 points in OSR, reduces CR by 9.5 points, and improves NDTW by 14.2 points; removing affordance conditioning reduces OSR by 13.5 points.Removing preference loss reduces OSR by 10.9 points, while removing the full failure-aware refinement stage causes a 30.1-point drop.Source: §4.3 · pp. 7–8, Table 3 · p. 7

5 Conclusion

PACE connects frozen zero-shot semantic planners with local execution for cross-floor VLN. The conclusion summarizes its affordance-conditioned decisions and preference refinement, and identifies other constrained transitions as future scope.

5 Conclusion

  • PACE grounds stair-transition semantics in traversable affordance poses and conditions short-horizon action generation on the learned representation.The framework is plug-and-play supervised local execution for cross-floor VLN.Source: §5 · p. 8
  • Failure-aware preference refinement is used to improve closed-loop correction.The conclusion describes refinement alongside affordance-conditioned decision making as part of PACE.Source: §5 · p. 8
  • Future work will extend PACE to other spatially constrained transitions, including doorways and narrow passages.Source: §5 · p. 8

A Failure-Aware Preference Refinement Details

The refinement procedure labels policy rollouts by deviation and recovery status, then builds preference pairs from short windows. Expert routes and recovery actions provide training-time signals unavailable during evaluation.

A Failure-Aware Preference Refinement Details

  • Rollouts begin from recorded expert states and randomized approach states; positional or heading errors above 0.5 m or 30° for two consecutive steps mark deviation.Failure is triggered immediately above 1.5 m or 60°; A* plans from failure toward a future expert-route anchor.Source: A Failure-Aware Preference Refinement… · p. 11
  • Recovery actions follow an A*-planned route toward an expert-route anchor, and recovery ends after both errors remain within 0.25 m and 30° for two consecutive steps.The expert route and A*-derived actions are privileged training signals and are unavailable during evaluation.Source: A Failure-Aware Preference Refinement… · p. 11
  • Each rollout’s contiguous same-role runs are split into sliding five-step windows, pooled within the refinement round, and rebuilt rather than accumulated across rounds.Each deviation window is paired with a normal window and, when available, a recovery window; positive pools wrap around when exhausted.Source: A Failure-Aware Preference Refinement… · p. 11
  • Recorded proposals, including terminal STOP, are rescored under teacher forcing using predicted affordance features; token averaging avoids systematic preference for shorter proposals.Temporal averaging assesses whether consecutive decisions sustain correction rather than relying on an isolated action.Source: A Failure-Aware Preference Refinement… · p. 11

B Implementation and Annotation Details

PACE combines gated RGB-D features and temporal context to estimate traversable affordances and actions, with deterministic stair-phase labels and masked pose supervision at terminal Exit states.

Architecture and optimization

  • The RGB-D streams are projected into a shared feature space and gated before temporal encoding, affordance estimation, and causal action decoding.The Transformer uses two temporal-encoder layers, one affordance-decoder layer, and two action-decoder layers; the model has 27.4M trainable parameters.Source: B Implementation and Annotation Details · pp. 11–12

Cross-floor splits and scene isolation

  • Cross-floor training and validation splits retain 925/261 R2R-CE-CF and 932/217 RxR-CE-CF episodes, with no scene overlap between splits.The retained episodes span 61/11 R2R scenes and 59/11 RxR scenes.Source: B Implementation and Annotation Details · p. 12

Pose and phase annotation

  • Stair runs are labeled from height changes: preceding poses are Approach, the pre-stair pose is Entry, the interval is Traverse, and the selected plateau pose is Exit.Runs with less than 0.5 m total height change are discarded; plateau confirmation uses F = 3 for R2R and F = 9 for RxR.Source: B Implementation and Annotation Details · p. 12
  • Exit states receive no affordance-pose label, so the pose-regression term is masked there while phase and action losses remain active.Equivalently, λp is zero when the target phase is Exit.Source: B Implementation and Annotation Details · p. 12

C Integration Details

PACE takes over from frozen navigators when staircase detection and VLM guidance satisfy an activation condition, then executes and replans locally before handing control back. Early handoffs caused by an Exit prediction can be recoverable through reactivation if the staircase remains visible.

Unified baseline adapter

  • A detected staircase activates PACE when the VLM predicts Up or Down, or when its relevance confidence reaches the activation threshold, even if direction is Unknown.Otherwise, the navigator retains control; its maps, memory, hidden states, and planner remain inaccessible to PACE and are not retrained.Source: C Integration Details · p. 12

Handoff and reactivation

  • PACE executes only the first action of each short-horizon proposal, then replans from the next RGB-D observation; proposal STOP does not end the episode or return control.Control returns when the phase head predicts Exit or a safety or budget limit is reached.Source: C Integration Details · pp. 12–13
  • An Exit prediction can return control before the stair exit is reached, but the navigator may reactivate PACE if the staircase remains visible and the VLM again passes the activation gate.Simulation takeovers are limited to 200 primitives and are interrupted after five consecutive forward collisions or six attempts with less than 0.05 m displacement.Source: C Integration Details · p. 13

D Expanded Evaluation Results

Expanded comparisons show mostly robust gains across metrics and navigator–benchmark pairs, alongside one robust NDTW decrease. Takeover-conditioned success-rate gains are positive across all reported pairs, while independently sampled VLM outputs limit attribution of variance.

Expanded benchmark comparisons

  • PACE yields robust gains in 9/12 SR, 12/12 OSR, 8/12 SPL, and 11/12 NDTW comparisons; the only robust decrease is NDTW for 3-step-Nav on RxR-CE-CF (∆= −0.051, 95% CI [−0.081, −0.022]).Seven of the remaining eight comparisons are inconclusive.Source: D Expanded Evaluation Results · p. 13

Evaluation caveat

  • Alone and +PACE runs use independently sampled VLM outputs, so repeated decoding seeds are needed to isolate VLM-induced variance.The reported analysis retains realized end-to-end variation rather than matched VLM decisions.Source: D Expanded Evaluation Results · p. 13

Takeover-conditioned analysis

  • Across the 12 navigator–benchmark pairs in the takeover analysis, every paired-bootstrap 95% confidence interval for takeover-conditioned SR differences was strictly above zero, with gains ranging from 9.68 to 44.12 percentage points.Source: Table 5 · p. 13, D Expanded Evaluation Results · p. 13

E Real-World Robot Deployment Details

Real-world deployment places robot sensing and control onboard while running PACE and the locally hosted VLM offboard. Stairwell trials expose both successful recovery from premature handoff and remaining sensitivity to viewpoint shifts and contact perturbations.

Robot control

  • The offboard controller requests fresh RGB-D observations and sends one discrete action at a time; forward motion uses a 0.25 m target with 10 Hz MPC, while turns are closed-loop 30° rotations.Each primitive has a 2 s execution budget; the reported control cycle is 2.2–2.5 s including inference.Source: E Real-World Robot Deployment Details · p. 13

Hardware and computation

  • The Go2 Edu carries a RealSense D435i and Livox Mid-360, while its Jetson Orin NX runs sensing, motion control, and recording; PACE inference runs offboard on an RTX 3070 Ti laptop.The laptop accesses the locally hosted Qwen VLM over HTTPS within a private network.Source: E Real-World Robot Deployment Details · p. 13, Table 7 · p. 14

Real-world observations

  • In an unseen two-floor stairwell, a premature handoff was recovered through navigator reassessment and PACE reactivation, while observed failures included oscillation and reduced railing clearance.The observations also indicate remaining sim-to-real sensitivity to viewpoint shifts and contact perturbations.Source: E Real-World Robot Deployment Details · p. 14

F Failure Case Analysis

PACE’s aggregate gains are concentrated in stair-transition failures, while qualitative cases show both successful transitions across navigators and remaining failures in local execution and upstream navigation.

Additional real-world examples

  • Additional real-world examples visualize executed trajectories; their point-cloud maps are not used for planning or as PACE inputs.Figure 5 shows two instruction-following trajectories.Source: Figure 5 · p. 14

Failure-rate decomposition

  • The stair-transition category marks failures coinciding with PACE takeover, not proof that stairs caused them; priority ordering makes the categories mutually exclusive.Each bar’s segments are normalized by all 478 aligned episodes and sum to that baseline’s overall failure rate.Source: F Failure Case Analysis · p. 14
  • 88.9% to 79.4% overall failure rate and 34.2% to 23.8% stair-transition failure rate across six baselines with PACE.The stair-transition reduction is 10.4 points, exceeding the 9.5-point overall reduction.Source: F Failure Case Analysis · pp. 14–15, Figure 6 · p. 15

Qualitative success cases

  • PACE supports successful cross-floor routes across all six navigators, including takeover after staircase entry or near the stair entrance.Figure 7 compares 3-step-Nav, Open-Nav, and STRIDER in one scene and includes cases from CA-Nav, VLN-Zero, and HSGM.Source: F Failure Case Analysis · p. 15, Figure 7 · p. 16

Failure cases

  • Remaining failures include harmful takeovers, ineffective collision recovery, poor upstream direction selection, and failures where PACE never activates.The underlying navigator’s unreasonable global direction in Open-Nav and STRIDER cases is beyond what local execution can correct.Source: F Failure Case Analysis · p. 15, Figure 8 · p. 16

Related papers

Loading 2609.32292v2…