AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding

Qize Yu; Jiadi You; Yuran Wang; Jiaqi Liang; Bowen Ping; Yang Tian; Yue Chen; Minghong Cai; Zeying Gong; Ruihai Wu; Yinchuan Li; Junwei Liang; Yingcong Chen

Source-linked AI summary

AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding

Qize Yu, Jiadi You, Yuran Wang, Jiaqi Liang, Bowen Ping, Yang Tian, Yue Chen, Minghong Cai, Zeying Gong, Ruihai Wu, Yinchuan Li, Junwei Liang, Yingcong Chen

arXiv:2606.06155v1Jun 4, 2026cs.ROcs.CVcs.MM

TL;DR

VLA models struggle to map vision-language semantics directly to 3D robotic actions, motivating structured intermediate representations. AffordanceVLA addresses this with affordance forecasting across object grounding, 2D interaction localization, and 3D geometric reasoning, achieving strong performance on LIBERO, CALVIN, and real-world experiments.

Problem
VLA models face a structural gap between vision-language semantic representations and the 3D physical representations required for robotic actions, making direct perception-action mapping challenging.
Method
AffordanceVLA uses structured affordance forecasting through Which2Act, Where2Act, and How2Act to integrate object grounding, 2D localization, and 3D geometric reasoning into VLA control.
Results
AffordanceVLA achieves strong, competitive performance on LIBERO, CALVIN, and real-world manipulation experiments, with strong generalization and robust reasoning.
Takeaways & Limitations
Structured affordances provide a task-oriented bridge between semantic understanding and embodied control across diverse manipulation settings.
Takeaways & Limitations
Performance margins narrow on LIBERO-Long, suggesting that extremely long-horizon sequential tasks may benefit from explicit long-term memory.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models leverage the rich world knowledge of pretrained vision-language models (VLMs) to enable instruction-following robotic manipulation. However, the structural mismatch between VLM semantic spaces and embodied control policies often hinders the learning of precise perception--action mappings. To address this challenge, we propose \textbf{AffordanceVLA}, a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation to establish a more precise and robust perception--action mapping. Specifically, we progressively model manipulation priors through three complementary components: 1) \textbf{Which2Act} for object-centric grounding via visual latent prediction to suppress distractions; 2) \textbf{Where2Act} for 2D interaction localization via affordance map estimation; and 3) \textbf{How2Act} for 3D geometric reasoning to guide manipulation policies. These affordance cues provide spatially grounded, semantically conditioned, and action-coupled intermediate representations, thereby naturally bridging vision, language and action. We integrate these modules into a Mixture-of-Transformer (MoT) architecture with specialized experts and train the model using a three-stage training strategy with a progressive data curriculum. To overcome the scarcity of dense affordance labels in robotic datasets, we also develop a robust automated data augmentation pipeline. Extensive experiments on simulation and real-world demonstrate that AffordanceVLA achieves strong performance across diverse manipulation scenarios.

1 Introduction

AffordanceVLA addresses the semantic-to-physical gap in VLA models by using structured affordance forecasting as a task-oriented bridge between perception and action. Its specialized experts, progressive training, and augmented supervision support strong simulation and real-world manipulation performance.

Motivation: VLA models struggle with direct language-and-vision-to-action mapping because semantic representations and robotic actions occupy different spaces.The paper identifies this modality gap as a fundamental challenge requiring an appropriate intermediate representation.
Motivation: Affordances provide intermediate targets that specify which object to manipulate, where to interact, and how to execute actions.They couple spatial grounding, language-conditioned semantics, and execution guidance.
Framework: AffordanceVLA models structured manipulation priors through Which2Act, Where2Act, and How2Act for object grounding, 2D localization, and 3D geometric reasoning.Which2Act predicts target visual latents to suppress distractions, while Where2Act estimates 2D affordance maps for precise interaction localization.
Framework: The framework combines three specialized MoT experts with progressive information fusion, three-stage training, progressive data curriculum, and automated affordance-label augmentation.The augmentation pipeline addresses the lack of dense affordance annotations in large-scale robotic datasets.
Results: AffordanceVLA achieves strong simulation and real-world performance, with competitive success rates, strong generalization, and enhanced robustness.The paper supports these claims with ablation, qualitative, and quantitative analyses.

2 Related Work

Related VLA work combines vision-language backbones with action modules, but direct action regression can lack explicit task-relevant scene reasoning. AffordanceVLA instead uses affordance as a structured, task-focused intermediate representation that captures what, where, and how while integrating VLM knowledge into control.

Vision-Language-Action Models: VLA models pair vision-language backbones with action modules to inherit web-scale visual-linguistic priors for flexible instruction following.They differ mainly in action representation and decoding, including discretized action tokens or continuous diffusion-based approaches.
Intermediate Representations: Directly regressing actions from raw observations can undermine robustness by omitting explicit reasoning about task-relevant scene structure.Prior work addresses this gap with auxiliary intermediate representations, including generative visual foresight and train-only structured supervision.
Affordance as an Intermediate: Existing intermediates can be overly redundant or too coarse, whereas affordance provides a spatially grounded, semantically conditioned, and action-coupled representation focused on manipulation.This distinguishes the approach from video-prediction-as-bridge and latent-action world models.
Affordance-Based Manipulation: Prior affordance methods show cross-task generalization across grasping, articulated-object, deformable-object, and broader scene interaction tasks.However, they commonly use sparse 2D/3D contact points with directions alongside external grasp generators or motion planners, creating brittle open-loop pipelines.
Affordance-Based Manipulation: AffordDP and CoA-VLA advance closed-loop control, but affordance remains largely an external cue that does not fully exploit pretrained VLM world knowledge.AffordanceVLA addresses this limitation by integrating affordance reasoning with the VLM-based control framework.

3 Method

AffordanceVLA bridges vision, language, and action by using structured affordance forecasting within a Mixture-of-Transformer architecture. Its specialized experts progressively ground instructions into object, interaction, and 3D geometric cues before generating physically coherent actions.

Architecture: The Mixture-of-Transformer architecture combines Understanding, Affordance Generation, and Action Experts to connect semantic alignment, affordance prediction, and control execution.A unidirectional Understanding–Affordance–Action attention flow coordinates the experts while preventing action information leakage into affordance prediction.
Action Generation: The Action Expert decodes semantic, affordance, and proprioceptive representations into smooth, temporally coherent, physically plausible action chunks.Intermediate affordance representations reduce the visual-reasoning burden on action generation and support precise physical execution.
Architecture: The Affordance Generation Expert predicts three structured manipulation priors: object-centric grounding, fine-grained interaction localization, and 3D spatial-geometric reasoning.These correspond to Which2Act, Where2Act, and How2Act, jointly providing task-relevant intermediates that unify vision, language, and action.
Affordance Modules: Which2Act reconstructs cropped target visual latents from bounding-box regions, isolating the interacting object while filtering background distractions.It uses a frozen pretrained encoder to extract zq ∈ R^C×H×W and optimizes predicted latent reconstruction with MSE.
Affordance Modules: Where2Act unfolds query tokens into 2D affordance maps that identify interactive regions and provide explicit contact-point guidance for low-level planning.A Transformer decoder uses spatial position embeddings as queries, producing spatial logits aligned with ground-truth masks through pixel-wise Binary Cross-Entropy.
Affordance Modules: How2Act performs 3D reasoning through conditional-diffusion shape generation and MLP-based regression of a 10-DoF layout vector covering rotation, scale, and translation.The two branches jointly represent target geometry and spatial arrangement for manipulation.

4 Experiment

AffordanceVLA achieves strong manipulation performance across standard, OOD, low-data, visually aliased, and long-horizon settings. Ablations show that structured, jointly optimized affordances and affordance-augmented co-training are central to these gains.

Overall performance: 95.8% average success rate is the highest among compared methods across four suites, while removing Stage II still yields 86.2%.The full model’s consistency supports the framework’s effectiveness and robustness; the Stage-II ablation quantifies the contribution of affordance-augmented co-training.
OOD generalization: 4.33 average length and 75.9% completion of five consecutive tasks demonstrate competitive zero-shot OOD performance.The result is attributed to structured affordance prediction filtering superficial scene correlations and emphasizing task-critical entities, interaction regions, and spatial relations.
Ablations: Jointly optimized affordances outperform data-only and frozen-representation controls, which reach LIBERO 92.4% and CALVIN 3.93, and LIBERO 67.1% and CALVIN 2.83, respectively.These controls indicate that data volume alone is insufficient and that affordance representations must adapt jointly with the embodied control policy.
Ablations: Removing Which2Act reduces CALVIN average length from 4.33 to 4.20, while removing Where2Act harms fine-grained manipulation and lowers LIBERO performance.The ablation tests object-centric grounding and precise 2D interaction localization as complementary components of the structured affordance representation.
Data efficiency: At 40% fine-tuning data, AffordanceVLA reaches ∼92% on LIBERO and above 4.0 average chain length on CALVIN, surpassing fully fine-tuned Pi0.The full model initially dips under severe distribution shift but then recovers rapidly, whereas No-Afd and the no-Stage-II model recover more slowly.
Real-world evaluation: 82.9% aggregate complex-task success exceeds Pi0’s 44.8%, including Drawer pick at 86.7% versus 46.7% and Drawer close at 100.0% versus 40.0%.Under identical visual observations with different instructions, the gap demonstrates stronger instruction-conditioned behavior under visual aliasing.

5 Conclusion · Appendix · A The Effectiveness of Affordance Subgoals

AffordanceVLA bridges VLM semantics and embodied 3D control by using affordances as intermediate representations, achieving competitive results across simulation and real-world evaluations. The appendix details supplementary experiments, including validation of Which2Act, Where2Act, and How2Act as effective task-relevant subgoals for action generation.

5 Conclusion: AffordanceVLA uses affordance forecasting to bridge VLM semantic representations and the 3D requirements of embodied control.The framework decomposes affordance forecasting into Which2Act and additional affordance components within a Mixture-of-Transformer architecture and progressive data curriculum.
5 Conclusion: The framework achieves strong, competitive performance on LIBERO, CALVIN, and real-world experiments, demonstrating generalization and robust reasoning.Future work will explore explicit temporal modeling and extensions to bimanual and deformable object manipulation.
Appendix: The supplementary material provides additional details and experimental results supporting the main manuscript.Its outline covers affordance subgoals, model variants, data-centric methodology, dataset and training details, inference latency, and real-world experiments.
A The Effectiveness of Affordance Subgoals: Section A.1 quantitatively validates the subgoal representations, while Section A.2 examines representation decoupling between the backbone and decoder.These subsections are listed as components of the appendix’s affordance-subgoal evaluation.
A The Effectiveness of Affordance Subgoals: Section A.3 qualitatively analyzes affordance grounding to complement the quantitative validation of the learned subgoals.The appendix outline identifies qualitative affordance grounding as the third component of Section A.
Appendix: The appendix also reports design-choice and methodology details, including Which2Act redesigns, model variants, data quality, annotation, datasets, training, latency, and real-world experiments.These topics are organized across Sections B through G of the supplementary material.
A The Effectiveness of Affordance Subgoals: Section A evaluates Which2Act, Where2Act, and How2Act to test whether their predictive heads extract crucial task-relevant information.The evaluation is conducted quantitatively on the Unseen PRISM subset, comprising 1,000 validation examples.

A.1 Quantitative Validation of Subgoal Representations · A.2 Representation Decoupling: Backbone vs. Decoder

AffordanceVLA’s Which2Act, Where2Act, and How2Act tokens capture task-relevant semantic and geometric information, outperform random baselines, and support downstream control despite weaker 3D reconstruction accuracy. A decoupling experiment further shows that affordance representations are assimilated by the backbone itself, not merely produced by an overfit decoder.

A.1 Quantitative Validation of Subgoal Representations: Which2Act and How2Act Shape are evaluated with Token Acc, while Where2Act is evaluated with threshold-free AUC-ROC.Token Acc measures element-wise agreement with ground-truth codebook indices; AUC-ROC is selected for robustness to vast zero regions.
A.1 Quantitative Validation of Subgoal Representations: All three affordance query tokens extract crucial environmental context and substantially outperform random baselines.Which2Act and Where2Act achieve near-perfect performance in the reported evaluations.
A.1 Quantitative Validation of Subgoal Representations: How2Act Shape has comparatively lower absolute accuracy because highly compressed tokens make high-fidelity 3D voxel reconstruction intrinsically difficult.The lower reconstruction accuracy does not imply that the token lacks useful task information.
A.1 Quantitative Validation of Subgoal Representations: The generated 3D representations still capture coarse object structure and anticipated interaction modality, providing effective explicit planning targets for the Action Expert.Removing Which2Act and Where2Act causes severe downstream performance degradation, including on the LIBERO average.
A.2 Representation Decoupling: Backbone vs. Decoder: The decoupling experiment freezes the Stage II backbone trained for 100k steps and evaluates it with decoders trained from 5k to 100k steps.This setup tests whether subgoal losses primarily optimize backbone representations rather than only a high-capacity decoder.
A.2 Representation Decoupling: Backbone vs. Decoder: Where2Act affordance maps dynamically adjust to different language instructions while the visual observation remains identical, demonstrating robust vision-language alignment.The qualitative visualization shows instruction-conditioned affordance prediction.
A.2 Representation Decoupling: Backbone vs. Decoder: The fully trained backbone retains meaningful affordance features with an under-trained decoder, with performance increasing stably and monotonically as decoder training aligns with the backbone.The result rules out catastrophic failure under weak-decoder evaluation and indicates that the backbone intrinsically assimilates affordance representations for action guidance.

A.3 Qualitative Analysis of Affordance Grounding · B Model Variants and Design Choice Details

The qualitative analysis visualizes Where2Act affordance maps and shows that they adapt to language instructions while remaining robust to semantically similar or divergent commands.

A.3 Qualitative Analysis of Affordance Grounding: Where2Act affordance maps are visualized to qualitatively illustrate the model’s grounding capabilities.The visualization appears in Fig. 7.
A.3 Qualitative Analysis of Affordance Grounding: Changing the language instruction for an identical visual observation dynamically alters the generated affordance map.This indicates vision-language alignment beyond reliance on visual salience biases.
A.3 Qualitative Analysis of Affordance Grounding: Semantically similar commands produce highly correlated affordance heatmaps, whereas divergent commands cause clean and precise shifts in affordance focus.The model therefore exhibits robustness across related and different instructions.

B.1 Design Insight … C Data-Centric Methodology

AffordanceVLA treats affordance as a structural bridge between visual perception and robotic execution, while emphasizing representation and data quality over data scaling alone. It refines Which2Act with continuous Flux VAE supervision and supports the architecture with curated data, automated annotation, and task-specific model variants.

B.1 Design Insight: VLA performance depends jointly on learned representation quality and training-data quality, rather than dataset quantity alone.This observation motivates AffordanceVLA’s design focus on representations and data quality.
B.1 Design Insight: Affordance provides a structural bridge linking visual perception with robotic execution.The framework centers its conceptual design on affordance as an intermediate representation.
B.1 Design Insight: Continuous Flux VAE supervision replaces discrete VQ-VAE supervision in Which2Act, eliminating codebook quantization artifacts and capturing finer-grained visual cues.The redesign targets high-fidelity affordance representations for precise manipulation.
C Data-Centric Methodology: Curated InternData-A1 and DROID datasets plus automated annotation provide accurate affordance knowledge and near-perfect affordance labels.This data-quality investment is intended to ground the model’s architectural capacity in reliable supervision.
B.1 Design Insight: The bridge-centric architecture and supporting technical details overcome performance ceilings that data scaling alone cannot overcome.The passage attributes this result to the synergy between architectural design and technical support, with validation in the data-efficiency analysis.
B.2 From VQ-VAE to Flux VAE: Which2Act Redesign: AffordanceVLA replaces VQ-VAE’s constrained discrete codebook with Flux VAE’s continuous latents, freeing Which2Act’s token count from predefined scale constraints.The Flux VAE produces zq ∈ R^B×16×32×32, totaling 16,384 floats from a 256 × 256 crop.
B.3 Model Variants: Three model configurations are evaluated, with optional wrist tokens generally providing marginal or occasionally negative improvement and therefore excluded from main results.Where2Act decodes tokens into heatmaps, while How2Act Shape mean-pools shape tokens to condition 3D diffusion for categorical shape priors.

C.1 Data Quality as the Performance Ceiling … C.2.6 Step 4: Rigorous Quality Verification.

The paper treats high-quality data and automated affordance supervision as prerequisites for reliable VLA representation learning. Its pipeline progresses from dataset curation and grounding-model adaptation through keyframe extraction, instruction decomposition, affordance generation, and stringent verification.

C.1 Data Quality as the Performance Ceiling: Training-data quality sets the performance ceiling, motivating use of photorealistic InternData-A1 simulation data and curated, high-quality DROID demonstrations.The authors report near-perfect annotation accuracy on A1 and DROID, linking clean observations to reliable grounding and downstream representations.
C.2 Affordance Annotation Pipeline: The automated annotation pipeline supplies missing affordance labels for robot datasets while deriving supervision from native annotations and SAM-based segmentation.Stage II and Stage III datasets lack bbox, heatmap, shape-token, and layout-token annotations; Stage I derivation varies by dataset.
C.2.1 Step 0: RexOmni Fine-Tuning.: RexOmni is fine-tuned on PRISM to adapt its general visual grounding and spatial-pointing abilities to robotic grasping geometry and physical requirements.PRISM provides large-scale robotic grasp-point and target-bounding-box annotations across diverse tabletop settings.
C.2.3 Step 2a: Instruction Decomposition via a Text LLM (Claude Opus 4.5).: Claude Opus 4.5 decomposes each long-horizon instruction into temporally ordered atomic sub-instructions aligned one-to-one with keyframes.Each sub-instruction describes a single primitive interaction such as grasp, place, push, or release without adding actions absent from the original instruction.
C.2.4 Step 2b: Per-Keyframe Affordance Annotation via a VLM (Qwen3-VL).: Qwen3-VL generates, for every keyframe, a specific contacted-object category and a spatial where-to affordance query, driving Which2Act and Where2Act supervision.The model receives the original instruction, keyframe sequence, and current image while following RexOmni-style detection and pointing conventions.
C.2.5 Step 3: Visual Grounding and Affordance Generation.: RexOmni converts detection categories into bounding boxes and affordance instructions into points, which are transformed into visual latents, constrained heatmaps, shape tokens, and layout tokens.The How2Act outputs comprise a shape token in R4096×8 and a layout token in R10, parameterized by rotation4, scale3, and translation3.
C.2.6 Step 4: Rigorous Quality Verification.: Quality verification requires 100% of generated points to lie within their bounding boxes and all 100 audit rounds to pass 30 samples each, totaling 3,000 inspected samples.Only batches passing both geometric consistency and the complete human audit are admitted into training.

D Dataset Details

Table 7 provides a summary of all datasets used across the paper’s training stages.

Table 7 summarizes the datasets used in the study.
The dataset summary covers all datasets.
The datasets are organized across training stages.

E Training Details · F Inference Latency

Training uses a staged curriculum on 16 NVIDIA H200 GPUs, progressing from selective expert training to end-to-end optimization with reduced-rate vision-encoder fine-tuning. Inference latency is reported as a breakdown for deployment on an NVIDIA RTX 5090 GPU.

E Training Details: Training is conducted on a cluster of 16× NVIDIA H200 GPUs, each with 141 GB HBM3e.
E Training Details: Table 8 summarizes hyperparameters across all training stages.
E Training Details: In Stage I, only the Affordance Generation expert, Affordance Query, and all decoders are trained.
E Training Details: The Understanding and Action experts remain frozen during Stage I.
E Training Details: In Stages II and III, all experts and decoders are jointly trained end-to-end.
F Inference Latency: Table 9 reports AffordanceVLA’s inference latency breakdown on an NVIDIA RTX 5090 GPU.The deployment device has 32 GB GDDR7 memory.

G Real-World Experiments Details and Extra Experiments

Real-world experiments use a 7-DoF Franka Panda with dual RealSense cameras and evaluate affordance-guided manipulation after sim-to-real training. The policy supports closed-loop recovery from disturbances and real-time control during diverse tasks.

Real-world platform: The robot uses a 7-DoF Franka Emika Panda, PGC-140 parallel jaw gripper, and two Intel RealSense D435 cameras streaming RGB at 15 Hz.One camera provides a fixed third-person view and the other is wrist-mounted; the robot connects via wired Ethernet.
Training and deployment: The model is pre-trained on curated DROID data, fine-tuned on in-house demonstrations, and replans after executing the first 5 steps of each predicted action chunk.Deployment uses single-frame observations to bridge the sim-to-real gap.
Task evaluation: Experiments cover diverse manipulation tasks, including the more complex “Clean all the rubbish” task designed to assess policy generalizability and robustness.The task set is intended to provide comprehensive evaluation beyond simpler manipulation scenarios.
Robustness evaluation: Closed-loop re-planning enables successful recovery from dynamic external disturbances, including a trial involving explicit human intervention.The recovery behavior is demonstrated during sequential execution of the “Clean all the rubbish” task.
Inference efficiency: ∼176 ms total inference time enables real-time control at ∼5.7 Hz on an RTX 5090, excluding communication delays.The robot connection uses wired Ethernet, but reported latency numbers exclude communication delays.