UniAfford: Token-Routed Multitask Learning for Generalizable 2D-3D Affordance Perception

• The paper argues that 2D image-space and 3D geometric affordance grounding are complementary manifestations of shared affordance understanding.A generalizable system should transfer object–affordance semantics across visual and geometric spaces and predict in image space, 3D space, or both.Source: §1 · p. 1
• Separate task definitions, supervision formats, datasets, architectures, and evaluation protocols leave cross-modal affordance learning fragmented.Input fusion alone does not unify pixel-level and point-level learning.Source: §1 · p. 2
• Token Router for Tasks assigns contextual MLLM states to task branches without predefined marker generation, letting branch-specific dense objectives shape shared representations.Text states receive language-head supervision, while routed states are supervised by downstream decoders.Source: §1 · pp. 2–3
• UniAfford-Data organizes pixel-level 2D annotations, point-level 3D annotations, and language instructions under a shared object–affordance taxonomy with semantic-level pairing.UniAfford uses an MLLM semantic hub and modality-aware routing to condition SAM-style 2D and SONATA-based 3D decoders.Source: §1 · pp. 2–3
• UniAfford achieves strong OOD zero-shot generalization on 2D and 3D benchmarks without target-specific fine-tuning and state-of-the-art branch-wise performance under modality-isolated protocols.Ablations validate joint 2D–3D supervision, token routing, and decoder coupling; diagnostics find object–affordance semantics in routed states.Source: §1 · p. 3

2.1 2D AND 3D AFFORDANCE GROUNDING

• Image-space methods use pixel-level segmentation, while 3D methods ground actionable regions on point clouds; UniAfford jointly uses both supervision under a common taxonomy.Prior work uses separate datasets, annotation formats, and evaluation protocols.Source: §2.1 · p. 3

2.2 LANGUAGE-GUIDED AFFORDANCE GROUNDING

• Language-guided methods connect text to image masks or point-wise predictions, but primarily target one output space; UniAfford uses shared MLLM states for both branches.Examples include LISA (Lai et al., 2024), AffordanceVLM (Wu et al., 2025a), DAG, and SeqAfford (Yu et al., 2025).Source: §2.2 · p. 3

2.3 CROSS-MODAL 2D–3D LEARNING AND ROUTING

• Cross-modal fusion methods emphasize 3D grounding, whereas UniAfford routes contextual hidden states directly to branches, separating task dispatch from predefined marker generation.Text states receive language-modeling supervision, while routed states receive branch-specific dense supervision.Source: §2.3 · pp. 3–4

3.1 TASK DEFINITION

• Given an instruction specifying an object and affordance plus an image, point cloud, or both, the task predicts the target affordance in the corresponding spatial output.Unavailable input modalities and their outputs are omitted.Source: §3.1 · p. 4

3.2 UNIAFFORD-DATA

• UniAfford-Data combines pixel-level masks from RAGNet (Wu et al., 2025a) and ReasonAff with point-wise annotations from PIADv2 (Yang et al., 2024) and AGPIL (Zhu et al., 2025a).The dataset organizes both annotation types under a shared object–affordance taxonomy.Source: §3.2 · p. 4, Table 1 · p. 5
• Semantic pseudo-pairs match image and point-cloud instances by shared object–affordance labels, not instance-level spatial correspondence.Image-only, point-cloud-only, and paired samples remain available, with each branch supervised when its observation and annotation are present.Source: §3.2 · p. 4

4.1 OVERVIEW

• UniAfford routes MLLM response states to text, 2D affordance, or 3D affordance branches, whose states condition pixel- and point-level decoders.The SAM-style image and SONATA-based point decoders receive their respective routed affordance states.Source: §4.1 · p. 5

4.2 UNIFIED MULTIMODAL ENCODING

• Text, image, and point-cloud pathways form a dynamically concatenated prefix that conditions the shared MLLM to generate autoregressive responses.The available modalities are combined in the prefix; the Token Router then filters response hidden states for spatial localization.Source: §4.2 · p. 5

4.3 MODALITY-AWARE TOKEN ROUTER

• The modality-aware router predicts text, image, or point-cloud assignments for response states, using soft probabilities for supervision and hard assignments to select branch states.A dense branch needs its input and annotation during training, but only its input at inference; branch-specific projections map selected states into image- and point-cloud-affordance query spaces.Source: §4.3 · p. 6

4.4 AFFORDANCE DECODERS

• The 2D decoder compares routed image queries with dense SAM image features to form a coarse heatmap, which becomes a mask prompt for spatial refinement.The 3D decoder uses scaled similarity between routed point-cloud queries and SONATA-based point features to produce point-wise affordance logits.Source: §4.4 · p. 6

4.5 TRAINING

• Training combines language modeling, 2D and 3D dense prediction, and routing losses, with dense losses shaping selected MLLM representations but not hard routing assignments.The dense losses use focal and Dice terms for 2D and binary cross-entropy and Dice terms for 3D; routing uses token-level cross-entropy, existence, and sparsity losses.Source: §4.5 · p. 6

5 EXPERIMENTS

• The experiments assess OOD zero-shot transfer, standalone branch performance, and whether routing, joint 2D–3D learning, and decoder coupling improve grounding and routed-state semantics.Language-head diagnostics find meaningful object–affordance semantics in routed states used for dense prediction.Source: §5 · p. 7, §5.4 · p. 9

5.1 EXPERIMENTAL SETUP

• The protocols test mixed-training OOD transfer without target-specific adaptation and branch-wise grounding with each modality trained and evaluated separately.The experiments report gIoU, cIoU, P50, and P50–95 for 2D prediction, with additional saliency-transfer metrics, and AUC, mIoU, SIM, and MAE for 3D.Source: §5.1 · p. 7

5.2 H1: OOD ZERO-SHOT GENERALIZATION

• 27.52 gIoU and 25.22 cIoU on AGD20K, plus 83.55 AUC, 14.67 mIoU, 0.565 SIM, and 0.102 MAE on GEAL*, outperform zero-shot baselines.On AGD20K, UniAfford leads methods without explicit reasoning-chain generation on gIoU and cIoU, and achieves the highest SIM at 0.37.Source: Table 2 · p. 7, §5.2 · pp. 7–8

5.3 H2: BRANCH-WISE AFFORDANCE GROUNDING ABILITY

• On ReasonAff, UniAfford reaches 71.19 gIoU versus 67.41 and 73.63 cIoU versus 62.72; on PIAD, mIoU rises from 9.73 to 14.25.It also achieves the best AUC and MAE and tied-best SIM on PIAD, and outperforms GREAT across all four metrics on PIADv2.Source: Table 3 · p. 8, §5.3 · p. 8

5.4 H3: EFFECTIVENESS OF KEY DESIGN CHOICES

• Under controlled same-subset evaluation, learned routing raises 2D gIoU from 62.28 to 68.79 and 3D mIoU from 17.46 to 34.56 versus fixed-anchor routing; joint training raises these metrics from 41.41 to 68.79 and 30.07 to 34.56, respectively.Replacing similarity-based coupling reduces gIoU to 36.97 for the 2D variant and mIoU to 14.31 for the 3D variant.Source: Table 4 · p. 9, §5.4 · p. 9

6 CONCLUSION

• UniAfford unifies 2D and 3D affordance grounding through shared learning from image-only, point-cloud-only, and semantically paired samples, with routed MLLM states linking functional semantics to modality-specific decoders.The conclusion reports benefits from token routing, joint supervision, and similarity-based decoder coupling.Source: §6 · p. 9

7 LIMITATIONS AND FUTURE WORK

• Large pretrained backbones increase training and inference costs; semantic pairing lacks instance-level spatial correspondence, and evaluation covers affordance grounding rather than closed-loop manipulation.Future work includes more efficient backbone and decoder designs, larger-scale data with instance-level pairing, real-world robotic evaluation, and broader multitask dense prediction.Source: §7 · pp. 9–10

A.1 UNIFIED DATA LAYOUT AND INDEXING

• UniAfford-Data uses shared semantic labels as an index and sample identifiers to link instructions with observations and annotations; image and point-cloud bindings do not imply the same physical instance.The dataset retains image-only, point-cloud-only, and semantically paired samples.Source: Table 5 · p. 13, §A.1 · p. 13

A.2 DATA SOURCING AND PREPROCESSING

• The dataset combines image masks and point-wise annotations from existing sources, normalizes object and affordance labels, and pairs different instances by matching both labels without transferring spatial annotations.Images and masks are resized and normalized; point clouds are sampled to a fixed count with corresponding point labels preserved.Source: §A.2 · pp. 13–14

A.3 INSTRUCTION GENERATION AND ROUTING SUPERVISION

• Instructions specify the target object and affordance; response anchors provide routing supervision, while dense branch supervision depends on which observations and annotations are available.Anchor positions are excluded from ordinary language-modeling loss.Source: §A.3 · p. 14

A.4 DATA SPLITS

• Each split stores modality-specific object–affordance indices, with instruction records and image or point-cloud sample identifiers tracking associated observations and annotations.Instruction entries may be identifiers or records with optional image and point-cloud bindings.Source: §A.4 · p. 14
• Sample is a lightweight partitioned subset for prototyping, debugging, and ablations, whereas Final is the full-scale version used for large-scale training and validation.These dataset-scale names are distinct from the protocols used in modality-isolated experiments.Source: §A.4 · p. 14

A.5 DATASET VISUALIZATION

• Figure 3 pairs RGB images with pixel-level masks and point clouds with point-wise annotations, organized by shared object–affordance semantics rather than instance-level geometric alignment.The annotations illustrate two spatial supervision formats within UniAfford-Data.Source: §A.5 · p. 15, Figure 3 · p. 15

B.1 ROUTING IMPLEMENTATION DETAILS

• The router assigns branch roles to contextual response states; assigned states become ordered image or point-cloud queries, and inference activates decoders for the number of queries predicted.Branch availability is observation-dependent at inference, while training availability requires both the observation and its annotation; unavailable branches are masked from assignment.Source: §B.1 · pp. 15–16

B.2 TRAINING OBJECTIVES

• Language modeling excludes routing-anchor targets, while available image and point-cloud branches receive focal-plus-Dice and binary-cross-entropy-plus-Dice supervision, respectively.Routing objectives supervise assignment and query allocation; dense losses supervise selected representations through the corresponding decoders.Source: §B.2 · pp. 17–18

C.1 EVALUATION PROTOCOLS

• The mixed-training protocol tests cross-dataset transfer without target-specific adaptation, whereas modality-isolated evaluation separately trains and tests each branch using its target modality and task instructions.Zero-shot here means cross-dataset transfer, not absence of all target categories from the training taxonomy.Source: §C.1 · p. 18

C.2 EVALUATION METRICS

• The evaluation reports 2D region-overlap and distribution metrics, alongside 3D ranking, overlap, distribution-agreement, and point-wise-error metrics.For 2D, gIoU averages image-level IoUs while cIoU aggregates intersections and unions; P50–95 averages the percentage above IoU thresholds 0.50–0.95. For 3D, higher AUC, mIoU, and SIM and lower MAE are better.Source: §C.2 · p. 19

C.3 IMPLEMENTATION DETAILS

• Training uses Qwen3-VL with LoRA adapters, while the original MLLM weights and visual and geometric encoders remain frozen; updates are limited to adapters, routing layers, and dense decoders.The implementation uses AdamW with warmup and cosine decay; images are resized to 1024 × 1024 and point clouds sampled to 2048 points unless otherwise specified.Source: §C.3 · pp. 19–20, Table 6 · p. 20

C.4 LANGUAGE-HEAD DIAGNOSTICS OF ROUTED STATES

• Joint training broadens decoded-token categories from 39 for ReasonAff-only training to 238 on ReasonAff, 398 on GEAL, and 426 on AGD20K, with readouts including object and interaction terms.The readouts characterize linguistically accessible content in states used for spatial prediction, not an additional zero-shot result or the causal basis of routing.Source: §C.4 · pp. 20–21, Figure 4 · p. 21

C.5 EXTENDED ABLATIONS AND CROSS-MODAL LEARNING GAINS

• 41.41 gIoU for 2D-only UniAfford exceeds the RAGNet baseline’s 31.44 gIoU on the same 2D training subset, a 9.97-point improvement without added 3D supervision.This is a complete-system comparison under shared 2D training data.Source: §C.5 · p. 22, Table 7 · p. 22
• Joint training raises 2D gIoU from 41.41 to 68.79 and 3D mIoU from 30.07 to 34.56 relative to the corresponding single-branch variants.The 3D variants retain the same SONATA backbone configuration, so the 4.49-point mIoU gain does not require a larger geometric backbone.Source: §C.5 · p. 22, Table 7 · p. 22
• Learned routing improves 2D gIoU from 62.28 to 68.79 and 3D mIoU from 17.46 to 34.56 over fixed-anchor selection with the same backbones and dense decoders.This comparison isolates learned state selection within the proposed architecture.Source: §C.5 · p. 22, Table 7 · p. 22
• Together, the comparisons show system performance under shared training data, joint-supervision benefits with unchanged branch backbones, and gains from learned state selection under matched components.The authors frame the joint gains as support for studying 2D and 3D affordances within a common learning framework.Source: §C.5 · p. 22

C.6 DETAILED COMPUTATIONAL COST ANALYSIS

• 19,658.40 GFLOPs per sample is approximately 27.6% fewer than Affordance-R1, while throughput is 8.20 versus 1.05 samples/s, or approximately 7.8× higher.UniAfford’s end-to-end latency is 122.01 ms per sample; AffordanceNet remains faster at 74.30 ms.Source: §C.6 · p. 22, Table 8 · p. 23
• 1,930.87 GFLOPs and 66.27 ms per sample are reported for 3D inference, exceeding the computational cost of GREAT and IAGNet.The MLLM accounts for 1,903.81 GFLOPs and 55.11 ms; router and decoder costs are 1.38 GFLOPs and 0.07 ms, and 7.11 GFLOPs and 3.63 ms, respectively.Source: §C.6 · pp. 22–23, Table 9 · p. 23
• At batch size 1, allocated memory is approximately 10.26 GiB on loading, with an inference peak of 11.60 GiB and peak reserved memory of 13.78 GiB.These measurements are reported for the stated standard profiling configuration.Source: Table 10 · p. 23, §C.6 · p. 23

D.1 2D ZERO-SHOT QUALITATIVE RESULTS ON AGD20K

• On AGD20K, UniAfford’s “scissors hold” and “knife hold” masks concentrate on graspable regions, while the shown RAGNet predictions extend further into blade regions.The H1 comparisons use UniAfford trained on UniAfford-Data without target-specific fine-tuning; the examples illustrate cross-dataset transfer.Source: §D.1 · pp. 23–24, Figure 5 · p. 24

D.2 3D ZERO-SHOT QUALITATIVE RESULTS ON GEAL*

• On GEAL*, UniAfford’s “knife grasp” response concentrates on the interaction-compatible region, whereas GREAT and IAGNet respond strongly to both blade and handle.Across the illustrated examples, UniAfford’s responses are described as more selective and spatially coherent; red intensity represents predicted scores and blue marks highest-confidence regions.Source: §D.2 · pp. 24–25, Figure 6 · p. 25
• UniAfford does not have the highest overlap in every illustrated 3D example, though its “earphone grasp” and “bottle pour” predictions remain coherent and functionally plausible by visual inspection.The qualitative examples complement aggregate quantitative results reported in Table 2.Source: §D.2 · p. 26