PaperlayerView exact revision on arXiv

Source-linked AI summary

Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control

Shunlei Li, Longsen Gao, Jin Wang, Chang Che, Xi Xiao, Jiuwen Cao, Yingbai Hu, Hamid Reza Karimi

arXiv:2508.05342v2cs.ROcs.AI

TL;DR

Robots trained from human videos often rely on low-level imitation that does not generalize across objects, layouts, or embodiments. GF-VLA instead fuses information-theoretic interaction graphs with language-conditioned planning and bimanual role assignment. Across dual-arm assembly variants, it reports accurate graph extraction, interpretable planning, and 90% overall task success, while retaining calibration and static-workspace assumptions.

  • Problem

    Low-level trajectory imitation from human videos has limited generalization across object types, spatial layouts, and manipulator configurations, motivating task-level policy reasoning.

  • Method

    GF-VLA extracts information-theoretic cues into temporally ordered hand-object and object-object graphs, fuses them with a language-conditioned transformer, and generates behavior trees, Cartesian commands, and cross-hand assignments.

  • Results

    90% overall task success was achieved across novel letter-assembly and tower-construction variants, alongside over 95% graph accuracy, 94% grasp success, and 89% placement accuracy.

  • Takeaways & Limitations

    Information-theoretic behavior graphs combined with LLM planning supported single-demonstration policy reuse, interpretable execution, and adaptation to shifted viewpoints, randomized configurations, and distractors.

  • Takeaways & Limitations

    Performance remains constrained by degraded temporal segmentation in ambiguous contacts, reduced ordering accuracy for parallel or symmetric subtasks, and assumptions of calibrated camera-arm alignment and a largely static workspace.

Abstract

from arXiv · show

Teaching robots dexterous skills from human videos remains challenging due to the reliance on low-level trajectory imitation, which fails to generalize across object types, spatial layouts, and manipulator configurations. We propose Graph-Fused Vision-Language-Action (GF-VLA), a framework that enables dual-arm robotic systems to perform task-level reasoning and execution directly from RGB and Depth human demonstrations. GF-VLA first extracts Shannon-information-based cues to identify hands and objects with the highest task relevance, then encodes these cues into temporally ordered scene graphs that capture both hand-object and object-object interactions. These graphs are fused with a language-conditioned transformer that generates hierarchical behavior trees and interpretable Cartesian motion commands. To improve execution efficiency in bimanual settings, we further introduce a cross-hand selection policy that infers optimal gripper assignment without explicit geometric reasoning. We evaluate GF-VLA on four structured dual-arm block assembly tasks involving symbolic shape construction and spatial generalization. Experimental results show that the information-theoretic scene representation achieves over 95 percent graph accuracy and 93 percent subtask segmentation, supporting the LLM planner in generating reliable and human-readable task policies. When executed by the dual-arm robot, these policies yield 94 percent grasp success, 89 percent placement accuracy, and 90 percent overall task success across stacking, letter-building, and geometric reconfiguration scenarios, demonstrating strong generalization and robustness across diverse spatial and semantic variations.

1. Introduction

GF-VLA addresses the gap between semantic reasoning and physical interaction modeling by combining information-theoretic scene graphs with vision-language-action reasoning for dual-arm policy transfer. The framework targets interpretable, generalizable manipulation from human demonstrations.

  • Motivation: Current robotic manipulation methods remain vulnerable to sensor noise, occlusion, physical contact, and rigid sensing assumptions in unstructured environments.These limitations restrict robustness beyond controlled settings.
  • Motivation: Current VLA frameworks struggle to model structured, dynamic relationships between objects and agents during precise physical interaction.This limitation affects planning under ambiguous instructions, unseen arrangements, and contact-rich coordination.
  • Proposed framework: GF-VLA combines temporally structured scene graphs from multimodal demonstrations with vision-language-action reasoning for semantically grounded task planning.The framework uses entropy and mutual information to encode dynamic interactions and supports transfer from limited demonstrations to unseen objects and tasks.
  • Proposed framework: Information-theoretic scene graphs explicitly encode temporal and spatial physical interactions that existing perceptual frameworks leave insufficiently represented.The graphs are constructed from multimodal human demonstrations with semantic segmentation.
  • Contribution: GF-VLA is evaluated as a unified paradigm integrating structured physical interaction modeling with vision-language-action reasoning for robust and generalizable manipulation.The authors position the framework as a benchmark-setting approach across challenging dual-arm scenarios.
  • Interpretability: Chain-of-Thought prompting provides explicit subgoal decomposition for more understandable and execution-oriented robotic policies.The claimed benefit is improved policy understandability and execution fidelity.

2. Related works

Related VLA research emphasizes action generation and generative efficiency, whereas GF-VLA adds information-theoretic behavior graphs as an intermediate symbolic layer for policy reasoning and generalization.

  • Vision-language-action models: VLA models learn mappings from multimodal inputs to robot control commands, with recent work refining action representations and generation efficiency.Examples include discrete action tokens, diffusion-based generation, diffusion transformers, and mixture-of-experts designs.
  • Information theory: Information theory provides a formal basis for quantifying uncertainty in signals through entropy over probability distributions.The paper uses this foundation to motivate information-based scene representations.
  • GF-VLA: GF-VLA introduces information-theoretic behavior graphs that encode the temporal and semantic structure of task dynamics as an intermediate symbolic representation.The graphs are integrated with visual observations and language instructions for interpretable policy generation.
  • GF-VLA: The graph-based knowledge layer supports symbolic abstraction, policy generalization, and local replanning in dynamic multi-step dual-arm manipulation.The paper contrasts these capabilities with prior emphasis on action formats or generative efficiency.

3. Information-Theoretic Scene Graphs

GF-VLA uses sliding-window information measures to identify task-relevant motion and interactions, then encodes them in temporally structured scene graphs for dual-arm policy reasoning.

  • Entropy-based motion analysis: Static object segments minimize Shannon entropy, whereas movement across multiple spatial bins yields positive entropy.This distinction supports using entropy as a motion indicator.
  • Entropy-based motion analysis: Entropy over sliding windows identifies temporal regions with significant positional change from quantized object trajectories.Localized motion produces lower entropy, while expansive movement spans more bins and increases entropy.
  • Hand-object coordination: Mutual information between hand and object position signals measures their kinematic dependency and identifies coordinated motion.The measure captures both linear and nonlinear statistical relationships.
  • Temporal scene graphs: Scene graphs represent hands and objects as 6D-pose nodes connected by directed interaction edges across temporal frames.The graph structure encodes the system state and relationships among scene entities.
  • Interaction representation: The framework distinguishes interaction patterns such as coupled motion, docking, and object-object relations for subsequent policy generation.These graph interactions connect information-theoretic motion cues to dual-arm action selection.

4. Graph-Fused VLA (GF-VLA)

GF-VLA transfers human demonstrations into dual-arm robot policies by combining information-theoretic scene representations with vision-language-action reasoning. Its unified architecture links hierarchical planning, self-verification, and executable manipulation commands.

  • Policy generation: GF-VLA processes human demonstrations into structured behavior graphs and temporal keyframes that encode spatial and semantic task elements.A Policy Agent reasons over these representations to produce subtasks and a behavior trace.
  • Policy execution: The framework fuses demonstration-derived policies with live RGB input and spoken instructions to update robot behavior for the current scene.The resulting policy preserves human intent while adapting to scene context.
  • Unified architecture: A unified dual-head transformer performs high-level CoT planning and self-verification alongside low-level motion control.The LLM Head handles semantic planning, while the Action Head generates executable commands.
  • Policy reasoning: Behavior trees decompose a root policy into n subtasks, with each action node carrying reasoning and verification criteria.The letter-R example uses coordinated dual-arm grasping and placement for structural assembly.
  • Policy reasoning: Chain-of-Thought reasoning decomposes multi-stage manipulation goals into interpretable subgoals and supports plan consistency and auditability.Each intermediate action includes a rationale and sensor-based success check.
  • Fine-tuning: LoRA independently fine-tunes the LLM and Action Heads while retaining a shared multimodal encoder.The LLM Head is trained from annotated human demonstrations, while the Action Head uses dual-arm robot trials.

5. Experiments

The experiments assess representation quality, policy reasoning, manipulation generalization, and end-to-end execution in dual-arm block tasks. Evaluations use video demonstrations, structured annotations, and robot trials spanning varied shapes, layouts, and instructions.

  • Experimental design: Four experiments evaluate video-to-representation quality, LLM policy generation, fine-tuned manipulation, and end-to-end execution.The studies progress from symbolic task representation to real-world dual-arm performance.
  • Experimental setup: The experimental setup includes a top-mounted RGB-D camera and a dual-arm platform executing coordinated motions in a shared workspace.The platform uses UR5e and UR10e arms with different grippers for bimanual trials.
  • Task representation assessment: The representation evaluation measures graph accuracy and subtask-boundary detection from RGB-D demonstrations.Graph equality requires matching topology and object identities within a spatial tolerance, while TSA uses a 0.5-second temporal tolerance.
  • Task planning assessment: LLM policy evaluation tests subtask coverage, temporal ordering, Chain-of-Thought interpretability, and verification correctness on held-out demonstrations.The test set contains 125 videos distinct from the 125 training videos.
  • Task manipulation assessment: Manipulation evaluation measures grasping, stable placement, pose error, instruction compliance, and transferability across varied shapes, spatial relations, and instructions.The tasks target robust grasping, coordinated dual-arm motion, and precise placement.
  • Task manipulation assessment: Plan Transferability Rate measures successful reuse of a policy from one human demonstration on novel task variants without retraining.It is defined as successful transfer executions divided by total transfer attempts.

6. Results and analysis

GF-VLA achieved high graph, planning, and dual-arm execution performance across increasingly complex manipulation tasks. Results also show that information-theoretic graphs, explicit reasoning, and dual-head outputs materially support task success and generalization.

  • Task representation: 98.5% GRA and 95.6% TSA were achieved in single-hand manipulation, while letter-block and tower tasks retained strong graph accuracy despite greater spatial and temporal complexity.Letter-block assembly reached 97.2% GRA and 93.9% TSA; tower construction reached 96.8% GRA and 93.1% TSA.
  • Task planning: 98% plan coverage, 95% ordering accuracy, and 96% verification correctness were achieved for single-hand manipulation, with CoT interpretability rated 4.7/5.Performance declined modestly for letter-block and tower tasks, where ordering, coverage, and verification were more affected by symmetry, occlusion, and overlapping transitions.
  • Block manipulation: 94% grasp and 90% placement success were achieved for ambiguous spatial relations, compared with 92% and 87% for absolute 6D pose and 91% and 85% for relative pose.Instruction compliance remained high across these settings, while pose error increased with spatial precision and relational reasoning demands.
  • Ablation: Removing the information-theoretic scene graph reduced average TSR by 0.49, while removing explicit reasoning reduced TSR by 0.17 and removing the dual-head design reduced it by 0.37.These ablations identify structured scene representation, intermediate reasoning, and separate language-control outputs as important components of the framework.

7. Discussion

Across four experiments, GF-VLA linked stable scene representations, interpretable planning, robust dual-arm execution, and generalization from single demonstrations. The main remaining boundaries concern ambiguous temporal structure, calibration, and workspace assumptions.

  • Scene representation: Over 95% graph representation accuracy and over 93% subtask segmentation accuracy were achieved from RGB-D streams.The graphs provided a symbolic interface between raw perception and LLM-based task reasoning.
  • Task planning: LLM-generated plans aligned strongly with expert ground truth, while CoT reasoning achieved an average interpretability score of 4.4/5 across symbolic and geometric tasks.The reasoning traces were reported as functionally correct, transparent, and human-readable.
  • Execution: 94% grasp success and 89% placement accuracy were maintained across four bimanual task categories, with average instruction compliance of 4.5/5.Stable coordination between heterogeneous UR5e and UR10e grippers supported policy decomposition and motor-level role allocation.
  • Generalization: 90% overall task success and 86% policy transferability were achieved when policies from single demonstrations were applied to novel letter-assembly and tower variants.The system adapted to shifted viewpoints, randomized configurations, and distractors without additional retraining.
  • Limitations: Complex visual interactions reduced segmentation quality and ordering accuracy, while execution still assumed pre-calibrated camera-arm alignment and a largely static workspace.Future work targets trainable graph extraction, temporal memory, deformable or articulated objects, and verification-driven recovery.
  • Discussion: GF-VLA bridges perception, language, and action to produce generalized, interpretable, and physically grounded dual-arm behavior from minimal demonstration.The reported generalization spans task forms, objects, and language conditions, suggesting applicability to collaborative manipulation settings.

8. Conclusion

GF-VLA converts a single human video into structured, interpretable, executable dual-arm policies by fusing information-theoretic interaction graphs with VLA reasoning. The resulting system generalizes across assemblies and viewing conditions, while retaining clear calibration, workspace, and temporal-modeling limitations.

  • Method: GF-VLA extracts Shannon-information cues, builds temporally ordered hand-object and object-object graphs, and fuses them with a language-conditioned transformer.The framework produces behavior trees and Cartesian commands, while cross-hand selection resolves bimanual role assignment.
  • Integrated findings: The four experiments support a full perception-planning-execution pipeline in which scene graphs, language-guided plans, robust execution, and single-demonstration transfer work together.The reported evaluation covers symbolic and geometric manipulation variations.
  • Outcome: 6D pose error was reported as halved relative to imitation-learning and language-only baselines in stacking, letter-building, and tower-reconfiguration trials.The conclusion attributes this result to fusing symbolic interaction structure with VLA reasoning.
  • Limitations: Temporal segmentation and ordering remain weaker for ambiguous, dynamic, parallel, or symmetric interactions, while execution assumes calibrated camera-arm alignment and a mostly static workspace.Proposed extensions include temporal memory, end-to-end graph extraction, deformable-object handling, self-correction, and failure recovery.
Loading 2508.05342v2…