PaperlayerView exact revision on arXiv

Source-linked AI summary

Lightweight Visual Reasoning for Socially-Aware Robots

Alessio Galatolo, Ronald Cumbal, Alexandros Rouchitsas, Katie Winkle, Didem Gürdür Broo, Ginevra Castellano

arXiv:2603.03942v1cs.RO

TL;DR

Robots need to interpret dynamic human behaviour, but existing VLMs remain limited in integrating visual and textual reasoning for multimodal interaction. The paper introduces a lightweight language-to-vision feedback module that reprocesses visual input under language context, improving scene description and intention recognition across evaluated models while producing mixed navigation results.

  • Problem

    Existing VLM research often addresses perception, instruction following, and interaction capabilities in isolation, leaving multimodal human-robot reasoning insufficiently integrated.

  • Method

    A lightweight module projects image-token language-model hidden states back into the vision encoder input, enabling a second pass conditioned on textual reasoning.

  • Results

    The method improves scene description and human intention recognition across model families, while navigation gains are modest or mixed.

  • Takeaways & Limitations

    Language-to-vision feedback provides a lightweight architectural route for improving cross-modal reasoning on robotics-centred tasks.

Abstract

from arXiv · show

Robots operating in shared human environments must not only navigate, interact, and detect their surroundings, they must also interpret and respond to dynamic, and often unpredictable, human behaviours. Although recent advances have shown promise in enhancing robotic perception and instruction-following using Vision-Language Models (VLMs), they remain limited in addressing the complexities of multimodal human-robot interactions (HRI). Motivated by this challenge, we introduce a lightweight language-to-vision feedback module that closes the loop between an LLM and the vision encoder in VLMs. The module projects image-token hidden states through a gated Multi-Layer Perceptron (MLP) back into the encoder input, prompting a second pass that reinterprets the scene under text context. We evaluate this approach on three robotics-centred tasks: navigation in a simulated environment (Habitat), sequential scene description (Mementos-Robotics), and human-intention recognition (our HRI dataset). Results show that our method improves Qwen 2.5 (7B) by $3.3\%$ (less distance), $+0.057$ description score, and $+2.93\%$ accuracy, with less than $3\%$ extra parameters; Gemma 3 (4B) and LLaVA OV 1.5 (4B) show mixed navigation results but gains $+0.111,+0.055$ and $+10.81\%,+4.79\%$ on the latter two tasks. Code is available at https://github.com/alessioGalatolo/VLM-Reasoning-for-Robotics

I. INTRODUCTION

Robots in human-shared environments must reason about dynamic human behaviour, but current multimodal approaches often integrate visual and textual information only shallowly. The paper proposes a lightweight feedback module that lets language interpretation modulate visual perception, reporting gains across robotics-centred tasks.

  • Robots must anticipate dynamic contexts, respond to spontaneous human behaviour, communicate effectively, and make context-dependent interaction decisions.
  • Existing LLM and VLM research often treats instruction following, perception, and interaction capabilities in isolation rather than integrating them holistically.
  • Multimodal reasoning methods commonly use visual information as context for textual reasoning, limiting deeper integration between the two modalities.
  • The proposed module closes the loop by enabling the language model to modulate the vision encoder, an underexplored architectural principle in robotics VLMs.
  • The method reports empirical gains on scene description and intention recognition, with modest gains on navigation, supported by ablations of image reuse, MLP removal, and modality order.

III. METHODOLOGY

The paper adds a lightweight visual reasoning module to pretrained vision-language architectures, connecting language understanding back to visual encoding. The approach is designed for compatibility with widely used VLMs while requiring minimal additional parameters.

  • The method connects the language understanding component to the visual encoding process to enable cross-modal reasoning.
  • The approach targets architectures that combine a pretrained vision encoder with a pretrained or fine-tuned language model through alignment training.
  • The module is compatible with LLaVA-OneVision, Qwen 2.5 VL, and Gemma 3, and may extend to other architectures because of its minimal requirements.

B. Visual reasoning module

The visual reasoning module uses language-model feedback to modify the image representation before a second model pass. It is trained with a gated MLP and patch unmerger while keeping the base language and vision models frozen.

  • B. Visual reasoning module: The module extracts image-token hidden states after a first multimodal pass, projects them into encoder input space, and adds the result to the embedded image.
  • B. Visual reasoning module: A second forward pass processes the original prompt and image together with the feedback-conditioned image representation, enabling reinterpretation under textual context.
  • B. Visual reasoning module: The visual reasoning module uses a gated MLP whose input and output dimensions match the language model hidden dimension, with a hidden dimension twice as large.
  • B. Visual reasoning module: A patch unmerger maps the language-model representation back to the number of image patches expected by the vision encoder.
  • B. Visual reasoning module: Training computes loss only from the second pass and backpropagates through the language model output, vision encoder, and visual reasoner while freezing the language model and vision encoder.
  • B. Visual reasoning module: Trainable parameters remain below 1.7% of the original model, or below 3% when LoRA layers are included.

D. Training data

The visual reasoning module is trained on the general Visual-CoT dataset rather than robotics-specific data. This choice avoids evaluation bias toward the base model and preserves broader applicability.

  • Training uses Visual-CoT image-question pairs with associated reasoning outputs.
  • A general dataset was selected because robotics-specific training could bias evaluation against the untrained base model and narrow VLM applicability.

E. Inference

The method performs an initial reasoning pass, feeds the resulting visual hint back into the encoder, and then generates a response using the original and modified images. Evaluation covers navigation, sequential scene understanding, and human intention recognition.

  • Inference: The initial pass generates a visual reasoning hint whose transformed representation modifies the vision encoder embedding before a second response pass.The LLM receives both the old and new image for final generation.
  • Inference: The evaluation compares three VLMs—Qwen 2.5 VL 7B, Gemma 3 4B, and LLaVA OneVision 1.5 4B—against plain baselines.All benchmarks use greedy decoding without sampling.
  • Inference: The benchmarks measure robot navigation in Habitat, sequential scene descriptions in Mementos-Robotics, and human intention recognition.The navigation task uses natural-language goals and parsed robot actions; Mementos provides sequential images with scene descriptions.

A. Human intention recognition dataset

The authors construct a new human-robot interaction dataset because existing resources lack sufficient recording quality, behavioral variability, or accessible data. It captures behavior-specific audiovisual interactions from a robot-like perspective.

  • A. Human intention recognition dataset: The new dataset addresses limitations in existing robotics datasets, including missing behavior annotations, restricted recording quality, and limited behavioral variability.The design targets human-intention recognition in human-robot interaction.
  • A. Human intention recognition dataset: Audio-video recordings were collected during a human-robot interaction study with 10 participants.Participant demographics and prior robot experience are reported in the study description.
  • A. Human intention recognition dataset: Participants interacted with the Furhat social robot as a tourism assistant under not-rushed, somewhat-rushed, and very-rushed conditions.The interaction duration was shortened to naturally induce different urgency levels.
  • A. Human intention recognition dataset: The study aims to elicit natural behavior while specifying participants’ intentions, producing rich audiovisual data for analyzing interaction strategies.The setup grounds intentions in a controlled conversational scenario.

1) Annotation and benchmark adaptation:

The dataset annotations are converted into four-option intention-recognition questions associated with individual frames, while the broader evaluation reports task-specific metrics and compares the method with plain models.

  • 1) Annotation and benchmark adaptation:: 188 annotated events cover five behaviors, including waiting, interrupting, signaling intent with two urgency levels, and interacting while another person waits.These categories represent distinct participant behaviors toward the social robot.
  • 1) Annotation and benchmark adaptation:: Each caption becomes two multiple-choice questions addressing the inactive and intervening persons, with the correct description paired with sampled distractors.Position markers are normalized and then restored for each instance.
  • 1) Annotation and benchmark adaptation:: The final dataset contains 376 samples, each pairing one 360p frame with a four-option question and no textual scene information.Each sample has a single associated frame.
  • 1) Annotation and benchmark adaptation:: Evaluation reports final distance for navigation, LLM-as-a-judge description scores from 1 to 5 for Mementos, and accuracy for intention recognition.The intention benchmark uses multiple-choice questions.
  • 1) Annotation and benchmark adaptation:: Qwen improves across all three tasks, reducing navigation distance from 7.787 to 7.530, raising Mementos from 2.261 to 2.318, and increasing intention accuracy from 34.04% to 36.97%.Gemma and LLaVA improve on description and intention recognition but worsen slightly on navigation.
  • 1) Annotation and benchmark adaptation:: The largest relative gains occur on scene description and human intention recognition, especially for models with lower initial performance.Navigation improvements are less significant or sometimes negative for Gemma and LLaVA.

A. Ablation study

Ablations show that retaining the original image and the MLP-based visual reasoner is important for performance, while changing modality order also affects results. The complete method outperforms the tested altered settings.

  • A. Ablation study: Removing the original image in the second pass or removing the MLP causes marked drops across benchmarks, although intention accuracy remains competitive.Without the original image, intention accuracy is 34.31%; without the MLP, it is 37.50%, versus 36.97% for the full method.
  • A. Ablation study: Image-before-prompt input reaches 28.46% intention accuracy, whereas prompt-first input reaches 25.53%, and both remain below the complete method.The prompt-first variant also scores lower on the other two benchmarks.
  • A. Ablation study: The authors conclude that the original second-pass image and MLP visual reasoner are necessary for best performance, while modality order also matters.Image before the prompt outperforms the other tested settings.

B. Baselines

The method’s two-pass design improves performance but adds substantial compute, while remaining modest in memory and parameter growth. Results also indicate gains are not explained by simply duplicating visual input.

  • Ordering of modalities:: 30% performance degradation occurs when swapping the image and prompt order for models trained with the reverse ordering.The authors attribute this sensitivity to the structure preferred during VLM pretraining.
  • Baselines: Providing baselines with the original image twice causes greater performance degradation, indicating that gains are not due to simply adding more visual input.The duplicated-image test uses the same template as the proposed method.
  • Resource consumption: Less than 3% additional parameters and memory growth from 15.9 to 16.32 GB accompany the method’s computational overhead.The overhead is measured on an NVIDIA RTX 3090 during Qwen evaluation for human-intention recognition.
  • Resource consumption: Two forward passes increase average TFLOPs from 7.06 to 20.39 and reduce throughput from 4.24 to 1.27 samples per second.The increased latency is primarily attributed to the extra inference pass rather than substantial model-size growth.
  • Resource consumption: The method remains deployable on a single consumer-grade GPU, with throughput above one sample per second described as sufficient for low-frequency robotic perception.The passage specifically mentions human-intention recognition and navigation as deployment scenarios.

A. Interpretable cross-modal feedback loops

The approach closes a language-to-vision feedback loop that lets the language model modulate visual processing without modifying the base models. Improvements are consistent on scene description and intention recognition but less reliable for navigation.

  • A. Interpretable cross-modal feedback loops: The compact MLP and unmerging mechanism reuse language output to alter visual embeddings instead of embedding vision once passively.The approach does not require backpropagation through the vision encoder or modifications to the base models.
  • Performance across tasks: The method consistently improves scene description and human intention recognition, especially when high-level visual semantics are critical.Navigation relies more heavily on structured outputs and spatial understanding, whereas the latter tasks require contextual parsing or subtle social cues.
  • Performance across tasks: Navigation gains are less consistent, with slight declines for Gemma and LLaVA attributed to limitations in producing well-structured outputs.The authors distinguish this issue from failures in visual reasoning.
  • Image reuse in forward passes:: Using the original image in both passes is essential, since ablating it reduces performance across all tasks.The authors characterize visual reinterpretation as more stable than visual replacement.

2) Ordering of modalities:

The ablations show that both modality ordering and feedback-modulated visual input matter. Duplicating the same image performs worse, supporting closed-loop reinterpretation rather than brute-force repetition.

  • 2) Ordering of modalities:: Performance unexpectedly degrades when the image follows the prompt for models initially trained with the reverse ordering.The authors report that their intended causal-masking arrangement is not optimal for these backbones.
  • 2) Ordering of modalities:: Duplicating the same image for baselines performs worse than using a modulated second image.This comparison supports the necessity of feedback-modulated visual input rather than simply increasing visual input.
  • VII. CONCLUSION: Cross-modal feedback is presented as a design choice that strategically closes the loop between perception and interpretation, rather than merely adding data or capacity.The conclusion frames this as a challenge to the dominant feedforward paradigm in vision-language integration.
  • VII. CONCLUSION: The method highlights architectural asymmetry and feedback as underused principles in multimodal models.The paper connects this design principle to embodied cognition while emphasizing its limited realization in VLMs.
Loading 2603.03942v1…