PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous Conditioning of VLAs with Augmenting EMG and Visual Task Descriptors

Edward W. Staley, Connor O. Pyles, Rahul Hingorani, Frank Camargo, Griffin Milsap, Jared Markowitz, Matthew S. Fifer, Michael Wolmetz

arXiv:2610.01794v1cs.CVcs.RO

TL;DR

Language may be insufficient to specify nuanced tasks in cluttered scenes. This paper tests continuous task conditioning through EMG in proprioception and visual annotations in images, and reports stronger performance than descriptive text prompts, particularly in cluttered scenes.

  • Problem

    Language may be insufficient for specifying nuanced tasks in cluttered scenes, motivating task information through other modalities.

    Source: §I · p. 1

  • Method

    The experiments replace fully descriptive task text with an underspecified prompt and add continuous EMG conditioning to proprioception or visual annotations to images.

    Source: §I · p. 1, §III · p. 2, 1) EC-VLA Task · p. 3

  • Results

    Continuous conditioning generally yields stronger performance than descriptive text prompts, with larger improvements in cluttered scenes.

    Source: §I · p. 1, §IV · p. 3, §V · p. 4

  • Takeaways & Limitations

    The results indicate that language may be less important as a downstream task descriptor than for learning latent representations during pretraining.

    Source: §V · p. 4

    The experiments provide updated conditioning at each state, and future work should examine the guidance needed and possible degradation with new users or a secondary tracking and segmentation system.

    Source: §V · p. 4

Abstract

from arXiv · show

Vision-Language-Action (VLA) models rely strongly on language for describing task information, despite having multimodal inputs. We hypothesize that other modalities in the state space may present opportunities for supplemental task conditioning, which may be particularly relevant in cluttered or otherwise ambiguous scenes. We introduce two tuned models to test this hypothesis: (1) an electrophysiology-conditioned VLA (EC-VLA) that incorporates 8-channel electromyography envelopes as continuous conditioning input concatenated to the proprioceptive vector, and (2) a visually-annotated VLA (VA-VLA) that incorporates visual segmentation annotations to the image inputs. On a cube-selection task evaluated across three participants, EC-VLA matches a language-prompted baseline in uncluttered, in-distribution conditions and substantially outperforms it in cluttered, out-of-distribution scenes. Similarly, VA-VLA shows modest improvements over a language-prompted baseline in in-distribution scenes with substantial improvement in cluttered, out-of-distribution trials. Together, these results provide strong evidence for the potential benefit of task-conditioning beyond language.

I. INTRODUCTION

Language is the standard task descriptor for tuned VLA policies, but may be insufficient in cluttered or nuanced scenes. This work tests continuous task conditioning through proprioceptive and image modalities and reports stronger performance, especially in clutter.

A. Vision Language Action Models

  • Tuned VLA policies use proprioception, camera views, and a task description in language, with many parameters initialized from an LLM.The paper situates its work among action-expert and earlier autoregressive VLA architectures.Source: §I · p. 1, A. Vision Language Action Models · p. 1

I. INTRODUCTION

  • Language may be insufficient for cluttered scenes and nuanced tasks that people could more naturally specify with gestures, pictures, or gaze.Source: §I · p. 1
  • The experiments test task conditioning in the proprioceptive vector and image modality, alongside generic task information in text.The proposed comparison is against tuned models using standard descriptive text inputs.Source: §I · p. 1
  • Continuous task-conditioning modalities generally outperform descriptive text prompts, with the largest gains in cluttered scenes.The experiments place task information in the proprioceptive vector or image input while providing only generic task information in text.Source: §I · p. 1

B. Electrophysiological Signals and Robot Policies

Prior work uses electrophysiological signals for robot intent, feedback, or shared control, while visual task cues include keypoints, trajectories, and segmentation masks. This work’s EC-VLA directly takes EMG as state information, and its visual experiment explores supplying masks without language specifying the target identity.

B. Electrophysiological Signals and Robot Policies

C. Visual Annotations and Robot Policies

  • Visual prompting approaches include imagery with keypoints, annotated keypoints, drawn motion curves, and segmentation-style masks.RoboGround combines masks with language, whereas this work explores supplying visual prompts without language specifying object identity.Source: C. Visual Annotations and Robot Policies · p. 2

III. METHODS

The methods tune pretrained action-expert VLA models with task details supplied through an augmented state or image modality and compare them with fully descriptive text prompts. They evaluate action-chunk prediction and task completion, including cluttered scenes absent from training.

III. METHODS

  • The experiments compare augmented-conditioning models with the standard tuning paradigm of fully descriptive text prompts.Source: §III · p. 2
  • The tuned model predicts action chunks from current observations decomposed into language, images, and proprioception, using a flow-matching procedure.The action chunk spans H actions from the current timestep, and the flow-matching loss transports normally distributed noise toward possible action chunks.Source: §III · p. 2
  • Augmented demonstrations replace descriptive task text with a constant underspecified prompt and add informative conditioning in either proprioception or images.Separate models are tuned on augmented and baseline datasets for comparison.Source: §III · p. 2
  • Both experiments test cluttered scenes absent from training and grade trials with a four-stage partial-credit task-completion score.The scoring scheme is adapted from MolmoAct.Source: §III · p. 2, Table I · p. 2

A. Experiment 1: Electrophysiology-Conditioned VLA

Experiment 1 adds continuous EMG signals to the proprioceptive state while using a generic task prompt, testing whether human wrist gestures can guide cube selection. The authors train participant-specific EC-VLA models and compare them with a descriptive-language baseline.

A. Experiment 1: Electrophysiology-Conditioned VLA

  • EC-VLA concatenates an 8-dimensional EMG signal with the standard proprioceptive vector while using a constant underspecified prompt.The resulting state is qEC = [q; EMG].Source: 1) EC-VLA Task · p. 2
  • The task asks a SO-101 arm to pick a specified cube from four identical cubes and place it in a receptacle.The baseline uses descriptive text naming the cube; the underspecified prompt simply says to pick up the cube and place it in the box.Source: 1) EC-VLA Task · pp. 2–3
  • Eight additional EMG inputs encode human wrist-gesture intent to correct left, correct right, or select a block.The system also uses overhead and wrist-mounted RGB views and a six-dimensional proprioceptive vector.Source: 1) EC-VLA Task · p. 3
  • The authors train three participant-specific EC-VLA models and one baseline, evaluating each on uncluttered and cluttered trials.Each EC-VLA model receives continuous EMG from its corresponding participant during evaluation; Figure 2 shows scene views and gestures used in training and testing.Source: 1) EC-VLA Task · p. 3, Figure 2 · p. 3

2) EC-VLA Evaluation:

The EC-VLA evaluation uses participant-specific models on uncluttered and cluttered trials. The VA-VLA experiment uses annotated images and an underspecified prompt for simulated pick-and-place tasks.

2) EC-VLA Evaluation:

  • Each EC-VLA model is evaluated on 30 trials: 20 uncluttered in-distribution trials and 10 cluttered trials with added objects.Continuous EMG during inference is provided by the participant corresponding to each EC-VLA model.Source: 2) EC-VLA Evaluation · p. 3

1) VA-VLA Tasks and Datasets:

  • The VA-VLA experiment uses an underspecified constant prompt and visually annotated images for 16 simulated RoboCasa pick-and-place tasks.The descriptive-text baseline names the object and destination, while the constant prompt does not.Source: 1) VA-VLA Tasks and Datasets · p. 3
  • The VA-VLA datasets comprise roughly 1,600 demonstration episodes, and the visual annotations highlight target objects in green and destinations in blue.The setup uses three RGB views and a 16-dimensional proprioceptive vector; train and evaluation episodes randomize layouts, interiors, and object assets.Source: 1) VA-VLA Tasks and Datasets · p. 3

2) VA-VLA Evaluation:

The VA-VLA evaluation compares annotated-image conditioning with a descriptive-text baseline across randomized pick-and-place tasks. The authors report larger gains in cluttered trials than in typical settings.

2) VA-VLA Evaluation:

  • VA-VLA is evaluated on 16 tasks over 50 randomized episodes each, for 800 episodes total.The clutter evaluation covers eight modified tasks with three additional random objects; tight-space tasks were not tested with clutter.Source: 2) VA-VLA Evaluation · p. 3

IV. RESULTS

  • Continuous conditioning yields marginal gains of a few percent in typical settings and more substantial gains in cluttered scenes compared with descriptive text.Table II provides the full breakdown, including partial task-completion values.Source: §IV · p. 3
  • Figure 3 shows portions of trials at failure, two partial-credit stages, and success, alongside an aggregate task-completion score.Its VA-VLA values cover only tasks that can add clutter; Table II presents evaluation results for tuned baselines and augmented models.Source: Figure 3 · p. 4, §IV · p. 4, Table II · p. 4

A. EC-VLA Results

EC-VLA improves cube-picking outcomes most clearly in cluttered trials, while its success rate remains constant in standard trials. The cluttered results also show fewer trials failing to touch the target cube.

A. EC-VLA Results

  • 20% success rate increase in cluttered trials for EC-VLA, while success rate remained constant in standard trials.The models also achieved a higher mean task score than the tuned baseline, especially in cluttered trials.Source: A. EC-VLA Results · p. 4
  • 50% of baseline cluttered trials did not touch the target cube, compared with 6% for EC-VLA.The most common failure mode involved correct target identification but unsuccessful grasp execution.Source: A. EC-VLA Results · p. 4
  • The most common failure mode involved correct target identification but unsuccessful grasp execution.The authors examined recovery from failed grasps, with participant-model details reported in the appendix.Source: A. EC-VLA Results · p. 4

B. VA-VLA Results

VA-VLA generally outperformed the tuned π0.5 baseline, particularly in cluttered trials. The conclusion notes modest gains in-distribution and identifies updated conditioning and deployment robustness as future questions.

B. VA-VLA Results

  • 47% of cluttered trials were complete failures for baseline π0.5, versus 25% for VA-VLA.The failures were averaged across tasks; all tasks improved in cluttered scenes, though the broader trends did not hold for every task individually.Source: B. VA-VLA Results · p. 4

V. CONCLUSION AND FUTURE WORK

  • VA-VLA showed minor performance increases in in-distribution scenes and strong increases in out-of-distribution scenes with ambiguity or clutter.The conclusion notes that these findings suggest language may be less important as a downstream task descriptor than for learning latent representations during pretraining.Source: §V · p. 4
  • The experiments updated conditioning at each state; future work should assess guidance requirements and possible degradation with new users or secondary tracking and segmentation systems.The authors also suggest exploring combinations of modalities.Source: §V · p. 4

A. Tuning Details and Hyperparameters

The experiments used different tuning setups for EC-VLA and VA-VLA, with EC-VLA trained on participant demonstrations containing EMG signals. Those signals were featurized as envelopes and added to proprioception.

1) EC-VLA Tuning Details:

  • EC-VLA tuning froze the backbone and trained the action expert and state projection for 40k updates, using batch size 64 and AdamW.The learning rate cosine-decayed from 1e-4 to 2.5e-6 over 30k steps; weight decay was 1e-10 and gradient norms were clipped to 10.0.Source: 1) EC-VLA Tuning Details · p. 6

2) VA-VLA Tuning Details:

  • VA-VLA tuning trained the full π0.5 architecture, including its VLM backbone and action expert, for 70k updates on 2xH200s.It used batch size 64, action chunks of 50, mild image augmentation, and AdamW with a learning rate cosine-decayed from 2.5e-5 to 2.5e-6 over 30k steps.Source: 2) VA-VLA Tuning Details · p. 6

B. EC-VLA Data Collection

  • Three participants each provided 105 balanced-target demonstrations with wrist gestures recorded as EMG; rectified, 2Hz low-pass-filtered envelopes formed an 8D vector concatenated with 6D proprioception.Flexion, extension, and relaxation represented left, right, and selection intent, respectively.Source: B. EC-VLA Data Collection · p. 6

C. EC-VLA Participant Models and Recovery Analysis

Participant-specific EC-VLA models had similar completion scores and successful corrections, while the baseline never successfully recovered. The VA-VLA imagery setup contrasted annotated and unaltered camera inputs.

C. EC-VLA Participant Models and Recovery Analysis

  • All three participant-specific EC-VLA models had similar completion scores and successful grasp corrections; the baseline never recovered successfully.Figure 4A shows partial scores by model; Figure 4B analyzes recovery attempts and their success.Source: Figure 4 · p. 6, C. EC-VLA Participant Models and… · p. 6

D. VA-VLA Task Imagery

  • Experiment 2 used left, wrist, and right camera views; VA-VLA annotated target objects in green and destinations in blue, while cluttered trials added objects.The baseline used unaltered images paired with a text description; VA-VLA used generic text and visual annotations.Source: Figure 5 · p. 6

E. VA-VLA Task-level Results

The task-level results section provides success rates and task-completion scores for all 16 Experiment 2 tasks.

E. VA-VLA Task-level Results

Related papers

Loading 2610.01794v1…