PaperlayerView exact revision on arXiv

Source-linked AI summary

What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory

Amir-Hossein Shahidzadeh, Seungjae Lee, Eadom Dessalene, Shanthosh Raaj Mohanram Mageswari, Soroush Etemad, Furong Huang, Cornelia Fermüller, Yiannis Aloimonos

arXiv:2609.38494v1cs.ROcs.CV

TL;DR

Fixed multimodal histories and fusion structures do not make differing skill-specific sensory needs explicit. SKILLFORMER conditions fusion of recent visual and tactile observations and sparse event memory on the queried skill, and reduces progress error across three manipulation tasks. The study evaluates this representation through skill-progress estimation rather than closed-loop control.

  • Problem

    Common manipulation representations use fixed histories and fusion across skills despite differences in sensory needs, and short histories may omit task-relevant earlier events.

    Source: §1 · p. 2

  • Method

    SKILLFORMER uses language-described skill queries to condition fusion over modality-specialized short-term observations and sparse memory of predicted skill-completion events.

    Source: §1 · p. 2

  • Results

    Across three tasks, skill-conditioned fusion reduces overall skill-progress MSE by 37–80% versus vision alone, and event memory reduces it by 59–92% versus recent observations alone.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Progress estimation provides an interpretable way to probe task-relevant representations of geometric state, contact dynamics, and interaction history.

    Source: §1 · p. 2

    The evaluation uses predefined language-described skills, annotated skill boundaries, one sensing platform, and unseen episodes from three collected task domains; closed-loop control remains future work.

    Source: D LIMITATIONS · p. 16

Abstract

from arXiv · show

Robotic manipulation integrates vision, touch, and language, whose importance shifts across stages: vision guides reaching, while touch, through its evolution over time, decides grasping, alignment, and contact. Yet existing multi-modal manipulation policies typically use fixed temporal contexts and fusion strategies, despite shifts in what each modality contributes across different skills. We study how vision and touch should be combined at the level of primitive skills, asking what each skill needs from each sensor, and propose a skill-conditioned representation in which the queried skill conditions fusion over modality-specific short-term observation tokens while attending to a sparse event memory that retains terminal observations from the last $K$ executed skills. Evaluated by skill progress estimation on three contact-rich tasks, it reduces slip-detection delay by 87% against fine-tuned SOTA progress models, twist-completion delay by 67.5% against a vision-only ablation, and progress error on a blind search task by 92% through sparse event memory. Gains concentrate exactly where completion is defined by contact or task history. More broadly, our results suggest that observation formation not only policy architecture is a central challenge in multi-modal representation. Project Website: http://what-to-attend-what-to-keep.github.io/

1 INTRODUCTION

The paper argues that manipulation skills require different sensory modalities and temporal contexts, while common fixed observation formulations leave those needs implicit. It introduces SKILLFORMER to condition fusion and sparse event memory on language-described skills, evaluating the representation through progress estimation across three tasks.

1 INTRODUCTION

  • Different stages draw on different sensory and temporal cues: vision guides reaching, while tactile patterns help distinguish contact states and their evolution.The paper notes that similar skills can have distinct tactile patterns, such as aligning bulb and bottle threads.Source: §1 · p. 2
  • Fixed short histories and shared fusion structures leave skill-specific sensory needs implicit and may omit earlier events needed to distinguish task history.Simply extending history can increase attention cost quadratically with sequence length, while repeated visual frames may obscure sparse tactile changes.Source: §1 · p. 2
  • SKILLFORMER combines modality-specialized encoders, asymmetric short-term observation windows, and sparse event-triggered memory conditioned jointly by the queried skill and progress objective.The formulation treats observation formation—what to attend to and what to keep—as a design question alongside policy architecture.Source: §1 · p. 2
  • Progress estimation probes task-relevant representations across active and inactive skill queries on lamp replacement, bottle capping, and blind object search.The study varies tactile input, event memory, history length, and fusion mechanism.Source: §1 · p. 2
  • Skill-conditioned fusion reduces overall skill-progress MSE by 37–80% versus vision alone, while event memory reduces it by 59–92% versus recent observations alone.The work also introduces three datasets comprising 429 demonstrations of lamp replacement, bottle capping, and blind cube search.Source: §1 · p. 3

2 BACKGROUND

The background situates the work at the intersection of visuotactile manipulation, skill-conditioned temporal representations, and progress-based evaluation. It distinguishes the paper’s focus on organizing features and histories around skills from backbone selection and downstream reward use.

Multimodal representations for manipulation

  • Visuotactile manipulation combines scene-level visual information with local contact observations, and modality usefulness motivates studying feature space, temporal context, and fusion.The paper builds on CLIP visual and language features and FeelAnyForce tactile features, using DINOv2 as a control for tactile specialization.Source: §2 · p. 3

Skill conditioning and temporal representations

  • Language-described skills provide a structured interface, while short windows and longer sequence context offer different temporal information for manipulation.The paper evaluates sparse memory of skill-terminal observations against recent and uniformly sampled histories.Source: §2 · p. 3

Progress supervision and representation evaluation

  • Prior work connects visual representations to reward and task structure; this work uses progress supervision to diagnose multimodal representations, including with frozen-feature probes.Comparisons with existing progress models assess progress estimation, while probes test task-relevant information beyond the original predictor’s needs.Source: §2 · p. 3

3 STRUCTURING SKILL-CONDITIONED REPRESENTATIONS WITH EVENT MEMORY

SKILLFORMER represents a queried primitive skill using recent visual and tactile observations together with sparse memory of earlier skill events. Progress supervision trains the representation and supplies predicted terminations for constructing memory.

3.1 SKILL-CONDITIONED FORMULATION

  • Progress targets distinguish continuous skills with linearly normalized time, discrete skills with a rise near termination, and conditional skills whose terminal outcome can be success or failure.The skill vocabulary is predefined, and demonstration instances have annotated boundaries from which progress targets are derived.Source: §3.1 · p. 4
  • The representation combines recent multimodal observations with up to K preceding skill-termination windows, and skill-conditioned progress links observation fusion to terminal observations retained for later interactions.Primitive skills have language-described objectives and observable termination conditions; reaching, tightening, and alignment illustrate different sensory requirements.Source: §3.1 · p. 4

3.2 MODALITY-SPECIFIC FEATURES AND HISTORIES

  • Separate visual and tactile encoders produce projected frame tokens over modality-specific histories, with temporal, sensor-identity, and event-order embeddings.The figure depicts RGB and two tactile streams encoded over their own horizons, alongside a language query.Source: §3.2 · pp. 4–5, Figure 3 · p. 5

3.3 SKILL-CONDITIONED ATTENTION

  • A transformer jointly processes the skill query, current observation tokens, and retained event tokens, with a learned state token aggregating information for the queried skill.Attention is bidirectional over current and retained observations and uses no observations after the current time.Source: §3.3 · p. 5

3.4 LONG- AND SHORT-TERM MEMORY VIA SPARSE EVENTS

  • When a skill event is detected, its ending observation window is appended to chronological memory, which retains at most K recent events and otherwise remains unchanged.The newly stored window is available from the next frame; the experiments use K=1 for lamp and bottle tasks and K=3 for blind search.Source: §3.4 · p. 6

3.5 TRAINING THE SKILL PROGRESS PREDICTOR

  • A progress head maps the skill-conditioned representation to predicted progress, trained against annotated targets with squared error.Training begins without sensory event memory, then introduces predicted event windows; progress thresholds trigger memory, using 0.95 for continuous skills and 0.25 for discrete and conditional skills.Source: §3.5 · p. 6, §3.4 · p. 6

4 CONTACT-RICH DATASETS AND TASKS

The three datasets pair contact-rich manipulation with complementary sensing and history requirements, using a synchronized wrist-camera and dual-tactile setup across 429 episodes.

4 CONTACT-RICH DATASETS AND TASKS

  • Lamp replacement and bottle capping combine vision-dominant reaching or alignment with touch-dependent contact skills; blind cube search additionally requires inspection history.The tasks test short-term contact dynamics and longer-term context retained through sparse events.Source: §4 · pp. 6–7
  • Table 1 reports progress-estimation results across visual- or tactile-dominant skills.Source: Table 1 · p. 7
  • The datasets contain 429 episodes: 134 for lamp replacement, 151 for bottle capping, and 144 for blind cube search, with about 10% held out by episode.Demonstrations use synchronized RGB and dual-tactile streams from a wrist camera and two fingertip sensors.Source: §4 · p. 7

5 SKILLFORMER REMEMBERS TASK-RELEVANT EVENTS

SKILLFORMER is evaluated against language-conditioned baselines and ablations for progress estimation and event detection. It improves estimates for contact-dependent skills, uses retained events to distinguish task history, and remains robust to several test-time perturbations.

5 SKILLFORMER REMEMBERS TASK-RELEVANT EVENTS

  • The evaluation queries every skill over held-out episodes, measuring progress with MSE and Pearson r and event timing with Delay@0.9.Comparisons include two reward-model baselines and ablations of tactile input, event tokens, fusion, tactile horizon, and visual features.Source: §5 · pp. 7–8
  • Relative to Vision Only, SKILLFORMER reduces tightening and cap-sliding MSE by 37% and 79%, respectively.On bulb tightening, its correlation is 0.899 versus 0.370 for Robometer∗ and 0.635 for ReWiND∗.Source: §5 · p. 8
  • For bulb tightening, SKILLFORMER detects completion four frames late; for Inspect Box 2, K=3 preserves the positive outcome while w/o Events and K=1 confuse inspections.The paper also reports clearer progress-state separation for Reach Box 2 in the SKILLFORMER token projections.Source: §5 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8
  • Replacing FAF with generic visual pretraining raises align-cap MSE from 6.4 to 12.9, approaching Vision Only’s 18.4.The passage attributes tactile benefit to suitable temporal context and specialized features.Source: §5 · p. 9
  • Under perturbations, SKILLFORMER’s error stays near clean-input levels on bottle capping and lamp replacement, and remains below every ablation on blind search.Event jitter of ±5 frames leaves performance essentially unchanged across all three tasks; blind search is more sensitive to strong hue shifts and blur.Source: §5 · p. 9, Figure 7 · p. 9
  • Closed-loop control remains future work, although the authors report improved progress estimation and event detection across three manipulation tasks.The conclusion suggests progress could provide policy-learning rewards and the representation could condition action generation on observations and retained history.Source: §5 · p. 9

A.1 TASKS

Tasks are decomposed into language-addressable primitive skills, allowing progress estimation to be queried by skill rather than by a fixed long-horizon sequence.

A.1 TASKS

  • Bottle capping, for example, is decomposed into reaching the cap, grasping it, reaching the bottle, aligning, and sliding the cap.Source: §A.1 · p. 12
  • The same representation is reused across task phases by changing only the queried skill description.Source: §A.1 · p. 12
  • Skill-level progress estimation separates primitive objectives from the particular long-horizon task sequence, so skills can compose longer behaviors.Source: §A.1 · p. 12

A.1.1 LAMP REPLACEMENT

Lamp replacement and bottle capping combine visually guided reaching with contact-dependent phases, but their completion cues differ: tightening relies on tactile force, while sliding can end in slip.

A.1.1 LAMP REPLACEMENT

  • Lamp replacement has five skills, and twist completion is marked by increased tactile force once the bulb is fully tightened.A sequence of tactile frames distinguishes normal rotation from terminal tightening; the light-on visual cue is disabled.Source: §A.1.1 · p. 12, Figure 8 · p. 12
  • Both tasks make touch informative during repetitive contact, but lamp twisting uses rising resistance at thread completion while cap sliding tests brief slip detection.Source: §A.1.1 · p. 12, §A.1.2 · p. 12

A.1.2 BOTTLE CAPPING

  • Bottle capping switches from RGB-led reaching and coarse alignment to tactile-dependent grasping and sliding, where contact or slippage marks terminal interaction.The cap may remain visually close to the bottle while its tactile signal changes sharply due to contact and slippage.Source: §A.1.2 · p. 12

A.1.3 BLIND CUBE SEARCH

Blind cube search tests whether progress representations can track tactile object identification across sequential inspections and insertion. The queried skill must remain part of the representation because identical observations can imply different progress for different skills.

A.1.3 BLIND CUBE SEARCH

  • The robot inspects candidate boxes, withdraws after a distractor, and carries a detected cube to insert it into a puzzle box.The hidden cube and distractors are distinguished by tactile deformation patterns because the object is not visible.Source: §A.1.3 · p. 12
  • Table 5 defines the skills used for progress supervision, while Table 4 reports blind cube search results by skill..Source: Table 5 · p. 13
  • Progress depends on both the observation and the queried skill, so the same RGB and tactile input can indicate different progress values.A frame near the bottle opening can mean high progress for reach bottle but low progress for slide cap.Source: §A.1.3 · p. 13
  • SKILLFORMER keeps the skill description as a token alongside RGB, tactile, event-memory, and output tokens for mutual updating through self-attention.This preserves explicit skill conditioning when adjacent skills have visually similar observations or interpret touch differently.Source: §A.1.3 · p. 13

B EXPERIMENTS DETAILS

The experiments compare models that isolate language conditioning, temporal horizon, touch, and event memory. They cover three manipulation tasks and report blind cube search progress results by skill.

B EXPERIMENTS DETAILS

  • The comparison isolates language conditioning, temporal horizon, tactile sensing, and event memory; Table 4 presents blind cube search results across skills.The figure and table passages identify task and result scope but do not provide unseen table values.Source: B EXPERIMENTS DETAILS · p. 13, Table 4 · p. 13
  • Training uses batch size 45 for 100 epochs, AdamW at 10^-4, and pretrained-encoder fine-tuning at 10^-5.Transformer models use hidden dimension 512, 8 attention heads, and 4 layers; active-skill sampling uses α = 0.8.Source: B EXPERIMENTS DETAILS · p. 13
  • Figure 9 depicts lamp replacement, bottle capping, and blind cube search as primitive-skill sequences with distinct tactile patterns and needs for sensory history or interaction memory.Its orange-bordered insets show egocentric RGB views, and red-bordered insets show tactile readings from the two fingers.Source: Figure 9 · p. 14

B.1 BASELINES EXPLAINED

The baselines vary language conditioning, observation horizons, tactile encoding, and event memory to test their roles in progress estimation. Two pretrained progress models are also adapted to the lamp and bottle demonstrations.

B.1 BASELINES EXPLAINED

  • The FiLM baseline modulates visual and tactile features with the queried skill, using the main model’s RGB and tactile horizons but no event memory.It tests feature-wise affine language modulation against joint token attention over language, sensory inputs, and memory.Source: §B.1 · p. 14
  • SKILLFORMER (h=2) restricts both RGB and tactile inputs to two frames while retaining tactile inputs and event memory.It tests whether modality-specific horizons help, particularly for tactile patterns that emerge over multiple frames.Source: §B.1 · p. 14
  • The Vision-only baseline removes both tactile streams and estimates progress from RGB observations and the queried skill.It tests visual-only performance on contact-rich skills whose terminal conditions involve force, slippage, or tactile shape.Source: §B.1 · p. 14
  • The No FAF encoder baseline replaces the tactile-specialized encoder with generic ImageNet-pretrained DINOv2 while keeping the architecture and tactile inputs.This comparison tests whether contact-aware tactile features matter for reliable progress estimation.Source: §B.1 · p. 14
  • The No event memory baseline removes memory tokens but keeps current RGB, tactile, and language tokens unchanged.It tests whether terminal observations from prior skills help resolve ambiguity near transitions.Source: §B.1 · p. 14
  • Robometer-4B is adapted with LoRA on lamp replacement and bottle capping demonstrations, using per-frame subtask progress labels and trained progress, success, and preference heads.Training uses 48-frame samples and selects the checkpoint with the best held-out progress correlation.Source: §B.1 · p. 15
  • ReWiND is trained jointly on the task demonstrations and Open X-Embodiment, with video rewinding on 80% of training clips and mismatched-instruction negatives on 20%.Clips contain up to 16 frames; training runs for 20,000 steps and selects by held-out per-frame correlation.Source: §B.1 · p. 15

C FURTHER DISCUSSIONS ON ROBUSTNESS

The robustness tests perturb appearance, geometry, temporal sampling, and access to touch or event memory. Their effects vary by task: contact dynamics matter on assembly tasks, while search progress depends strongly on retained interaction history.

C FURTHER DISCUSSIONS ON ROBUSTNESS

  • At half brightness, MSE changes from 3.4 to 3.8 for lamp replacement and 4.6 to 4.7 for bottle capping, but rises from 15.1 to 25.3 for blind search.The passage associates this difference with greater visual-appearance sensitivity when interpreting search interactions and spatial context.Source: C FURTHER DISCUSSIONS ON ROBUSTNESS · p. 15
  • At 108° hue shift, lamp and bottle MSE remain 4.2 and 5.1, while blind-search MSE rises to 54.4.This tests color dependence while preserving spatial arrangement; the search result alone does not establish that color identifies boxes.Source: C FURTHER DISCUSSIONS ON ROBUSTNESS · p. 15
  • At 15 px blur, bottle MSE stays near clean performance at 4.7 versus 4.6, while lamp rises from 3.4 to 6.8 and blind search from 15.1 to 54.4.SKILLFORMER’s bottle result contrasts with FiLM’s bottle MSE increase from 11.5 to 30.9 at 7 px.Source: C FURTHER DISCUSSIONS ON ROBUSTNESS · p. 15
  • Frame reversal raises lamp MSE from 3.4 to 6.6 and bottle MSE from 4.6 to 11.7, compared with blind search from 15.1 to 16.2.Reversal preserves the frames but changes their order, supporting the importance of ordered contact dynamics on assembly tasks.Source: C FURTHER DISCUSSIONS ON ROBUSTNESS · p. 15
  • Bottle MSE rises from 4.6 to 68.4 under H-flip, then falls to 12.4 with tactile-stream swapping; lamp MSE rises from 3.4 to 8.9, unchanged by swapping.H-flip mirrors camera and tactile images while keeping sensor slots; image transformations do not guarantee physically equivalent mirrored interactions or identify a particular attention mechanism.Source: C FURTHER DISCUSSIONS ON ROBUSTNESS · pp. 15–16
  • Tactile blackout raises lamp MSE from 3.4 to 17.8 and bottle MSE from 4.6 to 63.4, while blind-search MSE changes from 15.1 to 14.2.Blackout removes touch from current and retained observations and measures sensitivity to removing touch at deployment, not retrained performance.Source: C FURTHER DISCUSSIONS ON ROBUSTNESS · p. 16
  • Blind-search MSE rises from 15.1 to 171.4 when event memory is blacked out, while lamp and bottle MSE rise to 6.4 and 13.2, respectively.Search requires remembering inspection outcomes because similar current views can belong to different branches; this test removes history at deployment rather than training without it.Source: C FURTHER DISCUSSIONS ON ROBUSTNESS · p. 16
  • Perturbing event timestamps by ±5 frames changes performance little across tasks, whereas memory benefits do not prevent failures when geometric perturbations disrupt learned sensory relationships.Each memory entry contains an observation window, which may retain overlapping observations under nearby timestamps; the tolerance claim concerns small offsets, not missing or incorrect events.Source: C FURTHER DISCUSSIONS ON ROBUSTNESS · p. 16

D LIMITATIONS.

The evaluation focuses on representation quality through progress estimation and assumes predefined language-described skills with annotated boundaries. Its robustness findings are limited to unseen episodes in three task domains on one sensing platform, not transfer to new objects, environments, embodiments, or sensors.

D LIMITATIONS.

  • SKILLFORMER is evaluated for representation quality through progress estimation, not for downstream control or task success.Measuring downstream policy improvements remains an important next step.Source: D LIMITATIONS · p. 16
  • The formulation assumes a predefined graph of language-described skills and annotated skill boundaries for progress supervision.Future work should learn skill segmentation and progress labels with weaker supervision.Source: D LIMITATIONS · p. 16
  • Experiments use one sensing platform and unseen episodes from three collected task domains, establishing robustness only within this setting.Transfer to unseen object categories, environments, embodiments, and sensor configurations remains unestablished.Source: D LIMITATIONS · p. 16

Related papers

Loading 2609.38494v1…