PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Conditions for Diffusion models in Robotic Control

Heeseong Shin, Byeongho Heo, Dongyoon Han, Seungryong Kim, Taekyung Kim

arXiv:2510.15510v2cs.CVcs.RO

TL;DR

Frozen pre-trained visual representations can remain task-agnostic, while textual conditions that work in other vision domains provide minimal or negative gains in control. ORCA conditions a pre-trained diffusion model with learnable task and visual prompts, without fine-tuning the model. It achieves state-of-the-art results across robotic control tasks.

  • Problem

    Frozen visual representations remain task-agnostic, and selecting a suitable representation for each control task can require cumbersome task-by-task inspection.

    Source: §1 · pp. 1–2

  • Method

    ORCA conditions a pre-trained diffusion model with learnable task prompts and frame-specific visual prompts, learned during downstream policy training.

    Source: §1 · p. 2, §5 · p. 5

  • Results

    ORCA establishes state-of-the-art performance across all 12 tasks in three evaluation suites, outperforming all baselines by a significant margin.

    Source: §6.4 · p. 7

  • Takeaways & Limitations

    The authors conclude that properly conditioned diffusion models have potential to provide task-adaptive visual representations for robotic control.

    Source: §7 · p. 8

    The findings are based on Stable Diffusion v1.5 and may not directly apply to DiT or video diffusion architectures.

    Source: C. Limitations · p. 12

Abstract

from arXiv · show

While pre-trained visual representations have significantly advanced imitation learning, they are often task-agnostic as they remain frozen during policy learning. In this work, we explore leveraging pre-trained text-to-image diffusion models to obtain task-adaptive visual representations for robotic control, without fine-tuning the model itself. However, we find that naively applying textual conditions - a successful strategy in other vision domains - yields minimal or even negative gains in control tasks. We attribute this to the domain gap between the diffusion model's training data and robotic control environments, leading us to argue for conditions that consider the specific, dynamic visual information required for control. To this end, we propose ORCA, which introduces learnable task prompts that adapt to the control environment and visual prompts that capture fine-grained, frame-specific details. Through facilitating task-adaptive representations with our newly devised conditions, our approach achieves state-of-the-art performance on various robotic control benchmarks, significantly surpassing prior methods.

1. Introduction

ORCA learns task-adaptive visual representations by conditioning a pre-trained diffusion model with task and frame-specific visual prompts, without fine-tuning the diffusion model. It achieves state-of-the-art performance on robotic control tasks, while ordinary textual conditions yield minimal or negative gains.

1. Introduction

  • Frozen pre-trained visual representations remain task-agnostic, making it difficult to know which representation will suit each control task.Fine-tuning can also overfit to specific imitation-learning scenes and reduce generalization.Source: §1 · pp. 1–2
  • Generated text captions yield minimal gains and sometimes reduce control performance, unlike their reported benefits in other vision tasks.The authors attribute the difference to the domain gap between web-collected training images and specialized control environments.Source: §1 · p. 2
  • ORCA replaces text prompts with learnable task prompts and frame-specific visual prompts, trained end-to-end with behavior cloning.The visual prompts incorporate information from each frame to capture fine-grained visual state.Source: §1 · p. 2
  • ORCA, a task-adaptive framework leveraging conditioned diffusion models for robotic control, achieves state-of-the-art performance in robotic control tasks [57], surpassing VC-1.Source: §1 · p. 2

2. Related Work

Prior robotic-control representations often remain frozen and task-agnostic, while diffusion representations have shown strength on varied vision tasks. ORCA focuses on conditioning diffusion models specifically for robotic control.

Pre-trained visual representations for robotic control

  • Frozen pre-trained visual encoders have advanced robotic manipulation, but related approaches keep representations task-agnostic.Examples include R3M, VIP, MVP, VC-1, and SCR.Source: §2 · p. 2

Diffusion models as pre-trained visual representations

  • Diffusion representations have achieved strong results in several vision tasks, including semantic correspondence, segmentation, and 3D reconstruction.Prior work also used textual or learned conditions for vision tasks, including segmentation and depth estimation.Source: §2 · pp. 2–3
  • ORCA shifts the use of diffusion-model conditions from general image-domain vision tasks to robotic control.The motivation figure contrasts frozen task-agnostic representations with a task-adaptive approach using diffusion conditions.Source: §2 · p. 3, Figure 2 · p. 3

3. Preliminaries

Conditional diffusion models learn to reverse a noising process, with conditions injected into a latent-space denoising network. For control representations, intermediate U-Net features are extracted instead of using its predicted noise.

3. Preliminaries

  • The noised latent z_t is formed from clean data z_0 and Gaussian noise according to a predefined schedule.The cumulative schedule term is ᾱ_t, and the noise is sampled from a standard Gaussian.Source: §3 · p. 3
  • A diffusion model trains a denoiser to predict added noise from a noised latent at timestep t, given condition C.Stable Diffusion uses a U-Net operating in the compressed latent space of an autoencoder.Source: §3 · p. 3
  • Stable Diffusion injects its conditioning signal into U-Net blocks through cross-attention layers.Its latent diffusion process operates in a compressed space learned by a VQGAN autoencoder.Source: §3 · p. 3
  • For visual representations, the image is encoded, noised at a fixed timestep, and passed through the denoising U-Net.The method extracts intermediate feature maps, primarily from earlier U-Net blocks, rather than relying on the network's predicted noise.Source: §3 · p. 3

4. Motivation

Text conditions can be poorly grounded in control environments, and control tasks require conditions that track fine-grained visual changes across frames. The authors therefore motivate task-grounded and frame-sensitive alternatives.

4. Motivation

  • Control tasks use dynamic video streams and require fine-grained visual information about agents and objects, motivating frame-sensitive conditions.Generating detailed text descriptions for every frame is challenging and inherits text-grounding limitations.Source: §4 · p. 4, §4.2 · p. 5

4.1. Exploring textual conditions for robotic control

  • Text conditions produce mixed results: they help Button-press and Reacher but degrade Cheetah-run performance.Button-press attention maps align with relevant image regions, whereas Cheetah-run maps are noisy; the authors attribute text-condition failures to a domain gap between real-world images and simulated control environments.Source: §4.1 · pp. 4–5

4.2. What do we need as conditions in control?

  • Conditions should incorporate visual information from each frame to adapt to changing visual states and capture fine-grained regions.The authors argue against global representations and extra optimization steps for this frame-wise conditioning goal.Source: §4.2 · p. 5

5. ORCA: Conditioning diffusion models for robotic control

ORCA conditions frozen diffusion features with learnable task prompts and frame-level visual prompts. Behavior cloning trains both prompt types alongside the policy.

5. ORCA: Conditioning diffusion models for robotic control

  • ORCA combines task-specific implicit descriptions with frame-level visual information through learnable task and visual prompts, without modifying the diffusion model.The prompts are learned during downstream control training.Source: §5 · p. 5
  • Task prompts are shared across observations and learn to focus on regions critical to the task, such as a button and robot arm or a running agent.The paper visualizes these regions in Button-press and Cheetah-run.Source: §5 · pp. 5–6
  • Visual prompts use dense encoder representations projected through a small convolutional layer to provide localized detail, including distinctions between an agent’s front and back legs.These representations complement the task prompts.Source: §5 · p. 6
  • Behavior cloning trains the policy and both prompt types from observation trajectories and their demonstrated actions.The diffusion-derived visual state representations are input to the policy, which predicts actions.Source: §5 · p. 6
  • ORCA’s prompt modules can be swapped across tasks without modifying the diffusion model, unlike fine-tuning, which requires a tuned model per task and collapses success rates by over 80% versus the frozen counterpart.The authors attribute fine-tuning’s degradation to overfitting limited imitation-learning data.Source: §5 · p. 6
  • The evaluation visualization covers 12 tasks: five each from MetaWorld and DeepMind Control, and two from Adroit.The tasks are evaluated in MuJoCo.Source: Figure 5 · p. 6

6. Experiments

Across 12 tasks in three evaluation suites, ORCA establishes state-of-the-art performance and outperforms all baselines. The experiments also compare conditioning and fine-tuning approaches and analyze prompt components, layer choices, and attention patterns.

6.1. Evaluation suites

  • Evaluation spans 12 tasks across DMC, MetaWorld, and Adroit, including continuous control, Sawyer-arm manipulation, and dexterous-hand tasks.DMC uses normalized scores; MetaWorld and Adroit use the best success rates from online evaluation trials.Source: §6.1 · p. 6

6.2. Implementation details

  • Experiments use Stable Diffusion v1.5 features, DINOv2 visual prompts, and 100 training epochs per task.They use two demonstrations for Adroit and five each for DMC and MetaWorld; proprioception is omitted only for DMC.Source: §6.2 · p. 6

6.4. Main results

  • Across all 12 tasks in three suites, ORCA establishes state-of-the-art performance, significantly outperforming all baselines; CoOp and TADP generally outperform text conditions.The results cover DeepMind Control and MetaWorld in Table 1 and Adroit in Table 2.Source: Table 1 · p. 7, Table 2 · p. 7, §6.4 · p. 7

6.3. Baselines

  • Comparisons include task-agnostic CLIP and VC-1, unconditional SCR, text-conditioned variants, and task-adaptive CoOp and TADP baselines.CoOp adds learnable prefix tokens, while TADP adds style information from a single fixed image.Source: §6.3 · p. 7

6.5. Analysis

  • On Adroit, ORCA improves performance over parameter-efficient fine-tuning at comparable learnable parameter counts; full fine-tuning uses approximately 30× more parameters.Full fine-tuning completely deteriorates SCR's performance.Source: Table 3 · p. 7, §6.5 · pp. 7–8
  • Task and visual prompts can behave differently by task, but using both yields consistent gains; early U-Net layers perform best, with down 1–3 plus mid selected.The component ablation evaluates the prompts on DMC, as does the layer-selection study.Source: Table 4 · p. 8, Table 5 · p. 8, §6.5 · p. 8
  • On Relocate, task prompts attend to the hand and target, while visual prompts shift attention from table to hand across action phases.The visual prompts illustrate frame-varying attention during the pick-up and movement sequence.Source: Figure 6 · p. 8, §6.5 · p. 8

7. Conclusion

ORCA uses learnable task and visual prompts to generate task-adaptive visual representations from text-to-image diffusion models for robotic control. It achieves state-of-the-art performance, underscoring the potential of properly conditioned diffusion models in this setting.

7. Conclusion

  • The paper introduces ORCA, a framework that uses learnable task and visual prompts to generate task-adaptive visual representations for robotic control.The conclusion identifies conventional text prompts as limited and presents ORCA as a conditioning approach for text-to-image diffusion models.Source: §7 · p. 8

A.1. Results with different diffusion backbones

The appendix tests ORCA across diffusion backbones, benchmarks, prompt components, and efficiency, and examines null-condition attention and broader use cases. Results highlight both the approach’s scope and architecture-specific limits.

A.1. Results with different diffusion backbones

  • SVD falls significantly behind SD 1.5 with a three-frame input rather than its native eight, while SD 3.0 underperforms U-Net models with or without text conditions.The released SVD version also lacks text conditioning, so using text or learnable prompts would require additional fine-tuning.Source: §A.1 · p. 9

A.2. Results on LIBERO-Long benchmark.

  • ORCA consistently outperforms baselines on three LIBERO-Long tasks and in shared-policy multi-task training.The multi-task results indicate that visual and task prompts can be learned jointly across tasks.Source: §A.2 · p. 9

A.3. Ablation on the choice of the visual encoder for visual prompt.

  • SigLIP, CLIP, and SD-VAE visual-prompt variants all gain over the no-vision-encoder baseline, though encoder choice varies in performance.SD-VAE results show that competitive performance is possible without an external vision encoder.Source: §A.3 · pp. 9–10, Table C · p. 10

A.4. Ablation on diffusion timesteps

  • Timestep results favor t = 0 overall, although t = 100 or t = 200 helps some tasks while significantly degrading others.Cheetah-run is given as an example of a task where later timesteps lower the overall score.Source: §A.4 · p. 10, Table D · p. 10

A.5. Comparison with stronger pre-trained visual representations

  • DINOv2, SigLIP, and Theia remain underwhelming on several tasks, including Reacher-easy for DINOv2 and Cheetah-run for SigLIP.The authors suggest incorporating ORCA into VLA models as a future research direction.Source: §A.5 · p. 10

A.6. Efficiency comparison

  • Layer selection removes around 500M parameters from the denoising U-Net, bringing its parameter count near the VC-1 encoder.Most parameters added to ORCA are frozen DINOv2 parameters; learnable parameters consist mostly of visual-prompt projection layers.Source: §A.6 · p. 10

A.7. Analysis on the null condition

  • The <bos> token attends consistently to background regions, while <eos> is less reliable at attending to salient objects.The authors attribute <bos> background affinity to text prompts’ foreground descriptions and note that causal text encoding transfers this behavior to the null condition.Source: §A.7 · pp. 10–11

A.8. Further discussions

  • The discussion contrasts specialist agents with billion-scale VLAs, whose adaptation to new embodiments often requires costly fine-tuning.The authors note that general image-text data for conditional diffusion models is easier to collect and scale than robot-interaction data for VLAs.Source: §A.8 · p. 11

B. Further Implementation Details

The appendix documents text-condition examples and baseline prompting schemes, then describes the compression layer used to form state representations.

B.1. Full description of the text conditions

  • The text-condition appendix provides full task descriptions, including a bin-picking caption with a learned style token.The descriptions are generated by Gemini 2.5 Pro; the example also illustrates the CoOp prefix-token format.Source: §B.1 · p. 11

B.2. Details of the baselines

  • The baselines include null-condition SCR, task-name and generated-caption text conditions, CoOp learnable prefix tokens, and TADP’s learned style token.TADP’s single style token represents global visual information rather than frame-wise information.Source: §B.2 · p. 11

B.3. Implementation details of the compression layer

  • A convolutional compression layer maps 2D features to 1D state representations, with compress dim set to 48.The same compression layer is used for compared CLIP and VC-1 baselines.Source: §B.3 · pp. 11–12

C. Limitations

The reported exploration uses pretrained Stable Diffusion v1.5, a U-Net-based diffusion model, and cautions against assuming the findings transfer directly to other architectures.

  • The exploration is based on pretrained Stable Diffusion v1.5, which uses a U-Net architecture.Source: C. Limitations · p. 12
  • The authors caution that the findings may not directly apply to DiT models or video diffusion models.Source: C. Limitations · p. 12
  • Further exploration of DiT and video diffusion models is left for future work.Source: C. Limitations · p. 12

D. Qualitative visualization on robotic control tasks

The qualitative figures compare ORCA with CLIP and VC-1 across DMC, MetaWorld, and Adroit tasks, using task-specific outcome displays.

Related papers

Loading 2510.15510v2…