JevVLA: Jev-Inspired Energy-Based Adaptation for Vision-Language-Action Policies

1 Introduction

• The adaptation problem is to improve action decisions while retaining a pretrained policy’s action-generation capabilities, since plausible actions may still fail in a particular scene.Full-parameter fine-tuning can require substantial computation and additional task-specific data.Source: §1 · p. 2
• JevVLA draws on Jev’s structured, probabilistic decision interface but independently develops the robotic action energy, training objectives, and gradient-guidance mechanism.The design separates action proposal by the base policy from preference evaluation by the adapter.Source: §1 · p. 3
• Held-out task success rises over matched base-policy controls by 1.28, 2.31, 1.60, 1.52, and 1.28 percentage points on MetaWorld MT50, VIMA, LanguageTable, LIBERO Plus, and CLIPort, respectively.Evaluations cover both candidate selection and continuous control.Source: §1 · p. 3
• One conditional scalar energy ranks discrete proposals by its values and corrects or guides continuous actions through its action gradients.The base policy remains frozen; each benchmark has fixed encodings and independently trained weights, with an identical 79,817-parameter core.Source: §1 · p. 3

2.1 Vision–language–action models

• JevVLA separates action proposal from evaluation, using one conditional scalar module for candidate selection and continuous guidance across differing policy interfaces.Earlier VLA work spans pick-and-place decisions, motion control, action chunks, and downstream adaptation methods.Source: §2.1 · pp. 3–4

2.2 Robot preference learning

• JevVLA turns branch outcomes, corrected actions, and demonstration distances into distinct ranking or gradient supervision signals for a separate energy adapter.Prior preference-learning work uses feedback at different time scales, including trajectory quality, terminal success, and local corrections.Source: §2.2 · p. 4

2.3 Energy-based action selection

• Candidate selection is limited by the proposal distribution and candidate set, while correctly ranking observed actions may not yield reliable local gradients between them.JevVLA assigns ranking and gradient supervision to the same energy architecture when adapting frozen policies.Source: §2.3 · p. 4

2.4 Inference-time action guidance

• JevVLA uses a differentiable scalar core’s values to compare candidates and its action gradients for bounded continuous correction, with gradient insertion and scaling set by the base-policy interface.Prior guidance methods vary with their signals, retrieval, and generator structures.Source: §2.4 · pp. 4–5

2.5 Structured decisions with Jev

• Jev’s structured decision perspective motivates explicit candidate preferences, while JevVLA independently develops the robot-specific energy adapter and does not reproduce Jev’s internal model.The shared core architecture is reused across benchmarks, but weights are trained separately.Source: §2.5 · p. 5

3.1 Structured action preferences

• Fixed context and action encoders connect frozen-policy proposals to the adapter, while the base policy and encoder parameters remain fixed during energy training.The energy represents a learned preference potential, not physical work.Source: §3.1 · p. 5, §3 · p. 5

3.2 Conditional energy core

• The conditional energy combines a quadratic potential with a nonlinear scalar residual, and lower energy indicates a preferred action.The quadratic component supplies coordinate-specific curvature and couples action coordinates through a low-rank term; the residual captures preferences the quadratic cannot express.Source: §3.1 · p. 5, §3.2 · p. 6

3.3 Learning action preferences

• Preference supervision can use executed outcomes, expert corrections, or demonstration-derived action preferences, while continuous-action gradients are trained from local correction targets.Gradient targets may be evaluated at perturbed actions or intermediate generator states; all parameter updates affect the scalar energy, not the frozen base policy.Source: §3.3 · pp. 7–8

3.4 Acting on the learned preference

• For finite candidates, JevVLA scores valid proposals relative to Direct and executes the maximum-score candidate; for continuous actions, it applies bounded energy-gradient corrections or guides intermediate generation.A challenger replaces Direct only when its energy advantage exceeds the margin; iterative guidance reevaluates the energy and gradient at each intermediate action.Source: §3.3 · p. 7, §3.4 · p. 8
• One energy core serves both interfaces: its values rank discrete proposals, while its gradients correct continuous actions or guide their generation.Each benchmark uses the same architecture with independently learned weights.Source: §3 · p. 5, Figure 2 · p. 6, §3.4 · p. 9

4.1 Experimental setup

• The 79,817-parameter core keeps base policies and encoders frozen, with validation-based selection and separate held-out testing.Available supervision includes execution feedback, expert corrections, or demonstration preferences; CLIPort uses strict complete-task success.Source: §4.1 · p. 9

4.2 Overall performance

• JevVLA raises held-out task success by 1.28, 2.31, 1.60, 1.52, and 1.28 pp on MetaWorld MT50, VIMA, LanguageTable, LIBERO Plus, and CLIPort, respectively.The comparisons use the corresponding frozen-policy controls and separately trained adapters.Source: §4.2 · pp. 9–10, Table 1 · p. 10

4.3 Where adaptation helps

• Task-level gains are uneven: VIMA improves on 6/13 families, MetaWorld on 18/50 tasks, CLIPort on 11/18 variants, and LanguageTable on 3/5 families.LIBERO Plus improves in six of seven perturbation categories; camera viewpoint changes gain 3.56 pp, while background texture changes lose 1.95 pp.Source: §4.2 · p. 11, §4.3 · p. 12

4.4.1 Learning to select actions

• On CLIPort’s fixed candidate set, JevVLA reaches 35.50%, above the nearest-alternative rule at 33.33% and minimum-norm rule at 32.89%.Uniform random selection has an expected success rate of 34.22%; candidate generation is fixed across these comparisons.Source: §4.4.1 · p. 13, Table 3 · p. 13

4.4.2 Structure of the energy function

• Removing the quadratic term gives 33.39%, the largest deficit among the three structural variants; removing the residual gives 34.17%, and removing low-rank coupling gives 34.67%.The ablations retain the frozen base policy and candidate generation; variants changing the training objective also change the combined structural and training effects.Source: §4.4 · p. 12, §4.4.2 · p. 13

4.4.3 Learning contextual preferences

• Random initialization, shuffled preference labels, and shuffled input features score 33.67%, 33.61%, and 33.28%, respectively, below Full.Removing global visual features gives 34.22%, equal to Direct; Full yields 82 rescues and 59 harms, for 23 net additional successes.Source: §4.4.3 · pp. 13–15, Figure 6 · p. 14

4.5 Comparison methods

• On MetaWorld, JevVLA records 1,928/2,500 successes, ahead of Direct, the linear probe, and RAS; on CLIPort, its 639/1,800 exceeds TabM’s 627/1,800.The MetaWorld probe still leads on basketball and coffee pulling; UA-CLIPort has no matched online Direct run for a paired comparison.Source: Table 6 · p. 16, §4.5 · p. 17, Table 7 · p. 17

5 Conclusion

• The same scalar energy ranks discrete proposals and uses action gradients to refine continuous actions or guide their generation, with benchmark-specific weights.The core has 79,817 trainable parameters.Source: §5 · p. 17
• Across five manipulation benchmarks, JevVLA improves aggregate task success over the corresponding frozen policies.The CLIPort ablations examine energy structure, preference supervision, and scoring inputs; execution examples illustrate corrections and candidate ranking.Source: §5 · p. 17

A Reproducibility details

• The width-128, rank-8 scalar energy has 79,817 trainable parameters, initialized and trained separately for each benchmark.Frozen policies, feature encoders, and benchmark-specific input/output codecs are outside this parameter count.Source: A Reproducibility details · p. 23
• Supervision varies by benchmark, using executed outcomes, expert corrections or alignment, scripted-oracle signals, or proximity to demonstrations.MetaWorld combines executed branch returns with scripted expert corrections; LIBERO Plus uses future-action proximity to official demonstration trajectories.Source: A Reproducibility details · p. 23
• For the selected CLIPort comparison, 1,032 examples fit the adapter, 258 select the checkpoint and margin, and 1,800 test episodes provide success counts.The test set has 100 examples for each of 18 task variants.Source: A Reproducibility details · p. 23
• Adapters use AdamW with weight decay 10^-4 and a gradient-norm cap of 1; optimization excludes the frozen base policies.Parenthesized epoch counts identify checkpoints selected after validation.Source: A Reproducibility details · p. 23
• The target-distribution temperature τy and native-score scale τ0 have distinct roles, while mixture masses ρk are normalized per anchor from measured evidence.CLIPort and LIBERO Plus have no native proposal prior; discrete selection does not use η, σ, or α, and continuous correction does not use margin m.Source: A Reproducibility details · p. 23
• Experiments used one server with eight A100 GPUs and two Xeon CPUs; small adapters fit on CPU, and no adapter required multi-GPU optimization.MetaWorld energy continuation used one A100.Source: A Reproducibility details · p. 23

B Task-level results

• The appendix tables report full task breakdowns, and task names are shortened by replacing separators with spaces.Source: B Task-level results · p. 23

B.1 VIMA L1

• Table 9 reports task counts and success rates for VIMA L1.Source: Table 9 · p. 24

• Table 10 presents CLIPort task counts and success rates.Source: Table 10 · p. 24

B.3 MetaWorld MT50

• The MetaWorld MT50 table presents task counts and success rates.Source: Table 11 · p. 25

B.4 LanguageTable

• The LanguageTable table presents task counts and success rates.Source: Table 12 · p. 26

B.5 LIBERO Plus: original tasks

• LIBERO Plus original tasks total 10,030; reported success rates are 78.63 and 80.15, with +1.52.Source: Table 13 · p. 26, §B.5 · p. 27

B.6 LIBERO Plus: perturbations

• The perturbations table reports task counts and success rates.Source: Table 14 · p. 27

B.7 LIBERO Plus: suites

• The suites table reports task counts and success rates.Source: Table 15 · p. 28

• Table 16 presents MetaWorld MT50 comparison methods by task.Source: Table 16 · p. 28
• Table 17 presents CLIPort online and learned comparison methods by task.Source: Table 17 · p. 29
• Table 18 presents CLIPort tabular model architectures by task.Source: Table 18 · p. 29