Source-linked AI summary
ConflictVLA-Bench: Benchmarking Behavioral Responses of Vision-Language-Action Models to Premise Conflicts
Liyu Hou, Yuan Wu, Yi Chang
TL;DR
Task competence benchmarks generally assume valid instructions and scenes, while failure under a premise conflict cannot distinguish disengagement from continued pursuit followed by execution error. ConflictVLA-Bench pairs conflict and premise-consistent rollouts and evaluates outcomes alongside execution processes. Across eight VLAs, conflict failures often retain target-directed behavior and early trajectory structure, so terminal failure alone does not establish disengagement or refusal.
Problem
Most VLA benchmarks measure completion when instructions and scenes provide valid premises, while conflict-task failure alone cannot show whether a policy stopped pursuing its objective.
Source: §I · p. 1Method
ConflictVLA-Bench pairs conflict rollouts with same-model premise-consistent references and uses outcome and process evidence to evaluate behavior after task premises fail.
Source: §I · p. 2Results
Across eight VLAs, mean Early20 trajectory similarity exceeds 0.650 in every model’s matched successful-reference, failed-conflict cases, and Failed Persistence recurs.
Source: §I · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
While Vision-Language-Action (VLA) models perform strongly on manipulation tasks, their responses to invalid task premises remain underexplored. Existing evaluations of premise conflicts often focus on terminal task outcomes, yet task failure alone cannot distinguish behavioral disengagement from continued pursuit followed by an execution error. We call the latter pattern Failed Persistence. To study this phenomenon, we introduce ConflictVLA-Bench, which pairs conflict rollouts with premise-consistent reference rollouts and evaluates both outcomes and execution processes. Built on LIBERO, the benchmark contains 2,826 prompt-conditioned conflict tasks spanning four conflict families, four structural configurations, and two prompt conditions. Across all eight VLAs, invalid premises reduce original goal completion by at least 17.3 percentage points, with the reduction reaching 56.2 percentage points for OpenVLA. Crucially, even when models succeed on premise-consistent tasks and fail on their matched conflict tasks, they often continue to approach the original targets, retain early trajectory structure, and show limited action magnitude suppression. Failed Persistence therefore recurs across the evaluated models. Explicit premise checking does not consistently produce selective and coordinated behavioral changes. These findings show that terminal failure alone establishes neither behavioral disengagement nor refusal and that outcomes alone are insufficient for VLA evaluation. Experimental data and additional details are available on the project page: https://github.com/EmbodiedAISurvey/ConflictVLA-Bench
I. INTRODUCTION
Valid-premise task competence does not establish that a VLA will stop pursuing an objective when its premises become invalid. ConflictVLA-Bench evaluates this response using matched rollouts and both outcome and process evidence.
I. INTRODUCTION
- Failure to complete a conflict task may reflect continued pursuit followed by execution error, so terminal failure alone does not establish disengagement or refusal.Possible later errors include perception, grounding, planning, or control errors.Source: §I · p. 2
- Failed Persistence names incomplete task execution accompanied by behavior still directed toward the invalid objective, such as target approach or similar early trajectories.Diagnosing it requires process evidence, including target-directed behavior, trajectory retention, and action suppression.Source: §I · p. 2
- ConflictVLA-Bench pairs each conflict rollout with a same-model reference for the same base task, initial state, and prompt condition, and diagnoses failures only when the reference succeeds.The protocol evaluates observable behavior and does not infer internal recognition or understanding of the conflict.Source: §I · p. 2
- The LIBERO-based benchmark contains 2,826 conflict tasks spanning four conflict families, four structural configurations, and two prompt conditions.The prompt conditions differ in whether an explicit instruction asks the model to check task premises.Source: §I · p. 2
- Mean Early20 trajectory similarity exceeds 0.650 for every model among matched cases where the reference succeeds and the conflict task fails, while Failed Persistence recurs across models.Models also often approach the original target and show limited action suppression; Failed Persistence does not describe every failure.Source: §I · p. 2
- Across eight representative VLAs, Failed Persistence recurs, while explicit premise-checking instructions do not consistently produce selective and coordinated behavioral changes.These findings are reported alongside lower reference-goal completion under conflict conditions.Source: §I · p. 2
II. RELATED WORK
Prior VLA benchmarks measure competence, robustness, diagnosis, and safety under complementary settings. ConflictVLA-Bench focuses on closed-loop behavior when a required task premise fails, using matched references to distinguish terminal outcomes from observable pursuit.
A. VLA Benchmarks and Diagnostic Evaluation
- ConflictVLA-Bench complements benchmarks of competence, robustness, diagnosis, and safety by testing reference-task behavior after a required premise becomes invalid.Its paired closed-loop evidence avoids interpreting task success or failure in isolation.Source: A. VLA Benchmarks and Diagnostic… · p. 2
B. Premise Evaluation and Conflict Response
- Prior premise-response studies use varied interfaces, including feasible counterfactual instructions, controlled contradictions, and explicit clarification or correction before acting.ConflictVLA-Bench instead evaluates general VLA actions without requiring a standardized refusal output.Source: B. Premise Evaluation and Conflict… · pp. 2–3
- Figure 2 depicts four conflict families, four structural configurations, two prompt conditions, matched rollouts, and joint use of outcome and process evidence.The matched-rollout and joint-evidence design is the key comparison for diagnosing behavioral disengagement versus Failed Persistence.Source: Figure 2 · p. 3
- Matched premise-consistent rollouts qualify demonstrated capability, while outcomes, target approach, trajectory retention, and action suppression characterize observable Failed Persistence.The protocol does not establish latent conflict understanding, language refusal, or general safety.Source: B. Premise Evaluation and Conflict… · p. 3
III. CONFLICTVLA BENCHMARK
The benchmark formalizes premise conflicts and pairs each conflict rollout with a matched premise-consistent rollout from the same task, initial state, and prompt condition. This pairing distinguishes failure to complete the original goal from evidence that behavior has disengaged from it.
A. Task Definition
- The rollout records the closed-loop sequence of states, observations, and executed actions across H steps.At each step, the policy acts from the instruction and its available visual and proprioceptive interaction history.Source: A. Task Definition · p. 3
- A task is premise-conflicting when no feasible execution can satisfy its language premises, initial-scene facts, and physical constraints together.Otherwise, it is premise-consistent.Source: A. Task Definition · p. 3
- Matched conflict and premise-consistent rollouts use the same underlying task, initial state, and prompt condition; the reference rollout is evaluator-only.The reference is not provided to the policy during conflict execution.Source: A. Task Definition · pp. 3–4
- A qualified conflict failure requires success on the matched premise-consistent task and failure to complete the original goal under conflict, but does not establish behavioral disengagement.Observable refusal requires demonstrated premise-consistent capability and coordinated suppression of behavior associated with the original task.Source: A. Task Definition · p. 4
B. Task Design
ConflictVLA-Bench extends LIBERO by invalidating premises in executable tasks while preserving the initial scene and retaining each original task as a matched reference. Its conflict suite covers four sources of premise invalidity.
B. Task Design
- The benchmark extends LIBERO by invalidating one or more execution premises while preserving the initial scene and retaining the original task as its matched premise-consistent counterpart.The shared environment and task protocol support comparisons across model families.Source: B. Task Design · p. 4
1) Conflict Suite:
- The four families target incompatible instructions, invalid object references, false initial spatial relations, and physically unrealizable object-action or object-destination combinations.Spatial-relation and physical-feasibility conflicts keep referenced entities present and individually groundable; their invalidity lies in the relation or requested interaction.Source: 1) Conflict Suite · p. 4
2) Structural Configurations:
The benchmark varies conflict structure along two factors: the number of manipulation steps and the number of conflict factors. Its four configurations combine these factors without imposing an empirical difficulty ordering.
2) Structural Configurations:
- L1 combines one manipulation step with one localized conflict factor, identifiable without relying on additional subgoals.Source: 2) Structural Configurations · p. 4
- L2 combines one manipulation step with multiple conflict factors affecting the same manipulation requirement.Source: 2) Structural Configurations · p. 4
- L3 combines multiple ordered manipulation steps with one conflict localized to a subgoal, while the remaining components preserve a plausible execution sequence.Source: 2) Structural Configurations · p. 4
- L4 combines multiple ordered steps with interacting conflict factors distributed across task components, so evidence and dependencies may span a longer sequence.The levels describe construction structure, not an assumed empirical difficulty ordering.Source: 2) Structural Configurations · p. 4
3) Prompt Conditions:
The benchmark compares standard prompts with explicit premise-checking prompts and controls for prompt effects using matched premise-consistent tasks. Its construction pipeline injects and validates conflicts before evaluating outcomes and processes.
3) Prompt Conditions:
- Under P0, the model receives the conflicting instruction in the standard task format, without being told of a conflict or prompted to suppress execution.P0 is the primary condition for measuring responses without prompt intervention.Source: 3) Prompt Conditions · p. 4
- Under P1, the prompt asks the model to check premise validity and suppress execution if it detects a conflict.Source: 3) Prompt Conditions · p. 4
- Both prompt conditions are applied to matched premise-consistent tasks, allowing comparison of conflict behavior while controlling for prompt-related changes to valid-task execution.Source: 3) Prompt Conditions · p. 4
4) Construction and Validation Pipeline:
- The construction pipeline parses the LIBERO instruction, goal specification, and scene configuration, then extracts entities, goal predicates, spatial constraints, and affordance constraints.Source: 4) Construction and Validation Pipeline · pp. 4–5
- It selects a conflict operator by family and structural configuration, injects the conflict while preserving the scene, then pairs each case with its valid counterpart under P0 and P1.Source: 4) Construction and Validation Pipeline · p. 5
- Automatic checks and two-stage manual validation remove invalid or unobservable conflicts and cases whose matched counterparts are no longer valid.The process retained 1,413 base cases, yielding 2,826 prompt-conditioned instances.Source: 4) Construction and Validation Pipeline · p. 5
C. Evaluation Metrics
- The evaluation reports complementary outcome and process measures using the evaluation set and outcome indicators defined earlier.Source: C. Evaluation Metrics · p. 5
1) Outcome Measures:
Outcome measures compare premise-consistent success with completion of the original goal under conflict, while process measures characterize target approach, trajectory similarity, and action-magnitude change. These process signals are interpreted jointly rather than as individual evidence of disengagement.
1) Outcome Measures:
- PCSR measures premise-consistent success, RGSR measures completion of the original goal under conflict, and ΔSR is their difference.For capability-qualified cases, FR measures failure among the qualified subset.Source: 1) Outcome Measures · p. 5
2) Process Measures:
- TAR compares target approach across matched rollouts and is computed only when their evaluator-defined target sets match and target distances are computable.Source: 2) Process Measures · p. 5
- TS compares end-effector trajectories using dynamic time warping with Euclidean local costs; Early20 TS evaluates the initial position and positions after up to 20 executed actions.TS uses complete trajectories; the stated DTW scale is σ = 0.10 m.Source: 2) Process Measures · p. 5
- AMS measures reduction in action magnitude relative to the matched premise-consistent rollout, using six continuous motion dimensions and excluding the gripper command.TAR, TS, and AMS characterize complementary observable behaviors and are interpreted jointly rather than individually as evidence of disengagement.Source: 2) Process Measures · p. 5
A. Experimental Setup
The evaluation covers eight representative VLA architectures on prompt-conditioned conflict tasks and matched premise-consistent tasks. Each conflict rollout is paired with a reference rollout under the same base task, initial state, and prompt condition.
A. Experimental Setup
- Eight representative VLAs span tokenized autoregressive, continuous-action, world model-based, and generalist robot foundation model families.The evaluated models are OpenVLA, π0-FAST, OpenVLA-OFT, π0, π0.5, VLA-JEPA, UniVLA, and GR00T N1.7.Source: A. Experimental Setup · p. 5
- Each model is evaluated on 14,130 conflict rollouts and 400 premise-consistent rollouts, for 14,530 total rollouts per model.The conflict tasks and premise-consistent tasks are each evaluated across five initial states.Source: A. Experimental Setup · p. 5
- Conflict rollouts are paired with reference rollouts from the same base task, initial state, and prompt condition; the main tables report P0 unless otherwise specified.P1 is analyzed separately in RQ4; Table I covers outcome and process measures (%) for eight VLAs.Source: A. Experimental Setup · p. 6, Table I · p. 6
B. Overall Results
Across eight VLAs, invalid premises reduce reference-goal completion, yet many conflict rollouts retain target approach and early trajectory structure. Outcome changes therefore do not by themselves establish behavioral disengagement.
B. Overall Results
- Across eight VLAs, invalid premises expose a mismatch between reduced reference-goal completion and continued closed-loop behavior.Conflict rollouts often retain substantial target approach and early trajectory structure, with generally limited action suppression.Source: B. Overall Results · p. 6
C. RQ1: Do Invalid Premises Change Reference Goal Outcomes?
- 17.3–56.2 percentage-point success-rate drops accompany invalid premises, with OpenVLA falling from 78.7% PCSR to 22.5% RGSR.π0.5, OpenVLA-OFT, and VLA-JEPA retain RGSR values of 77.2%, 71.1%, and 69.1%, respectively.Source: C. RQ1: Do Invalid Premises Change… · p. 6
D. RQ2: How Does Closed-Loop Execution Change under Premise Conflicts?
- Across matched P0 episodes, TAR ranges from 0.765 to 0.896 and Early20 TS from 0.784 to 0.886 despite conflicts.OpenVLA retains TAR of 0.765 and Early20 TS of 0.784 while its RGSR falls to 22.5%.Source: D. RQ2: How Does Closed-Loop Execution… · p. 6
- Early20 TS exceeds TS for every model, while AMS ranges from 0.067 to 0.276, indicating limited and uncoordinated action suppression.The trajectory measures are consistent with stronger early similarity followed by later divergence, but do not identify its internal cause.Source: D. RQ2: How Does Closed-Loop Execution… · p. 6
E. RQ3: Does Reference Goal Failure Imply Behavioral Disengagement?
Capability-qualified conflict failures often retain target-directed behavior and early trajectory structure, while response patterns vary by conflict type. Explicit premise-checking prompts do not consistently produce coordinated behavioral changes.
E. RQ3: Does Reference Goal Failure Imply Behavioral Disengagement?
- 23.0%–76.7% failure rates among capability-qualified pairs coexist with mean TAR of 0.586–0.741 and mean Early20 TS of 0.653–0.845.Episode-level distributions are broad: upper-quartile TAR ranges from 0.898 to 0.985, so not every failure should be labeled persistence.Source: E. RQ3: Does Reference Goal Failure… · p. 7
F. RQ4: How Do Conflict Type and Explicit Premise Checking Affect Behavioral Response?
- Instruction and Grounding conflicts retain higher average RGSR, TAR, and TS than Spatial and Physical conflicts, whose average ∆SR values are 38.7 and 45.0 percentage points.The ordering is not universal across models, and AMS remains low across all four conflict types.Source: F. RQ4: How Do Conflict Type and… · p. 7
- No model changes in the intended direction across all four process measures under explicit premise-checking prompts.For OpenVLA and π0-FAST, RGSR decreases by 5.4 and 20.8 percentage points while PCSR decreases by 20.2 and 38.8 percentage points, respectively.Source: F. RQ4: How Do Conflict Type and… · pp. 7–8, Figure 4 · p. 7
V. CONCLUSIONS
- Even among failed conflict rollouts paired with successful premise-consistent rollouts, mean early trajectory similarity exceeds 0.650 for every model.The study concludes that terminal failure alone does not establish refusal and that evaluation needs both outcomes and execution processes.Source: §V · p. 8