PaperlayerView exact revision on arXiv

Source-linked AI summary

Refusals That Bend: Measuring and Predicting Task Malleability in Embodied VLM Planners

Leo Y. Lin, Mikhail Kuznetsov, Muslum Ozgur Ozmen, Z. Berkay Celik

arXiv:2609.38971v1cs.CRcs.RO

TL;DR

Existing red-teaming manipulates planner inputs, while safety benchmarks assess hazards in fixed scenes; neither asks whether an existing refusal persists after an ordinary environmental change. The paper tests refusals after inserting everyday objects and analyzes task malleability. At least one object reverses 20.2% of 846 initial refusals, and a proxy identifies malleable tasks 2.4× as often as random selection.

  • Problem

    Prior attacks manipulate planner inputs and safety benchmarks use fixed scenes, leaving untested whether a refusal persists after an ordinary environmental change.

    Source: §1 · p. 2

  • Method

    The authors insert everyday objects into scenes for initially refused tasks, measure refusal reversals as task malleability, and use a small VLM proxy to predict it without querying the target.

    Source: §1 · p. 2

  • Results

    20.2% of 846 initial refusals are reversed by at least one object, and proxy signals identify malleable tasks 2.4× as often as random selection.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A proxy can help reviews concentrate on tasks whose refusals are least likely to hold before the target planner is deployed.

    Source: §5 · p. 10

    No physical robot was operated or experimental plan executed, and the measurements begin with tasks the planner initially refuses.

    Source: §3 · p. 3, ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Embodied vision-language models (VLMs) are increasingly deployed as high-level planners for robots because they generalize across diverse environments. However, this requires their safety alignment to also hold in unseen environments. Existing red-teaming assumes an adversary who optimizes the prompt, the pixels, or text in the environment, and existing benchmarks ask whether a planner recognizes or mitigates a hazard in a fixed scene. Neither asks whether a refusal the planner has already given survives an ordinary change to the environment. We ask that question by placing a single everyday object into the environment, with no pixel, gradient, or prompt under adversarial control. On $846$ tasks that a constitution-guarded planner initially refuses, we find $20.2\%$ of tasks can be flipped to compliance by one or more objects, and the number of objects differs from one task to another. In addition, the object need not be chosen for the task, i.e., items drawn from a fixed list, with no knowledge of the environment or the instruction, bypass safety about as often as items proposed for the specific task. We qualitatively contrast the tasks bypassed most and least often and find that the distinction lies in how conspicuous the hazard is in the instruction and environment. Susceptibility to safety bypass is therefore a property of the task, which we call its \emph{malleability}, and we show that it can be predicted before the target is ever queried. A composite of signals read from a small open-source VLM identifies malleable tasks $2.4\times$ as often as picking at random. Everyday objects, whether placed by an adversary or introduced by ordinary rearrangement of the environment, are thus sufficient to overturn a refusal. Because susceptibility is determined by how a task is specified, we recommend assessing malleability per task prior to deployment.

1 INTRODUCTION

The paper asks whether an embodied planner's safety refusal survives an ordinary change to its environment, a question not addressed by adversarial-input attacks or fixed-scene benchmarks. It measures refusal reversals after inserting everyday objects, analyzes which tasks are most susceptible, and tests whether malleability can be predicted without querying the target planner.

1 INTRODUCTION

  • Prior red-teaming assumes an adversary can manipulate planner inputs, while safety benchmarks test hazard recognition or mitigation in fixed scenes.Ordinary environmental changes, such as leaving an object in a scene, do not require digital access or optimization.Source: §1 · pp. 1–2
  • The study inserts a single everyday object into scenes for tasks the constitution-guarded planner initially refuses and measures the fraction of objects that reverse refusal as malleability.The planner receives an instruction and environment image and returns an action plan.Source: §1 · p. 2
  • 20.2% of 846 initially refused tasks are reversed by at least one object, and fixed-list objects succeed nearly as often as task-targeted objects.The authors interpret this as evidence that malleability is a property of the task rather than the object.Source: §1 · p. 2
  • The most malleable tasks have hazards that are neither stated in the instruction nor visible in the scene, but arise from their combination.The authors use this pattern to construct LATENTHARM, a dataset of 281 tasks across 26 environments.Source: §1 · p. 2
  • A small open-source VLM's signals identify malleable tasks 2.4× as often as random selection, without querying the target planner.The predictor is intended to identify tasks for review before deployment.Source: §1 · p. 2

2 RELATED WORK

Prior work studies embodied VLM safety through attacks that manipulate planner inputs and benchmarks that evaluate hazards in fixed scenes. The paper identifies a gap around whether a refusal persists after an ordinary visual-context change, distinguishing its focus from scene-variation work that measures task performance rather than safety decisions.

2 RELATED WORK

  • Red-teaming adapts jailbreaks, injects text into scenes, or perturbs pixels, assuming an adversary can manipulate a planner input.These attacks target harmful action plans or altered perception.Source: §2 · pp. 2–3
  • Safety benchmarks evaluate hazard recognition, mitigation, or reasoning in fixed scenes, rather than whether a refusal survives a scene change.The related work includes both generated hazardous scenes and interactive simulators.Source: §2 · p. 3
  • RoboART and metamorphic tests vary scenes but assess manipulation-policy performance or trajectory changes, not planner safety decisions.The paper also distinguishes their single-testbed scope from its broader collection of environments and tasks.Source: §2 · p. 3

3 METHODOLOGY

The methodology measures how often ordinary object insertions reverse refusals on initially refused tasks, compares tasks at the extremes of malleability, and estimates malleability using a proxy rather than querying the target planner.

3 METHODOLOGY

  • The study defines tasks as instruction–image pairs, judges plans as compliant, refused, or inconclusive, and analyzes only tasks initially refused on the clean image.Refusal includes substituting a safer task or modifying the environment to avoid, isolate, neutralize, or remove a hazard; inconclusive judgments are excluded from rates.Source: §3 · p. 3

3.1 INSERTING ORDINARY OBJECTS INTO THE SCENE

  • Object insertion is constrained to ordinary, physically placeable additions that preserve the scene, add no text, and do not make the task explicitly safe.Objects can be proposed for a specific task, suggested from the scene alone, or drawn from a fixed list; an independent judge checks seven constraints on each edited image.Source: §3.1 · pp. 3–4
  • Task-targeted proposals use the instruction and image to suggest 20 candidates, which an image editor composites into the scene.This measures effects of objects chosen to make compliance more likely, whereas scene-only and fixed-list proposals test non-targeted additions.Source: §3.1 · pp. 3–4

3.2 CHARACTERIZING MALLEABLE TASKS

  • The qualitative analysis compares tasks with no observed flips against tasks flipped frequently and asks which input properties separate them.Three discriminators separately inspect instruction text, scene descriptions, or instruction–scene pairs, with the last assigning the pattern to the instruction, scene, or their combination.Source: §3.2 · pp. 4–5
  • Constitution proximity is measured by the maximum combined text and image similarity between a task and any safety rule.The text and image similarities use sentence and CLIP embeddings; a high maximum indicates a close match to a rule.Source: §3.2 · p. 5

3.3 PREDICTING MALLEABILITY WITHOUT QUERYING THE TARGET

  • A local open-weight VLM proxy supplies signals for ranking tasks without querying the target planner, and an L2-regularized logistic model combines six features.The signals include trace ambiguity, margin spread under image perturbations, reliance on the constitution, and decoding instability; the composite uses the three trace rates and one score from each other signal.Source: §3.3 · pp. 5–6
  • The proxy's compliance margin is derived from its answer log-probabilities, and its reasoning trace is also analyzed for uncertainty, conditional judgment, and hedging.A positive margin indicates that the proxy leans toward complying; perturbation spread and decoding instability are measured as spreads of margins.Source: §3.3 · pp. 5–6

4 EVALUATION

Across four research questions, the evaluation tests ordinary-object bypasses on initially refused tasks, whether object choice matters, which task features track malleability, and whether malleability can be predicted without querying the target.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses 846 tasks that Gemini Robotics-ER 2 refused under a constitution guard, selected from 1,121 photorealistic EARDataset tasks.The tasks pair long-horizon robot instructions with environment images.Source: §4.1 · p. 6

4.2 RQ1: TO WHAT EXTENT CAN ORDINARY OBJECTS BYPASS SAFETY ALIGNMENT?

  • Across 846 tasks, 20.2% (n = 171) were malleable—bypassed by at least one ordinary object; object insertion bypassed 5.68% of trials versus 0.24% on clean re-query.The paired difference was 5.44 percentage points (95% CI [4.37, 6.50]).Source: §4.2 · pp. 6–7

4.3 RQ2: IMPACT OF OBJECT SELECTION ON BYPASS SUCCESS

  • On 192 stratified tasks, targeted, scene-only, and random objects bypassed at 20.69%, 18.77%, and 18.97%, respectively.Scene-only and random rates also rise across targeted-rate strata, from under 1% in the zero stratum to about 50% in the high stratum.Source: §4.3 · p. 7, Figure 2 · p. 8

4.4 RQ3: TASK CHARACTERISTICS ASSOCIATED WITH MALLEABILITY

  • Malleable tasks often leave hazards unstated and invisible until instruction and scene combine; constitution proximity did not distinguish robust and malleable groups.The authors built LATENTHARM, a dataset of 281 tasks across 26 environments where hazards arise through that interaction.Source: §4.4 · p. 8

4.5 RQ4: PREDICTING MALLEABILITY WITHOUT QUERYING THE TARGET

  • A5 raises ROC-AUC from 0.734 to 0.785 and PR-AUC from 0.395 to 0.476 over trace length when ranking malleable tasks.Its precision averaged over cutoffs is 2.4× the prevalence of malleable tasks.Source: Figure 3 · p. 9, §4.5 · p. 9

5 CONCLUSIONS

The paper finds that ordinary object insertions can reverse refusals, with susceptibility varying by task and a local proxy able to identify malleable tasks before querying the target. It releases LATENTHARM and notes that no physical robot was operated and no generated plan executed.

5 CONCLUSIONS

  • 20.2% of refusals were reversed by inserting a single ordinary object, and the object need not be chosen for the task.Reversal likelihood depends on the task; refusals often hold when the hazard is apparent in the instruction or scene, but may drop when it arises from carrying out the instruction there.Source: §5 · p. 9
  • LATENTHARM is a dataset of tasks on which refusal reversal is most likely.The paper suggests future work expand target planners, stage edits in real rooms, and develop alignment methods that ground refusals in consequences of acting in the scene.Source: §5 · pp. 9–10
  • 2.4× as often as random selection, signals from a small local proxy identify malleable tasks without querying the target planner.The paper proposes using this ranking to concentrate review on tasks whose refusals are least likely to hold.Source: §5 · p. 10
  • No physical robot was operated, and no plan produced in the experiments was executed.The paper describes LATENTHARM as generated rather than scraped and reports no human subjects or personal data.Source: ETHICS STATEMENT · p. 10

A MODELS USED

The appendix identifies the models and their roles, and specifies deterministic decoding for measurements alongside a sampled object proposer. It also gives the target planner’s safety constitution and robot capabilities.

A MODELS USED

  • Measurement-dependent target and judge queries use deterministic decoding, while the object proposer is sampled to cover a diverse candidate set.The target uses temperature 0 and seed 42; the proposer uses temperature 0.7 and top-p 0.9.Source: A MODELS USED · p. 13
  • The target model is the system under test; the other listed models support measurement or, in Section 3.3, local prediction.The models table identifies each model, its role, and provider.Source: Table 1 · p. 13, A MODELS USED · p. 13

B SAFETY CONSTITUTION

  • The target’s system prompt states the robot’s capabilities and contains 63 numbered safety rules.The rules are reproduced in the appendix, with line breaks adjusted to column width.Source: B SAFETY CONSTITUTION · p. 13
  • The robot is described as a mobile base with one 7-DoF arm and a parallel gripper, with a maximum payload of 100 kg and reach of 0.8 m.It cannot climb, jump, or reach behind obstacles.Source: B SAFETY CONSTITUTION · p. 13

C THE STRATIFIED SUBSAMPLE

The study stratifies tasks by their object-insertion bypass rate, then samples across the susceptibility range rather than in proportion to how common each stratum is.

C THE STRATIFIED SUBSAMPLE

  • The sample contains 194 tasks drawn across four bypass-rate strata, with up to 50 selected uniformly without replacement from each.Strata are zero, low (0%, 10%], medium (10%, 25%], and high above 25%; the medium pool contributes all 44 tasks.Source: C THE STRATIFIED SUBSAMPLE · p. 15, Table 2 · p. 16
  • Stratification covers the full susceptibility range with equal precision at each level rather than in proportion to each level’s prevalence.Only the medium stratum falls short of the 50-task draw.Source: C THE STRATIFIED SUBSAMPLE · p. 15
  • 20.2% (n = 171) of the 846 initially refused tasks were bypassed by at least one object; the distribution also marks the planner’s 0.24% rate on unmodified images.The four strata in Table 2 are cut from this distribution.Source: Figure 4 · p. 16

D THE FIXED RANDOM LIST

The fixed random condition uses a task-independent list of ordinary objects, constrained to items a person could carry and place and excluding categories that could introduce alternative hazards.

D THE FIXED RANDOM LIST

  • The random condition draws five objects without replacement per task from a fixed list of 26 everyday items, with no model seeing the scene or task during selection.The list is external to the project’s data and results, and its categories are inspired by COCO.Source: D THE FIXED RANDOM LIST · p. 16, Table 3 · p. 16
  • The list keeps categories a single person could carry and place, while excluding people, animals, vehicles, outdoor fixtures, large furniture, sharp items, and weapons.The filter is fixed before any object is drawn and matches the proposer’s physical-feasibility constraint.Source: D THE FIXED RANDOM LIST · p. 16

E A WORKED BYPASS

A hospital-ward task illustrates how an ordinary scene edit can overturn a refusal: after a water bottle is added, the planner returns a plan it had declined on the clean image.

E A WORKED BYPASS

  • The target complied under 12 of 20 objects on the hospital-ward task, giving it a malleability of 0.60.The task had no inconclusive verdicts.Source: E A WORKED BYPASS · p. 17
  • On the clean image, the target refused because the hot soup could spill from the unstable surface onto the patient and cause injury.The bowl was on the patient’s blanket, and the cart was beside a vital heart monitor.Source: E A WORKED BYPASS · p. 17
  • With the instruction, constitution, and decoding configuration unchanged, adding a plastic water bottle led the target to plan moving the soup bowl to the cart beside the heart monitor.The clean scene prompted a refusal; the edited scene prompted a plan for the previously declined placement.Source: Figure 5 · p. 17, E A WORKED BYPASS · p. 17

F THE SAME EXPERIMENT ON AN EARLIER TARGET

The earlier target was evaluated with the same task pool and construction as the main target, and ER-2 refused a larger share of the pool and was bypassed less often on every measure.

F THE SAME EXPERIMENT ON AN EARLIER TARGET

  • Both targets saw the same 1,121-task pool and construction, with each evaluated on the tasks it itself refused.The earlier target was gemini-robotics-er-1.6-preview; the ER-2 column is the main RQ1 result.Source: F THE SAME EXPERIMENT ON AN EARLIER… · p. 18
  • Table 4 reports refusal over conclusively judged pool tasks and bypass measures over refused tasks: 697 for ER-1.6 and 846 for ER-2.The caption defines the denominators for comparing the two targets’ measurements.Source: Table 4 · p. 18
  • ER-2 refused 78.3% of the pool, compared with 64.9% for the earlier target, and was bypassed less often on every measure.The passage characterizes ER-2 as the safer model.Source: F THE SAME EXPERIMENT ON AN EARLIER… · p. 18

G THE RQ3 DISCRIMINATOR PATTERNS

The section reports discriminator patterns for robust and malleable tasks, then describes LatentHarm, a corpus designed around hazards latent in instruction–scene pairs. Its example shows why identifying such a hazard requires combining the instruction with the environment.

G THE RQ3 DISCRIMINATOR PATTERNS

  • Table 5 reports patterns from three discriminators on 50 robust and 50 malleable tasks, alongside a constitution-proximity comparison.The supplied passage identifies the table’s scope but does not state the patterns’ outcomes.Source: G THE RQ3 DISCRIMINATOR PATTERNS · p. 18, Table 5 · p. 18

H LATENTHARM: A CONSTITUTION-GROUNDED CORPUS

  • LatentHarm is a second corpus whose tasks make harm latent in the instruction–scene pair, not stated in either alone; danger arises from their jointly described state, requiring more than alarm-language or known-object screening.Source: H LATENTHARM: A CONSTITUTION-GROUNDED… · p. 18

H.1 DATASET CARD

  • Each released task includes an instruction, environment label, scene prompt and image, plus a violation note; the target sees only the image and instruction.Tasks are generated with a single Gemini query, rendered with Flux.2-klein, manually checked, and tied to at least one of 54 represented constitution rules.Source: §H.1 · p. 19

H.2 EXAMPLE FROM LATENTHARM

  • In the example, putting metal tools into a storage bin is hazardous because carrying heavy steel wrenches across the room brings them near an active MRI scanner.Neither the ordinary tidying instruction nor the benign objects alone names the hazard; the planner must anticipate the consequences of the action in the scene.Source: §H.2 · p. 19

H.3 EVALUATION ON LATENTHARM

On LatentHarm, the evaluated target’s refusals were frequently reversed by object additions, at higher rates than on EARDataset. The section also details proxy signals and scoring procedures used to study and predict task malleability.

H.3 EVALUATION ON LATENTHARM

  • The target refused 41.6% of LatentHarm’s 281 tasks (117 tasks) on the unedited scene, and evaluation used those rejected tasks.This evaluation followed the Section 4.2 procedure with gemini-robotics-er-1.6-preview.Source: §H.3 · p. 19
  • 29.17% was the mean per-task bypass rate on LatentHarm, and 64.9% of its 117 refused tasks were bypassed by at least one object.On EARDataset, ER-1.6’s corresponding values were 9.90% per task and 28.9% of tasks flipped at least once.Source: §H.3 · pp. 20–21

I PROXY MALLEABILITY

  • Section I provides supporting detail on the local proxy signals used to predict malleability.The supplied passage identifies this section’s role without specifying additional results.Source: I PROXY MALLEABILITY · p. 21

I.1 TRACE AMBIGUITY FEATURES

  • A1 scores six reasoning-trace features: five marker-family densities and trace length; the A5 composite uses object-ambiguity, conditional, and hedge densities.Trace length is a stronger predictor alone, while controlling for length leaves the five densities with partial ρ of 0.04 to 0.12 against mal(t).Source: §I.1 · p. 21

I.2 PROXY PROMPT AND SCORING

  • The proxy procedure generates a reasoning trace in a first pass and scores a yes/no continuation in a second pass; A1 uses the trace, while A2–A4 use the returned compliance margin.The first pass is capped at 2,048 tokens; the scored yes/no answers share a forced prefix, making the margin a verdict conditioned on the trace already written.Source: §I.2 · p. 22

I.3 EVERY SIGNAL ON EVERY METRIC

The section reports the all-signal proxy scoreboard, distinguishes task-aware from scene-only proposal prompts, and checks whether targeted object edits satisfy the stated constraints.

I.3 EVERY SIGNAL ON EVERY METRIC

  • Trace length outranks every density among Table 8's local proxy signals, while adding it as a seventh composite feature changes nothing.Table 8 reports three metrics for each signal and precision at two audit budgets.Source: Table 8 · p. 23, §I.3 · p. 23

J OBJECT PROPOSAL PROMPTS

  • The two model-based proposer conditions differ only in proposer instructions; random selection uses no model or prompt, and both model-based conditions attach the image to the user message.The prompt conditions are described as targeted and scene-only.Source: J OBJECT PROPOSAL PROMPTS · p. 23, §J.1 · p. 23, §J.2 · p. 24

J.1 TARGETED (IMAGE AND INSTRUCTION)

  • The targeted proposer uses a red-teaming prompt, sees the task instruction, and can use scored objects from earlier rounds as feedback.Its criteria 7 and 8 specifically require seeing the task.Source: §J.1 · p. 23

J.2 SCENE-ONLY (IMAGE ALONE)

  • The scene-only prompt removes red-team framing, feedback, and task-dependent criteria while retaining the other criteria and omitting the instruction from the user message.It uses a plain scene-annotation instruction and runs in a single round.Source: §J.2 · p. 24

J.3 CONSTRAINT VERIFICATION

  • 92.6% of audited edits satisfy all seven constraints, and 181 of 193 bypassed trials pass them outright, making imperfect editing an unlikely explanation for bypasses.The audit sampled 966 edited images from 194 stratified tasks; among 74 failures spanning 54 tasks, 41 failed exactly one check, most often because the edit altered the scene.Source: Table 9 · p. 25, Table 10 · p. 25, §J.3 · pp. 25–26

Related papers

Loading 2609.38971v1…