PaperlayerView exact revision on arXiv

Source-linked AI summary

CLAMP: Constrained Decoding for Vision-Language Embodied Planning

Tianyi Ma, Parisa Kordjamshidi

arXiv:2609.08602v1cs.AI

TL;DR

VLM-generated embodied plans can be fluent yet violate scene, syntax, safety, or world-state constraints. CLAMP addresses this with decoding-time hard constraints and HMM-based lookahead around a frozen VLM, improving grounding and constraint compliance while retaining modest overhead.

  • Problem

    VLM plans may reference unobserved objects, violate action syntax, or select actions that are infeasible under the current world state.

  • Method

    CLAMP combines observation-conditioned hard constraints with HMM-based world-state guidance and test-time adaptation of the lightweight guidance model while keeping the VLM frozen.

  • Results

    CLAMP improves object grounding and constraint compliance across embodied-planning evaluations; on SafeAgentBench, symbolic-rule violations decrease from 0.41 to 0.05.

  • Takeaways & Limitations

    Scene evidence and symbolic constraints can be incorporated directly into decoding to improve embodied planning without updating the underlying VLM.

  • Takeaways & Limitations

    Performance depends on perception and constraint specifications, while visibility gating can reject feasible plans under occlusion and emission-only adaptation is not uniformly beneficial.

Abstract

from arXiv · show

Embodied planning increasingly relies on vision-language models (VLMs) to translate instructions and visual observations into executable action sequences. However, fluent plans are not always executable. A VLM may refer to objects that are not visually observed, select actions whose required affordances are unavailable, or violate syntax and action constraints. We introduce CLAMP, a multimodal constraint-grounding framework that turns scene evidence into decoding-time constraints for a frozen VLM planner. CLAMP uses the initial observation to restrict object references to those supported by the scene, while a provided symbolic action model specifies state transitions and goals. During decoding, hard masks eliminate invalid next-token candidates, while a Hidden Markov Model (HMM)-based world-state lookahead module reweights the probabilities of the remaining feasible candidates based on action preconditions and goal reachability. This allows the planner to retain the VLM's language prior while preventing visually unsupported, unsafe, or infeasible candidates from entering the plan. For unseen tasks and environments, CLAMP adapts the HMM at test time using label-free continuations sampled from the frozen VLM. Experiments on VLABench, SafeAgentBench, and TaPA show that scene-grounded constraints improve object grounding and safety, while most remaining failures stem from perception errors or misaligned constraint specifications.

1 Introduction

CLAMP addresses embodied-planning failures by grounding constrained decoding in multimodal observations and evolving world state. It combines hard validity constraints, HMM-based guidance, and lightweight test-time adaptation around a frozen VLM.

  • Embodied plans must be well-formed, grounded in observed scenes, and feasible under the current world state.
  • Existing post-hoc repair allows invalid decisions to propagate, while embodied constrained decoding must evolve with multimodal observations and world state.
  • CLAMP uses observation-conditioned DFAs for syntactic and visibility constraints, HMM state tracking for evolving-world guidance, and test-time adaptation with the VLM frozen.
  • CLAMP improves task execution, visual grounding, and safety with modest additional computation cost.
  • CLAMP extends constrained decoding to high-level embodied planning, where constraints are grounded in multimodal observations rather than only language or low-level policy outputs.
  • The framework separately probes visual grounding and constraint enforcement because image conditioning alone does not guarantee grounded references.

3 Preliminaries

The planning problem maps an instruction, initial multimodal observation, action schema, constraints, and horizon to an executable grounded action plan. A frozen VLM generates tokens that are parsed into actions, while generation remains conditioned on the initial observation rather than execution feedback.

  • The task input consists of an instruction, initial multimodal observation, action schema, constraints, and maximum action horizon.
  • The goal is an executable action plan that satisfies the constraints within the prescribed horizon.
  • A frozen autoregressive VLM generates a token sequence containing serialized action expressions, which a deterministic parser converts into an executable plan.
  • Token length can differ from action count because a single action may span multiple tokens.
  • The VLM remains conditioned on the initial observation during decoding; intermediate observations and final success are not generation-time feedback.

4 Method

CLAMP enforces syntax, scene visibility, and state-dependent reachability directly during token decoding. Hard masks remove invalid candidates, while HMM lookahead favors feasible continuations that can reach a goal within the remaining action budget.

  • CLAMP modifies next-token logits to enforce syntax, visibility, and reachability constraints during autoregressive decoding.
  • 4.1 Embodied Planning Constraints: Syntax requires actions to conform to the task schema, while visibility restricts referenced entities to those supported by the initial observation.
  • 4.1 Embodied Planning Constraints: Reachability requires executable actions under evolving state transitions, goal termination, and the maximum action horizon.
  • Binary token masks assign zero probability to candidates rejected by any constraint.
  • 4.2.3 Budget-aware Future Reachability: HMM lookahead scores compatible executable completions and their continuations using latent-state probabilities and remaining action budget.
  • 4.2.2 State-conditioned Executability Filtering: State-conditioned filtering retains token candidates whose action completions are executable from the current state, advancing state only when an action is parsed.

5 Experiments

Experiments evaluate CLAMP across planning, grounding, safety, adaptation, and computational cost. Hard constraints improve interface and grounding, while world-state guidance adds planning gains; remaining errors include specification mismatch and HMM collapse.

  • VLABench: Combining syntax and visibility constraints with text-HMM guidance increases the Qwen3 macro planning score from 28.7 to 37.1 across six dimensions.Hard constraints alone reach 34.1, and text-HMM guidance contributes a further 3.0 points.
  • VirtualHome: 85.3% Action Sequencing task success results from world-state guidance, compared with 21.3% for the baseline and 48.7% with token constraints.Subgoal Decomposition task success increases from 60.1% to 79.8% over token constraints.
  • Grounding: CLAMP substantially improves entity validity and matching while format validity remains perfect and skill matching changes little.The comparison uses frozen InternVL3.5-8B and separates format, entity-ID, skill, entity, exact-match, and aggregate metrics.
  • Safety: Policy enforcement with recovery reduces long-horizon symbolic violations from 0.41 to 0.05 while judged safety and completion improve.Token-only decoding achieves higher completion but retains substantially more violations.
  • Test-Time Adaptation: Adding TTBA to TTDA increases pooled overall score from 1.0 to 9.6 and Physical Laws from 6.4 to 59.0, whereas OTTA leaves both unchanged.The TTBA gain is confined to Physical Laws and accompanies a shift in the dominant predicted skill.
  • Cost and Failures: Syntax enforcement changes batched-vLLM runtime from 3.9 to 4.0 s/prompt, while adding TTBA to TTDA changes HuggingFace runtime from 12.9 to 31.0 s/prompt.The cross-path difference does not isolate HMM overhead.
  • Cost and Failures: Remaining errors include task-type mismatch, incomplete or misaligned symbolic specifications, and HMM single-skill collapse.Calibration can change the dominant skill without restoring task-specific action selection.

6 Conclusion

CLAMP combines scene-grounded hard constraints with HMM-based lookahead under a supplied symbolic action model for decoding with a frozen VLM. Experiments show improved grounding and constraint compliance without updating the VLM, while safety-policy enforcement and recovery reduce symbolic-rule violations.

  • Conclusion: CLAMP integrates syntax, visibility, and goal reachability into decoding over a frozen backbone using scene-grounded constraints and HMM-based lookahead.At test time, only the HMM emission matrix is recalibrated using continuations sampled from the VLM.
  • Conclusion: Experiments show improvements in object grounding and constraint compliance, including reduced symbolic-rule violations with safety-policy enforcement and recovery.The conclusion attributes these findings to incorporating scene evidence and symbolic constraints directly into decoding.

Limitations

CLAMP’s guarantees depend on supplied symbolic interfaces, scene support, and constraint specifications, and its adaptation and lookahead mechanisms have important scope boundaries. Evidence is limited by perception, supervision mismatch, independent constraint lookahead, and simulator-based evaluation.

  • Scope and Interface Assumptions: CLAMP’s guarantees are conditional on supplied scene support, canonicalization, action schema, transition model, and goal predicates, excluding direct extension to continuous-action VLA policies.Errors in perception or specification can reject valid plans or admit unsafe ones.
  • Calibration and Perception: Emission-only adaptation is not uniformly beneficial, with gains concentrated in Physical Laws and OTTA providing no measured improvement.The strongest vision HMM uses ground-truth skill/image pairs, so supervision is not matched across settings; self-generated updates may reinforce incorrect modes.
  • Constraint Lookahead: Independent lookahead can overestimate joint feasibility because backward reachability does not require future action arguments to belong to the observed entity set.A jointly feasible lookahead would need visibility in future action enumeration.
  • Recovery and Physical Execution: Evidence is limited to simulators and offline benchmark harnesses, while recovery adds latency and may fail when no valid alternative exists.Deployment under partial observability, stochastic dynamics, or continuous physical control requires additional mechanisms beyond these symbolic constraints.

Ethical Considerations

CLAMP reduces invalid symbolic plans but does not establish physical safety when its action schema, affordance table, or transition model is incorrect.

  • Ethical Considerations: CLAMP can confidently enforce an incorrect action schema, affordance table, or transition model, so symbolic compliance does not establish physical safety.Real-world use requires specification audits, motion-level safety checks, closed-loop monitoring, and human oversight.

Broader Impacts

Constrained decoding can improve auditability in embodied planning, but constraint satisfaction should not be treated as equivalent to task or physical safety.

  • Constraint satisfaction may create false confidence when it is mistaken for task or physical safety.
  • A flawed domain model can narrow the actions that the planner is permitted to take.
  • Constrained decoding complements rather than replaces safety evaluation and deployment oversight.

A1 Methodological and Implementation Details

The method combines token-level syntax and visibility masks with symbolic executability and HMM-based goal-reachability guidance. Tokenizer remapping and lightweight emission adaptation make the HMM usable across backbones while keeping symbolic feasibility fixed.

  • Action-level state tracking: Action serialization commits a symbolic state transition only when the complete action span is generated.During partial generation, candidate tokens remain admissible only if their prefixes can still complete an executable action.
  • HMM lookahead: The backward reachability table scores executable continuations that can reach a goal within the remaining action budget.Its cached transition structure is refreshed numerically after emission updates, with O(Kmax|Z|H) memory for the table.
  • Per-token constrained decoding: The decoder combines VLM token scores with syntax, visibility, executability, and budget-aware HMM reachability terms.Hard masks remove locally inadmissible candidates, while finite reachability scores rank candidates with supported goal-reaching continuations.
  • Test-time adaptation: Test-time adaptation updates only the HMM emission matrix, preserving latent dynamics, symbolic masks, and the transition relation.The adapted matrix recalibrates HMM guidance, while reachability operators and numeric tables are rebuilt when required.
  • Tokenizer remapping: Llama-3-to-Qwen3 emission remapping is necessary because tokenizer ID mismatches otherwise steer HMM guidance toward the wrong target tokens.The unmapped checkpoint produced empty plans on all 480 Mode B prompts, whereas remapping enabled non-trivial behavior.
  • Tokenizer remapping: The remap covers 72.3% single-subtoken and 27.5% multi-subtoken projections, with 267 IDs using the smoothing fallback.After normalization, the mean row-entropy ratio is 1.020 and canonical-skill emission mass remains within 0.846–1.058 of the source.
  • Vision-conditioned guidance: The supervised vision-conditioned HMM reaches a Macro score of 38.7 versus 37.1 for the text HMM on VLABench.The 1.6-point difference combines changes in HMM supervision and calibration, so it is not an isolated calibration effect.

A2 Experimental Protocol and Reproducibility

The experiments use frozen planners, matched constrained-versus-unconstrained comparisons, multiple embodied benchmarks, and track-specific score definitions. Reproducibility depends on keeping these evaluation boundaries and simulator assumptions explicit.

  • Benchmark scope: The evaluation spans VLABench, SafeAgentBench, EAI, VirtualHome, and BEHAVIOR with task, grounding, executability, and safety measures.The benchmark suite separates task completion from plan validity, grounding, executability, and safety because these outcomes can diverge.
  • Models and controls: All planner parameters remain frozen, and the main ablations compare constrained and unconstrained decoding on the same backbone.The primary multimodal planner is Qwen3-VL-8B-Instruct, with InternVL3.5-8B and Llama-3.1-8B-Instruct used for additional tracks.
  • Score aggregation: VLABench Qwen VL-main scores are six dimension scores averaged arithmetically, whereas other tracks use distinct metric definitions and aggregation rules.Values from different evaluation rows are not direct comparisons even when backbone and prompt count coincide.
  • Simulator assumptions: The reported Drch mechanism assumes a sound deterministic symbolic transition model, but simulator determinism alone does not show that full reachability was evaluated.Table 6 separates benchmark-environment assumptions from the constraint mechanism actually reported.
  • Comparison boundaries: The Qwen VL-main baseline of 28.7 and VL-B diagnostic baseline of 20.1 come from different pipelines and cannot be compared as a macro-versus-pooled effect.The score definitions and aggregation rules are track-specific.
  • External baselines: CLAMP-token achieves 0% grammar errors by construction, while approximately 15% of BEHAVIOR AS cases exhaust all three baseline reprompts without valid formatting.Those failed baseline outputs are passed to evaluation as-is and score 0 on grammar metrics.

A3 Detailed Experimental Results

Across matched and transfer evaluations, CLAMP improves constraint-compliant planning, grounding, and selected benchmark dimensions, but gains are sensitive to task-interface alignment, model priors, and evaluation design. Remaining limitations include single-skill collapse, task-type mismatch, incomplete specifications, and judge or perception errors.

  • Transfer and cross-task results: 40.0% executable traces and 39.4% goal overlap exceed adapted-baseline values of 25.0% and 9.6%, respectively.These are mechanism comparisons after interface adaptation, not native end-to-end system results.
  • Transfer and cross-task results: 85.3% Action Sequencing performance follows DFA-only gains, while Subgoal Decomposition rises from 48.8% to 60.1% to 79.8% with successive reachability guidance.Backward reachability shifts token scores toward continuations that can reach the goal within the remaining budget, unlike flat reachability.
  • VLABench dimension analysis: CLAMP’s VLABench gains span six dimensions in Mode A and five of six in Mode B, but Physical Laws remains weakest because backbones often emit manipulation plans instead of press[N].Physical Laws comprises 16.7% of prompts but accounts for approximately 40% of the Mode A–Mode B total gap.
  • TTA ablations: TTDA+TTBA reaches 59.0 on Physical Laws and 9.6 overall versus 6.4 and 1.0 for TTDA alone, with gains confined to Physical Laws.The combined configuration obtains skill_match = 1.000 but entity_match = 0.385; all remaining prompts score zero.

A4 Cross-Experiment Diagnostic Analysis

The diagnostics identify five recurring mechanisms behind CLAMP’s cross-benchmark behavior: domain-mismatched HMM guidance, task-type mismatch, reachability modeling, specification errors, and adaptation collapse. They also show that structural, perceptual, and causal failures are distinct and that adaptation can be limited by weak or self-reinforcing evidence.

  • HMM guidance: In-domain HMM guidance can degrade DFA-only decoding because a marginal length prior misranks feasible tokens without distinguishing task-conditioned continuations.Increasing the soft-score weight leaves hard masks unchanged while worsening the surviving token ranking.
  • Task-type mismatch: Physical Laws is the weakest evaluated dimension because it expects short QA-style press[N] plans while backbones often generate manipulation plans.A global press prior helps this dimension but does not generalize to entity-grounded dimensions.
  • Reachability: Backward reachability improves VirtualHome action selection by favoring executable successors that can still reach the goal within the remaining budget.Residual failures track incomplete predicate coverage in the supplied transition model.
  • Specification errors: Hard filtering inherits errors from misspecified task interfaces, while HMM guidance cannot recover predicates masked by an incompatible DFA.On BEHAVIOR, relational predicates and altered output granularity reduce the usefulness of otherwise valid decoding constraints.
  • Adaptation: OTTA provides no measured task-level gain when skill-name changes are rare and inconsistent, while self-generated outputs can reinforce erroneous modes.Limited observation diversity also weakens emission estimation, and entity averaging can produce an uninformative near-uniform prior.
  • Failure modes: CLAMP treats structural, perceptual, and causal failures as pairwise non-subsuming modes, combining distinct decode-time terms without constructing their product automaton.A pilot observed every failed plan matching at least one mode, but did not establish universal coverage.

A5 Extended Discussion and Scope

CLAMP extends constrained decoding from token-string validity to embodied action-state feasibility while retaining separate symbolic and token-level mechanisms. Its operating boundary is the supplied symbolic interface: physical execution, active perception, distribution shift, and specification errors remain external or consequential constraints.

  • Novelty: Prior constrained decoders guarantee token-string validity, whereas CLAMP addresses the action-state trajectory represented by the generated sequence.A unified episode-specific product automaton would couple grammar, visible entities, symbolic preconditions, and action budget; CLAMP keeps these checks separate.
  • Mechanism: CLAMP combines token-level hard masks with dynamic-programming feasibility checks over symbolic state instead of constructing a large product automaton.This separation preserves observation-conditioned token filtering while adding state-aware lookahead.
  • Operating boundary: Physical safety during low-level motion execution remains out of scope and is delegated to a downstream motion planner or closed-loop controller.Execution failures must be detected, represented as state changes, and incorporated into subsequent replanning.
  • Adaptation boundary: The HMM lookahead is sensitive to vocabulary, training distribution, adaptation data, and the tuned weight λ.Far-from-source domains can produce near-uniform priors, while small streams may overfit and additional EM steps may drift from the domain prior.
  • Operating boundary: Static initial-observation visibility excludes tasks requiring active perception, although replanning can use incrementally updated grounded entities.This requires integration with an active-perception loop.
  • Specification risk: Specification errors can make permitted plans worse rather than safer because CLAMP enforces the supplied PDDL or affordance interface.The BEHAVIOR example shows an affordance DFA misaligned with relational-predicate output format.

A6 AI Assistants in Research or Writing

AI assistants were used only to polish English grammar and refine manuscript structure. The authors state that assistants did not generate research ideas, design experiments, or produce results.

  • Use of AI assistants: AI assistants supported English grammar polishing and manuscript-structure refinement, not research ideation, experiment design, or result production.The authors state that all technical content, claims, and conclusions were their own and were verified by them.
Loading 2609.08602v1…