Source-linked AI summary
Think Only When Needed: Prompt-Authority Control for Selective Slow-Path Intervention in Vision-Language-Action Manipulation
Zhiruo Zhou, Zelin Li, Xiwen Chen, Jiazhuo Li, Chenwei Wang, Huiming Chen, Xiaojun Zhu
TL;DR
Frozen VLA policies can be harmed by retrieved text because prompt form, not only semantics, affects closed-loop control. TOWN-VLA separates candidate generation from prompt authorization, admitting canonical instructions or restoring the exact Base prompt. It improves matched LIBERO-Plus and PiPER success without retraining, while oracle-free admission remains a calibration challenge.
Problem
Raw appended text can cause prompt-form collapse at the frozen VLA control interface, motivating evidence that distinguishes prompt form from retrieved semantic content.
Method
TOWN-VLA uses fixed compatibility ranking and a Top-2 fail-closed authorization cascade to render a canonical instruction or restore the exact Base prompt.
Results
69.46% to 73.07%: TOWN-VLA adds 362 LIBERO-Plus successes and improves PiPER success from 52.7% to 78.7% without retraining the action generator.
Takeaways & Limitations
Prompt authority is an enforceable control primitive for frozen VLAs, with exact restoration making rejected intervention reversible.
Takeaways & Limitations
Oracle-free admission remains a calibration challenge, and the evaluation isolates the interface using a 48-entry same-domain memory and a controlled single-task, single-operator physical study.
Abstract
from arXiv · showhide
Retrieval can efficiently and effectively augment a frozen vision--language--action (VLA) policy without retraining, yet retrieved text becomes a control intervention once it enters the executed prompt. In a matched audit, raw appended text reduces mean success from 92.47\% to 3.00\%, while meaningful and length-matched meaningless appends both fail on all 500 states. This result identifies \emph{prompt-form collapse}: changing the instruction form, rather than adding useful semantics, can dominate execution. We introduce TOWN-VLA (Think Only When Needed), a prompt-authority interface that separates candidate generation from permission to alter the policy input. A fixed compatibility rule authorizes a canonical compact instruction; otherwise, the interface restores the original Base prompt exactly. Across 900 audited routes, every route follows this contract: 525 routes recover Base with matching hashes, and all 375 authorized prompts preserve the task signature. On a matched $4\times7$ LIBERO-Plus evaluation with 10{,}030 episodes per method, success rises from 69.5\% to 73.1\% ($+362$ episodes; 95\% CI 1.89--5.45 points), improving on six perturbation axes and all four suites. On a physical PiPER arm with a frozen \pizerofive{} checkpoint, success rises from 52.7\% to 78.7\% over 150 trials per method ($p=3.16\times10^{-6}$). Prompt authority is enforceable for a frozen controller; oracle-free admission calibration is the next deployment target.
Introduction
The paper identifies prompt-form collapse as a control failure in frozen VLAs and reframes retrieval augmentation as prompt-authority control. TOWN-VLA separates candidate generation from authorization, using canonical instructions or exact Base restoration to preserve the frozen policy interface.
- Motivation: 92.47% mean success falls to 3.00% when raw text is appended to a frozen VLA prompt.Meaningful and length-matched meaningless appends both fail on all 500 audited states, implicating prompt form rather than semantic quality.
- Problem formulation: TOWN-VLA treats retrieval augmentation as a prompt-authority problem rather than assuming retrieved strategy content should control the policy.The interface separates candidate quality from an explicit authorization decision at the frozen-policy boundary.
- Method: TOWN-VLA ranks candidates with a fixed compatibility rule and admits at most one canonical compact instruction through a Top-2 fail-closed cascade.If neither candidate passes, the interface retains the exact Base Policy prompt while the frozen Base Policy remains the sole action generator.
- Auditability: Across 900 routes, 525 unauthorized routes recover hash-identical Base prompts and all 375 authorized prompts preserve the task signature.These checks operationalize exact restoration for rejected routes and constrained rendering for accepted routes.
Related Work
Prior work improves VLA behavior through policy, data, memory, reasoning, routing, or runtime-assurance changes. This paper instead regulates external text at the frozen policy boundary, separating slow-path proposals from permission to alter the policy input.
- Frozen policies and inference-time interfaces: Frozen-policy research scales language-conditioned robot control through new policies, training data, embodiments, and action representations.TOWN-VLA differs by freezing the action generator and regulating external text through its existing instruction interface.
- External reasoning and fast–slow authority: External reasoning and fast–slow systems move semantic computation, reactive computation, or assistance closer to action selection.TOWN-VLA keeps its slow path external and treats it as a source of proposals rather than an automatic controller input.
- Inspectable memory at the policy boundary: External memory methods retrieve experience or action priors, whereas TOWN-VLA makes an external capsule inspectable and hashable before it crosses the policy boundary.The paper regulates whether and how such a candidate may alter the frozen policy input.
- Prompt brittleness beyond language generation: Prompt-brittleness studies show formatting and suffix sensitivity in open-loop language generation; this paper examines the resulting action consequences in a frozen closed-loop VLA.Its intervention is boundary regulation rather than prompt optimization.
- Selective intervention and runtime assurance: Assistance, routing, selective intervention, and runtime-assurance systems calibrate uncertainty, select experts, or supervise learned controllers.With a fixed action generator and fallback, TOWN-VLA implements rejection as exact Base-prompt restoration rather than controller switching.
- TOWN-VLA interface: Figure 2 presents TOWN-VLA as a pipeline that separates invocation, candidate ranking, prompt authorization, and frozen control.Logged scores, flags, and prompt hashes support route auditing, while Task-Prior is reserved for the oracle control.
Method
TOWN-VLA separates slow-path candidate preparation from authority to modify a frozen VLA's prompt. Its deterministic, fail-closed interface authorizes only canonical compatible instructions and otherwise preserves the Base prompt exactly.
- Interface contract: The frozen VLA remains the sole action generator while TOWN-VLA constrains only candidate-generated policy inputs.Policy parameters and the control loop remain unchanged throughout evaluation.
- Auditability: The implementation logs candidate identifiers, scores, checker reasons, route state, and the resolved-prompt hash for auditability.Unless explicitly labeled oracle-side, results use g = 1, so the main evaluation measures prompt authority independently of benchmark labels and Task-Prior routing.
- Compatibility-Reranked Capsule: The interface parses task and context text into object–target fields and computes deterministic overlap and mismatch signals.The same frozen parser supplies normalized object, relation, and target fields for route identity.
- Candidate preparation: Candidates are retrieved and reranked textually, but a hard checker alone grants authorization.Reranking changes inspection order without itself permitting prompt intervention.
- Top-2 Fail-Closed Cascade: The Top-2 cascade checks at most two candidates, short-circuits at the first eligible one, and falls back to the exact Base prompt when neither passes.Accepted context is rendered with a fixed template; fallback adds no retrieved text or rejection message.
- Runtime states: TOWN-VLA distinguishes bypass, inspected-but-unauthorized, and authorized runtime states, keeping slow-path computation separate from prompt authority.A single resolved prompt is fixed before rollout, and the slow path is not queried again during execution.
Experiments
The evaluation tests TOWN-VLA’s prompt-authority interface across robustness, prompt-form failure, authorization, selective computation, and transfer settings. Under matched protocols, it improves LIBERO-Plus and physical success while restoring rejected prompts exactly, but oracle-free admission remains unresolved.
- End-to-end robustness: 73.1% LIBERO-Plus success adds 362 successes over 10,030 episodes versus 69.5% for the matched OpenVLA-OFT Base Policy.The paired-cell 95% interval is 1.89–5.45 points; gains occur on six of seven axes and all four suites.
- Prompt-form failure: 92.47% mean Base success collapses to 3.00% with raw appended text across three fixed-schedule executions.The paired diagnostic reports a −88.67-point change for the first raw candidate, and increasing retrieval depth neither worsens nor mitigates collapse.
- Prompt-form failure: 499/500 states succeed with Base or exact restoration, while meaningful and length-matched meaningless appends both yield 0/500.Canonical wrong-object and wrong-target controls reach 497/500 and 496/500, respectively, separating prompt form from retrieved strategy content.
- Authorization and exact restoration: 525 of 900 routed cases recover exact Base prompts, while all 375 authorized prompts preserve the task signature.In the separate 900-route audit, 450 routes bypass retrieval, 75 are rejected after inspection, and 375 are authorized with SigEq = 1.
- Transfer and selective computation: 89.33/97.53 restricted-OOD success under spatial/object shift exceeds Base in both reported shift aggregates, while the bounded cascade scores 91.32% on five suites.The cascade is retained for exact restoration rather than claimed as an incremental success-rate mechanism over reranking.
- Selective computation: Oracle-free admission authorizes only 2/36 held-out cells, matching 91.81% Base success across those 40 episodes.CLIP authorizes 0/36 cells, and always-on intervention remains 1.53 points below Base.
Discussion
TOWN-VLA separates retrieval relevance from prompt authority by admitting only canonical guidance and making rejection reversible through exact Base-prompt restoration. The evaluation isolates this interface under controlled simulation and a limited physical study, while identifying broader admission and attribution as open targets.
- Canonical rendering constrains accepted intervention, while exact restoration makes rejected routes reversible.
- The overall gain is strongest when perturbations degrade the executed instruction and largely neutral when Base remains strong.
- The evaluation uses a 48-entry same-domain memory, text-only compatibility, and a controlled single-task, single-operator physical study.
- Natural extensions include larger cross-domain memories, visually conditioned admission, and broader blinded robot trials.
Conclusion
The paper identifies prompt-form collapse when retrieved text crosses a frozen VLA's input boundary and introduces TOWN-VLA to separate candidate generation from authorization. Exact Base restoration on rejected routes preserves control, while matched evaluations improve LIBERO-Plus and PiPER without retraining the action generator.
- Discussion: Prompt-form controls reveal severe prompt-form collapse under raw appends to a frozen VLA.
- Conclusion: TOWN-VLA separates candidate generation from authorization and restores the Base prompt exactly on rejected routes.
- Conclusion: 3.61 points: TOWN-VLA improves LIBERO-Plus under matched evaluation without retraining the action generator.
- Conclusion: 26.00 points: TOWN-VLA improves the PiPER real-robot platform under matched evaluation without retraining the action generator.
- Conclusion: Reliable oracle-free admission is identified as the next frontier for selective slow-path control.