Think Only When Needed: Prompt-Authority Control for Selective Slow-Path Intervention in Vision-Language-Action Manipulation

Motivation

• 92.47% mean success falls to 3.00% when raw text is appended to a frozen VLA prompt.Meaningful and length-matched meaningless appends both fail on all 500 audited states, implicating prompt form rather than semantic quality.Source: Introduction · p. 1

Problem formulation

• TOWN-VLA treats retrieval augmentation as a prompt-authority problem rather than assuming retrieved strategy content should control the policy.The interface separates candidate quality from an explicit authorization decision at the frozen-policy boundary.Source: Introduction · pp. 1–2

Method

• TOWN-VLA ranks candidates with a fixed compatibility rule and admits at most one canonical compact instruction through a Top-2 fail-closed cascade.If neither candidate passes, the interface retains the exact Base Policy prompt while the frozen Base Policy remains the sole action generator.Source: Introduction · p. 2

Auditability

• Across 900 routes, 525 unauthorized routes recover hash-identical Base prompts and all 375 authorized prompts preserve the task signature.These checks operationalize exact restoration for rejected routes and constrained rendering for accepted routes.Source: Introduction · p. 2

Frozen policies and inference-time interfaces

• Frozen-policy research scales language-conditioned robot control through new policies, training data, embodiments, and action representations.TOWN-VLA differs by freezing the action generator and regulating external text through its existing instruction interface.Source: Related Work · p. 2

External reasoning and fast–slow authority

• External reasoning and fast–slow systems move semantic computation, reactive computation, or assistance closer to action selection.TOWN-VLA keeps its slow path external and treats it as a source of proposals rather than an automatic controller input.Source: Related Work · p. 2

Inspectable memory at the policy boundary

• External memory methods retrieve experience or action priors, whereas TOWN-VLA makes an external capsule inspectable and hashable before it crosses the policy boundary.The paper regulates whether and how such a candidate may alter the frozen policy input.Source: Related Work · p. 2, Figure 2 · p. 3

Prompt brittleness beyond language generation

• Prompt-brittleness studies show formatting and suffix sensitivity in open-loop language generation; this paper examines the resulting action consequences in a frozen closed-loop VLA.Its intervention is boundary regulation rather than prompt optimization.Source: Related Work · p. 2

Selective intervention and runtime assurance

• Assistance, routing, selective intervention, and runtime-assurance systems calibrate uncertainty, select experts, or supervise learned controllers.With a fixed action generator and fallback, TOWN-VLA implements rejection as exact Base-prompt restoration rather than controller switching.Source: Related Work · p. 2

TOWN-VLA interface

• Figure 2 presents TOWN-VLA as a pipeline that separates invocation, candidate ranking, prompt authorization, and frozen control.Logged scores, flags, and prompt hashes support route auditing, while Task-Prior is reserved for the oracle control.Source: Figure 2 · p. 3

Interface contract

• The frozen VLA remains the sole action generator while TOWN-VLA constrains only candidate-generated policy inputs.Policy parameters and the control loop remain unchanged throughout evaluation.Source: Problem Setup and Interface Contract · p. 3

Auditability

• The implementation logs candidate identifiers, scores, checker reasons, route state, and the resolved-prompt hash for auditability.Unless explicitly labeled oracle-side, results use g = 1, so the main evaluation measures prompt authority independently of benchmark labels and Task-Prior routing.Source: Problem Setup and Interface Contract · p. 3, Runtime States and Interface Guarantees · p. 4

Compatibility-Reranked Capsule

• The interface parses task and context text into object–target fields and computes deterministic overlap and mismatch signals.The same frozen parser supplies normalized object, relation, and target fields for route identity.Source: Compatibility-Reranked Capsule · p. 3

Candidate preparation

• Candidates are retrieved and reranked textually, but a hard checker alone grants authorization.Reranking changes inspection order without itself permitting prompt intervention.Source: Compatibility-Reranked Capsule · pp. 3–4

Top-2 Fail-Closed Cascade

• The Top-2 cascade checks at most two candidates, short-circuits at the first eligible one, and falls back to the exact Base prompt when neither passes.Accepted context is rendered with a fixed template; fallback adds no retrieved text or rejection message.Source: Top-2 Fail-Closed Cascade · p. 4

Runtime states

• TOWN-VLA distinguishes bypass, inspected-but-unauthorized, and authorized runtime states, keeping slow-path computation separate from prompt authority.A single resolved prompt is fixed before rollout, and the slow path is not queried again during execution.Source: Runtime States and Interface Guarantees · p. 4, Task-Prior Admission as an Oracle… · p. 4

End-to-end robustness

• 73.1% LIBERO-Plus success adds 362 successes over 10,030 episodes versus 69.5% for the matched OpenVLA-OFT Base Policy.The paired-cell 95% interval is 1.89–5.45 points; gains occur on six of seven axes and all four suites.Source: Perturbation Breadth (Q1) · p. 5, Table 1 · p. 5, Table 2 · p. 5

Prompt-form failure

• 92.47% mean Base success collapses to 3.00% with raw appended text across three fixed-schedule executions.The paired diagnostic reports a −88.67-point change for the first raw candidate, and increasing retrieval depth neither worsens nor mitigates collapse.Source: Prompt-Form Collapse (Q2) · p. 5, Figure 3 · p. 5
• 499/500 states succeed with Base or exact restoration, while meaningful and length-matched meaningless appends both yield 0/500.Canonical wrong-object and wrong-target controls reach 497/500 and 496/500, respectively, separating prompt form from retrieved strategy content.Source: Prompt-Form Collapse (Q2) · p. 5, Table 3 · p. 5

Authorization and exact restoration

• 525 of 900 routed cases recover exact Base prompts, while all 375 authorized prompts preserve the task signature.In the separate 900-route audit, 450 routes bypass retrieval, 75 are rejected after inspection, and 375 are authorized with SigEq = 1.Source: Authorization and Exact Restoration (Q3) · p. 6

Transfer and selective computation

• 89.33/97.53 restricted-OOD success under spatial/object shift exceeds Base in both reported shift aggregates, while the bounded cascade scores 91.32% on five suites.The cascade is retained for exact restoration rather than claimed as an incremental success-rate mechanism over reranking.Source: Authorization and Exact Restoration (Q3) · p. 6, Table 5 · p. 6, Table 4 · p. 6

Selective computation

• Oracle-free admission authorizes only 2/36 held-out cells, matching 91.81% Base success across those 40 episodes.CLIP authorizes 0/36 cells, and always-on intervention remains 1.53 points below Base.Source: Selective Computation and Gate… · p. 6, Table 6 · p. 6

• Canonical rendering constrains accepted intervention, while exact restoration makes rejected routes reversible.Source: Discussion · p. 7
• The overall gain is strongest when perturbations degrade the executed instruction and largely neutral when Base remains strong.Source: Discussion · p. 7
• The evaluation uses a 48-entry same-domain memory, text-only compatibility, and a controlled single-task, single-operator physical study.Source: Discussion · p. 7
• Natural extensions include larger cross-domain memories, visually conditioned admission, and broader blinded robot trials.Source: Discussion · p. 7

Discussion

• Prompt-form controls reveal severe prompt-form collapse under raw appends to a frozen VLA.Source: Conclusion · p. 7

Conclusion

• TOWN-VLA separates candidate generation from authorization and restores the Base prompt exactly on rejected routes.Source: Conclusion · p. 7
• 3.61 points: TOWN-VLA improves LIBERO-Plus under matched evaluation without retraining the action generator.Source: Conclusion · p. 7
• 26.00 points: TOWN-VLA improves the PiPER real-robot platform under matched evaluation without retraining the action generator.Source: Conclusion · p. 7
• Reliable oracle-free admission is identified as the next frontier for selective slow-path control.Source: Conclusion · p. 7