PaperlayerView exact revision on arXiv

Source-linked AI summary

Privacy-Preserving Prompted Policy Search for Robotic Control

Ali Irshayyid, Feng Lin, Chong Li, Jun Chen

arXiv:2609.30554v1cs.ROcs.SYeess.SY

TL;DR

Cloud-based LLM policy optimization can expose raw policy parameters and reward histories, including proprietary control strategies. PP-ProPS keeps these values encoded on the client while guiding search with component feedback and bounded history; it outperforms Vanilla ProPS in seven of ten evaluated environments.

  • Problem

    Cloud-based LLM policy search requires repeatedly sharing raw policy parameters and rewards, which may reveal proprietary control strategies and policy evolution.

    Source: §I · pp. 1–2

  • Method

    PP-ProPS applies secret client-side transformations to parameters and reward feedback, adds component-level diagnostics, and limits prompt history with a bounded representation.

    Source: §I · p. 2, C. Component-Level Feedback and Reward… · p. 3, D. Bounded History Representation · p. 3

  • Results

    PP-ProPS outperforms Vanilla ProPS in seven of ten evaluated environments and remains comparable in the rest.

    Source: §I · p. 2

  • Takeaways & Limitations

    Encoded-space search remains effective for policies with up to 782 parameters, while bounded history supports open-weight LLMs and reduces prompt size and response time.

    Source: §I · p. 2

Abstract

from arXiv · show

Large language models (LLMs) have recently demonstrated promising capabilities as in-context policy optimizers for Reinforcement Learning (RL), enabling policy search driven by both numerical reward signals and natural language reasoning. However, deploying such methods in practice requires transmitting raw policy parameters and rewards history to cloud-based LLM APIs, exposing proprietary control strategies to third-party service providers. To address this issue, this paper introduces Privacy-Preserving Prompted Policy Search (PP-ProPS), a framework that enables LLM-guided policy optimization while keeping policy and environmental parameters confidential. PP-ProPS encodes policy parameters and reward values using secret client-side transformations before they are included in each API request, ensuring that the LLM provider observes only encoded policy parameters and scaled reward information. Furthermore, unlike Vanilla ProPS, the proposed framework does not require the true optimal episodic return to be known or disclosed to the LLM. Beyond protecting the optimization data, PP-ProPS improves the search process in two ways. First, it provides the LLM with individual reward components instead of only a single total return, offering more informative feedback about each candidate policy. Second, it uses a bounded history that prevents the prompt from growing indefinitely, improving search with high-dimensional policies and supporting the use of open-weight LLMs. The proposed PP-ProPS is evaluated on both continuous and discrete control problems spanning Multi-Joint dynamics with Contact (MuJoCo) locomotion, classic control, highway driving, and robotic arm manipulation. Compared to Vanilla ProPS, the proposed PP-ProPS outperforms ProPS in seven of the ten evaluated tasks, and surpasses conventional RL methods including PPO, SAC, and TRPO, in five of the six tasks.

I. INTRODUCTION

Cloud-based LLM policy search can expose proprietary parameters and reward histories. PP-ProPS keeps original values on the client while providing encoded search information and additional feedback to the LLM.

I. INTRODUCTION

  • Cloud LLM policy search repeatedly sends raw parameters and rewards, which may disclose proprietary control strategies and reveal how a policy evolves.Source: §I · pp. 1–2
  • PP-ProPS applies secret client-side transformations to policy parameters and reward feedback, so the external LLM receives only encoded parameters and scaled rewards.The original values and encoding variables remain local.Source: §I · p. 2
  • PP-ProPS outperforms Vanilla ProPS in seven of ten evaluated environments and remains comparable in the rest.The approach demonstrates policy search in encoded parameter spaces for policies with up to 782 parameters.Source: §I · p. 2
  • Component-level reward and diagnostic feedback improves average policy-search performance in nine of ten evaluated environments compared with ProPS's scalar episodic feedback.Source: §I · p. 2
  • A bounded top-K history prevents prompt growth with iterations; retaining ten highest-reward candidates reduces final prompt size by 80.2% and mean LLM response time by 74.9%.It maintains comparable policy-search performance and supports a 20B-parameter open-weight LLM.Source: §I · p. 2

II. METHODOLOGY

The methodology formulates policy optimization as iterative LLM-guided search, then describes how PP-ProPS keeps the optimization history encoded on the trusted client.

A. Problem formulation

  • Episodic RL is formulated with a parameterized policy πθ and cumulative return R(τ), with the objective of maximizing expected return.Source: A. Problem formulation · p. 2
  • ProPS uses an LLM to propose policy parameters from a prompt containing task information, constraints, the optimal objective value, and evaluated parameter-return history.Source: A. Problem formulation · p. 2
  • Each proposed policy is executed in the environment and its episodic return is added to the history, enabling repeated search without policy gradients, a value function, or an environment model.Source: A. Problem formulation · p. 2

II. METHODOLOGY

  • PP-ProPS transmits only encoded optimization history; original parameters, secret encoding variables, policy execution, and environment interactions remain on the trusted client.Source: A. Problem formulation · p. 2, §II · p. 2

B. Parameter Encoding

PP-ProPS uses a secret diagonal transformation to encode policy parameters and secret positive scaling to encode reward feedback. It also supplies component-level feedback without disclosing the true optimal return.

B. Parameter Encoding

  • A secret positive diagonal matrix is independently sampled for each run and held fixed across its LLM queries, preserving a consistent encoded coordinate system.Each diagonal coefficient is independently sampled.Source: B. Parameter Encoding · p. 3
  • The diagonal transformation preserves parameter layout and identities while hiding the true policy parameters from the LLM.The LLM can still use the task description, policy structure, and relationships observed in the optimization history.Source: B. Parameter Encoding · p. 3

C. Component-Level Feedback and Reward Encoding

D. Bounded History Representation

PP-ProPS limits prompt history to the K highest-reward candidates while retaining the complete encoded history locally. This bounds prompt size as optimization proceeds.

  • Unbounded history grows with both evaluated candidates and policy dimension, potentially making prompts prohibitive for long runs or high-dimensional policies.Source: D. Bounded History Representation · p. 3
  • Each prompt includes only the K highest-reward entries from the complete locally retained encoded history.Source: D. Bounded History Representation · p. 3
  • The selected entries are ordered from lowest to highest encoded reward, placing the best candidate last; history size is O(KD) rather than O(HiD).The positive reward scale preserves the original reward ordering, and the prompt history remains fixed in size as iterations increase.Source: D. Bounded History Representation · p. 3

E. Privacy-Preserving Prompted Policy Search

PP-ProPS combines client-side parameter and reward encoding with component-level feedback and bounded prompt history. Its evaluation spans multiple control domains and compares PP-ProPS-based configurations with other LLM-guided and standard RL methods.

E. Privacy-Preserving Prompted Policy Search

  • At each run’s start, the client generates secret parameter and reward encodings, evaluates random warm-up policies, and keeps original returns local to identify the best policy.Source: E. Privacy-Preserving Prompted Policy… · p. 4
  • Each iteration prompts the LLM with the top K candidates and task guidance, decodes and evaluates its proposal locally, and adds the encoded feedback; the true optimum is withheld.The decoded best policy can be deployed directly without encoding during execution.Source: E. Privacy-Preserving Prompted Policy… · p. 4

III. EXPERIMENTAL SETUP

  • Ten environment configurations span MuJoCo locomotion, classic control, highway driving, and robotic arm manipulation, with continuous and discrete action spaces.Source: §III · p. 4, Figure 3 · p. 5
  • All environments use linear policies; Ant and Humanoid observation configurations yield policy dimensions from 88 for Ant-Ultra to 782 for Humanoid-Torso.The policy dimension is D = (ns + 1)na.Source: §III · pp. 4–5
  • The four ProPS configurations vary parameter encoding, component-level feedback, reward encoding, and whether the maximum achievable return is disclosed.Vanilla ProPS uses original parameters and scalar rewards; PP-ProPS encodes parameters and scales reward-valued feedback while withholding the maximum return.Source: §III · p. 5
  • R2PO uses Search-LLM proposals and Critic-LLM revisions, with 10 warm-up policies and 25 two-stage iterations to match 50 LLM calls.Each initial and revised candidate is evaluated over 20 episodes.Source: §III · p. 5
  • ProPS variants use gpt-oss-20b and top-10 history; Fig. 4 shows five-run mean rewards and standard deviations after a 10-policy random warm-up.Source: §III · p. 5, Figure 4 · p. 5
  • Seven standard RL algorithms are evaluated with Stable-Baselines3 default hyperparameters; SAC, TD3, and DDPG apply only to continuous actions, and DQN only to discrete actions.Source: §III · p. 5

IV. RESULTS

The results compare episodic returns across methods under a stated evaluation protocol and show comparable reward-curve trends for PP-ProPS and unencoded configurations in two representative environments. The supplied results also note early termination from context overflow.

IV. RESULTS

  • PP-ProPS follows reward-curve trends comparable to unencoded configurations in CartPole and FetchReachDense despite receiving only encoded parameters and rewards.All methods use a 10-policy random warm-up; returns are reported as the mean of the best policy with standard deviations across five runs.Source: §IV · p. 5
  • Table II compares episodic rewards for Vanilla ProPS, R2PO, PE-ProPS, DF-ProPS, and PP-ProPS across the evaluated environments.The section reports mean episodic return of the best policy under each method's evaluation protocol, with standard deviations across five runs.Source: §IV · p. 5, Table II · p. 6
  • Three runs terminated early because of context overflow, and the reported policies are the best found before overflow.Source: §IV · p. 6
  • Figure 5 relates LLM response time for Ant-Ultra (D = 88) to retained history size and prompt size, highlighting bounded K = 10 and unbounded histories.Source: Figure 5 · p. 6

A. Bounded-History Analysis

The bounded-history analysis finds that retaining the ten highest-reward candidates substantially reduces prompt size and response time while preserving performance comparable to full history. Unbounded history yields only small return improvements across the three Ant policy dimensions.

A. Bounded-History Analysis

  • 0.84%, 0.03%, and 0.38% are the unbounded-history return improvements for Ant policies with D = 88, 136, and 224, respectively.The top-K representation therefore maintains performance comparable to complete history.Source: A. Bounded-History Analysis · p. 6
  • Table III compares bounded and unbounded history representations across the Ant policy dimensions.Source: Table III · p. 6
  • 80.2% lower prompt size and 74.9% lower mean LLM response time result from top-10 history, reducing the prompt from 114,000 to 22,000 characters.The bounded representation uses K = 10 and is adopted for other LLM-guided evaluations unless stated otherwise.Source: A. Bounded-History Analysis · p. 6

B. Effect of Parameter Encoding

Parameter encoding improves average returns for the three Ant configurations but has mixed effects across other tasks. The paper hypothesizes that encoding reduces reliance on pretrained assumptions about parameter magnitudes.

B. Effect of Parameter Encoding

  • For Ant-Ultra, encoding raises return from 942.75 to 998.64 and lowers standard deviation from 59.11 to 4.75.Across the three Ant configurations, average returns increase by approximately 1.7–5.9%, generally with reduced run-to-run variability.Source: B. Effect of Parameter Encoding · p. 6
  • Encoding may reduce bias from pretrained assumptions about parameter magnitudes by encouraging reliance on observed parameter–return relationships.The paper presents this mechanism as a hypothesis.Source: B. Effect of Parameter Encoding · p. 7
  • Encoding lowers average returns for all three Humanoid configurations and reduces FetchReachDense and Highway returns from −0.60 to −0.78 and 122.57 to 90.04, while CartPole and Acrobot remain nearly unchanged.Despite the Humanoid decreases, encoded in-context search remains feasible at dimensions up to 782; the authors suggest scalar feedback may be insufficient for tasks needing more behavioral guidance.Source: B. Effect of Parameter Encoding · p. 7

C. Effect of Component-Level Reward Feedback

Component-level and diagnostic reward feedback improves average returns in nine of ten environments. It supplies behavioral information that can distinguish policies with similar total returns but different behaviors.

C. Effect of Component-Level Reward Feedback

  • Component-level and diagnostic feedback improves average return in nine of ten environments, including Ant gains of 3.1–6.9% and approximately 6.2% gains for Humanoid-Ultra and Humanoid-Torso.Humanoid-Joints shows a small decrease that remains limited relative to reported variation.Source: C. Effect of Component-Level Reward… · p. 7
  • Forward motion, control effort, health, and contact penalties help the LLM distinguish policies that reach similar total returns through different behaviors.Source: C. Effect of Component-Level Reward… · p. 7
  • Highway return rises from 122.57 to 151.46, an approximately 23.6% improvement, while Acrobot, CartPole, and FetchReachDense also improve.The respective returns change from −77.60 to −71.60, 495 to 495.2, and −0.60 to −0.51.Source: C. Effect of Component-Level Reward… · p. 7

D. Performance of Complete PP-ProPS

The complete PP-ProPS configuration outperforms Vanilla ProPS in seven of ten environments despite encoding policy parameters and reward feedback and withholding the optimal return. Its performance varies by task, with gains in several environments and decreases across the Humanoid configurations.

D. Performance of Complete PP-ProPS

  • Seven of ten evaluated environments favor complete PP-ProPS over Vanilla ProPS, despite encoding parameters and rewards and withholding the optimal return.The complete configuration includes component-level feedback.Source: D. Performance of Complete PP-ProPS · p. 7
  • Approximately 7.3% for Ant-Torso and 26.1% for Highway are the largest reported gains over Vanilla ProPS.Source: D. Performance of Complete PP-ProPS · p. 7
  • PP-ProPS also improves Acrobot and FetchReachDense and reaches a CartPole return of 499.9 with the lowest variability among four configurations.Source: D. Performance of Complete PP-ProPS · p. 7
  • PP-ProPS outperforms R2PO in five of seven tasks with reported results, while performance decreases across all three Humanoid configurations.Source: D. Performance of Complete PP-ProPS · p. 7
  • For Ant, the trajectory prompt exceeds the context window before the first Critic-LLM query; three of five Humanoid-Torso runs also stop before completing all iterations.Those Humanoid-Torso returns use the best policy found before overflow; PP-ProPS completes all evaluations without exposing original parameters or reward scale.Source: D. Performance of Complete PP-ProPS · p. 7

E. Standard RL Baselines

Across the evaluated task groups, PP-ProPS configurations outperform standard RL baselines in five of six groups and remain competitive on Ant-Ultra. The conclusion also reports results for policies up to 782 parameters and identifies nonlinear policies and encoding as future work.

E. Standard RL Baselines

  • On Ant-Ultra, PPO achieves the highest return at 1000.16, while DF-ProPS, PE-ProPS, and PP-ProPS remain within approximately 0.6% of it.PE-ProPS has a standard deviation of 4.75, compared with 15.06 for PPO.Source: E. Standard RL Baselines · p. 7, Table IV · p. 8
  • PP-ProPS outperforms all evaluated RL baselines on Humanoid-Ultra, Acrobot, CartPole, Highway, and FetchReachDense.For FetchReachDense, DF-ProPS achieves −0.51 versus −7.39 for DDPG, the strongest evaluated RL baseline.Source: E. Standard RL Baselines · p. 7, Table IV · p. 8
  • Five of six task groups show PP-ProPS outperforming evaluated RL baselines, with a 0.6% decrease for the remaining group.The configurations remain competitive on Ant-Ultra.Source: §V · p. 7, E. Standard RL Baselines · p. 7, Table IV · p. 8

V. CONCLUSION

  • The conclusion reports effectiveness for policies containing up to 782 parameters, with final prompt size reduced by approximately 80.2% and mean LLM response time by 74.9%.Source: §V · p. 7
  • Future work will evaluate nonlinear control policies with physical experiments and investigate nonlinear encoding against parameter-recovery and information-inference attacks.Source: §V · pp. 7–8

Related papers

Loading 2609.30554v1…