Source-linked AI summary
A Schema Bounded Language Model for Refining Robot Policies Without Destabilizing Local Learning
Chongwen Dong, Mithun Paul Saint-Germain, Pinjari Asif, Carlo R. daCunha
TL;DR
The paper asks how persistent robot-specific LLM policies can be refined from shared completed-round information while local learning supplies every motion action. It presents a decentralized architecture combining round-level LLM policy refinement with robot-local tick-level control; in the fixed simulation, the complete configuration achieved success in all records and the lowest median and P90 completion times.
Problem
Because policy reasoning occurs at task boundaries while local control selects every tick, the design question is how shared completed-round information can refine persistent robot-specific LLM policies while local learning supplies motion actions.
Source: §I · p. 1Method
A decentralized two-tier architecture combines robot-local LLM policy ownership and UCB-guided round-level refinement with local Double DQN tick-level action selection and shared cross-LLM communication.
Source: A. System Model and Overall Framework · p. 2, §I · p. 2Results
All 90 robot–round records for C4 reached the goal, and C4 had the lowest observed median (42 ticks) and P90 (73.2 ticks) across the four configurations.
Source: D. C4: Complete Multi-Timescale… · p. 10, §VI · p. 10Takeaways & Limitations
The strongest observed advantage was configuration-level rather than uniform across robots or rounds.
Source: §VI · p. 11Within-configuration records share persistent policy and learning states, so the summaries are descriptive rather than estimates based on independent trials.
Source: F. Primary Metrics and Analysis · p. 8
Abstract
from arXiv · showhide
This paper addresses navigation by composite heterogeneous robots in a decentralized system when policy reasoning and local control operate at different update levels. In a NetLogo--Python implementation, three robots share motion dynamics but use different LLM backends. Each robot independently combines a large language model (LLM) policy agent, an Upper Confidence Bound (UCB) bandit, and a Double Deep Q-Network (Double DQN) controller; no central LLM generates team actions. LLM inference is confined to round-level policy generation and refinement rather than tick-level action selection. The robots perform cross-LLM communication through a shared round summary containing policies, outcomes, and learning feedback. UCB performs refinement-mode selection, and the policy-conditioned Double DQN performs tick-level action selection from navigation variables, active policy parameters, and the LLM action prior. Each of the four configurations was evaluated over 30 rounds. In the fixed simulation, the complete configuration reached the goal in all 90 correlated robot--round records and achieved the lowest median completion time (42 ticks) and P90 (73.2 ticks); its median was 25.0--39.1\% lower than those of the other configurations. These observations provide descriptive, configuration-level evidence from the evaluated configurations.
I. INTRODUCTION
The introduction identifies a mismatch between round-level policy reasoning and tick-level motion control, then proposes a decentralized architecture for refining robot-specific policies while local controllers select actions.
- LLM inference is confined to round boundaries for policy generation and refinement, while robot-local controllers select motion actions at every tick without per-action LLM calls.This separates policy-level reasoning from the local control loop.Source: §I · pp. 1–2
- Prior work studied multi-robot dialogue, task allocation, decentralized planning, and policy revision –, alongside hierarchical language reasoning and execution,; this paper asks how shared round information can refine persistent robot-specific policies.Source: §I · p. 1
- The proposed two-tier architecture combines robot-local policy ownership, UCB-guided refinement, and Double DQN control, with one LLM assigned to each robot.Each robot retains its own policy and local learning and control mechanisms.Source: §I · pp. 1–2
- A shared board mediates cross-LLM exchange of completed-round policies, outcomes, rationales, DQN feedback, and UCB advice while preserving robot-local policy ownership.Source: §I · p. 2
II. RELATED WORK
Related work distinguishes multi-robot systems by where decision authority resides, when language reasoning enters control, and how reinforcement learning supports execution. The proposed system gives each robot its own LLM policy and learning state, with shared communication limited to completed-round information.
II. RELATED WORK
- Prior systems distribute decision authority across centralized, decentralized, and hybrid organizations, and communication topology can affect coordination performance and token use.The reviewed examples include centralized task assignment, locally coordinated heterogeneous robots [5], hybrid motion planning, capability-aware allocation, modular embodied agents, and embodiment-aware planning.Source: §II · p. 2
- Dedicated LLMs generate and refine robot-specific policies at round boundaries, while robot-local controllers select actions at each tick.This contrasts with approaches that query an LLM before decentralized neural-policy execution or separate LLM reasoning from sensorimotor feedback,.Source: §II · p. 2
- The system combines persistent one-LLM-per-robot policy ownership with completed-round cross-LLM communication, while each robot retains its own policy and learning states.The shared board transports completed-round information; it does not generate team policies or robot actions centrally.Source: §II · p. 2
- Unlike approaches where RL learns the principal policy from language-provided specifications, RL here assists LLM policy refinement and supplies tick-level action selection and execution feedback.UCB1 selects the refinement mode for the next LLM update; robot-local Double DQN draws on DQN and Double Q-learning,.Source: §II · p. 2
III. FRAMEWORK
The framework combines robot-local, policy-conditioned control with round-level LLM policy refinement guided by UCB and a shared board. LLMs operate only at initialization and eligible round boundaries, while local controllers select tick-level actions.
A. System Model and Overall Framework
- Each robot pairs its dedicated LLM policy agent and UCB refinement selector with a policy-conditioned Double DQN; decision authority and persistent control state remain robot-local.The shared board carries completed-round policies, outcomes, rationales, DQN feedback, and UCB advice, but each LLM generates only its own robot’s next policy.Source: A. System Model and Overall Framework · pp. 2–3
- The coupled rollout loop applies each active policy through NetLogo, trains each robot’s Double DQN from local rewards and next states, and processes policy updates after all turtles terminate.At round boundaries, the system records outcomes, credits UCB, selects the next refinement mode, and may query the dedicated LLMs concurrently before resetting the task.Source: A. System Model and Overall Framework · p. 3
B. Round-Level Policy Refinement and Cross-LLM Communication
- Each robot’s LLM generates an initial policy and is called once after each completed nonfinal round, totaling exactly K calls per robot over K rounds.No LLM is called during tick-level control or after the final round; UCB selects the refinement mode at eligible boundaries.Source: B. Round-Level Policy Refinement and… · p. 3
- Network parameters, replay memory, exploration rate, UCB state, policy histories, and lifetime statistics persist across rounds, while only round-specific DQN action counts and loss statistics are cleared.At eligible boundaries, the system publishes completed-round records through the shared board and installs sanitized LLM policies before resetting the turtles.Source: B. Round-Level Policy Refinement and… · p. 3
TURN_ONLY, TURN_AND_FORWARD,
The design separates round-level LLM policy generation and refinement from tick-level Double DQN action selection. LLM recommendations act as soft priors, while robot-local UCB1 guides refinement-mode selection.
TURN_ONLY, TURN_AND_FORWARD,
- The DQN treats the LLM-recommended action as a soft prior, then maps its selected category to policy-conditioned translation and bounded goal-directed rotation.The recommendation is conditioned on distance and signed goal-angle error; the DQN may select a different category.Source: TURN_ONLY, TURN_AND_FORWARD · p. 3
- Invalid or missing policy fields retain defaults, while failed initialization loads the complete default policy and failed later refinement preserves the active policy.This ensures an executable policy before the first rollout without allowing malformed later output to replace an established controller.Source: TURN_ONLY, TURN_AND_FORWARD · pp. 3–4
- Each robot’s LLM receives the shared board but only its own policy, history, and next advice; concurrent calls load returned policies only onto their corresponding robots.Mission, rationale, expected DQN improvement, and cross-LLM learning summary are recorded for the next board but excluded from tick-level action selection.Source: TURN_ONLY, TURN_AND_FORWARD · p. 4
C. UCB-Guided Policy Refinement
- UCB1 provides each robot a local exploration–exploitation rule over five interpretable LLM refinement modes.Source: C. UCB-Guided Policy Refinement · p. 5
SMALL_MUTATION, EXPLORE_NEW,
UCB selects round-level policy-refinement advice, while each robot’s policy-conditioned Double DQN selects tick-level actions using navigation, policy, and LLM-prior inputs. The passages specify how refinement modes are credited and how the local controller is conditioned and trained.
SMALL_MUTATION, EXPLORE_NEW,
- UCB’s natural-language refinement arms retain, adopt shared components, mutate, explore, or combine policies; untried arms follow a fixed order, and arms select neither actions nor LLM backends.After all arms are observed, UCB selects the next refinement mode; the evaluated implementation fixes c = 1.4 for every robot.Source: SMALL_MUTATION, EXPLORE_NEW · p. 5
- The initial LLM policy receives no arm credit; credited rollouts update arm counts and empirical means, with successful-rollout credit scaled from zero at failure to one at the optimistic movement lower bound.Credit decreases with successful completion time, and accumulated tick-level DQN reward is excluded because the bandit credits the round-level refinement mode.Source: SMALL_MUTATION, EXPLORE_NEW · p. 5
D. Policy-Conditioned Local Double DQN Control
- Each robot’s Double DQN has two 64-unit ReLU hidden layers, five action-value outputs, and a 15-dimensional state combining navigation, active-policy, and LLM-prior components.The navigation feature layout includes distance and signed angle error; policy conditioning distinguishes categorical actions when policy parameters change their physical magnitudes.Source: D. Policy-Conditioned Local Double DQN… · pp. 5–6
- A fifth state entry stores the preceding distance in replay transitions as a one-step progress signal, then synchronizes with current distance for the next online action.This preserves the same feature layout across online selection, replay storage, and minibatch training.Source: D. Policy-Conditioned Local Double DQN… · pp. 5–6
- In C4, the LLM-prior bias is β = 0.05 and guides exploitation without preventing Q-values from selecting another action; C1 and C2 vanilla DQN controllers omit it.The prior affects online-network next-action selection but is not added to the target-network value.Source: D. Policy-Conditioned Local Double DQN… · p. 6
- Double DQN training uses Adam at learning rate 10−3, batch size 32, replay capacity 5000, and target-network synchronization every 100 successful updates; exploration decays from one to a 0.05 floor.Learning begins after the replay buffer reaches 32 transitions, and exploration decays by 0.997 after each successful mini-batch update.Source: D. Policy-Conditioned Local Double DQN… · p. 6
IV. EXPERIMENTAL DESIGN
The experiment compares four decentralized robot-controller configurations under fixed simulation conditions, using round-level policy updates and tick-level control. Its analysis emphasizes descriptive success and completion-time measures while accounting for persistent learning states and fixed robot–LLM pairings.
A. Evaluation Objectives
- RQ1–RQ4 assess joint communication/UCB effects, learned versus direct tick control, policy-conditioned Double DQN versus vanilla DQN, and adaptation in C1 and C4.Because each backend remains paired with the same robot, robot-wise outcomes characterize instantiated composite robots rather than rank LLM backends.Source: A. Evaluation Objectives · pp. 6–7
B. Controller Configurations
- C1 disables shared communication and UCB, C2 enables both, C3 replaces DQN with direct regional preferences, and C4 uses a 15-dimensional policy-conditioned Double DQN.C1–C2 changes communication and UCB jointly; C2–C3 removes learned tick control, while C2–C4 is bundled and cannot isolate Double DQN effects.Source: B. Controller Configurations · p. 7, Table II · p. 7
C. Common Simulation Setup
- Three robots with fixed starts, a shared goal, five-action motion interface, and physical transition rules are evaluated for 30 rounds, yielding 90 correlated robot–round records per configuration.Each round allows up to 1000 ticks; policy and learning states persist across rounds, so records are repeated measurements rather than independent replicates.Source: C. Common Simulation Setup · p. 7
D. LLM Backends and Simulation View
- Three dedicated LLM backends generate robot-specific policies, but fixed pairings with start poses and learners make agent-level differences properties of composite robots, not isolated models.The endpoints include service-specific quantized Llama variants with undisclosed serving and quantization parameters; published model characteristics motivate backend diversity but do not predict navigation performance [20] [22].Source: D. LLM Backends and Simulation View · p. 7, C. Common Simulation Setup · p. 7
E. Evaluation Protocol and Metrics
- Positions, headings, and round-local counters reset each round, while enabled UCB statistics, DQN state, policy history, and best-policy records persist; policies stay fixed during rollouts as tick controllers choose actions.The protocol records goal attainment and successful completion ticks; primary metrics are success rate and successful-tick distributions summarized by median, IQR, mean, and P90.Source: E. Evaluation Protocol and Metrics · pp. 7–8
F. Primary Metrics and Analysis
- Successful completion-time summaries use successful records only, while success rate uses all 90 robot–round records; one C2 Llama-3.1 composite-robot timeout is coded as Tmax = 1000.The primary efficiency statistic is median and IQR for right-skewed finish ticks; round traces retain timeouts, and early-versus-late comparisons describe horizon change rather than inferential effects.Source: F. Primary Metrics and Analysis · p. 8
- LBSE summarizes success-weighted efficiency relative to the shared movement lower bound Li = 20, while WDC0.10 summarizes the average cost of the slowest 10% of 90 observations.Both are descriptive configuration-level measures; LBSE aligns with the UCB efficiency signal, and WDC0.10 is a finite-sample upper-tail mean, not population CVaR or an independent-sample significance test.Source: F. Primary Metrics and Analysis · p. 8
V. RESULTS AND ANALYSIS
Across four bundled configurations, C4 had the strongest aggregate completion-time and normalized-efficiency results, while C3 had the lowest worst-decile cost. The evidence is descriptive and limited to the fixed simulation; the configurations do not isolate single-factor effects.
F. Evidence Scope
- The comparisons are descriptive repeated measurements, not independent replicates or evidence for significance tests or causal attribution, and apply only to the fixed NetLogo world, starts, goal, and LLM-to-robot mapping.Rounds share persistent policies, UCB statistics, replay buffers, network weights, exploration states, and shared-board information; the configurations also differ in multiple implementation choices.Source: §V · p. 8, E. Cross-Configuration Comparison · p. 10, F. Evidence Scope · p. 10
A. C1: Local LLM–DQN Baseline
- C1 reached all 90 goals, with a 69-tick median, 91.5-tick mean, and 105.8-tick P90; its pooled median fell from 98 ticks in rounds 1–5 to 60 in rounds 26–30.The largest successful value was 644 ticks, leaving a pronounced upper tail despite the faster late segment.Source: A. C1: Local LLM–DQN Baseline · p. 8, Figure 5 · p. 9
B. C2: Cross-LLM Communication, UCB, and Vanilla DQN
- C2 achieved 89/90 goals (98.9%); among successful observations, its median was 56 ticks, mean 72.7, and P90 89.2, with early and late medians of 87 and 48 ticks.The Llama-3.1 composite robot timed out in round 12. Its lower median relative to C1 is associated with jointly adding cross-LLM communication and UCB, which this comparison does not separate.Source: B. C2: Cross-LLM Communication, UCB,… · pp. 8–9, Figure 5 · p. 9
C. C3: Cross-LLM Communication and UCB Without DQN
- C3 reached all 90 goals and had a 57-tick median, 64.0-tick mean, and 89.0-tick P90, but its median rose from 46 ticks in rounds 1–5 to 78 in rounds 26–30.Its pooled median was close to C2’s, but its late-round trajectory differed; final-round completion times were 56, 146, and 184 ticks across the three robots.Source: Figure 5 · p. 9, C. C3: Cross-LLM Communication and UCB… · p. 9
D. C4: Complete Multi-Timescale Configuration
- C4 reached all 90 goals and had the lowest observed median (42 ticks), mean (56.3 ticks), and P90 (73.2 ticks); its median declined from 57 ticks early to 41 late.It still showed isolated 232- and 288-tick observations; the Phi-4 composite robot had a 35-tick median and 7-tick IQR, which does not establish an intrinsic ranking of LLM backends.Source: Figure 5 · p. 9, D. C4: Complete Multi-Timescale… · p. 10
E. Cross-Configuration Comparison
- C4’s median was 39.1%, 25.0%, and 26.3% lower than C1, C2, and C3, respectively; its P90 was 17.8% below C3’s 89.0 ticks, while success rates ranged only from 98.9% to 100%.The complete configuration reached 90/90 goals; C1 and C3 also reached 90/90, while C2 reached 89/90. These are configuration-level comparisons, not single-factor estimates.Source: Figure 6 · p. 9, Table III · p. 10, E. Cross-Configuration Comparison · p. 10
- C4 had the highest LBSE, 39.0%, 27.3%, and 27.2% above C1, C2, and C3, but C3 had the lowest worst-decile cost at 127.3 ticks, 16.1% below C4’s 151.8 ticks.C4 therefore had the strongest aggregate normalized efficiency in this run but did not dominate every slow-tail criterion.Source: Table IV · p. 10, E. Cross-Configuration Comparison · p. 10
VI. CONCLUSION
The paper presents a decentralized, multi-timescale robot system that keeps LLM policy reasoning at round boundaries and action selection at the tick level. In the fixed task, C4 had the strongest aggregate completion results, but the evidence is configuration-level and does not establish causal effects or generalization.
VI. CONCLUSION
- The system combines robot-local LLM policy ownership and UCB-guided round-level refinement with tick-level Double DQN control conditioned on navigation state, active policy, and regional-action prior.Cross-LLM communication occurs at round boundaries; physical actions are selected without repeated LLM calls.Source: §VI · p. 10
- Across the four configurations, C4 combined complete observed success with the lowest pooled median and P90 in the fixed task; C3 remained competitive in pooled median but worsened over the evaluation horizon.The strongest observed advantage was configuration-level rather than uniform across robots or rounds.Source: §VI · pp. 10–11
- Because configurations bundle implementation differences and share persistent policy and learning states, the results neither isolate each module’s causal contribution nor demonstrate generalization to other environments or physical robots.The 90 robot–round records per configuration are interpreted as repeated measurements, not independent replicates.Source: §VI · p. 11
A. Future Work
- Future experiments will use independent seeded repetitions, separate evaluation rollouts, single-factor ablations, RL-only and per-tick LLM baselines, and UCB-coefficient sensitivity analysis.Ablations will isolate cross-LLM communication, UCB advice, policy-conditioned state, Double DQN, reward design, and LLM-prior bias.Source: A. Future Work · p. 11
- Further studies will measure LLM latency, token use, response validity, and fallback frequency while varying assignments, communication quality, obstacles, and task geometry and testing physically heterogeneous platforms.These extensions are needed before claims about statistical reliability, computational efficiency, scalability, or real-world deployment.Source: A. Future Work · p. 11