PaperlayerView exact revision on arXiv

Source-linked AI summary

A Schema Bounded Language Model for Refining Robot Policies Without Destabilizing Local Learning

Chongwen Dong, Mithun Paul Saint-Germain, Pinjari Asif, Carlo R. daCunha

arXiv:2609.05133v1cs.ROcs.AI

TL;DR

The paper addresses decentralized navigation when LLM policy reasoning and local robot control operate at different update levels. It combines robot-local LLM policies, UCB-guided round-level refinement, shared-board communication, and policy-conditioned Double DQN tick-level control. In the fixed simulation, the complete configuration achieved success in all 90 robot–round records and the lowest median and P90 completion times, with descriptive configuration-level evidence.

  • Problem

    The paper asks how persistent robot-specific LLM policies can be refined from shared completed-round information while local learning supplies every motion action.

  • Method

    Each robot combines a dedicated LLM policy agent, UCB refinement, and policy-conditioned Double DQN control, with LLM inference confined to round boundaries and communication through a shared board.

  • Results

    All 90 robot–round records under C4 reached the goal, while C4 achieved the lowest observed median completion time of 42 ticks and P90 of 73.2 ticks.

  • Takeaways & Limitations

    The observed advantage was configuration-level: C4 combined complete observed success with the lowest pooled median and P90 in the fixed task, without uniformly lowest agent-level medians.

Abstract

from arXiv · show

This paper addresses navigation by composite heterogeneous robots in a decentralized system when policy reasoning and local control operate at different update levels. In a NetLogo--Python implementation, three robots share motion dynamics but use different LLM backends. Each robot independently combines a large language model (LLM) policy agent, an Upper Confidence Bound (UCB) bandit, and a Double Deep Q-Network (Double DQN) controller; no central LLM generates team actions. LLM inference is confined to round-level policy generation and refinement rather than tick-level action selection. The robots perform cross-LLM communication through a shared round summary containing policies, outcomes, and learning feedback. UCB performs refinement-mode selection, and the policy-conditioned Double DQN performs tick-level action selection from navigation variables, active policy parameters, and the LLM action prior. Each of the four configurations was evaluated over 30 rounds. In the fixed simulation, the complete configuration reached the goal in all 90 correlated robot--round records and achieved the lowest median completion time (42 ticks) and P90 (73.2 ticks); its median was 25.0--39.1\% lower than those of the other configurations. These observations provide descriptive, configuration-level evidence from the evaluated configurations.

I. INTRODUCTION

The paper targets decentralized heterogeneous-robot navigation where LLM policy reasoning and local motion control operate at different update levels. It proposes robot-local policy ownership, cross-LLM communication, and separate round-level refinement and tick-level execution.

  • Motivation: The design problem is retaining LLM reasoning at policy boundaries while providing a separate mechanism for tick-level action selection.
  • Architecture: The system uses a decentralized two-tier architecture combining LLM-guided policy refinement with local DQN-based control.
  • Architecture: Each robot owns its LLM policy, UCB-guided refinement, and Double DQN controller, with no central LLM generating team actions.
  • Temporal separation: LLMs generate and refine robot-specific policies at round boundaries, while robot-local controllers select actions at every tick without per-action LLM invocation.
  • Experimental system: The implementation evaluates three robots with shared embodiment and motion rules but different LLM backends, policies, learning states, and trajectories.

B. Round-Level Policy Refinement and Cross-LLM Communication

The event-driven schedule separates persistent round-level policy refinement from local tick-level learning and execution. UCB evaluates completed rollouts and selects refinement advice before each subsequent non-final round.

  • Schedule: Each dedicated LLM generates one initial policy, then receives one refinement call after every completed non-final round.
  • Schedule: Evaluating K rounds requires exactly K LLM calls per robot, with no LLM invocation inside the tick-level loop or after the final round.
  • Update cycle: After terminal states, the system records rollout history, credits UCB, selects advice, constructs the shared board, and installs sanitized outputs.
  • Policy representation: The structured policy contains a numerical motion vector and four regional action preferences for far/near and aligned/misaligned states.

TURN_ONLY, TURN_AND_FORWARD,

The local controller converts structured LLM policy outputs into executable motion while preserving the separation between soft language guidance and tick-level Double DQN control. The implementation also shares completed-round information while keeping persistent state robot-local.

  • Action selection: The LLM-recommended action is a soft prior for the DQN rather than a compulsory motion action.
  • Action selection: The active policy maps DQN-selected categories to bounded translation and rotation, including FORWARD, TURN_ONLY, and turn-and-forward actions.
  • Controller: The system uses a 15-D input and five action categories for local control.
  • Communication and ownership: Each robot retains local turtle, policy, UCB, and Double DQN state, while the shared board carries completed-round information between robots.
  • Policy safety: Malformed policy outputs fall back to a complete default at initialization or the previously active control policy during later refinement.
  • Communication: The shared board aggregates outcomes, policy fields, rationales, DQN feedback, and UCB advice; textual fields do not enter tick-level action selection.
  • Update cycle: After all turtles terminate, tick-level learning and UCB updates precede shared-board construction and parallel LLM regeneration on non-final rounds.

SMALL_MUTATION, EXPLORE_NEW,

UCB selects among refinement modes rather than motion actions or model backends. Its reward credits completed round-level policy refinements using success and completion-time information.

  • Refinement modes: The refinement modes retain, borrow, mutate, explore, or combine policy components from local and shared information.
  • Credit assignment: The initial LLM policy is not associated with a UCB arm and is therefore not credited.
  • Credit assignment: Successful rollout credit is based on completion time relative to an optimistic movement lower bound, while failure receives zero credit.
  • Credit assignment: The bandit credits the round-level refinement mode rather than individual tick-level actions.
  • Mode selection: After all refinement arms have been observed, UCB selects the next mode using the stated selection rule.
  • UCB role: UCB selects only the next refinement mode; the robot’s LLM remains responsible for numerical and categorical policy synthesis.

D. Policy-Conditioned Local Double DQN Control

The local controller uses Double DQN to select tick-level actions while conditioning decisions on navigation state, active policy parameters, and LLM action priors.

  • Controller architecture: Each robot uses an online and target network in a local Double DQN controller.The controller is implemented per robot and supports tick-level action selection.
  • State representation: The 15-dimensional state combines navigation, active-policy, and LLM-prior components.Each block contributes five entries to the network input.
  • State representation: Policy conditioning prevents identical categorical actions from appearing equivalent after an LLM changes physical step or turn magnitudes.The active policy parameters therefore affect how the controller interprets actions.
  • Action selection: C4 adds an LLM-prior bias with β = 0.05 during behavior-policy selection, while learned Q-values can still select another action.The prior guides exploitation without overriding learned action differences.
  • Double DQN update: Double DQN selects the next action with the online network and evaluates it with the target network.The prior bias affects online action selection but is not added to the target-network value.
  • Training: Training uses Smooth L1 loss over sampled replay transitions, with Adam optimization and periodic target-network synchronization.The implementation uses batch size 32, replay capacity 5000, and synchronization every 100 successful updates.
  • Reward design: The reward combines independently activated navigation, goal, and timeout components during each transition.Exact-goal arrival and timeout are evaluated independently, with their terminal contributions canceling when simultaneous.
  • Procedure: Algorithm 1 specifies the end-to-end procedure for the evaluated controller configuration.The procedure integrates policy-conditioned action selection, replay training, and the implemented reward design.

A. Evaluation Objectives

The evaluation compares four controller configurations to examine communication, UCB refinement, local tick-level learning, and policy-conditioned Double DQN effects.

  • Evaluation questions: The study evaluates the joint effect of cross-LLM communication and UCB refinement-mode selection.This is the focus of RQ1.
  • Evaluation questions: RQ2 compares learned tick-level local control with direct execution of LLM-generated regional rules.The comparison holds the same round-level components fixed.
  • Evaluation questions: RQ3 compares policy-conditioned Double DQN in C4 with state-only vanilla DQN in C2.This isolates the stated controller contrast at the evaluation-question level.
  • Evaluation questions: RQ4 characterizes round-level adaptation and robot-specific outcomes in C1 and C4.Robot-wise results describe instantiated robot configurations rather than ranking LLM backends.

B. Controller Configurations

The evaluation fixes the simulation and compares four configurations that vary communication, UCB refinement, and tick-level control while preserving dedicated LLMs per robot.

  • Configuration design: Each configuration retains one dedicated LLM per robot and follows the same round-level schedule.The world, assignments, action vocabulary, and motion interface are fixed across configurations.
  • Configuration design: C1 uses robot-local history, disables the shared board and UCB, and applies a five-dimensional vanilla DQN.C2 adds cross-LLM communication and UCB refinement-mode selection while retaining vanilla DQN.
  • Configuration design: C3 retains round-level LLM–UCB components but directly executes the current policy’s regional action preference without DQN.C4 instead uses the 15-dimensional policy-conditioned Double DQN.
  • Comparison boundaries: C1–C2 jointly change cross-LLM communication and UCB, preventing their effects from being separated.The C2–C4 comparison is also bundled because controller and training differences change together.
  • Evaluation protocol: Each configuration runs for 30 rounds with at most 1000 ticks per round, producing 90 correlated robot–round records.Persistent policy and learning states mean these records are analyzed as repeated measurements rather than independent replicates.
  • Common simulation setup: Figure 4 depicts a representative rollout with three robots, a common goal, runtime action labels, and local best-tick records.The robots are paired with Llama3.1, Phi4, and Llama3.3 in the displayed setup.
  • Evaluation protocol: Round starts reset positions, headings, and local counters, while UCB statistics, DQN state, policy history, and best-policy records persist.The round-level policy remains fixed during each rollout while the tick controller produces physical actions.
  • Evaluation metrics: The simulator records goal attainment and successful completion ticks, summarizing success and completion-time distributions with median, IQR, mean, and P90.Round-wise traces and early-to-late medians describe adaptation over the fixed horizon.

F. Primary Metrics and Analysis

The evaluation uses success, completion-time, adaptation, efficiency, and upper-tail metrics, with summaries interpreted descriptively because persistent states create correlated robot–round records rather than independent trials.

  • Metric definitions: Success rate is computed over all 90 robot–round records, while completion-time summaries use only successful records.There are three robots and 30 rounds; Tmax = 1000 is treated as a timeout rather than a successful completion tick.
  • Metric definitions: The median with interquartile range is the primary efficiency statistic because successful finish-tick distributions are right-skewed.Round-wise traces use the median of the three recorded values, retaining timeouts at Tmax.
  • Adaptation: Positive I_c denotes fewer ticks in the final five rounds than in the first five, describing change across the evaluation horizon rather than an inferential effect.Early and late performance pool successful observations from rounds 1–5 and 26–30, respectively.
  • Efficiency: LBSE measures successful completions relative to the optimistic movement lower bound, with larger values indicating completions closer to that bound.The fixed start coordinates give L_i = 20 for all three robots, and LBSE_c lies in [0, 1].
  • Efficiency and upper-tail analysis: LBSE is an implementation-aligned diagnostic rather than independent validation because it averages the same efficiency signal used internally by UCB.The empirical worst-decile completion cost separately summarizes the slowest 10% of observations, but remains a descriptive finite-sample statistic rather than population CVaR or an independent significance test.

V. RESULTS AND ANALYSIS

The four bundled configurations show configuration-level differences in completion speed, temporal adaptation, and slow-tail behavior, with C4 achieving the strongest central completion-time results but not dominating every tail criterion. Because configurations change multiple components jointly, these comparisons do not isolate single-factor causal effects.

  • A. C1: Local LLM–DQN Baseline: C1 reached all 90 goals but had a 69-tick median and pronounced upper tail, including a 644-tick successful observation.Its pooled median declined from 98 ticks in rounds 1–5 to 60 ticks in rounds 26–30.
  • B. C2: Cross-LLM Communication, UCB, and Vanilla DQN: 98.9% success was recorded for C2, whose successful-observation median was 56 ticks and P90 was 89.2 ticks.C2 combined cross-LLM communication and UCB with vanilla DQN, but those additions were not separated.
  • C. C3: Cross-LLM Communication and UCB Without DQN: C3 achieved 90/90 successes and a 57-tick pooled median, but its late median increased to 78 ticks after an early median of 46 ticks.Its slowest observations concentrated near the end of the horizon, including final-round values of 56, 146, and 184 ticks.
  • D. C4: Complete Multi-Timescale Configuration: 42 ticks was C4’s lowest observed median, alongside a 56.3-tick mean and 73.2-tick P90 across 90 successful robot–round records.Its early median decreased from 57 ticks to 41 ticks in the final five rounds, despite isolated 232- and 288-tick observations.
  • E. Cross-Configuration Comparison: C4’s median was 39.1%, 25.0%, and 26.3% lower than C1, C2, and C3, while its P90 was 17.8% below C3’s 89.0 ticks.C1, C3, and C4 reached 90/90 goals, whereas C2 reached 89/90.
  • E. Cross-Configuration Comparison: C4 had the highest LBSE, but C3 had the lowest worst-decile cost: 127.3 ticks versus C4’s 151.8 ticks.C4 therefore had the strongest aggregate normalized efficiency in this run without dominating every slow-tail criterion.

F. Evidence Scope

The evaluation provides descriptive, configuration-level evidence that C4 achieved complete observed success with the lowest pooled median and P90 in the fixed task. Because records share persistent learning states and configurations bundle implementation differences, the results do not establish causal module effects or generalization.

  • Evidence scope: The 90 robot–round records per configuration are repeated measurements because persistent policies and learning states carry across rounds.Shared states include policies, UCB statistics, replay buffers, network weights, exploration states, and shared-board information.
  • Observed configuration-level result: C4 combined complete observed success with the lowest pooled median and P90 completion times in the fixed task.This indicates better aggregate central and upper-tail completion behavior at the configuration level.
  • Observed configuration-level result: The strongest observed advantage was configuration-level rather than uniform across robots or rounds.C3 remained competitive in pooled median completion time, while agent-level medians were not uniformly lowest under C4.
  • Evidence scope: The bundled configurations do not isolate each module’s causal contribution or demonstrate generalization beyond the fixed environment and robot mapping.Future work proposes seeded repetitions, separate evaluation rollouts, single-factor ablations, additional baselines, and broader environment and platform tests.
Loading 2609.05133v1…