PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboFL: Federated Expert Assembly for World Action Models

Rongyu Zhang, Ruizhi Fan, Yunfan Lou, Hengyu Fang, Shenli Zheng, Chenrui Wu, Yili Jin, Li Du, Dan Wang, Yuan Du, Shanghang Zhang

arXiv:2609.34968v1cs.ROcs.AI

TL;DR

Robot interaction data is scarce and institutionally siloed, while federating task-specialized adapters raises aggregation and routing challenges. ROBOFL assembles client-trained adapters as server MoE experts and uses cross-path consensus for routing and adapter conversion. It reports the highest overall success among evaluated federated methods and strong real-robot results, alongside reduced client communication.

  • Problem

    Robot demonstrations are scarce and institutionally siloed, and federated MoE aggregation can dilute specialized updates or disrupt expert assignments.

    Source: §1 · p. 2

  • Method

    ROBOFL installs client-trained LoRA adapters as server MoE experts, uses FARD to align action routing with understanding-generation consensus, and uses PCEA to convert complete expert updates into a compact adapter.

    Source: §1 · pp. 2–3

  • Results

    Across simulation and real-robot experiments, ROBOFL had the highest overall success among evaluated federated methods; on six Franka tasks it reached 59.17% against 46.94% for centralized InternVLA.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    ROBOFL combines structured server-side expert assembly with single-adapter client training and communication cost.

    Source: §5 · p. 10

    Experiments use one backbone and aligned MoT architecture, share embodiment, sensors, and action space, and use a single simulation seed and fixed 20-trial robot protocol.

    Source: E LIMITATIONS · p. 24

Abstract

from arXiv · show

Vision-language-action and world-action models are increasingly popular, yet remain bottlenecked by physical interaction data that is scarce, institutionally siloed, and task-heterogeneous. A natural federated solution is to let each client adapt a shared foundation model through parameter-efficient fine-tuning, avoiding the exchange of full-model updates. However, federating these adapters is nontrivial, as naive aggregation can entangle incompatible updates, while incorporating MoE-style routing into federated aggregation may dilute specialization and destabilize expert selection. We present RoboFL, which instantiates MoSAIC (Mixture of Slotted Adapters) for federated world-action learning. MoSAIC directly installs locally trained LoRA adapters as the expert branches of a server MoE. Server-side routers learn token assignments over these prior-informed branches while jointly refining routing and expert parameters. Foresight-to-Action Routing Distillation (FARD) aligns routing across the model's three paths, while Path-Consensus Expert Aggregation (PCEA) converts complete expert updates into a compact global adapter for personalized redistribution. Experiments on RoboTwin 2.0, RLBench, and a real-world Franka robot arm show the superiority of RoboFL with structured expert assembly, as it outperforms centralized PEFT InternVLA-A1 by 12.23% on the Franka arm, while reducing per-round client communication by up to 86.81% relative to MoE-based federated VLA baselines.

1 INTRODUCTION

Robot interaction data is scarce, institutionally siloed, and task-heterogeneous, making federated learning attractive but complicating aggregation of specialized updates. ROBOFL assembles client-trained adapters as server experts, uses cross-path routing and aggregation, and reports gains across simulation and real-robot evaluations.

Motivation

  • Robot demonstrations are costly to collect and may expose private environments or proprietary procedures, while institutionally siloed task families create strongly non-IID client data.Under task-level skew, local optima can drift apart and naive averaging can blend transformations suited only to particular task families.Source: §1 · p. 2
  • Client-trained MoE branches and routers can specialize to narrow local task distributions, so averaging may dilute task knowledge and disrupt useful expert assignments.The paper identifies separating local specialization from shared routing-policy learning as the central challenge.Source: §1 · p. 2

ROBOFL

  • ROBOFL trains one LoRA adapter per client, installs uploaded adapters in fixed server expert slots, and learns routing while refining experts, keeping multi-expert optimization server-side.The fixed client-to-slot mapping preserves parameter correspondence across rounds while clients retain single-adapter training and communication.Source: §1 · pp. 2–3
  • FARD distills understanding-generation routing consensus into the action router, while PCEA uses three-path consensus to weight complete expert updates and reconstruct a rank-constrained global adapter.The global adapter is redistributed for personalized client training.Source: §1 · p. 3

Evaluation

  • 83.12% overall on RoboTwin 2.0, 4.2% above federated averaging and 1.52% above centralized InternVLA-A1; Franka results were 59.17% versus 46.94% for centralized InternVLA.ROBOFL also cut client communication by up to 86.81% against MoE-based federated VLA baselines.Source: §1 · p. 3

2 RELATED WORKS

Prior work develops VLA and world-action models, MoE-based robotic policies, and federated adaptation methods. ROBOFL differs by assembling single client adapters as server experts and using cross-path consensus for routing and adapter conversion.

Vision-language-action and world-action models

  • VLA and world-action models connect multimodal knowledge with robot control and future-state prediction, but generally assume centralized training rather than institutionally distributed task data.Related directions include multi-robot policies, flow-based control, action tokenization, spatial representations, and future-state prediction.Source: §2 · p. 3

VLA and WAMs with MoE and FL

  • MoE and federated VLA work explores specialized computation and robotic federated training, while ROBOFL separates local adapter specialization from server-side routing and refines uploaded experts and routers on server data.The passage notes that narrow client data can produce inconsistent expert assignments.Source: §2 · p. 3

Federated PEFT and MoE

  • Federated PEFT and MoE methods study low-rank aggregation, client-specific experts, and personalization; ROBOFL installs single-adapter updates as server experts and converts complete updates using cross-path consensus.The resulting compact adapters support personalized redistribution.Source: §2 · p. 3, Figure 2 · p. 4

3 METHOD

The method combines a three-path world-action model with server-side assembly of client-trained LoRA experts. FARD guides action routing with understanding-generation agreement, while PCEA converts server-refined expert updates into a rank-constrained adapter for clients.

3.1 PRELIMINARY

  • The understanding, generation, and action paths share masked self-attention; the action path predicts action-chunk velocity from noisy actions, time, context, and robot state.The paths retain path-specific projections and MLPs, and action-loss averaging excludes padded steps when labels are available.Source: §3.1 · p. 4

3.2 MOSAIC: MIXTURE OF SLOTTED ADAPTERS

  • Clients upload LoRA factors into fixed client-associated server slots without averaging, and the server jointly refines expert factors and selection routers.The server maintains one complete expert branch per client and uses mixed-task data for refinement.Source: §3.2 · p. 5

3.4 PCEA: PATH-CONSENSUS EXPERT AGGREGATION

  • PCEA uses detached routing evidence to weight complete server-refined expert residuals and convert them into a rank-constrained global adapter.It shares weights across matched U/G/A modules; the resulting static adapter conversion does not guarantee equivalence to the input-dependent MoE.Source: §3.2 · p. 5, §3.4 · p. 6

3.3 FARD: FORESIGHT-TO-ACTION ROUTING DISTILLATION

  • FARD distills understanding-generation routing agreement into the action router, down-weighting unreliable signals through a reliability measure.Gradients pass through the action-distribution argument rather than the teacher or reliability.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

ROBOFL performs strongly on RoboTwin 2.0 and real-world Franka tasks, while remaining the best federated method on the smaller RLBench suite. Its evaluation also examines client communication, routed behavior, and successful real-world rollouts.

4.1 EXPERIMENTAL SETTINGS

  • The evaluation partitions 50 RoboTwin tasks across eight client task-family groups, eight RLBench tasks across four client partitions, and six Franka tasks across four clients and a server.The Franka split creates an extreme non-IID data scenario.Source: §4.1 · p. 6

4.2 QUANTITATIVE RESULTS FOR SIMULATION

  • 83.12% overall success on RoboTwin 2.0 exceeds FedAvg by 4.20% and surpasses centralized PEFT with InternVLA-A1 and Motus; ROBOFL is best or joint-best on 15 displayed tasks.Table 1 covers 25 challenging tasks selected where FedAvg is below 80%, while Overall reports all 50 tasks.Source: §4.2 · p. 7, Table 1 · p. 7
  • 41.25% on RLBench makes ROBOFL the best federated method, though its margin over centralized PEFT reverses relative to RoboTwin.The authors suggest structured expert assembly is most effective in larger, task-heterogeneous regimes rather than smaller, low-data suites.Source: §4.2 · p. 8, Table 2 · p. 8
  • Per-round client communication falls by up to 86.81% and client parameters by up to 86.95% versus MoE-based FL methods FedMoE and FedVLA.With one rank-16 adapter per client, ROBOFL matches FedAvg at 40.85M trainable parameters and 311.63 MiB per-round communication.Source: §4.2 · p. 8, Table 3 · p. 8

4.4 REAL-WORLD APPLICATIONS WITH FRANKA

  • 59.17% overall success on six Franka tasks exceeds centralized InternVLA by 12.23% and federated ForgeVLA and FedVLA by 23.61% and 25.56%.The comparison averages three evaluation rounds of 20 trials per task.Source: Table 4 · p. 9, §4.4 · p. 9

4.3 ABLATION STUDIES

  • Instruction swaps yield strong pooled route-to-action coupling; scrambling displaces FARD routing more but action less than vanilla ROBOFL.The probes hold initial observation and robot state fixed; direct interventions act only on the action-router distribution.Source: §4.3 · pp. 8–9, Figure 4 · p. 9

4.4 REAL-WORLD APPLICATIONS WITH FRANKA

  • Successful Franka rollouts show approach, sustained contact, and sequenced state change across six tasks, including seating rather than striking a bottle and sustaining cap rotation.The authors describe the behaviors as closed-loop and input-dependent.Source: Figure 5 · p. 9, §4.4 · p. 10

5 CONCLUSION

RoboFL learns world-action models from task-siloed robot data without centralizing raw trajectories. Its combined methods achieve the highest overall success rates among evaluated federated methods while retaining single-adapter client training cost.

5 CONCLUSION

  • Across simulation and real-robot experiments, RoboFL’s combined methods deliver the highest overall success rates among evaluated federated methods.Client training remains at single-adapter cost.Source: §5 · p. 10
  • MoSAIC installs locally trained LoRA adapters as server MoE experts and jointly refines routing and expert parameters.This supports learning from task-siloed robot data without centralizing raw trajectories.Source: §5 · p. 10
  • FARD aligns action routing with understanding-generation consensus, while PCEA consolidates expert updates into a rank-constrained global adapter for personalized redistribution.Source: §5 · p. 10

AI USE STATEMENT

The authors used AI-based assistants for writing and coding support. The authors state that they verified and take responsibility for the paper’s research and results.

AI USE STATEMENT

  • AI-based writing and coding assistants helped draft and polish portions of the text and support implementation and analysis scripts.Source: AI USE STATEMENT · p. 10
  • The authors carried out and verified the problem formulation, method design, experimental execution, data collection, and interpretation of results.Source: AI USE STATEMENT · p. 10
  • The authors take full responsibility for the paper’s content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work studies federated world-action learning for robotic manipulation using two simulation benchmarks and real-robot evaluations. Its protocol keeps raw trajectories local, but the authors do not claim formal privacy guarantees.

ETHICS STATEMENT

  • The work studies federated learning of world-action models for robotic manipulation.Source: ETHICS STATEMENT · p. 10
  • Simulation experiments use RoboTwin 2.0 and RLBench, while real-robot evaluations collect no personally identifiable information.Source: ETHICS STATEMENT · p. 10
  • The protocol keeps raw trajectories at each institution and exchanges only adaptation parameters, reducing exposure from centralized pooling.Source: ETHICS STATEMENT · p. 10
  • The authors disclaim formal privacy guarantees such as differential privacy or secure aggregation, and note that updates may still leak information in adversarial settings.They position the method as an architectural alternative to raw-data centralization, not a complete privacy solution.Source: ETHICS STATEMENT · p. 10
  • The authors note that broad manipulation policies can be misused and state that they followed the ICLR Code of Ethics.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement points to appendices and tables for settings, task assignments, interfaces, resources, and method formulations. It also specifies planned supplementary releases and fixed evaluation protocols.

REPRODUCIBILITY STATEMENT

  • Training and evaluation settings are in Table 5 and Appendix A; task-silo assignments for eight RoboTwin 2.0 clients and four RLBench clients are in Tables 6 and 8.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendix A specifies observation, action, and optimization interfaces, client resources, logical communication accounting, and architecture profiles behind memory measurements.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The MoSAIC, FARD, and PCEA formulations and implementation are described in the Method section and Appendix A.1.The statement identifies adapter slotting, router integration, personalized redistribution, and consensus-weighting rules.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors plan to release source code, launch configurations, evaluation scripts, preprocessing and normalization steps, benchmark packages, and scripts for resource and ablation figures as supplementary materials.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Simulation results use one fixed evaluation seed, and real-robot results use a fixed scripted protocol with 20 trials per task.Differences below a few points and per-task swings on the eight-task RLBench suite are described as not statistically established.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 SHARED IMPLEMENTATION DETAILS

This section specifies how RoboFL trains and deploys compact adapters across a task-silo federation, including the server-compute assumption and benchmark-specific implementation settings. It also frames centralized PEFT as a stronger reference: RoboFL matches or exceeds it on RoboTwin 2.0 and Franka, but trails it on low-data RLBench.

A.1 SHARED IMPLEMENTATION DETAILS

  • Clients upload complete LoRA factors and trainable action projections and MLP weights; frozen backbone weights are excluded, and shared action-head weights are averaged separately.Server MoE checkpoints are saved before global adapter conversion and personalized redistribution.Source: §A.1 · p. 15
  • The server participates in training with a residual task group and routed expert refinement, while clients continue training one adapter with per-round steps matched to the server.The server groups cover 7/50 RoboTwin, 2/8 RLBench, and 2/6 Franka tasks; the authors state this leaves most of each task distribution with clients.Source: §A.1 · p. 15
  • RoboFL matches or exceeds the stronger centralized PEFT reference on RoboTwin 2.0 and Franka, but trails it on low-data RLBench.The paper describes centralized PEFT as observing all task families in one optimizer without task silos, non-IID client drift, or communication rounds.Source: §A.1 · p. 15
  • At deployment, each client uses one rank-16 LoRA adapter; the eight-expert server MoE stays server-side, matching the references in family, rank, and scaling.Clients do not hold routers or multiple experts; the authors attribute gains to server-side composition rather than increased deployed capacity.Source: §A.1 · p. 15
  • All federated baselines follow RoboFL's training procedure, and the implementation records batch-size conventions and launcher defaults separately from task assignments and evaluation semantics.For RLBench, the stated defaults are launcher defaults rather than reconstructed checkpoint settings.Source: §A.1 · p. 15, §A.3 · p. 17, §A.4 · p. 18

A.2 HYPERPARAMETER AND DESIGN CHOICES

  • The personalization coefficient equally interpolates global and local models, while small auxiliary weights preserve the action loss as primary and FARD warms up over 10 rounds.The stated weights are λaux = 0.001 and λFARD = 0.01; the paper says the reliability weight is uninformative before the action router stabilizes.Source: §A.2 · p. 16

A.3 ROBOTWIN 2.0

  • RoboTwin uses task-disjoint client and server assignments, frame-level shuffled loading, and joint targets derived from the configuration at each action queue's start.Execution horizon is independent of prediction length; accepted randomized trials need not be consecutive candidate seeds because unstable or expert-unsuccessful scenes are excluded.Source: §A.3 · pp. 16–17, Table 5 · p. 16, Table 6 · p. 17

A.4 RLBENCH

  • RLBench uses task-disjoint evaluation assignments and a shuffled loader; queued translations add to the current end-effector position, while Euler angles specify absolute world-frame orientation.Invalid actions and failures terminate episodes and count as failures; expert actions and future observations are not supplied to the policy.Source: §A.4 · pp. 17–18, Table 8 · p. 18

B.1 IMPLEMENTATION DETAILS

The Franka implementation partitions six real-robot tasks across four clients and a separate server, then evaluates policies in closed-loop trials on the physical platform. The setup uses two camera views and an absolute-joint-target action interface.

B.1 IMPLEMENTATION DETAILS

  • Six Franka tasks are partitioned task-disjointly among four clients and a separate server, with each client owning one task family and the server retaining two.Evaluation uses real-robot inference on the physical platform.Source: §B.1 · p. 18, Table 9 · p. 19
  • The policy consumes external-stand and wrist camera views, masks the third view slot, and outputs eight-dimensional absolute joint targets executed directly by the robot.Auxiliary tactile and wrench channels are not supplied; queued targets are not added to the state at the start of the queue.Source: §B.1 · p. 19
  • Each method receives three rounds of 20 trials per task; an operator starts and resets trials, and only reaching the task goal counts as success.The policy runs closed loop on the physical robot; otherwise the episode terminates and counts as a failure.Source: §B.1 · p. 19

B.2 FRANKA SINGLE-ARM PLATFORM

  • The platform is a Franka Research 3 with a Robotiq 2F-85 gripper and two RealSense D435i cameras, one wrist-mounted and one externally mounted above the workspace.The workspace includes a perforated board and task objects for bottle adjustment, stamping, cup stacking, bottle screwing, pouring, and test-tube placement.Source: §B.2 · p. 19

C.1 COMPLEMENTARY QUANTITATIVE ANALYSIS FOR ROBOTWIN 2.0

On RoboTwin 2.0, the full-benchmark results rise as server-side expert assembly is augmented with routing distillation and consensus-weighted conversion; RoboFL is also strongest among federated methods on RLBench. Qualitative rollouts and failure cases characterize the behaviors and remaining limits, while the evaluation uses single-seed simulation results.

C.1 COMPLEMENTARY QUANTITATIVE ANALYSIS FOR ROBOTWIN 2.0

  • On the complementary saturated RoboTwin subset, ROBOFL is best or joint-best on 16 of 25 tasks and records 100% observed success on nine.The main table emphasizes a lower-success subset where FedAvg is below 80%, while the appendix preserves full task coverage.Source: §C.1 · pp. 19–20
  • 83.12% overall success on RoboTwin 2.0 follows 81.94% for vanilla MOSAIC and 82.42% with FARD; FedAvg reaches 78.92% and FedMoE 75.12%.The paper reports gains of 3.02% over FedAvg, 0.48% from adding FARD, and 0.70% from adding PCEA.Source: §C.1 · p. 20, Table 10 · p. 20

C.2 QUANTITATIVE ANALYSIS FOR RLBENCH

  • On the eight-task, single-view RLBench suite, ROBOFL reaches 41.25%, the best among federated methods, but trails centralized InternVLA at 52.75% and Motus at 52.50%.The authors hypothesize that this lower-data suite offers less task diversity and complementary evidence for expert composition.Source: §C.2 · pp. 20–21, Table 11 · p. 21

C.3 QUALITATIVE ANALYSIS

  • RoboTwin rollouts show approach, contact or grasp, then transport or placement; difficult handle-based or occluded tasks require slower, more tentative approaches.The reported remaining failures concentrate in fine-grained contact and long-horizon placement rather than gross reaching or scene understanding.Source: §C.3 · p. 21
  • RLBench rollouts show contact-and-push sequences for closing, closed grasps before relocation, and narrow-target alignment for tool-use and placement.The illustrated episode per task was selected for visual clarity, not as the longest or most representative rollout.Source: §C.3 · p. 21

C.4 FAILURE ANALYSIS

  • Failures include incomplete object engagement, stalled or abandoned motion, and misplaced end states; RLBench failures instead end in motion-planning errors.The authors identify absent client skills, ambiguous routing, and lossy conversion as bottlenecks for future instrumentation.Source: §C.4 · p. 22

D STATISTICAL REPORTING PROTOCOL

  • Simulation results use one fixed training seed and one evaluation seed, with 100 RoboTwin and 50 RLBench trials per task; the real-robot protocol is fixed and scripted.The paper says multi-seed training with confidence intervals would strengthen evidence for smaller effects.Source: D STATISTICAL REPORTING PROTOCOL · p. 23

E LIMITATIONS

The study's evidence is limited to one backbone and aligned MoT architecture, task heterogeneity with a shared embodiment and interface, and narrow evaluation protocols.

E LIMITATIONS

  • All experiments use one backbone, InternVLA-A1-3B, and one aligned MoT architecture.The study does not evaluate other backbones or aligned MoT architectures.Source: E LIMITATIONS · p. 24
  • Clients vary in task distribution but share embodiment, sensor suite, and action space, leaving cross-embodiment and cross-modal heterogeneity open.Source: E LIMITATIONS · p. 24
  • Simulation results use a single evaluation seed, and real-robot results use a fixed scripted protocol with 20 trials per task.Source: E LIMITATIONS · p. 24
Loading 2609.34968v1…