LIBERO-PeRM: Benchmarking Personalized Robotic Manipulation

1 Introduction

• PeRM lacks a shared formulation and standardized evaluation, and existing benchmarks often omit fine-grained manipulation preferences and multi-preference conflicts.Fragmented definitions and evaluation approaches make methods difficult to compare and failure modes hard to diagnose.Source: §1 · p. 2
• LIBERO-PeRM defines observable motion, skill, and user-profile preferences that are language-expressible, automatically evaluable, reusable across tasks, and do not change task goals.The benchmark includes 37 single-preference instances and 18 two-preference combinations.Source: §1 · p. 2
• Its procedural pipeline generates preference-conditioned task variants in shared layouts and produces explicit satisfaction metrics and subtask-level metadata.Metadata includes language instructions, goal images, and physical attributes.Source: §1 · p. 2
• The benchmark comprises five suites and 312 preference-conditioned tasks with pipeline-generated, human-reviewed demonstrations.The suites target single-preference learning, scaling, conflicts, composition, and comprehensive preference reasoning.Source: §1 · p. 2
• The experiments investigate zero-shot preference following, learning from demonstrations, acquisition as demonstrations increase, generalization across layouts and tasks, and composition and conflict handling.These questions organize the study of capabilities needed for reliable personalization.Source: §1 · p. 3
• Experiments show that current VLA policies struggle with zero-shot personalization, while SFT makes preferences learnable.Mixed training improves no-preference task success over sequential adaptation and contextual-prompt task success and preference satisfaction over preference-only LoRA, but weakens motion-preference learning.Source: §1 · p. 3

Robotic Manipulation Benchmark

• Existing manipulation benchmarks largely focus on task completion, whereas LIBERO-PeRM separates task success from preference satisfaction.LIBERO is widely adopted for high-quality demonstrations and standardized data generation, with emphasis on lifelong learning and task generalization.Source: §2 · p. 3

Personalized Robotic Manipulation

• Prior personalization work spans varied settings and preference representations, leaving the scope of personalized manipulation unclear.LIBERO-PeRM responds with an operational taxonomy of observable preferences and five preference-aware task suites.Source: §2 · p. 4

Vision-Language-Action (VLA) Models

• VLA benchmarks mainly measure task success, leaving preference alignment underexplored despite its relevance to users specifying how tasks should be executed.LIBERO-PeRM evaluates personalization with preference-injected trajectories, natural-language instructions, and ground-truth satisfaction matrices.Source: §2 · p. 4

3.1 Preference Formulation

• The benchmark treats preferences as constraints on how a task is completed, covering trajectory-level motion, skill execution, and user-profile choices while preserving the task goal.Motion examples include avoiding a region; skill preferences can require handle-outward placement; user profiles can select a coaster as the support target.Source: §3.1 · pp. 4–5

3.2 Preference-Conditioned Data Generation Pipeline

• The three-stage pipeline selects preferences and modifies layouts, generates preference-conditioned subtasks and trajectories, then derives satisfaction metrics and metadata.Preference type determines whether the subtask plan, skill selection, or trajectory generation is altered; satisfaction can be scored from trajectory quantities, waypoint checks, or executed plans.Source: §3.2 · pp. 5–6, Figure 3 · p. 5

3.3 Task Suites

• Five task suites comprise 312 tasks, targeting single-preference learning, scaling and generalization, conflicts, composition, and contextual preference reasoning.The tasks include preference-conditioned and no-preference conditions, with more than 100 trajectories per task on average and 40.8K rollout episodes in total.Source: §3.3 · pp. 6–7

3.4 Evaluation

• Evaluation reports task success and preference satisfaction, while conflict scoring prioritizes task completion and uses the higher satisfaction score for preference-preference conflicts.Generalization is assessed in-domain, across task or layout changes, and at the preference level; conflict scores do not evaluate conflict detection, preference weighting, or when to ask for guidance.Source: §3.4 · p. 7

4.1 Experimental Setup

• The experiments use six representative VLA policies for evaluation and mainly use π0.5 for non-zero-shot experiments.Table 2 evaluates zero-shot preferences on LIBERO-SinglePref; Table 3 reports fine-tuned results, including mean and standard deviation over ten evaluation runs with fixed checkpoints.Source: §4.1 · pp. 7–8, Table 2 · p. 8, Table 3 · p. 8

4.2 Experimental Results

• Off-the-shelf policies fail to follow preferences reliably: π0.5 has high Motion and Skill SR but much lower preference satisfaction.π0.5 reaches 97.50% Motion SR and 100.00% Skill SR, with 24.38% and 25.00% SA, respectively.Source: §4.2 · p. 8
• Mixed training improves contextual-prompt robustness over preference-only LoRA on LIBERO-ComprePref, reaching 72.83±0.95% SR and 70.07±0.91% SA.Preference-only LoRA reaches 44.00±0.71% SR and 42.64±0.69% SA in those columns.Source: §4.2 · p. 8
• The results compare strategies for jointly learning preference-conditioned and preference-agnostic behavior: Motion Preference SR decreases from 90.00±2.85% to 66.88±3.56%, so mixed training is not established as uniformly superior.Source: §4.2 · p. 9
• At fixed 2.25K scale, L1–L3 share a demonstration budget; L1-to-L3 improvement shows broader layout coverage improves in-domain preference learning more than denser demonstrations within fewer layouts.L4 uses the same layout setting as L2 but adds matched no-preference missions; its improvement over L2 suggests benefit beyond increasing layout diversity, likely helping separate task execution from preference-conditioned behavior.Source: §4.2 · p. 9
• Under generalization, task success and preference satisfaction remain challenging together; performance is especially low on unseen layouts with different tasks.Unseen-layout performance benefits from increased data scale, while generalization to both unseen layouts and different tasks remains limited.Source: §4.2 · p. 9
• After separate training on PA and PB, the policy composes PA+PB with 25.0% Eval SR/SA, but decomposition yields 37.5% Eval SR and 0.0% Eval SA.Decomposition tests inferring PB from PA+PB and PA.Source: §4.2 · p. 10, Table 4 · p. 10
• Conflicting preferences reduce performance: preference-preference conflict-time SR/SA are 66.00%/47.75%, with Pref-Dom. at 11.25%.For task-preference conflicts, Follow Task is 33.3%, Follow Preference is 29.2%, and 37.5% of trials are incomplete.Source: §4.2 · p. 10

5 Conclusion

• LIBERO-PeRM provides five suites that evaluate single-preference learning, scaling, conflicts, composition, and comprehensive preference reasoning.The benchmark contains 312 preference-conditioned tasks with human-reviewed, pipeline-generated demonstrations.Source: §5 · p. 10
• Current VLA policies struggle with zero-shot personalization, while supervised fine-tuning makes preferences learnable.Mixed training better preserves no-preference behavior than sequential adaptation and improves contextual-prompt robustness, but weakens motion-preference performance relative to preference-only LoRA.Source: §5 · p. 10
• Preference learning benefits more from layout diversity and same-layout no-preference data than denser demonstrations, but generalization to unseen layouts and different tasks remains difficult.Composition is only partially observed, decomposition fails, and conflicts often degrade execution rather than resolve.Source: §5 · p. 10
• The benchmark covers predefined, language-expressible preferences in simulation, excluding implicit inference, non-linguistic signals, and open-ended or evolving preferences.No real-robot or longitudinal user study is included, and generated demonstrations may retain pipeline- and skill-library-specific biases.Source: §5 · p. 10

A Additional Information

• The appendix adds experimental details and results supporting the main findings in Section 4.2.Source: A Additional Information · p. 18

A.1 Ethics Statement

• The benchmark uses procedurally generated preference-conditioned tasks, layouts, trajectories, and evaluation signals in simulation, with human review of data and preference plausibility.No personal data or real-world user logs are used or released as policy-training data.Source: §A.1 · p. 18
• LIBERO-PeRM is intended for controlled research evaluation, not as a substitute for human evaluation in safety-critical or real-world personalized deployment.Its preference metrics capture observable execution outcomes in simulation, not open-ended, private, or highly subjective preferences.Source: §A.1 · p. 18
• The dataset and open-source code are released under the MIT license, with attribution and upstream dependency terms to be preserved.Source: §A.1 · p. 18

A.2 Training Details

• For each 4,000 rollout episodes, training uses roughly 45,000 optimization steps on eight A6000 Ada GPUs at a 5×10^-5 learning rate.Source: §A.2 · p. 18
• All-Pref is one checkpoint jointly trained on Motion, Skill, and User-Profile tasks, not an ensemble or an average of category-specific checkpoints.Source: §A.2 · p. 18
• Each fixed checkpoint is evaluated in 10 independent runs, so the reported mean ± standard deviation reflects randomized evaluation rollouts rather than training seeds.Source: §A.2 · p. 18

• Each preference must be language-expressible, grounded in existing LIBERO scenes, satisfiable without changing the task goal, automatically measurable, and reusable across tasks or layouts.Source: B Preference Selection Criteria · p. 18
• The taxonomy includes motion preferences such as speeding up, skill preferences such as grasping an object by its handle, and user-profile preferences such as preferring dairy products.Source: B Preference Selection Criteria · p. 18
• Preferences requiring scene-external information, unsafe or infeasible behavior, or subjective judgments that cannot be mapped to observable execution outcomes are excluded.Source: B Preference Selection Criteria · p. 18
• Table 6 lists all 37 single-preference instances in LIBERO-SinglePref.Source: Table 6 · p. 19

• Three model-assisted stages generate layouts, plan preference-conditioned actions, and inspect rollout frames for task success and visually observable preference compliance.Source: C Model-Assisted Generation and… · p. 20
• Layout and plan candidates undergo generate-and-trial simulation, with proposals rejected for format, physical, task-success, or preference-constraint violations and failures fed into revisions.Source: C Model-Assisted Generation and… · p. 20
• Independent VLM judgments assess visible properties such as contact location and gripper orientation; motion speed, acceleration, smoothness, and clearance use trajectory-based metrics instead.Source: C Model-Assisted Generation and… · p. 20
• The reported VLM failure rate in each stage was below 23%, and inconsistent image judgments were flagged for re-evaluation.Source: C Model-Assisted Generation and… · p. 20

D Human Review and Initial User Validation

• Contributors supplied reference demonstrations, inspected generated demonstrations and labels, and rated preference plausibility and human-likeness; cross-review resolved disputed cases, with 11.8% initial reviewer disagreement.Source: D Human Review and Initial User… · p. 21
• Fifteen participants rated 37 preferences on a 10-point scale for household plausibility and demonstration human-likeness; this is initial evidence, not a substitute for real-robot, longitudinal, or larger-scale studies.Source: D Human Review and Initial User… · p. 21, Table 7 · p. 21

E A Worked Example of ComprePref

• ComprePref conveys a specified preference through contextual language while keeping the task goal, scene, and evaluator fixed; it does not establish recovery of an individual's unobserved preference history.Source: E A Worked Example of ComprePref · p. 22

F Adding a New Preference

• Adding a preference requires one reference demonstration, a natural-language metric description, and about three to five validation trials per applicable task; roughly five tasks took about 20 minutes on average.The estimate applies to preferences supported by the current scene representation, skill library, and programmable metrics; new skills or non-observable feedback require additional work.Source: F Adding a New Preference · p. 22

• The five suites include 312 tasks across SinglePref (100), PrefScale (100), PrefConflict (48), and PrefCompose (64); ComprePref reuses tasks without adding mission-preference combinations.Source: G Benchmark Release and Maintenance · p. 22
• Release documentation covers installation, dataset and configuration descriptions, and generation, training, and inference examples; Croissant metadata includes two personal-information and synthetic-data fields.Source: G Benchmark Release and Maintenance · p. 22
• The authors plan to maintain the code, datasets, and leaderboard, improve documentation using community feedback, and expand rollout coverage and generation diversity.Source: G Benchmark Release and Maintenance · p. 22