Learning to Fold: prizewinning solution at LeHome Challenge 2026 (1st place online, 2nd offline)

Ilia Larchenko

Source-linked AI summary

Learning to Fold: prizewinning solution at LeHome Challenge 2026 (1st place online, 2nd offline)

Ilia Larchenko

arXiv:2606.27163v2Jun 25, 2026cs.ROcs.AIcs.LG

TL;DR

Bimanual garment-folding policies must generalize to unseen garments and infer garment type when it is not provided at evaluation. This work augments a VLA policy with shared action and value predictions for training and decision-making, achieving 79.63% overall success and first place online, followed by second place in the real-world final.

Problem
The challenge is generalizing to unseen garments while inferring garment type because evaluation garments are random and unlabeled.
Method
The approach equips a flow-matching VLA policy with auxiliary value-related heads whose shared predictions support action learning, failure detection, and candidate selection.
Results
79.63% overall success earned 1st place among 62 teams online, while the system placed 2nd in the real-world final.
Takeaways & Limitations
Keeping value, Q, and a lightweight world-model substitute inside the policy provides one model to train and serve while sharing representations with the action head.
Takeaways & Limitations
The author does not consider the approach sample-efficient and suspects comparable results could be reached with a much smaller dataset.

Abstract

from arXiv · show

I describe my solution to the LeHome Challenge 2026, an ICRA 2026 competition on bimanual garment folding. The system placed 1st of 62 teams in the online (simulation) round and 2nd in the real-world final. It improves a vision-language-action (VLA) policy with a reinforcement-learning loop. The policy is its own value function: the same network that predicts actions also predicts success, progress, and a few task-relevant future quantities, and those predictions drive advantage estimation, live failure detection, and candidate selection. The work mostly recombines existing RL ideas with engineering and optimization contributions that can be used together as one recipe or individually: AWR + RECAP combined for flow-matching VLA; an asynchronous distributed training / rollout pipeline through HuggingFace Hub; inference-time hyperparameters optimization via Thompson sampling; a sim-to-real recipe with camera-alignment tooling, heavy augmentation and DAgger-like HIL data collection.

1 Introduction

The LeHome Challenge 2026 evaluates bimanual garment folding under binary geometric success criteria, with unseen garments and sim-to-real transfer creating major challenges. The solution combines reinforcement learning, value prediction within the policy, data collection, inference optimization, and transfer tooling, achieving 1st online and 2nd in the real-world final.

Challenge setting: The task uses two 6-DOF arms, a 12-dimensional action space, and three RGB cameras to fold four garment types on a table.Simulation runs at 30 Hz and the real round at 20 Hz; overhead depth was available but unused.
Challenge setting: Success is binary and geometric, requiring 5 keypoint conditions for tops and 4 for pants; the same conditions provide dense intermediate rewards.Pairs that should meet must fall below distance thresholds, while pairs that should remain apart must stay above them.
Challenges: Unseen garments, deformable cloth, sparse rewards, and unavailable evaluation hardware make behavior-cloning robustness, generalization, and transfer central difficulties.The real transfer path was sim → the author’s robot → the organizers’ robot.
System contributions: The system combines AWR and RECAP-style advantage conditioning, asynchronous HuggingFace Hub workers, policy-integrated value heads, and Thompson-sampling inference optimization.The policy predicts success, completion, garment type, keypoint distances, future distances, and an action-conditional success residual used as a Q-function.
Results: 1st of 62 teams was achieved online with 79.63% overall success, 6.1 points ahead of second place; the system finished 2nd in the real-world final.The sim-to-real sprint used organizer data, teleoperation/DAgger, augmented simulation replays, heavy augmentation, motion-velocity alignment, and camera-overlay calibration.

2 RL Training

RL training combines AWR and RECAP-style conditioning for flow-matching VLAs with an asynchronous HuggingFace Hub pipeline connecting training, rollouts, and manual DAgger. The system samples high-advantage actions more often, conditions on advantage, and continuously trains on newly available data without synchronization barriers.

Asynchronous pipeline: The system has independent training, rollout, and manual DAgger components that communicate only through HuggingFace Hub.Rollout workers pull the newest checkpoint, upload episodes with collection-time values, and scale by adding machines; the DAgger station uploads teleoperated corrections.
Asynchronous pipeline: There are no synchronization barriers: training uses arrived data, collection uses the newest checkpoint, and background synchronization prevents network blocking.Each trainer iteration downloads new rollouts, recomputes advantages, trains approximately 1000 steps, and uploads checkpoints every approximately 500 steps.
AWR + RECAP: AWR oversamples high-advantage frames, while RECAP-style conditioning supplies advantage as an input and enables classifier-free guidance at inference.The two methods use shared primitives, complement each other, and can be combined as standalone RL approaches.
Advantage-weighted sampling: Sampling rather than loss weighting applies AWR while retaining an unweighted flow-matching MSE, so computation concentrates on desired actions and effective batch utilization remains 100% of weight mass.Auxiliary heads remain unbiased through inverse-sampling importance weights, while BC and DAgger frames are prioritized toward garments with lower success rates.
Dataset mixing: RL rollout datasets decay by 0.98 per training iteration, floored at 0.1, so fresh on-policy data dominates while stale data fades.BC retains a fixed sampling rate, and harder garments receive more samples based on overall garment success rate.

3 Data Collection

The collection system is engineered around Isaac Sim’s 30 s episodes, shared policy serving, early termination, and recovery to maximize useful simulation throughput. It combines replay and hard-mining strategies with augmentation and DAgger-style correction, with real-robot DAgger becoming especially valuable.

Collection infrastructure: 30 s per Isaac Sim episode makes simulation speed the collection bottleneck, motivating engineering focused on extracting more useful episodes per sim-hour.The paper identifies simulation speed as the bottleneck of the whole system.
Collection infrastructure: 3–5 sim processes per machine connect as thin clients to one shared, stateless policy server, with action chunks cached client-side.This design accommodates the competition environment’s inability to support multiple scenes in one process.
Collection infrastructure: Early termination stops successful episodes immediately, while stuck detection and watchdog restarts trim hopeless episodes and recover from hung simulations.The stuck detector uses state, action, and predicted-value variance.
Targeted rollout strategies: Replay and hard-mining restore saved physics states to multiply scarce successes and retry failures when predicted success visibly collapses.Success states are saved at step 5, while failure states are captured after an EMA-smoothed success prediction drops by more than 0.12 from its running maximum above 0.25.
DAgger data collection: Real-robot DAgger became one of the project’s most useful tools, whereas the simulation teleoperation version was limited because teleoperation was difficult and the policy outperformed the author.The loop teleoperates briefly from saved failure or semi-success states, then saves completed folds as demonstrations.

4 Policy Architecture

The policy extends a BEHAVIOR-1K/π0.5 architecture with LeHome-specific token layout, garment-type and advantage conditioning, multi-signal AdaRMS, XSA, and smooth action-target normalization. Its image-group query forces prediction heads to rely on pixels, while the architecture is presented without proper ablations establishing which components are critical.

Base architecture: The base stack uses frozen SigLIP-So400m/14 vision, a Gemma-2B prefix transformer, and a Gemma-300M flow-matching expert emitting 30-step action chunks.Three RGB cameras are resized to 224×224; each chunk represents 1 s at 30 Hz with 12-dimensional joint deltas.
LeHome additions: LeHome additions comprise token-layout changes, garment-type input, advantage conditioning, multi-signal AdaRMS, XSA, and smooth per-timestamp action-target normalization.Auxiliary prediction heads are described separately in Section 5.
Token layout: The current query is placed with images, making state, garment type, and advantage invisible to prediction heads and forcing them to operate from pixels alone.This prevents value heads from overfitting to proprioception and prevents the garment-type head from copying its input token.
Garment-type conditioning: The garment-type token is supplied from ground truth during training but predicted at episode start during evaluation and reused thereafter.Late-training garment-type accuracy exceeded 99%, with errors mostly occurring after folding had already failed in very messy states.
Multi-signal AdaRMS: AdaRMS adds advantage and garment-type signals to every action-expert layer, with zero-initialized vectors preserving pre-change behavior at step 0.The advantage term uses the same per-sample mask as the advantage token.
XSA and limitations: XSA removes each token’s direct self-value path and was retained after initial pretrained-model loss disruption despite the absence of a proper ablation.The author reports improved training and validation loss at negligible compute overhead, but explicitly relies on limited evidence for the choice.

5 Auxiliary Prediction Heads

The policy uses current-frame and future-prediction auxiliary heads to make the same model serve as a value function, Q-like predictor, and cheap reward-relevant world-model analogue. These heads predict success, progress, garment state, and action-conditioned future outcomes, supplying training and inference signals while sharing the VLM representation.

Current-frame heads: Current-frame heads predict success, completion, garment type, checkpoint reachability, and garment-specific keypoint-distance ratios from images.The success and completion predictions provide value and progress signals; garment type supports inference-time bootstrapping, while checkpoint predictions are legacy and unused in the final solution.
Design rationale and uses: Shared heads simplify training and serving, reduce compute, enrich the shared representation, and provide signals for advantage computation, failure detection, garment bootstrap, and best-of-N action selection.All current-frame heads share one VLM forward pass, and the auxiliary gradients flow into the VLM backbone with small loss weights.
Current-frame heads: The keypoint-distance head outputs 21 normalized distances, activates only the current garment-type slice, and NaN-masks the remaining slices.The slices are top_long [0:5], top_short [5:10], pant_long [10:17], and pant_short [17:21].
Future-prediction heads: Future heads predict success, completion, and keypoint distances at t+30, with FAST predictions used only during training and FM predictions conditioned on the denoised action tokens.The FM head provides the cheap world-model analogue by predicting reward-relevant garment state rather than pixels or a full latent state.
Future-prediction heads: The FM head also predicts a success residual against the stop-gradient image-only success estimate, functioning as an action-conditioned Q/advantage-like signal.Its raw linear output uses MSE loss, while the stop-gradient prevents this head from pulling the value-like success head.

6 Reward Design and Advantage Computation · 6.1 Dense reward from the success checker · 6.2 Success probability as a value function

The method densifies sparse binary success rewards with checkpoints and gradual progress signals while withdrawing rewards on failure so episode returns remain binary. It uses the VLA’s success-probability head as its value function, with a learned correction coefficient and prediction-bias corrections for advantage computation.

6 Reward Design and Advantage Computation: Binary episode success is too sparse for efficient reinforcement learning, so the final reward combines value prediction, partial progress, completion prediction, and group-relative baselines.The simulator’s shaped garment-niceness reward was not used.
6.1 Dense reward from the success checker: Intermediate checkpoints reuse the challenge’s existing keypoint-distance success conditions, adding garment-specific fold checkpoints without new keypoints or success definitions.Tops and long pants receive an extra intermediate checkpoint; short pants are treated differently in the described design.
6.1 Dense reward from the success checker: 0.5 reward is assigned at the intermediate fold checkpoint, while full success brings cumulative reward to 1.0.For tops, the first 0.5 is allocated proportionally as the primary proximity distance closes; a 60% gap reduction earns 0.3 reward.
6.1 Dense reward from the success checker: Failure withdrawal removes accumulated reward and spreads the deduction after the last reward maximum, ensuring total return equals the binary success indicator.This preserves temporal credit from intermediate checkpoints while keeping the episode-level objective aligned with true success.
6.2 Success probability as a value function: Because total return equals success, the VLA predicts success probability through an auxiliary image-only query head instead of using a separate value network.The shared query token attends only to image tokens, preventing state-shortcut overfitting while reducing pipeline complexity and compute.
6.2 Success probability as a value function: Literal return −V subtraction cancels checkpoint rewards and is suboptimal with imperfect predictions, motivating a CUPED-like correction with an estimable coefficient.The coefficient approaches full subtraction for perfect predictors and zero for completely random predictors.
6.2 Success probability as a value function: αs = 0.5 was used in the final setup, while post-factum per-garment coefficients ranged roughly from 0.4 to 0.8, making the fixed choice close to the training median.The same attenuation partially cancels checkpoint rewards rather than removing them completely, preserving positive advantage at checkpoint moments.
6.2 Success probability as a value function: EMA smoothing, a K=30-frame tail interpolation toward known outcomes, and 20× loss weighting on the final 20 successful-episode frames correct noisy and systematically low ending predictions.These corrections target raw prediction noise and underprediction near successful episode endings.

6.3 Completion prediction

The success head is noisy, saturates on high-success garments, and drifts during training, so the method adds a completion head as a second progress signal. The completion head is trained with MSE on t/T using successful episodes only and is described as more stable than success probability.

Motivation: The success head overfits to particular states, provides little signal above 90% success, and drifts because the network also predicts actions.These weaknesses motivate adding a separate completion-based progress signal.
Completion head: The completion head is trained with MSE on the target t/T using successful episodes only.It is added specifically to complement the success head’s progress information.
Completion head: Completion is described as far more stable than success probability.The passage states that it barely changes, but the supplied text is truncated before completing that description.

6.4 GAE over both heads

Both success and completion signals are aggregated with generalized advantage estimation (GAE) offline before each training iteration. The success estimator uses an αs-dampened value baseline with terminal value pinned to the true outcome, while completion uses potential-based shaping over an EMA-smoothed prediction.

GAE aggregation: Both signals use GAE with γ = 0.999 and λ = 0.99, computed offline before each training iteration.The aggregation is performed in a separate pass before training begins.
Success GAE: Success GAE uses the exact TD residual of an αs-dampened value baseline, with terminal value pinned to the true outcome y.The baseline is defined as V̂_t = αs …, and the terminal value is S̄_T = y.
Completion shaping: Completion shaping adds a potential-based term over the EMA-smoothed completion prediction C̄_t, using potential Φ_t = αc C̄_t and terminal C̄_T = y.The shaping term is defined over the completion prediction rather than the success signal.

6.5 Stale rollouts: segment baselines and blending

The method addresses stale rollouts by recording success predictions at collection time, decaying old data, and blending policy-based GAE with an outcome-only baseline. Segment-level advantages compare checkpoint progress and failures against per-garment empirical baselines, while completion shaping remains valid on old rollouts.

Stale-rollout handling: Success and completion predictions are recorded during rollout collection and never recomputed later, avoiding stale predictions as the model evolves and overfits past data.This preserves on-policy and unseen-state predictions for later training.
Stale-rollout handling: Each rollout dataset’s sampling share decays by 0.98 per training iteration, while BC and DAgger datasets retain fixed shares.The blend weight is w = min(sampling share, 1).
Stale-rollout handling: As predictions become stale, the advantage shifts from GAE toward a GRPO-style relative-success signal based only on outcomes.The objective baseline therefore does not depend on the policy’s stale predictions.
Blending and normalization: Completion shaping Φc remains at full strength for all data because the completion head is policy-stable and uses potential-based shaping rather than a value baseline.Consequently, it does not require the Rcum correction used by the success channel.
Segment baselines: Episodes split at the first checkpoint use segment returns compared with per-garment empirical baselines and scaled by G(n)/n, where G(n) = 1−(γλ)^n.For episodes missing a checkpoint, the fallback is (R−SR) G(T)/T, with baselines garment mean → type mean → 0.5.
Segment baselines: Reaching the checkpoint and then failing produces positive advantage before the checkpoint and negative advantage afterward.This property comes from the segment-return comparison and checkpoint-aware baseline construction.

6.6 Precision boost · 6.7 How the advantage is used · 6.8 Intuitive summary

The method sharpens successful garment folds with a margin-based precision bonus, uses advantages for prioritized sampling and inference conditioning, and combines progress, success, completion, and rollout freshness in a GAE-like estimate. Its intuitive summary emphasizes objective progress and predicted future outcomes, with stale rollouts degrading toward sparse relative-success feedback.

6.6 Precision boost: 20% of successful episodes per garment receive a fixed bonus ΔA = 0.3 on every frame, ranked by final-frame tightness margins.The tightness score is the minimum per-condition margin, measuring distance inside proximity or spread thresholds.
6.6 Precision boost: The tightness score uses the worst per-condition final-frame margin: 1 −d(i) for proximity conditions and d(i) −1 for spread conditions.This distinguishes barely passing folds from folds that satisfy success conditions tightly.
6.7 How the advantage is used: Advantages drive AWR-style prioritized sampling with P(frame i) ∝ eclip(Ai, −2, 2) and RECAP-style conditioning of the action expert.The RECAP-style conditioning enables classifier-free guidance at inference.
6.8 Intuitive summary: An action’s advantage is high when it makes objective task progress measured by the keypoint distances defining success.This is the first component of the paper’s simplified intuitive summary.
6.8 Intuitive summary: An action’s advantage is also high when predicted eventual success increases or predicted completion increases conditioned on eventual success.These predictions provide complementary future-outcome signals in the simplified summary.
6.8 Intuitive summary: As rollouts become staler, advantage degrades gracefully toward a sparse, outcome-only relative-success signal, aggregated with a GAE-like backward pass using γλ ≈0.989.The freshness mechanism preserves a usable signal while shifting stale data toward outcome-based feedback.

7 Inference-Time Optimization

Inference-time performance is controlled by deployment-time choices for chunk execution, guidance, and candidate sampling rather than by changing model weights. The final recipe combines overlap-based chunking, per-garment guidance tuning, and best-of-N selection, with warm-up garment-type estimation on the client.

Chunk execution: The fixed H = 30-action chunk is consumed through execution length ne, playback stretch k, and anchor length na, which control re-planning, speed, and continuity.The constraint ne + na ≤ H applies to the retained anchor.
Chunk execution: The trailing na actions softly anchor the next denoised chunk, preserving trajectory consistency while allowing later flow steps to self-correct.The anchor also supports mode stickiness and smooth trajectories.
Guidance: Guidance runs conditional and unconditional action-expert passes, shares the expensive prefix computation, and uses a per-garment scale α of 7–9 in the final submission.Guidance doubles only the cheap action-expert cost because the vision-language prefix runs once per chunk.
Candidate selection: 2–3-candidate rollouts consistently beat single-candidate ones despite effectively zero correlation between predicted and actual outcomes.The method samples independent flow-matching seeds, ranks candidates by averaged predicted ∆success, retries with a larger batch when all candidates have ∆success < 0, and found that more than 3 candidates did not help.
Garment-type bootstrap: The client bootstraps garment type with a warm-up prediction, refines it by majority vote, freezes the voted type, and uses shorter early re-planning windows.The late-training garment classifier is >99% accurate, making the warm-up prediction usually correct while the vote provides insurance.

8 Online Round Results

The policy ranked 1st among 62 teams in the online simulation round, achieving a 79.63% overall success rate and leading the second-placed team by 6.1 pp. Performance was broad across garment types, but failures remained tied to precision, simulation physics, and the lack of recovery behavior.

Evaluation setup: 80 garments formed the online leaderboard set, covering 20 garments per type across seen and unseen categories.The simulation track used Isaac Sim, binary fold-success scoring, and four garment types.
Leaderboard performance: 79.63% overall success rate placed the final policy 1st among 62 teams, 6.1 pp ahead of the second-placed team.The online leaderboard averaged success equally across four garment types and evaluated 80 garments 10 times each.
Leaderboard performance: 74.5% / 70.0% / 80.5% / 93.5% were the per-type scores, with outright top scores on short tops, long pants, and short pants.Short pants were strongest at 93.5%, versus 91.0% for the next-best policy; long tops ranked third behind 77.0% and 76.5%.
Evaluation setup: ∼4.3M frames came from about 12,500 policy rollout episodes across ∼140 collection sessions in the final rollout dataset.The pipeline continuously pruned and re-weighted the pool, while older rollouts decayed or were removed.
Remaining failures: No recovery caused the policy to fail outright after mistakes led to out-of-distribution states, reflecting limited exploration and recovery training.Other failures involved near-correct folds missing keypoint thresholds and simulator-induced grasp failures or garment slips.

9 Sim-to-Real Transfer (Final Round)

The final round required transferring an Isaac Sim policy to a real bimanual robot under a sim-to-real-to-real gap, with only a little more than one week available. The approach combined setup alignment, distribution diversity, selective fine-tuning, and heavy augmentation, achieving 2nd place in the final sim-to-real round.

Final-round setting: More than one week was available for sim-to-real, making the transfer a hackathon-style sprint rather than a careful study.Most experiments and ideas concerned simulation, while the real-world stage focused on obtaining a working transfer and quickly fine-tuning it.
Transfer challenges: The transfer gap included camera and calibration differences, robot mechanics and backlash, garment physics, task-execution variations, and an unseen evaluation robot.The setup effectively required sim → my robot → their robot transfer, with organizer data collected under yet another physical configuration.
Transfer recipe: The recipe used an earlier strong simulation checkpoint, retained transferable heads, mixed organizer, teleoperation/DAgger, and simulation data, aligned setups, applied heavy augmentation, and resampled motion intensity.The two overarching levers were making environments more similar and broadening the training distribution so the gap fell inside it.
Implementation lessons: A subtle LeRobot 0.4.x state-representation change caused a data bug, with follower arm joints switching from normalized −100 . . . 100 encoding to degrees.The conventions were related by per-joint rescaling but looked nearly identical on screen, making the mismatch difficult to detect.
Final-round setting: 2nd place was achieved in the final sim-to-real round after transferring the policy from Isaac Sim to a real bimanual robot.The final round evaluated folding real garments on a real robot, unlike the preceding simulation round.

10 Discussion

The winning system combined practical engineering choices around a VLA that also served as its value function, while revealing persistent limitations in exploration, recovery, sim-to-real transfer, and round-specific tooling. The discussion identifies inference-time tuning, augmentation, and human interventions as effective, and proposes unifying the online and real-world pipelines.

What helped: The policy’s success head replaced a separate critic, while completion, keypoint, and ∆success predictions drove advantage estimation and best-of-N selection.These signals came from the same forward pass as action prediction.
What helped: Inference-time tuning of execution length, guidance scale, candidate count, and related parameters via cheap Thompson sampling improved performance without extra training.The optimization was performed online after training on the same checkpoint.
What helped: Performance on unseen garments was only slightly below trained garments, with misses concentrated on items genuinely far from the training distribution.One ambiguous long-pants instance was classified as short pants and stopped early in one of two rollouts.
What helped: Deliberate rig randomization and heavy augmentation made the policy relatively robust to camera, lighting, color, focal-length, table, and robot-base changes.Camera-overlay alignment kept the varied data distribution centered on the real rig.
What remained difficult: Automatic exploration remained unsolved: perturbations generally pushed flow-matching action chunks off-manifold and degraded performance instead of producing useful recovery moves.Human intervention was the only reliable recovery source, working on the real robot but not in simulation.
What should change: The online round used RL machinery, whereas the sim-to-real round used plain BC; a unified real-side reward/value pipeline could combine their complementary strengths.The proposed pipeline includes advantage conditioning, best-of-N, prioritized real-rollout sampling, and value-weighted DAgger interventions.