HABILIS Brain 0: Geometry-Change Supervision for Vision-Language-Action and Residual Flow Recovery

• Static current-depth supervision does not directly describe geometric transitions such as gripper approach, drawer motion, contact, or object pose change toward a goal.Source: §1 · p. 1
• GC-VLA predicts changes between current and future spatial representations rather than reconstructing future observations or complete future latent states.Future observations construct training targets; at deployment, the geometry-change representation is predicted from current observations and instruction.Source: §1 · p. 1
• Geometry pretraining can use robot demonstrations and action-free egocentric human video before embodiment-specific robot-action conversion.The paper motivates this interface because robot-action supervision is unavailable for human video and inconsistent across embodiments, action spaces, and data sources.Source: §1 · p. 1
• The four-stage pipeline trains GC-VLM, adds an ActionExpert with detached action-flow gradients, jointly adapts trainable VLM components, then applies GCRF to frozen GC-VLA.Inference uses current instruction, RGB observations, and robot state; future observations and offline target encoders are not used by the deployed policy.Source: §1 · pp. 1–2
• 99.55% success on LIBERO is reported for GC-VLA with GCRF, compared with 95.20% for GC-VLA.Both are reported under the paper's evaluation protocol.Source: §1 · p. 2

• GC-VLA follows the VLM-to-continuous-action design of π0 [1], using geometry change over a fixed prediction horizon as VLM-side supervision.RT-1 and RT-2 [2] [3] scale transformer VLA policies, and OpenVLA [13] is an open generalist VLA trained on diverse robot demonstrations.Source: §2 · p. 2
• Unlike future-state approaches, GC-VLM supervises future–current geometry change and conditions inference-time action generation on representations predicted from current observations.Geometry-aware objectives and future-state prediction provide task-relevant structure for robot policies [36].Source: §2 · p. 2
• GC-VLA uses a continuous-action viewpoint related to Flow Matching and FlowPolicy [38], while Stage 4 corrects the flow velocity rather than the final action chunk.Diffusion Policy models action chunks through conditional denoising.Source: §2 · p. 2
• GCRF applies a bounded residual to the action-generation velocity field, connecting residual adaptation with guided flow methods [41] and Residual Flow Steering [29].Source: §2 · p. 2

3.1. Geometry-Change Targets

• The training label is geometry change over the prediction horizon, represented on a 10 × 10 spatial grid quantized into 100 token positions for each view.The target offset is HGC = round(0.5f) frames; head views capture scene-level change, wrist views provide local gripper–object observations, and missing-camera slots are excluded by a validity mask.Source: §3.1 · p. 3

3.2. Architecture

• GC-VLM adds 12 transformer blocks to the 36-layer Molmo2-ER backbone, and the continuous ActionExpert consumes learned visual-language and geometry-change representations.The GeometryReader pools geometry-change hidden states with learned queries and exposes them to selected ActionExpert blocks through cross-attention residual updates; its zero-initialized output projection makes it a no-op at initialization.Source: §3.2 · p. 3

3.3. Leakage Prevention

• Future frames construct geometry-change labels during training and do not enter the evaluation path, where rollouts use only current observations.Source: §3.3 · p. 3

3.4. GCRF: Geometry-Conditioned Residual Flow

• GCRF keeps the base VLM and ActionExpert frozen and uses a binary router to apply one bounded residual velocity correction within the base flow solver.The correction is held constant over one replan's ten integration steps, while base velocity is reevaluated at each updated flow state; subsequent replans may produce a different correction.Source: §3.4 · pp. 3–4

3.2. Architecture

• Figure 1 summarizes GC-VLM prediction from current observations, ActionExpert introduction during alignment, and the Stage 2-to-Stage 3 gradient-routing change.Future observations are used only for offline target generation.Source: Figure 1 · p. 4

4.1. Pretrained Geometry Models and Target Generation

• Depth Anything 3 pseudo-depth and DINOv2 features are used offline to construct geometry-change targets, with neither pretrained model executed during GC-VLA inference.Targets use current and future frames separated by H = round(f · 0.5), then quantize the continuous DINOv2 feature difference into 100 codes per view across three fixed view slots.Source: §4.1 · pp. 4–5

4.2. Stage-wise Training Data and Optimization

• Stage 1 geometry pretraining initializes GC-VLM from Molmo2-ER and trains it on current–future frame pairs with offline geometry-change targets, requiring no robot-action labels and using no ActionExpert.Source: §4.2 · p. 5
• Stage 2 trains a continuous ActionExpert on a common 14-dimensional bimanual end-effector delta representation while stopping action-flow gradients at the VLM interface.Unavailable arm dimensions are masked; Stage 3 removes the detach boundary so action-flow gradients update trainable VLM components with the ActionExpert.Source: §4.2 · p. 5
• The stage-specific training objective combines token cross-entropy with action flow-matching where enabled, and Stage 4 freezes GC-VLA for residual post-training from closed-loop feedback.Source: §4.2 · p. 5

4.1. Pretrained Geometry Models and Target Generation

• Figure 2 shows head and wrist views, current and future RGB, offline pseudo-depth, DINO feature-difference magnitude, quantized tokens, and target overlays.The overlay visualizes the training target rather than model attention.Source: Figure 2 · p. 6

4.3. Unified End-effector Action Representation

• Each arm is represented by translation, a three-component rotation increment, and a gripper command, with fixed left- and right-arm positions.The gripper command is not a gripper-state difference.Source: §4.3 · p. 6
• Unavailable arm dimensions are masked, and invalid dimensions are excluded from the action loss.Source: §4.3 · p. 6
• Source adapters resolve pose conventions, units, arm ordering, gripper semantics, and timing before converting demonstrations into adjacent end-effector transitions.Joint-position sources require forward kinematics with the corresponding robot model and frame calibration; sources without validated conversion are excluded from action training.Source: §4.3 · p. 6

4.4. Data

• Stage 1 pretrains geometry change on diverse human and robot video, without requiring action annotations.Stage 2 uses physically validated action-bearing robot caches mapped to a common masked action interface; LIBERO is excluded from both broad pretraining mixtures.Source: §4.4 · p. 6

4.5. Training Data

• Stage 1 uses human and robot videos for geometry supervision, while Stage 2 uses robot demonstrations with validated end-effector action conversion.Source: §4.5 · p. 6
• LIBERO is introduced for Stage 3 coupled adaptation, and Stage 4 uses closed-loop rollouts of the frozen policy and residual candidates.LIBERO is retained as the canonical closed-loop evaluation benchmark.Source: §4.5 · p. 6, §4.4 · p. 6

4.6. Action Conversion and Temporal Alignment

• Action preprocessing decodes each source, converts it to the canonical end-effector frame convention, normalizes translation and gripper units, and constructs adjacent end-effector transitions.Joint-state sources undergo validated forward kinematics before conversion; resulting left- and right-arm targets are concatenated and masked.Source: §4.6 · pp. 6–7
• Action chunks comprise consecutive transitions, and action and geometry supervision are aligned with each source’s validated timestamps and sampling rate.At canonical LIBERO inference, the policy predicts a horizon-10 action chunk.Source: §4.6 · p. 7

4.7. Optimization Summary

• Geometry-token losses mask unavailable view slots, while action losses mask unavailable action dimensions.Source: §4.7 · p. 7
• Stage 2 detaches the flow objective at the VLM–ActionExpert interface, whereas Stage 3 enables action-flow gradients through trainable VLM components.Source: §4.7 · p. 7, Table 2 · p. 7

4.8. Stage 4: GCRF—Geometry-Conditioned Residual Flow

• The residual policy is optimized with a continuous value head using an explicit no-op latent, with closed-loop contexts and terminal binary success labels.The backbone and ActionExpert remain frozen during this stage.Source: §4.8 · p. 7
• Figure 3 pairs current RGB, future RGB, and a target overlay showing the spatial magnitude of DA3–DINO feature difference before quantization.Frame selection uses a nominal 0.5-second horizon, with the realized interval depending on frame discretization; future RGB constructs training targets only.Source: Figure 3 · p. 8
• Table 3 compares action-chunk prediction from alternative representations under a matched state baseline and data split.Source: Table 3 · p. 8
• The value head learns from binary outcomes, while centered value differences relative to no intervention weight sampled residuals and successful samples impose a margin over no-op.Source: §4.8 · p. 8
• A residual is selected only when its policy-mean value exceeds the no-intervention value by margin m; otherwise the selected latent is zero.The binary intervention router is trained separately from closed-loop outcome supervision.Source: §4.8 · p. 8

4.9. Evaluation Protocol and Metrics

• Canonical LIBERO evaluation covers four suites, ten tasks per suite, and fifty fixed initial states per task, reporting episode success rates and their mean over suites.The reported run uses one fixed-initialization evaluation with seed 1000 and batch size five; closed-loop success is the primary policy metric.Source: §4.9 · p. 8

5.1. Geometry-Change Representation Probes

• Matched frozen GC representations improve manipulation-phase decoding and modestly improve action-chunk prediction relative to current-depth features.The action-chunk gain has a 95% confidence interval of [+0.004, +0.037] and one-sided sign-flip p = 0.027.Source: §5.1 · p. 9, Table 4 · p. 9
• On the matched dynamic-patch benchmark, DA3–DINO feature differences achieve AUROC/AP of 0.937/0.762, outperforming affine-aligned direct depth and VQ depth on change discrimination.Static-region false-positive rate is 0.027 for DA3–DINO differences, versus 0.092 for affine depth and 0.093 for VQ depth; direct depth remains competitive in some in-domain action probes.Source: §5.1 · p. 9
• Wrist information complements head features in a diverse-task action probe: their aligned combination reaches R2 = 0.654, versus 0.467 head-only and 0.492 wrist-only.Shuffling wrist features reduces R2 to 0.379; the pseudo-3D trajectory pilot did not produce a contract-valid SE(3) target and was not a controlled policy ablation.Source: §5.1 · p. 9

5.2. Direct-Training Geometry-Change Ablation

• Replacing current-depth supervision with geometry-change supervision raises closed-loop success from 82.1% to 86.1%, adding 80 successful episodes.Neither variant used the Stage 1–2 pretraining curriculum; the subsequent 95.20% result combines the target-design benefit with dedicated geometry pretraining and staged adaptation.Source: §5.2 · pp. 9–10

5.4. GCRF Evaluation

• 99.55% success on LIBERO with GCRF compares with 95.20% for GC-VLA alone, with the GC-VLA base parameters fixed.The evaluated GCRF configuration applies the selected bounded velocity correction at all ten flow integration steps within each replanning cycle.Source: Table 6 · p. 10, §5.4 · p. 10

5.3. Stage 1–3: Canonical LIBERO Evaluation

• The Stage 1–3 GC-VLA policy achieves 95.20% success under the canonical LIBERO evaluation protocol.Table 7 groups published comparison methods by adaptation regime, but notes that differing data and evaluation implementations preclude a controlled held-out-cohort comparison.Source: §5.3 · p. 10, Table 7 · p. 10

5.5. Mechanism Analysis: GC Reader and Residual Placement

• On the four-task diagnostic panel, reader removal has no Holm-significant planned contrast, while all-step residual application scores 97.5% versus 86.5% for either alternative.The reader intervention retains the geometry-change backbone representation; residual-placement results support distributed in-flow application on this panel, not comparisons between independently trained residual policies.Source: §5.5 · p. 11, Table 8 · p. 11

5.6. LIBERO-PRO Four-Axis Perturbation Evaluation

• 51.09% macro-average on the four reported LIBERO-PRO perturbation axes is comparable to π0.5 at 53.35%, with performance varying substantially by axis.GC-VLA+GCRF scores 93.00% on language, 71.85% on object, 18.05% on position, and 21.45% on task perturbations; this is a perturbation-robustness test within related task families, not evidence of embodiment or environment transfer.Source: §5.6 · pp. 11–12, Table 9 · p. 12

6. Limitations

• 99.55% on LIBERO characterizes the combined procedure, including closed-loop post-training on the benchmark distribution, rather than pretraining scale alone.Source: §6 · p. 12
• LIBERO-PRO measures perturbation robustness within related simulated task families.Source: §6 · p. 12
• The combined-system result does not isolate GCRF’s causal contribution without a matched base-policy comparison.Source: §6 · p. 12

7. Conclusion

• 99.55% success on LIBERO is reported for the frozen GC-VLA policy augmented with GCRF under the reported evaluation protocol.Source: §7 · p. 12
• GC-VLA learns action-relevant visual representations through multiview geometry-change supervision, followed by detached action alignment and coupled policy adaptation.Source: §7 · p. 12
• 95.20% success on LIBERO is reported for the resulting GC-VLA policy before GCRF augmentation.Source: §7 · p. 12
• GCRF uses a binary intervention router and one bounded residual velocity policy to augment the frozen policy.Source: §7 · p. 12