PaperlayerView exact revision on arXiv

Source-linked AI summary

Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence

Ying Chen, Weizhen Li, Zhe Hu, Zhenjiang Li, Rui Jiang, Zhifeng Gu, Lihuang Fang, Jiangping Liu, Lei Yi, Jie Chen

arXiv:2608.06756v1cs.AI

TL;DR

Embodied VLMs must repeatedly reason about changing scenes, actions, events, and task states, but existing post-training is often organized around isolated objectives. Capek 0.5 organizes four execution-centric capabilities and consolidates specialists into one model, improving most matched benchmark rows while transferring to closed-loop simulation.

  • Problem

    Robot execution continually changes the scene and physical state, while existing pipelines organize heterogeneous capabilities around isolated tasks, datasets, or benchmark objectives.

  • Method

    Capek 0.5 trains specialists for four execution-centric capability families, then consolidates them through TIES weight merging and routed MOPD.

  • Results

    Capek 0.5-35B-A3B and Capek 0.5-2B improve 28 of 34 and 30 of 34 matched benchmark rows, respectively, while transferring to simulated closed-loop environments.

  • Takeaways & Limitations

    The unified model retains the four specialized capabilities in one inference-time checkpoint and supports evaluation beyond isolated benchmarks in simulated embodied environments.

  • Takeaways & Limitations

    State verification is trained with teacher-side simulator information, while the model receives only the instruction and causal visual evidence available at the selected checkpoint.

Abstract

from arXiv · show

Vision-language models are increasingly serving as the reasoning core of embodied agents. Robot execution is inherently iterative: each action reshapes the scene and physical state, continually renewing what must be perceived, reasoned about, and verified. Meeting these demands requires complementary capabilities that differ in supervision signals, prediction formats, and verification criteria. Existing approaches typically develop these capabilities against isolated, task-specific objectives, leaving open how they should be organized and integrated around execution as a whole. We present Capek 0.5, an embodied vision-language model built around an execution-centric capability taxonomy. Rather than organizing training by datasets or tasks, the taxonomy groups embodied capabilities according to their functional roles throughout execution and comprises four capability families: Spatial Reasoning, Temporal Understanding, Action Guidance, and State Verification. Each capability is first acquired by a dedicated specialist through reinforcement learning with verifiable rewards from a shared backbone, and the specialists are then consolidated into a single inference-time model through weight-space merging followed by routed policy-space distillation. We instantiate Capek 0.5 at the 2B and 35B-A3B scales and evaluate it from three complementary perspectives: comprehensive benchmark suites including Capek-StateBench, a new benchmark for state verification; a controlled study of capability retention from specialists to the unified model; and closed-loop evaluation in simulated embodied environments. Capek 0.5 improves the large majority of matched benchmark rows over its initialization, retains all four specialized capabilities in one checkpoint with quantified losses, and transfers to closed-loop embodied task execution.

1 Introduction

Capek 0.5 organizes embodied reasoning around the iterative execution cycle, where actions continually change the scene and require renewed spatial, temporal, action, and state reasoning. It acquires these capabilities through specialized, verifiable objectives and consolidates them into unified models evaluated through benchmarks, retention analysis, and closed-loop simulation.

  • Motivation: Robot execution is inherently iterative: each action changes the scene and physical state, renewing the evidence needed for subsequent decisions.Effective execution requires interpretable, verifiable intermediate predictions that inform later actions.
  • Capability taxonomy: Capek 0.5 introduces an execution-centric taxonomy spanning Spatial Reasoning, Temporal Understanding, Action Guidance, and State Verification.The taxonomy groups capabilities by their functional roles throughout execution and uses typed, verifiable output formats.
  • Model construction: Capek 0.5 consolidates capability specialists into one inference-time model through TIES initialization followed by routed MOPD.The approach separates capability acquisition from consolidation to address heterogeneous supervision signals and output formats.
  • Evaluation: Capek 0.5 is instantiated as Capek 0.5-2B and Capek 0.5-35B-A3B and evaluated with benchmark suites, controlled capability-retention analysis, and simulated closed-loop environments.The evaluation separately analyzes capability acquisition, consolidation, and composition, including Capek-StateBench for physical- and task-state verification.
  • Evaluation: 28 of 34 matched benchmark rows improve for Capek 0.5-35B-A3B, while 30 of 34 improve for Capek 0.5-2B, against protocol-matched Qwen initializations.These results are reported across the two model tracks’ matched benchmark comparisons.

2 Overview

Capek 0.5 uses 2B and 35B-A3B vision-language models with a shared multimodal autoregressive interface and post-training formulation. It separates capability acquisition across four specialists from integration into one unified inference model.

  • Model Overview: Capek 0.5 comprises 35B-A3B and 2B models built on the Qwen vision-language series with ViT–LLM architectures.Both tracks use independently initialized dense and MoE backbones at different capacity scales.
  • Model Overview: All capabilities use text generation without task-specific decoding heads, emitting coordinates, temporal spans, or structured state fields.Outputs follow the format associated with each capability family.
  • Capability Integration: The four capability families are acquired by independently post-training same-origin specialists under capability-aligned formats.This separation reflects differences in output formats and reward geometries.
  • Capability Integration: Specialist task vectors are composed with TIES, then consolidated into a unified student through routed MOPD on student-generated prefixes.Specialist routes are used only during training, not inference.

3 Capability-Oriented Data Construction

The data construction organizes supervision by four execution roles—Spatial Reasoning, Temporal Understanding, Action Guidance, and State Verification—rather than by source dataset alone. Each family aligns its formats and evidence with the capability specialist it trains, including causal checkpoint evidence for verifying physical state and task progress.

  • Capability taxonomy: Four capability-aligned supervision families cover recurring embodied information needs: spatial geometry, temporal events, action guidance, and state verification.The families correspond to Spatial Reasoning, Temporal Understanding, Action Guidance, and State Verification.
  • Capability taxonomy: Records are organized by primary execution role and output format, while multiple formats within a family remain complementary outputs for one specialist.Action Guidance combines pointing, affordance localization, and trajectory prediction without creating separate specialist branches; standardization harmonizes serialization across families.
  • Spatial Reasoning: Spatial supervision combines single-view geometry with cross-observation reasoning over relations, metrics, viewpoints, and object correspondence.Its corpus draws from SenseNova-SI-8M, VSI-590K, MindCube, and EmbSpatial-SFT, spanning spatial relations, metric reasoning, and multi-view reasoning.
  • Temporal Understanding: Temporal supervision combines video reasoning with temporal localization to model event content, order, causality, boundaries, and scene changes over time.The data include closed-form video reasoning and temporal grounding across short clips and long-form videos.
  • Action Guidance: Action Guidance unifies visual grounding, pointing, affordance localization, and trajectory prediction because each produces action-oriented visual targets despite differing output formats.The supervision covers object regions, interaction points or regions including negative cases, waypoint sequences, and embodied instruction grounding.
  • State Verification: State Verification uses execution-aligned physical and task levels, with minimal local evidence for action-result predicates and causal task prefixes for progress and next-action estimation.Teacher-side checkpoint selection may use trajectory metadata and future observations, but the model receives only the instruction and causal visual evidence available at the checkpoint.

4 Post-Training

Capek 0.5 uses a two-stage post-training paradigm that independently acquires four execution-centric capabilities from a shared backbone before consolidating them into one policy. Specialists are trained with capability-specific verifiable rewards and GRPO, then combined through TIES merging and routed Multi-Teacher On-Policy Distillation.

  • Two-Stage Post-Training: Post-training separates independent capability specialization from consolidation into a single inference-time model.The four specialists are optimized from the same backbone, then consolidated through weight-space merging followed by on-policy distillation.
  • Capability Specialization: Four specialists cover Spatial Reasoning, Temporal Understanding, Action Guidance, and State Verification, each trained with its own data and rewards.All specialists retain identical architecture and tokenizer, enabling parameter-compatible consolidation despite heterogeneous output formats.
  • Capability Specialization: GRPO trains every specialist using task-specific parsers and verifiers under a unified reward formulation.Rewards combine optional format validity with task accuracy, while group-relative normalization removes the need for a learned value function.
  • Capability Consolidation: TIES first initializes the unified student by resolving conflicts among specialist updates, after which routed MOPD transfers their behaviors using student-generated trajectories.TIES masks the 80% lowest-magnitude entries within each task vector, with E = 4, τ = 0.8, and λ = 1.0.

5 Experiments

Section 5 evaluates Capek 0.5 at 35B-A3B and 2B scales across four embodied capability families, general-retention controls, consolidation variants, and closed-loop environments. It introduces Capek-StateBench to assess physical- and task-state verification, capabilities underrepresented in existing benchmarks.

  • Evaluation scope: Capek 0.5-35B-A3B and Capek 0.5-2B are evaluated on general and embodied benchmarks spanning Spatial Reasoning, Temporal Understanding, Action Guidance, and State Verification.The evaluation also includes general multimodal and language benchmarks as regression controls for retained capabilities.
  • Consolidation and closed-loop evaluation: The study compares four capability specialists, Mix-RL, TIES initialization, MOPD-only student, and released TIES+MOPD model to measure capability retention during consolidation.Closed-loop embodied interaction is evaluated with EmbodiedBench and VIGIL.
  • Capek-StateBench: Capek-StateBench addresses post-execution verification through complementary physical-state and task-state tracks.Capek-StateBench-P evaluates observable object states and spatial relations, while Capek-StateBench-T predicts current task progress and the immediate next action from execution history.
  • Evaluation protocol: Offline evaluation assigns each benchmark to one capability group by dominant prediction type while retaining its original protocol and native metric.The 35B-A3B track is the main comparison, while the 2B track uses the same benchmark rows as a compact-capacity counterpart.
  • Capek-StateBench: State Verification reports normalized exact-match accuracy for physical states and progress, next-step, and joint scores for task states.Diagnostic metrics include progress mean absolute error, terminal versus non-terminal performance, label-space breakdowns, and output validity.

Appendix A.1 gives the prompts and output formats used for the different benchmark types.

Capek 0.5 improves execution-facing capabilities across 35B-A3B and 2B benchmark tracks, retains specialized skills during consolidation, and transfers these gains to closed-loop embodied execution. The results span spatial reasoning, temporal understanding, action guidance, and state verification while largely preserving general abilities.

  • 5.2 Overall Benchmark Results: Capek 0.5 improves 28 of 34 matched rows over Qwen3.6 on the primary 35B-A3B track, with gains concentrated in execution-facing capabilities.Action Guidance improves on all 10 rows, including PixMoPointsEval, Where2Place, and NaviTrace.
  • 5.2 Overall Benchmark Results: VABench-trace RMSE decreases by 31.13 pixels and ShareRobot-Trajectory DFD decreases by 0.1128, extending gains beyond recognition-style grounding to action-oriented paths.These metrics are distance-based, so lower values indicate improvement.
  • 5.2 Overall Benchmark Results: Both State Verification rows improve, with Capek-StateBench-P increasing by 11.00 points and Capek-StateBench-T by 2.45 points.Temporal Understanding improves on all five evaluated rows, while Spatial Reasoning improves on 6 of 9 rows.
  • 5.2 Overall Benchmark Results: Capek 0.5-2B improves 30 of 34 matched rows, including all 9 Spatial Reasoning rows, all 10 Action Guidance rows, both State Verification rows, and 4 of 5 Temporal Understanding rows.Its largest reported gains include VABench-point by 28.87 points, PIO by 24.59 points, and NaviTrace by 54.67 points.
  • 5.3 Consolidation Analysis: Matching experts improve all eight signature benchmarks from a shared backbone, including (+12.85) on MindCube, (+10.16) on VSI-Bench, (+12.60) on EgoTempo, and (+14.20) on StateBench-P.The specialized experts provide the source checkpoints for consolidation.
  • 5.3 Consolidation Analysis: TIES+MOPD achieves the strongest overall consolidation trade-off, leading on MindCube, VABench-trace, and StateBench-P while ranking second on the other five benchmarks.Relative to MOPD alone, it improves seven of eight results while giving up only (0.45) points on VABench-point.
  • 5.4 Embodied Agent Evaluation: Capek 0.5 achieves the highest average success rate on EB-HAB and EB-ALF, outperforming scale-matched Qwen3.6 by 17.0 and 4.6 percentage points, respectively.Long-horizon success increases by 26.0 points on EB-HAB and 6.0 points on EB-ALF.
  • 5.4 Embodied Agent Evaluation: Capek 0.5 raises overall VIGIL world completion from 37.4% to 38.6% and benchmark success from 28.8% to 32.2%, with pixel grounding increasing from 76.0/52.8 to 85.6/72.8 in W/B.Approach-and-interact success rises from 18.4% to 28.8%, while sequential manipulation rises from 3.2% to 5.6%.

6 Conclusion and Future Work

Capek 0.5 unifies four execution-centric embodied reasoning capabilities by specializing and consolidating experts from a shared checkpoint. Evaluations span benchmarks, capability retention, and closed-loop simulated execution, while future work targets tool-mediated agentic embodiment.

  • Conclusion: Capek 0.5 unifies Spatial Reasoning, Temporal Understanding, Action Guidance, and State Verification experts into one inference-time model.Experts are derived from a shared checkpoint and consolidated through weight-space merging followed by routed policy-space distillation.
  • Conclusion: The post-training recipe matches each expert’s training to its native supervision signals and output contracts, organizing capabilities around recurring evidence needs in the execution cycle.
  • Conclusion: Evaluation covers public benchmarks at 2B and 35B-A3B, specialist-to-unified capability retention, and closed-loop execution in simulated embodied environments.Together, these levels assess broad embodied capability, retention, and capability composition during interactive task execution.
  • Future Work: Future work aims to build an agentic embodied brain that perceives, orchestrates, and acts through perception, executable-code, and action tools.The proposed system would use code for long-horizon decomposition and recovery, expose robot skills through action tools, and treat tool invocation as a first-class execution-cycle action.

A Training and Evaluation Details

Training uses a shared systems stack and frozen specialist configurations, while policy-space integration distills the TIES-merged checkpoint through routed expert supervision. The integration run specifies fixed batch, optimization, sequence-length, hardware, and teacher-serving settings.

  • Training infrastructure: Training uses verl and Megatron-LM with vLLM rollouts and bfloat16 precision, while the 2B track shares verifier and optimization interfaces with branch-specific manifests.The 35B-A3B specialist configurations are recorded in frozen GRPO manifests; 2B settings are recorded separately.
  • Policy-space integration: The integration run initializes from the TIES-merged checkpoint and trains for one epoch with global batch size 240, learning rate 1×10−6, and two student rollouts per prompt.Maximum prompt and response lengths are 10,240 and 5,120, respectively.
  • Policy-space integration: The run uses 12 actor GPUs with TP2/EP4 and rollout TP2, serving the four TIES expert checkpoints as frozen MOPD teachers.Each training contract is routed to its responsible expert.

A.1 Evaluation Prompts · A.2 Capek-StateBench Protocol · A.3 Trajectory Reward and VABench-Trace Score

The appendix fixes evaluation-time prompts and scoring settings, defines the two Capek-StateBench tracks and their aggregation rules, and distinguishes trajectory-reward computation from the public VABench-trace metric. These protocols enforce standardized parsing, validity handling, and benchmark-specific evaluation procedures.

  • A.1 Evaluation Prompts: Public benchmarks use fixed media inputs, questions, prompts, parsers, and scorers across models, with Capek 0.5 and Qwen configured for 128K context and up to 64 video frames.Both use temperature 0.7, top-p 0.95, top-k 20, and at most 16,384 output tokens.
  • A.1 Evaluation Prompts: Evaluation prompts enforce benchmark-specific JSON outputs, including normalized points, waypoint labels, feasible paths, localization answers, and binary judge responses.Examples include coordinates normalized between 0 and 1000 and outputs containing point_2d entries labeled as waypoints.
  • A.2 Capek-StateBench Protocol: Capek-StateBench-P contains 500 examples and scores physical-state verification by exact match, with invalid or ambiguous answers receiving zero and PSV reported as mean accuracy.The track uses local image or video evidence and normalizes predictions to the state label licensed by each question.
  • A.2 Capek-StateBench Protocol: Capek-StateBench-T contains 213 task-condition and 287 primitive-skill records, estimating progress from current visual evidence while excluding future observations.Progress is defined by satisfied BEHAVIOR-1K goal conditions for task-condition records and completed annotated child skills for primitive-skill records.
  • A.2 Capek-StateBench Protocol: The two Capek-StateBench-T definitions remain separately typed, and next-action evaluation targets the immediate annotated continuation rather than an optimal policy action.The primary result is the mean over all 500 records, with breakdowns retained by annotation type, scenario type, and terminal status.
  • A.3 Trajectory Reward and VABench-Trace Score: Trajectory reward rescales valid nonempty predicted and reference paths to [0,1]^2, arc-length-resamples them to a common length, and compares them with an endpoint-preserving monotone coupling.Invalid or empty trajectories receive zero; resampling removes waypoint-count dependence, while discrete Fréchet distance preserves path order and worst-case alignment.
  • A.3 Trajectory Reward and VABench-Trace Score: VABench-trace separately reports global trajectory RMSE over valid waypoint pairs after benchmark preprocessing and resampling, in padded-square pixel coordinates.The public metric is lower-is-better and remains in benchmark pixel scale; the training reward kernel is not used to transform table entries.

B Qualitative EmbodiedBench Rollout

A successful long-horizon EmbodiedBench rollout demonstrates Capek 0.5’s execution-centric interface across grounding, manipulation, appliance interaction, and final state verification. The episode completes 20 steps without an invalid action.

  • Long-horizon rollout: The rollout resolves the required tool and object, slices an apple, stores the knife in a bowl, heats the slice in a microwave, and places it on the dining table.This sequence combines object grounding, state-changing manipulation, appliance interaction, and final task-state verification within one rollout.
  • Execution trace: 20 successful steps are executed with no invalid action.Each selected keyframe is paired with the model action executed at that step.

C Qualitative VIGIL Rollouts

A representative VIGIL rollout shows the model identifying and repeatedly attempting to activate a toaster, diagnosing failed attempts from unchanged visual and task-state evidence. The episode succeeds at Step 8 and closes with a matching success report at Step 10.

  • Representative rollout: The model approaches a table, identifies a toaster among nearby objects, and repeatedly attempts to activate it.Several early activation calls do not change the task state.
  • Failure diagnosis: The model attributes failed activations to being slightly out of interaction range or targeting imprecision.This distance diagnosis is inferred from unchanged visual and task-state evidence rather than returned explicitly by the environment.
  • Episode outcome: The successful activation occurs at Step 8, and Step 10 closes the episode with a matching success report.The rollout frames pair each observation with the model’s exact function call.

D Qualitative Static QA Rollouts

Qualitative static QA rollouts demonstrate Capek 0.5 across spatial reasoning, state verification, and manipulation trajectory prediction. The examples show correct lighter-state tracking and next-action prediction, alongside a completion underestimate and object-centric placement guidance.

  • Physical-state verification: The model tracks the lighter across chronological observations and correctly labels its final state as off.The first image shows a flame, while the second shows no flame and a red/orange tip.
  • Task-state verification: For the packing task, the model identifies opening the refrigerator as the immediate next action but estimates completion at 75% instead of the 83.33% ground truth.Three of four listed conditions are satisfied; the bottle of tea remains outside the packing box.
  • Spatial reasoning: Across bathroom views, the model identifies a small glass shelf on the wall inside the shower area as a place for shampoo.The shelf is shown holding small bottles, supporting its use for toiletries.
  • Grounding and trajectory: For placing the yellow object into the silver pot, the model identifies the objects and emits an eight-point object-centric trajectory.The rollout sequences approach, grasp, lift, movement over the pot, lowering, placement, and release.

D.1 Static QA Failure Cases

Static-QA failures show that the model can identify plausible visual evidence while still selecting incorrect execution targets, spatial relations, or chronological answers. The errors include misplaced action points, incorrect destination surfaces, and misread ordering cues.

  • Representative failure pattern: The model often recognizes relevant objects but commits to an incorrect execution target or spatial relation.These failures remain interpretable because the model identifies plausible visual evidence before making a brittle decision.
  • Toy-rabbit placement: 0.0000 acc: the model grounds the rabbit’s destination near the drawer front instead of the drawer’s top surface.The parsed model point is [237, 868].
  • Infographic chronology: The model answers Iron Man instead of the ground-truth first movie, Captain America: The First Avenger, after reading the Phase One release label.It follows the infographic’s visible “2008 Iron Man” label while ignoring the chronological ordering cue.
Loading 2608.06756v1…