PaperlayerView exact revision on arXiv

Source-linked AI summary

FineART: Fine-Grained Annotated Robotic Trajectory Dataset and Vision-Language-Action Model for Bimanual Manipulation

Jade Choghari, Pepijn Kooijmans, Mansi Agarwal, Yusuf Umut Ciftci, Aseem Doriwala, Catherine Weaver, Mouli Sivapurapu, Kai Yang, Thomas Wolf, Jackson Lee, Pragna Mannam

arXiv:2609.36416v3cs.ROcs.AIcs.CVcs.LG

TL;DR

Long-horizon bimanual tasks need dense, persistent subtask supervision, which existing manipulation datasets provide only in limited form. FineART supplies that supervision and FineART-VLA predicts subtasks to guide actions; reported evaluations show improved instruction following and transfer across tasks and embodiments.

  • Problem

    Long-horizon bimanual manipulation needs dense, persistent subtask supervision, while existing datasets offer limited support for it.

    Source: §1 · p. 2

  • Method

    FineART provides densely annotated bimanual episodes, and FineART-VLA uses a single end-to-end policy to predict subtasks and execute actions.

    Source: §1 · p. 2

  • Results

    Dense subtask training raises spatial-instruction success from 32.0% to 100.0%, alongside reported gains on long-horizon tasks and transfer across embodiments.

    Source: §1 · p. 2, §5 · p. 13

  • Takeaways & Limitations

    FineART-VLA completes long-horizon tasks absent from the target embodiment’s fine-tuning set, while experiments indicate gains across task distributions and embodiments.

    Source: §5 · p. 13

    FineART uses a single robot platform, and the policy is trained for only about 10% of an epoch.

    Source: §5 · p. 13

Abstract

from arXiv · show

Robots operating in real-world environments must often execute complex, multi-step bimanual tasks over long horizons rather than single, isolated actions. Current manipulation datasets provide limited support for this capability: although single-arm datasets reach hundreds of thousands of trajectories, they typically provide only one high-level instruction per episode, while existing bimanual datasets provide subtask annotations for only part of their recorded hours. We present FineART, a densely annotated bimanual manipulation dataset comprising 40,543 episodes (1,718 hours) and 533,913 subtasks across 151 tasks. We also introduce FineART-VLA, a vision-language-action policy that predicts its own next subtask to guide its actions. Mid-training on FineART's subtask annotations improves FineART-VLA's success at following spatial instructions from 32.0% to 100.0%. With step-by-step human subtask guidance, success on unseen long-horizon tasks increases from 16.0% to 76.0%. Furthermore, FineART-VLA matches baseline performance on a new robot with 10x less fine-tuning data and generalizes zero-shot to unseen tasks. We open-source the full dataset, model weights, and training code.

1 Introduction

Long-horizon bimanual tasks require coordinated actions and persistent subtask supervision, but existing datasets and policies provide limited support for this setting. FineART pairs dense annotations with an end-to-end policy, and subtask training improves spatial instruction following and long-horizon task success.

1 Introduction

  • Robots in households and workplaces must perform long-horizon tasks involving dozens of coordinated two-handed actions, which humans manage by sequencing subtasks.The paper argues that robots need both long-horizon bimanual data and dense, persistent language supervision marking subtask boundaries.Source: §1 · p. 2
  • Existing datasets often pair episodes with one high-level instruction or annotate subtasks for only a fraction of recorded hours, while prior pipelines use broad labels or generate subtasks online.The paper identifies the absence of persistent, dense subtask training for long-horizon bimanual manipulation within a single end-to-end policy.Source: §1 · p. 2
  • FineART provides 533,913 subtask labels across all 40,543 episodes, paired with persistent supervision in a single end-to-end policy.The dataset comprises 1,718 hours, averaging 152.5 seconds per episode.Source: §1 · p. 2
  • 32.0% to 100.0% spatial-instruction success and 16.0% to 76.0% long-horizon success are reported for dense subtask training versus a matched policy without it.The long-horizon evaluation uses human subtask guidance.Source: §1 · p. 2

2 Related Work

Prior work has expanded robot-learning scale and introduced language-guided hierarchies, but persistent subtask labels across long bimanual episodes remain limited. FineART combines dataset-wide annotation with a single policy for subtask prediction and action execution.

2.1 Large-Scale Robot Learning Datasets

  • FineART is the only dataset described as combining bimanual manipulation, multi-minute episodes, and dense sub-episode labels across its full duration rather than a curated subset.The comparison spans trajectory counts, task counts, and annotated hours across prior datasets.Source: §2.1 · p. 3

2.2 Data Curation and Annotation Pipelines

  • FineART annotates every episode with fixed subtask training targets, unlike prior approaches that assign broad video labels or generate subtasks online.A closer prior method used an intermediate language-motion layer, but its hierarchy was produced online.Source: §2.2 · p. 3

2.3 Language-Conditioned Policies

  • FineART uses one model for subtask prediction and action execution, rather than separate planner and controller models.The policy builds on a π0.5 flow-matching backbone and knowledge insulation; related work also explored dense language annotation and subtask-conditioned diffusion policies.Source: §2.3 · p. 4

3 FineART Dataset

FineART provides full temporal supervision for a large bimanual dataset, with multi-stage episodes, a reusable primitive vocabulary, contact-rich skills, and detailed object descriptions. Its collection and annotation design supports long-horizon manipulation across varied objects and tasks.

3 FineART Dataset

  • FineART contains 40,543 teleoperated episodes, 1,718 hours, and 185.5 million frames across 151 tasks, with full temporal supervision across all hours and tasks.The paper describes it as the largest subtask-annotated robot dataset to its knowledge.Source: §3 · p. 4

3.1 Data Collection Protocol

  • Annotations include whole-episode summaries with execution metadata and contiguous, nonoverlapping temporal segments with free-text task descriptions.Human annotators and an independent audit team contributed to annotation and quality control.Source: §3.1 · pp. 4–5

3.2 Dataset Composition: Long-Horizon Multi-Stage Tasks

  • Episodes average 13.17 subtasks, have a median duration of about 2.0 minutes, and include a long tail exceeding 10 minutes.The dataset is analyzed against ABC-130K, a prior single-embodiment dataset with timestamped subtask text.Source: §3.2 · p. 5

3.2.1 Subtask Composition

  • FineART allocates 1,326.2 hours (77.2%) to pick-and-place and semantic organization, with 391.8 hours (22.8%) in contact-rich operations.The contact-rich tail includes textile manipulation, tool-mediated assembly, flexible-object routing, compliant alignment, and mechanical insertion; the paper tests the tail by ablating it.Source: §3.2.1 · p. 5

3.2.2 Object Diversity

  • FineART annotations identify 657 object classes across everyday categories, though the annotation-based comparison reflects both collection diversity and annotator naming specificity.The collection varies objects and spatial configurations; some extracted labels could not be semantically mapped from the noun alone.Source: §3.2.2 · p. 6

3.2.3 Descriptive Specificity

  • Color modifiers appear in 81.1% of FineART object mentions versus less than 0.01% in ABC-130K, and FineART disambiguates same-class instances 81.4% versus 44.0%.FineART contains 110,299 distinct subtask strings and 23,589 object-referring expressions, compared with 7,977 and 534 in ABC-130K.Source: §3.2.3 · p. 6

4 Experiments

The experiments mid-train FineART-VLA for 300k steps and evaluate subtask supervision and knowledge insulation on ALOHA, then fine-tune across embodiments on YAM. Results cover long-horizon instruction following, long-tail data, spatial grounding, and transfer to new tasks and embodiments.

4.2.1 Does dense subtask training and knowledge insulation (KI) improve OOD generalization?

  • Subtask training and knowledge insulation each improved overall performance, and policies with both showed the highest OOD generalization.Saucer and type-routed tool sorting were OOD because the dataset contained neither saucers nor type-routed tool sorting.Source: Figure 4 · p. 8, §4.2.1 · p. 9
  • Shelf placement reached 96.0% without subtask training but with KI versus 72.0% with both; towel folding benefited more from subtask training, while partial-ID tasks benefited most from KI alone.The authors suggest the comparatively worse partial-ID performance with subtask training may reflect over-sensitivity to language conditioning.Source: §4.2.1 · p. 9

4.2.2 How much does long-tail data contribute to overall performance?

  • Excluding long-tail task categories reduced overall success from 24.7% to 14.0%; every evaluation task except shelf placement degraded substantially.Run 2.1 retained pick-and-place and sorting categories, covering 106 of 151 task labels and 71.0% of valid frames.Source: §4.2.2 · p. 9

4.2.3 Does subtask training provide spatial grounding and long-horizon instruction following?

  • On the put-donut-into-left/right/middle-bin task, subtask training achieved 100% SR, versus 32% success without subtask training; all 17 failures involved placing the object in the incorrect bin.Source: §4.2.3 · p. 9
  • With human-provided subtask guidance, Run 1.3 reached 76.0% success versus 16.0% for Run 1.4, while both autonomous policies achieved 0.0% success on prepare breakfast.The autonomous policies reached 54.9% and 54.3% progress, respectively; the comparison holds guidance constant between the two runs.Source: §4.2.3 · p. 10

4.2.4 Observation: policies exhibit recovery and complex bimanual behaviors

  • Rollouts included self-correction, recovery after a fallen cup, and bimanual coordination; these behaviors were observations, not estimates of their frequency.In one flower-in-vase rollout, the left arm steadied the vase while the right inserted the flower.Source: §4.2.4 · p. 10, §4.3 · p. 11

4.3.1 Does mid-training on FineART reduce embodiment-specific downstream data requirements?

  • FineART-VLA reached 26.4% SR with 25 YAM fine-tuning episodes per task, versus 18.4% SR with 250 episodes and no mid-training.The checkpoints shared π0.5 initialization and differed only in FineART mid-training.Source: §4.3.1 · p. 11

4.3.4 Does fine-tuning on non-relevant YAM tasks enable transfer to unseen long-horizon tasks?

  • 28.0% SR and 78.3% PR on unseen YAM prepare-breakfast tasks, and 12.0% SR and 60.0% PR on unseen sort-tools tasks, followed fine-tuning on five unrelated tasks.On prepare breakfast, the policy also emitted an Adjust subtask annotated in FineART but never performed in YAM fine-tuning.Source: §4.3.4 · pp. 12–13

5 Conclusion

FineART is presented as a large bimanual dataset with dense subtask labels, and FineART-VLA experiments indicate gains across task distributions and embodiments. The authors identify limits in platform coverage, training duration, and which target tasks support cross-embodiment transfer.

5 Conclusion

  • FineART provides 533,913 subtask labels across all 40,543 episodes and 151 tasks, spanning contact-rich skills beyond ordinary pick-and-place.The authors describe it as the largest open-source subtask-annotated manipulation dataset to their knowledge.Source: §5 · p. 13
  • Experiments indicate gains on in-distribution and out-of-distribution tasks and across embodiments, with long-tail tasks supporting performance beyond object variety alone.With minimal target-embodiment fine-tuning, FineART-VLA completed long-horizon tasks absent from that fine-tuning set; the authors say it retained sequential task planning while learning target-robot kinematics.Source: §5 · p. 13
  • The dataset covers one robot platform, and the policy was trained for about 10% of an epoch; the tasks driving cross-embodiment transfer remain unclear.The authors propose longer training, more base models, and a task, annotation, and embodiment sweep as future work.Source: §5 · p. 13

A FineART-VLA: Architecture and Mid-Training Details

FineART-VLA combines subtask generation and action prediction in a single architecture. The appendix points to its full computational path and specifies shared settings for the subtask and knowledge-insulation ablations.

A FineART-VLA: Architecture and Mid-Training Details

  • Figure 8 summarizes the architecture path, Figure 9 the attention masks for the two training shapes, and Table 5 the configuration used for the Table 2(a) runs.The appendix identifies these as implementation details for the architecture and ablation setup.Source: A FineART-VLA: Architecture and… · p. 17
  • The 2B-parameter Gemma backbone with SigLIP vision attends bidirectionally to camera views and the task string, then decodes a subtask followed by FAST-tokenized actions.A separate 300M-parameter action expert predicts continuous action chunks with flow matching and attends to stop-gradient copies of backbone keys and values under knowledge insulation.Source: Figure 8 · p. 17
  • The action expert conditions on the subtask for 70% of samples, while the backbone predicts the subtask on the remaining 30%; the two cases use different attention patterns.The action expert always conditions on the 14-dimensional joint state, discretized into 256 bins and embedded as text in the prompt.Source: A FineART-VLA: Architecture and… · p. 17
  • All four subtask and knowledge-insulation ablation runs share the Table 5 mid-training configuration and use 1.0 learning-rate multipliers for the LM head, backbone, and action expert.The runs differ only in whether subtask training and knowledge insulation are enabled.Source: A FineART-VLA: Architecture and… · p. 18, Table 5 · p. 18

A.1 Task-Balanced Sampling

Task-balanced sampling addresses the dataset’s wide variation in task sizes by sampling a task first and then a valid frame within it. The paper uses an intermediate temperature setting to reduce domination by the largest tasks.

A.1 Task-Balanced Sampling

  • FineART’s 151 tasks range from dozens to tens of thousands of episodes, so uniform pooled-frame sampling would concentrate training on over-represented tasks.The paper uses temperature sampling with α = 0.5 for every run.Source: §A.1 · p. 18
  • At α = 0, tasks are sampled uniformly; at α = 1, sampling is proportional to each task’s valid-frame count, and α = 0.5 interpolates between these extremes.After selecting a task, the sampler chooses uniformly among that task’s valid frames.Source: §A.1 · p. 18
  • The implementation samples hierarchically by drawing from the 151-task distribution, then selecting a frame from that task’s precomputed valid-frame index.This avoids constructing a categorical distribution over 185.5 million frames, exceeding the stated 224-entry torch.multinomial cap.Source: §A.1 · pp. 18–19

A.2 Knowledge Insulation

Knowledge insulation (KI) blocks continuous-action-loss gradients from reaching the VLM backbone while leaving forward computation unchanged. The backbone still receives gradients from token-level losses.

A.2 Knowledge Insulation

  • KI preserves the forward computation but blocks the continuous action loss from updating VLM parameters.The stop-gradient operator is an identity in the forward pass and a constant under differentiation; the action loss is read from the action stream.Source: §A.2 · p. 19
  • Applying the stop-gradient split at every backbone layer prevents action-loss gradients from reaching any VLM parameters through this path.The split blocks gradients to keys and values, including parameters in earlier embeddings and layers that produced the VLM representation.Source: §A.2 · pp. 19–20
  • The VLM backbone continues to receive full gradients from the text and FAST-token losses; KI insulates it only from the continuous action loss.Both losses use outputs from the VLM stream, and their computational graphs do not pass through the stop-gradient node.Source: §A.2 · pp. 19–20
  • Block-causal masks keep generated or predicted suffixes from feeding back into the prefix, independently of KI.FAST-token and continuous-action outputs are mutually invisible; the action stream reads VLM keys and values through stop-gradient copies.Source: §A.2 · p. 20, Figure 9 · p. 20

A.3 Optimization Objective

The optimization and deployment recipe combines language, FAST-token, and flow-matching objectives, then supports flat, hierarchical, or operator-provided instruction conditioning. Its runtime can update instructions during execution, while autonomous replanning does not detect subtask completion.

A.3 Optimization Objective

  • All three loss terms are computed on every action chunk, and the recipe adds a 10^-4 z-loss to the text logits.Subtask-context dropout is available in the recipe but is set to 0 for every reported run.Source: §A.3 · p. 21
  • Mid-training used 300K steps at global batch 64, about 10% of one epoch over 185.5M frames, taking roughly a week on one 8×H100 node.The four subtask/KI ablations ran on separate nodes concurrently, so their checkpoints were ready within that week.Source: §A.3 · p. 21

A.4 Inference

  • Deployment offers flat task-string conditioning, hierarchical model-generated subtasks, or interactive operator-supplied subtasks, all using the same action expert and flow-matching head.Hierarchical mode regenerates a subtask every N chunks; interactive mode advances when an operator judges the previous subtask done.Source: §A.4 · p. 21

A.5 Language Runtime and Subtask Execution

  • In hierarchical and interactive modes, a subtask spans multiple action chunks; the runtime accepts instruction updates without reloading the model.Autonomous replanning uses the original goal for each query, and the runtime does not itself detect subtask completion.Source: §A.4 · p. 21, §A.5 · p. 22

A.6 Efficient Training and Inference

The implementation reduces training and inference overhead through shared computation, batching, caching, and optional optimized kernels. FlashRT kernels are inference-only and disabled by default.

A.6 Efficient Training and Inference

  • Training reuses the visual-language prefix across multiple noise and timestep draws, with five draws by default, while vectorizing action operations and averaging flow losses.Attention masks isolate action sequences and prevent access to FAST action targets.Source: §A.6 · p. 22
  • Additional training optimizations include sparse and bucketed cross-entropy, per-layer vision activation checkpointing, and fused AdamW updates.With KI and no language or FAST supervision, flow-only batches can omit the backbone backward graph.Source: §A.6 · p. 22
  • Inference reuses visual-language prefix caches during autoregressive subtask generation and action denoising; optional calibrated FP8 MLP kernels are disabled by default.The optional FlashRT kernels change numerical precision and are inference-only; Liang Su contributed implementation and optimization work underlying the section.Source: §A.6 · p. 22

B Evaluations

The evaluation summary covers 3,400 rollouts across 23 checkpoints on ALOHA and YAM.

B Evaluations

B.1 Procedure

Evaluation fixes 25 initial conditions for each task and scores checkpoint progress by subtask completion, with task-specific ordering rules.

B.1 Procedure

  • 25 predetermined position-and-orientation initial conditions are used for each task in the benchmark suites.Figure 12 illustrates the conditions for the in-distribution task put cup on shelf.Source: §B.1 · p. 23, Table 7 · p. 23, Figure 12 · p. 25

B.2 Rubric

  • Checkpoint progress rate is based on the percentage of subtasks completed.Source: §B.2 · p. 23
  • Subtasks are ordered so each must finish before the next begins, except that Prepare breakfast has no prescribed order after its first subtask.Source: §B.2 · p. 23, Table 8 · p. 24
Loading 2609.36416v3…