FLORA: An Open Model Suite for Scaling Robotics Data Engines

1. Overview

• Timestamped captions support language conditioning and data-mixture balancing, process rewards measure within-episode progress for RL, and outcome rewards monitor fleets and evaluate policies.These labels serve pretraining, post-training, and deployment, respectively.Source: §1 · p. 2
• Flora combines captioning, segmentation, and reward models to produce timestamped captions, process rewards, and outcome rewards for robotics datasets.User-defined rubrics also enable flexible reward modeling at test time.Source: §1 · p. 2
• Flora has labeled up to 200,000 hours of data per day across 512 GB300 GPUs and is open-sourced for scaling custom robot datasets and exploring new applications.Examples include physical auto-research and human-to-robot transfer.Source: §1 · p. 2

2. Capabilities and Results

• Table 1 and Figure 3 summarize Flora’s competitive scores across timestamped-caption and reward-labeling tasks.The table caption describes Flora as competitive with GPT-6 Astra while beating other baselines; Figure 3 covers six tasks.Source: Table 1 · p. 3, Figure 3 · p. 4

2.1. Timestamped Action Captions

• Flora leads WGO-Bench segmentation with 50.3% mean F1 and ranks second in joint F1 at 34.5%, while its 64.7% gold-window caption accuracy trails Astra and Gemini 3.8 Flash.Against Refiner, Flora improves mean segmentation F1 by 16.9 points and joint F1 by 7.3 points, with similar gold-window caption accuracy (64.7% vs. 64.3%).Source: §2.1 · p. 4, Figure 4 · p. 5
• On DenseCaptionBench, Flora leads all three Human100 metrics at 54.6% segmentation F1, 66.5% oracle caption quality, and 37.4% joint F1, but ranks second to Astra across Robot200 metrics.Robot200 joint F1 is 51.0% for Flora, compared with 57.9% for Astra and 33.6% for the next-best baseline.Source: §2.1 · p. 5

2.2. Reward Labeling

• Flora-Rewarder achieves a RoboRewardBench MAE of 40.0, versus 55.4 for GPT-6 Astra and 66.5 for RoboReward, a 27.8% reduction relative to Astra.The advantage comes from partially completed tasks; reported errors are multiplied by 100.Source: §2.2 · p. 5
• On RobometerBench, Flora-Rewarder trails Astra in OOD trajectory ranking at 72.0 versus 82.3, ties Astra at 98.9 ID VOC, and reaches 99.6 OOD VOC.The paper says the VOC results exceed reported specialized reward baselines on their evaluated metrics.Source: §2.2 · p. 5
• On DenseRewardBench, Flora-Rewarder leads all four metrics with 46.2 macro F1 and recalls 53.3% of failure steps, while labeling about 11% of normal steps as failures.No baseline exceeds 11.7% failure-step recall, and GPT-6 Astra detects no failures.Source: §2.2 · p. 6
• On RubricBench, Flora-Rewarder reaches a pooled L1 of 6.5, outperforming nine of ten general-purpose models, while Astra achieves 3.6.These are system-level comparisons on a benchmark of 40 teleoperated robot episodes from 10 tasks.Source: §2.2 · p. 6

2.3. Annotation Efficiency

• 57.1% quality at $0.72 per source-video hour versus Astra’s 53.7% at $88.49; Flora’s estimated cost is approximately 125× lower on the fixed 50-episode subset.The comparison uses an eight-GB300 deployment and estimates GPU cost at $10 per hour.Source: §2.3 · p. 6

3. Qualitative Examples

• Flora matches 34 of 37 reference intervals, versus four for Astra, with F1@0.5 scores of 82.9 and 18.2, respectively.The examples also show Flora identifying the intended wire insertion and right-to-left handover, while Astra misdescribes those operations.Source: §3 · p. 6
• For the falling-component example, only Flora-Rewarder assigns the failure label.Both models describe the component falling.Source: §3 · p. 6
• In the bottle task, Flora-Rewarder scores toppling without knocking the bottle off the table as partial progress (3/5), while Astra gives 1/5.Source: §3 · p. 6

4. Methods

• Flora aligns action descriptions with execution quality through localized captions and reward labels for steps, episode completion, or user-defined rubrics.One reward model provides dense, outcome, and rubric labels; prompts select the reward type.Source: §4 · p. 6, §4.2 · p. 8

4.1. Timestamped Captions

• Overlapping video windows yield episode-level action intervals, which are captioned using neighboring context while preserving the intended action even when it fails.The segmenter predicts window-relative boundaries and continuation flags; compatible fragments are stitched while concurrent actions remain separate. Neighboring frames help resolve object identity and local intent, but captions stay restricted to the target interval.Source: §4.1 · pp. 7–8

4.2. Reward Labels

• Flora-Rewarder supports outcome completion scores, retrospective step-level execution labels, and user-defined rubric grades through prompt-selected reward modes.Outcome scores range from complete failure to complete success; dense labels distinguish failure, suboptimal, and normal behavior at each 0.5-second step. Rubrics assign scores per task step.Source: §4.2 · pp. 8–9

4.3. Training

• Training combines open, licensed, and in-house data with human annotations, supervised fine-tuning, and GRPO for segmentation and captioning.The released reward model uses SFT only; training details include joint reward-family training, a verifiable segmentation reward, and reference-grounded caption-quality rewards.Source: §4.3 · p. 9

• Flora provides timestamped captions, process rewards, and outcome rewards for robotics datasets and has been used to train GR00T models.Source: §5 · p. 10
• Flora is competitive with GPT-6 Astra on captioning benchmarks while improving on reported specialized reward models on RoboReward and Robometer.Source: §5 · p. 10
• The authors hope open-sourcing Flora will help researchers scale custom robot datasets and explore physical auto-research and human-to-robot transfer.Source: §5 · p. 10

A. Complete Benchmark Results

• The benchmark tables retain evaluated models and metric breakdowns, scale values by 100, and mark best displayed results, including ties.Evaluation protocols are in Appendices B and C.Source: A. Complete Benchmark Results · p. 13

A.1. Timestamped captions: WGO-Bench

• WGO-Bench reports subset scores and equal-weight average percentages, comparing off-the-shelf models using a common pipeline with Flora's default pipeline.Source: Table 2 · p. 13

A.2. Timestamped captions: DenseCaptionBench

• DenseCaptionBench evaluates 100 human, 100 parallel-gripper, and 100 dexterous-hand episodes under a shared point-quality protocol.Off-the-shelf models use the common pipeline, while Flora uses its production pipeline.Source: Table 3 · p. 13

A.3. Public reward benchmarks

• Flora-Rewarder has the lowest reported MAE on the public reward benchmarks and exceeds specialized reward baselines on their reported metrics.Table 4 reports metrics multiplied by 100; published specialized-model scores retain their original protocols.Source: Table 4 · p. 14

A.4. Dense and rubric rewards

• Flora-Rewarder achieves the best values on all four DenseRewardBench metrics and grades rubrics more accurately than nine of ten general-purpose models.Astra has a lower pooled L1 score than Flora-Rewarder on RubricBench.Source: Table 5 · p. 14, §A.4 · p. 15

B. Timestamped Captioning: Evaluation Details

• Evaluation covers localization, reference-interval caption quality, and joint annotation on complete episodes, using the configurations reported in the main text.The supplementary protocol specifies benchmark construction, visual inputs, scoring, and pipeline comparisons.Source: B. Timestamped Captioning: Evaluation… · p. 15

B.1. DenseCaptionBench construction

• DenseCaptionBench contains 300 episodes, and its reference intervals may include idle gaps or concurrent intended actions.Initial selection uses existing annotations and metadata, not model outputs or scores, and imposes no event-count, duration, or event-density quota; human cohort coverage is conditioned on source and service availability.Source: Table 6 · p. 15, §B.1 · p. 15

B.2. Inputs and inference settings

• Off-the-shelf systems use two stages: segment timestamped contact sheets, then caption each interval independently from before, target, and after blocks.Flora uses a native production workflow with different processors, prompts, window schedules, and inference compute, so these are complete-system comparisons.Source: §B.2 · p. 16

B.3. DenseCaptionBench scoring and aggregation

• Caption quality weights action, identity, coordination, and detail requirements at 0.50, 0.25, 0.20, and 0.05, respectively.Joint annotation matches temporally eligible predictions and references one-to-one to maximize total caption quality.Source: §B.3 · p. 17

B.4. Output validity and incomplete measurements

• The Astra judge is also an evaluated system, creating a stated generator–judge self-preference limitation.Scores also depend on reference requirements and sampled visual evidence; camera timestamp agreement does not independently verify hardware synchronization.Source: §B.3 · p. 17

B.5. WGO-Bench protocol

• WGO-Bench comparisons use single-view videos, episode instructions, tIoU ≥0.75 matching, and a public text-only caption rubric.Scoring copies snap predicted outer endpoints to reference edges, while native Flora runs use its production workflow without benchmark-specific event merging or reference-boundary tuning.Source: §B.5 · p. 18

Protocol

• The common pipeline, Refiner, and LeRobot share Gemini 3.5 Flash, while Pegasus and trained Flora retain their own backbones.Refiner and LeRobot retain their native prompts and visual plans, and systems use different native visual representations, prompt sequences, and compute.Source: §B.6 · pp. 18–19

Backbone sensitivity

• Refiner’s mean joint F1 rises from 27.2% with Gemini 3.5 Flash to 36.4% with Gemini 3.8 Flash, above trained Flora’s 34.5%.Flora retains the highest mean segmentation score, while Refiner leads caption accuracy and joint F1 in the Gemini 3.8 comparison; pipeline rankings depend on the generation backbone.Source: §B.6 · p. 19, Table 9 · p. 20

API substitution

• With Gemini 3.5 Flash, API substitution reaches 19.4% segmentation F1, 52.1% oracle accuracy, and 11.1% joint F1.Gemini 3.8 Flash substitution reaches 21.2%, 72.1%, and 16.4%, respectively; neither substitution matches the trained system.Source: §B.6 · p. 19
• The Gemini 3.8 reruns retain one source exclusion and include its 20 reference targets in the denominators.Unavailable captions receive no semantic match credit, although valid segmentation outputs remain eligible for temporal scoring.Source: §B.6 · p. 19

C. Reward Evaluation Protocol

• General-purpose baselines receive Flora-Rewarder’s task instructions and frames with the same prompts and answer formats, but reasoning compute is not equated across models.Specialized reward-model results retain their published protocols; RynnValue ranks 976 trajectories rather than 567, and the Robometer row reports its best variant.Source: C. Reward Evaluation Protocol · p. 21
• On RoboRewardBench, Flora-Rewarder predicts scores from min(64, max(2, floor(10T))) uniformly sampled frames in score-only mode, with greedy decoding and thinking disabled; MAE uses the 1–5 scale and is multiplied by 100.Qwen3.5-397B-A17B receives 32 frames, while other baselines receive up to 64. Kimi K3’s MAE covers 2,815 of the 2,831 episodes.Source: C. Reward Evaluation Protocol · p. 21
• RobometerBench ranks 567 trajectories from six OOD datasets using equally weighted dataset-level Kendall’s τa and measures value-order correlation on successful ID and OOD trajectories.Flora-Rewarder derives progress from expected dense-reward quality, with failure, suboptimal, and normal weights of 0, 0.5, and 1.Source: C. Reward Evaluation Protocol · p. 21
• DenseRewardBench compares labels at 1 Hz across 4,861 time steps from 40 held-out episodes, using 508 shared windows of 64 frames at 4 FPS.Predictions are stitched by Eq. (22), dropping the last 4 seconds of each non-final window; Flora-Rewarder decodes greedily with thinking disabled.Source: C. Reward Evaluation Protocol · p. 21
• RubricBench uses 40 teleoperated episodes from 10 tasks, with no episode or task name shared with the rubric training pool.Flora-Rewarder uses nine samples and returns their medoid; the reported error is expected over random draws of nine from 32 samples per episode.Source: C. Reward Evaluation Protocol · p. 21

D.1. Evaluation cohort and accounting

• The 50-episode quality cohort contains 20 human, 15 parallel-gripper, and 15 dexterous-hand episodes, totaling 4,483.54 seconds; its equal-weighted scores differ from full Table 3 evaluation.Figure 7 intervals use paired episode-bootstrap draws and operational scores, assigning unavailable contributions no credit; they exclude generation, judge, reference, and service uncertainty.Source: §D.1 · p. 22
• The quality comparison is not fully reference-version matched, and API costs are known-usage subtotals because 106 of 7,062 attempts remain unpriced.API usage is priced at public standard rates from October 4, 2026 per source-video hour, excluding training, reference construction, oracle captioning, and judging.Source: §D.1 · p. 22
• Requested reasoning settings are not compute matched across providers, and Pareto comparisons use displayed mean estimates rather than statistical dominance.Figure 2 compares Flora with maximum-effort Astra, Claude Opus 5, and Gemini 3.8 Flash; its dashed line is the empirical API-only Pareto frontier.Source: §D.1 · p. 22

D.2. Native deployment model

• The native cost model assumes a 9B segmenter, a 27B captioner, and a reference workload of 660 windows and 890 captions rather than measured RL-pair invocations.The eight-GB300 deployment model assumes data parallelism and batch size 16; its conditional rate is 1.23495 episodes/s, with 0.70804–1.74862 engineering scenarios.Source: §D.2 · p. 22
• $0.7224 per source-video hour is the Flora cost estimate, versus $88.4922 for maximum-effort Astra, a 122.49 ratio summarized as approximately 125×.The $0.5102–1.2600 Flora range is an engineering scenario, not a confidence interval; GPU pricing is assumed and storage and deployment overheads are excluded.Source: §D.2 · p. 22

D.3. Conditional throughput comparison

• API throughput estimates use fixed October 2026 quota scenarios, not verified entitlements or universal provider limits; Figure 7’s throughput is modeled, not measured, and points are not budget matched.Its Flora rate assumes 12 independent eight-GPU replicas, with vertical bars showing episode-bootstrap 95% intervals.Source: §D.3 · p. 23, Table 10 · p. 23, Figure 7 · p. 24

E. Additional Qualitative Evidence

• Flora captures nearly all reference intervals for a dense action where Astra matches only a few, but can fragment coarse events and match fewer references.On mutually observed dexterous-hand episodes recall is similar but Flora is less precise; the aggregate parallel-gripper ordering reverses on mutually observed episodes.Source: E. Additional Qualitative Evidence · p. 27, Figure 10 · p. 27
• Flora preserves wire insertion and battery adjustment details that Astra misidentifies, but also substitutes placement for reaching and napkin folding for dish wiping.On jointly judged human windows, Flora covers coordination requirements more often than Astra (58.3% versus 21.2%), while action coverage is similar; differing inputs and references limit causal attribution.Source: E. Additional Qualitative Evidence · p. 27, Figure 10 · p. 27
• Flora-Rewarder labels falling wings and missed grasps as failures where Astra calls them suboptimal, but also labels about 11% of normal steps as failures.Astra’s rationales describe the failures despite its less severe labels.Source: E. Additional Qualitative Evidence · p. 28, Figure 11 · p. 28
• Flora-Rewarder’s advantage is in partially completed tasks; on complete failures and successes, Astra is at least as accurate.Examples include a toppled bottle that remained on the table and a completed fold Astra scored 3 instead of 5.Source: E. Additional Qualitative Evidence · p. 28, Figure 11 · p. 28

F.1. Temporal reconstruction and visual context

• Overlapping segmenter windows emit central-core intervals with context guards; projection maps local timestamps to episode time, and stitching joins compatible fragments while preserving concurrent actions.Guard-only fragments do not create events, shifted windows refine unresolved boundaries, and unobserved endpoints retain observed extent without inventing a transition.Source: §F.1 · p. 29

F.2. Supervised fine-tuning

• The segmenter and captioner are initialized from Qwen3.5-9B and Qwen3.5-27B, respectively, and trained on timestamped manipulation annotations with token-level cross-entropy.Segmentation targets serialize intervals and continuation flags; captioning targets pair an action instruction with its interval and visual context.Source: §F.2 · p. 29

F.3. Temporal segmentation reinforcement learning

• GRPO trains the segmenter with verifiable reward based on one-to-one temporal matching, F1, precision, and boundary error.The reward clips predictions and references to the emission core, discards guard-only fragments, and gives both-empty sets reward one; checkpoint selection uses a separate validation set and guarded-group constraint.Source: §F.3 · p. 30

F.4. Captioner reinforcement learning

• The captioner’s reference-grounded reward scores intended action, object identity, manipulator coordination, relevant detail, coverage, and factual error.Reference instructions are unavailable to generation; validation weights embodiments equally, and segmenter and captioner checkpoints are selected independently.Source: §F.4 · p. 30

F.5. Pipeline schematics

• The pipeline schematics show window fragments projected and stitched into episode intervals before independent captioning with before, target, and after visual evidence.Synchronized camera mosaics preserve source-time alignment and temporal order across the context blocks.Source: Figure 12 · p. 31, Figure 13 · p. 31

G. Reward Modeling Implementation and Training

• Outcome, dense, and rubric interfaces respectively score episode outcomes, label temporal execution quality, and grade user-defined task steps against allowed values.Dense outputs include ordered quality labels and rationales; rubric grades are compared with human grades by L1 error.Source: G. Reward Modeling Implementation and… · p. 32

G.1. Video Input

• Outcome and dense prompts use synchronized multi-view mosaics, while rubric prompts receive separate camera videos; outcome and dense timestamps refer to absolute episode seconds.Mosaics use up to four views, with camera roles determining placement.Source: §G.1 · p. 32

G.2. Dense Reward

• Dense labels derive from human-annotated bimanual episodes, with success and normal mapped to normal and overlapping actions resolved by severity.Each 64-frame, 4-FPS clip yields 32 ordered two-frame groups; non-normal ranges receive visible-evidence rationales, and evaluation episodes are excluded from training and validation.Source: §G.2 · pp. 32–33

G.3. Outcome Reward

• Outcome supervision combines public, licensed, and internal recordings, retaining native five-point labels and mapping binary outcomes to 5 or 1.Binary success and failure map to 5 and 1; trajectory ranking uses the expected score from the decoded score-token distribution.Source: §G.3 · pp. 34–35

G.4. Rubric-Derived Reward

• Rubrics specify ordered task steps, maxima, and allowed score levels; human grades define targets, and evaluation episodes and task names are held out.Internal policy-evaluation rubrics contain 2–10 steps, while external-episode rubrics contain 1–4.Source: §G.4 · p. 35

G.5. Co-Training

• The released reward model uses SFT to co-train one Qwen3.5-27B model on shuffled outcome, dense, and rubric examples.Training updates the language model, visual merger, and visual encoder, using a lower learning rate for the visual encoder.Source: §G.5 · pp. 35–36

G.6. Interface Examples

• The examples show three reward interfaces, including rubric-based step grading with discrete allowed values for each step.The sample rubric concerns apple cutting.Source: Figure 14 · p. 36, §G.6 · p. 36, Table 11 · p. 36

H. Related Work

• Related systems cover robotics annotation, open-weight VLM annotators, and reward modeling; Flora targets accurate, consistent, low-cost annotation of robot and human data.The cited reward-modeling progression includes episode-level, frame-level, and process-reward approaches.Source: H. Related Work · p. 37