PaperlayerView exact revision on arXiv

Source-linked AI summary

From Association to Decision Value: Testing Paired Instruction Dependence for VLA Corpus Selection

Nurmukhambet Serikbayev, Abdul Basit, Muhammad Shafique

arXiv:2610.29646v1cs.AI

TL;DR

Selecting robot demonstration corpora requires ranking candidates by downstream policy performance, rather than relying only on association with average outcomes. The paper tests a same-state, goal-swap audit and evaluates its predictive and selection value prospectively; the unsigned switch statistic adds uncertain incremental proper-score value in this setting.

  • Problem

    A corpus's downstream value is usually known only after fine-tuning a VLA, so an upstream diagnostic needs to rank candidates well enough to select better-performing corpora.

    Source: §1 · pp. 1–2

  • Method

    The paper measures paired instruction dependence by changing the goal at a fixed state, then prospectively compares nested predictors with and without this audit statistic.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    +0.059 nats (95% corpus-bootstrap CI [−0.027, 0.133]) is the Qwen2.5-ID mean log-score gain from adding Ad to M0.

    Source: §5.1 · p. 6

  • Takeaways & Limitations

    The unsigned switch statistic mixes both-correct and both-wrong responses, while correctness-resolved alternatives show larger exploratory held-out gains under the tested predictor specification.

    Source: §7 · p. 9

    The audit requires counterfactual demonstrator access and is tested on one simulator, one tabular demonstrator family, short corpora, and a largely maturity-driven population; broader generalization remains untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

Choosing among robot demonstration corpora is a ranking problem: an upstream diagnostic must preserve which candidates produce better downstream policies, not merely correlate with average performance. We study paired instruction dependence, the rate at which a demonstrator changes its first decision when only the goal changes. Across 136 independently seeded quadruped-navigation corpora, predictions developed on 56 corpora are evaluated prospectively on 80 held-out corpora. The audit is reliable and associated with downstream paired correctness, and lowers mean absolute prediction error when added to a correctness-aware baseline, but the primary proper-log-score increment is +0.059 nats (95% corpus-bootstrap CI [-0.027, 0.133]) and the prespecified top-20% comparison does not establish an incremental selection gain. An exploratory response decomposition suggests a structural source of this predictive-ranking mismatch: under the tested predictor specification, separating both-correct and both-wrong responses yields larger held-out gains than their unsigned switch statistic. Seed-to-seed agreement indicates that the held-out endpoint retains recoverable rank signal. These results show that counterfactual response structure can be reliable and predictive while providing limited incremental decision value in this setting when compressed into an unsigned statistic, motivating diagnostics that preserve outcome-relevant structure and are validated against the decisions they are intended to support.

1 INTRODUCTION

Choosing demonstration corpora requires a diagnostic that ranks candidates by downstream policy quality, not merely one associated with average performance. This study tests whether same-state instruction-dependent response changes add predictive and selection value beyond marginal correctness.

1 INTRODUCTION

  • The practical question is whether an upstream measurement can rank corpora well enough to select those producing better policies without paying the cost of fine-tuning every candidate.The downstream value of a corpus is usually known only after a VLA is fine-tuned on it.Source: §1 · pp. 1–2
  • Paired instruction dependence, Ad, measures how often a demonstrator changes its first decision when only the goal changes at the same observation.The controlled intervention tests whether the response depends on the instruction, a distinction relevant when language-conditioned success can conceal shortcut policies.Source: §1 · p. 2
  • Ad is unsigned: both-correct and both-wrong response pairs can each contribute to switching, so the question is whether it adds value after marginal correctness is known.For binary left/right responses, Ad is their sum; HALT makes that identity approximate in these data.Source: §1 · p. 2
  • A signal can lower mean prediction error yet fail to preserve candidate ordering near a selection cutoff, motivating decision-focused evaluation.The paper evaluates this distinction through prespecified top-k selection as well as prediction criteria.Source: §1 · p. 2
  • The prospective test uses 136 seeded corpora, developing predictors on 56 and evaluating them on 80 held-out corpora, with M1 adding Ad to correctness-aware M0.The study separately evaluates reliability, association, proper probabilistic prediction, and top-k selection, with the corpus as the inference unit.Source: §1 · p. 2

2 RELATED WORK

The paper situates corpus selection among work on language-conditioned robot learning, counterfactual grounding, data valuation, and proxy selection. It tests an interpretable corpus-level diagnostic prospectively against prediction and selection criteria.

Vision–language–action learning

  • The work builds on language-conditioned robot learning from task-specific imitation through pretrained embodied models and heterogeneous robot corpora, including physical VLM-to-VLA transfer.It separates language-level navigation decisions from low-level legged control, as NaVILA does.Source: §2 · p. 2

Language grounding and counterfactual evaluation

  • The intervention holds the state fixed while changing the goal, but measures a demonstrator and predicts a separately trained learner; it distinguishes unsigned response change from correctness-resolved structure.The paper places this question alongside tests of instruction-insensitive shortcuts and language grounding.Source: §2 · pp. 2–3

Robot-data selection and valuation

  • Unlike end-to-end selectors such as DataMIL, this work tests whether an interpretable pretraining-corpus diagnostic adds value at downstream selection, conditioning on marginal demonstrator correctness.Its unit is the entire candidate corpus.Source: §2 · p. 3
  • The study addresses which fixed demonstration corpus should train a policy, amid evidence that robot policies vary with demonstration composition and scale.Prior approaches include marginal contribution, training dynamics, gradients, standardized retraining, and policy-aware datamodels.Source: §2 · pp. 2–3

Proxy selection and decision-aware evaluation

  • Prior proxy-selection work distinguishes absolute prediction quality from rank preservation and selection utility; this study extends that distinction to prospective robot-corpus selection.The evaluation includes prespecified decisions, a proper-score test, a seed-based resolution benchmark, and practical selector comparisons.Source: §2 · p. 3

3 METHOD

The method audits demonstrator responses by changing the goal at a fixed state, then compares nested predictors and their held-out ranking and selection behavior. Its design separates audit, training, and evaluation data and prespecifies the primary prediction test.

3.1 SAME-STATE GOAL-SWAP AUDIT

  • The same-state goal-swap audit queries a demonstrator twice on identical observation and action history, changing only between goals requiring opposite branches.The resulting Ad is an unsigned response statistic, not a correctness or downstream-value measure.Source: §3.1 · p. 3
  • The audit requires counterfactual access to a queryable demonstrator, so it does not apply to fixed corpora with only one instruction per state.The stated access model fits settings where a state can be replayed under alternate goals before committing to a large collection.Source: §3.1 · p. 3

3.2 WHAT DOES THE SWITCH RATE CONTAIN?

  • For binary left/right responses, Ad is the unsigned sum of both-correct and both-wrong cases, while marginal correctness carries their signed difference; HALT breaks the exact sum identity in these data.This decomposition makes the test about whether unsigned joint response structure improves prediction beyond marginal correctness and corpus composition.Source: §3.2 · pp. 3–4

3.3 NESTED PREDICTION TEST

  • Nested beta–binomial predictors model paired learner correctness, with M1 adding Ad while both models use the same demonstrator responses and control for corpus features and learner seed.The shared ridge penalty prevents the primary comparison from adding a hyperparameter search; response decomposition and separate tuning are exploratory.Source: §3.3 · p. 4

3.4 PREDICTION, RANKING, AND SELECTION

  • The primary estimand is equal-corpus improvement in held-out proper log score, with a corpus-cluster bootstrap interval and a prespecified rule requiring its lower bound above zero.For selection, each model ranks 80 held-out corpora by predicted mean correctness and selects K = 16; no held-out outcome enters fitting, tuning, or ranking.Source: §3.4 · pp. 4–5

3.5 RESOLUTION AND COMPARATOR ANALYSES

  • Cross-seed ranking serves as an empirical resolution benchmark, alongside demonstrator and proxy selector comparisons and exploratory tests excluding the two least mature checkpoints.These analyses do not alter the prespecified primary test.Source: §3.5 · p. 5

4 EXPERIMENTAL SETUP

The experiments use independently seeded quadruped-navigation corpora and separate audit, training, and evaluation observations. They test fine-tuned VLA adapters in simulation, controlled scene shift, and limited physical settings under a prespecified inference protocol.

4.1 CORPUS POPULATION

  • The study evaluates 136 independently seeded corpora, each built from a separately trained tabular Q-learning demonstrator; the 56 development and 80 held-out corpora share no demonstrator seeds or scene realizations.Corpora span checkpoints from 10 to 50,000 episodes, with 16 held-out corpora per level.Source: §4.1 · p. 5
  • Each corpus has disjoint audit, training, and in-distribution evaluation packets, and first-branch correctness ranges from .375 to .813 across corpora.All 6,528 attempted training rollouts passed structural-validity checks, so acceptance is constant across corpora and cannot explain their ranking.Source: §4.1 · p. 5
  • The audit costs a median 4.18 GPU-min per corpus, while a Qwen2.5 fit and in-distribution evaluation average 11.46 GPU-min for one seed, or 2.74× the audit cost.The primary setup fine-tunes rank-8 LoRA adapters, with evaluations in-distribution and under controlled scene shift.Source: §4.1 · p. 5, §4.2 · p. 5

4.2 VLA TRAINING AND EVALUATION

  • Controlled shift crosses two spawn/target geometries with two lighting settings while holding camera, warehouse, objects, prompt, action space, and controller fixed.The in-distribution endpoint uses 48 paired states per corpus and requires both goals to receive the required branch.Source: §4.2 · p. 5
  • Closed-loop simulation covers 16 held-out corpora selected across Ad ranks, while stationary physical-image evaluation tests 16 adapters and 48 physical runs describe two deployed adapters.The 48 runs are descriptive because terminal state was not externally adjudicated and parts of the recorded protocol departed from the initial design.Source: §4.2 · pp. 5–6, Figure 3 · p. 6

4.3 INFERENCE PROTOCOL

  • The inference protocol prespecifies predictors, held-out corpora, and the top-20% operating point, and uses corpus-level resampling to preserve nested and paired structure.The primary endpoint and selection rule remain unchanged after reveal; decomposition, resolution, proxy, calibration, and closed-loop correlation analyses are exploratory.Source: §4.3 · p. 6

5 RESULTS

Across held-out corpora, paired instruction dependence was reliable and associated with learner correctness, but its incremental predictive and selection value was limited. Exploratory analyses found larger gains from correctness-resolved responses, while seed agreement showed that the endpoint retained recoverable rank signal.

5.1 RELIABILITY, ASSOCIATION, AND INCREMENTAL VALUE DIVERGE

  • Across all 136 corpora, Ad had Spearman ρ = .467 with Qwen2.5 paired correctness, and audit reliability was .802.On the 80 held-out corpora, the association was ρ = .356 and reliability was .766.Source: §5.1 · p. 6
  • Adding Ad to M0 yielded a Qwen2.5-ID mean log-score gain of +.059 nats (95% corpus-bootstrap CI [−.027, .133]); the interval crosses zero.Figure 5 reports heterogeneous and uncertain incremental gains; every corpus-bootstrap 95% interval crosses zero, and every mean lies below the prespecified magnitude reference.Source: §5.1 · p. 6, Figure 5 · p. 7

5.2 UNSIGNED SWITCHING HIDES CORRECTNESS-RESOLVED STRUCTURE

  • +.104 nats for ad and +.110 for bd exceeded the +.059 gain for Ad in exploratory held-out tests.These correctness-resolved parameterizations selected correctness of .846 and .842, versus .811 for M1; the post-hoc comparison does not establish an information-theoretic advantage.Source: §5.2 · pp. 7–8

5.3 THE ENDPOINT CONTAINS RESOLVABLE RANK SIGNAL

  • Qwen2.5 learner seeds had rank agreement ρ = .567; selecting 16 corpora with one seed yielded .874 correctness when evaluated with the other.The ranking gap also persisted after excluding the 10- and 50-episode checkpoints: M0 had Spearman .349 and selected correctness .821, versus M1's .172 and .791.Source: §5.3 · p. 8

5.4 PRACTICAL BASELINES LOCATE THE AUDIT’S DECISION VALUE

  • Training-trajectory first-branch correctness improved proper score by +.156 nats but did not improve top-20% selection over M0.Qwen3.5 had the highest observed rank correlation among tested selectors (ρ = .715) and selected .866 Qwen2.5 correctness, but its measured fit averaged 13.41 GPU-min.Source: §5.4 · p. 8, Table 2 · p. 8

5.5 LOCAL RESPONSE AND NAVIGATION SUCCESS DIVERGE

  • Across 768 simulated episodes, first-command correctness was .829 and target reach was .655, a 17.4-point gap.On 48 physical Go2 runs, the first choice was correct in 47, but strict terminal success was not estimable under the recorded protocol.Source: §5.5 · p. 9

6 SCOPE AND LIMITATIONS

The study’s claims are specific to a controlled corpus population and a fixed learner comparison. Generalization beyond the tested setting remains untested, and exploratory alternatives do not establish why they differ.

6 SCOPE AND LIMITATIONS

  • The audit was tested on one simulator, one tabular demonstrator family, short corpora, and a largely maturity-driven population.Different demonstrator families, corpus-generation mechanisms, and larger real-world corpora remain untested.Source: §6 · p. 9
  • Correctness-resolved alternatives yielded larger exploratory gains under the tested predictor specification, but were not prespecified and do not identify the cause of the difference.This includes alternatives evaluated after separate development-set penalty tuning.Source: §6 · p. 9
  • Corpus value is defined relative to the fixed learner architecture and fine-tuning recipe used in each comparison.Qwen3.5 is a related backbone, not an independent architecture family.Source: §6 · p. 9

7 CONCLUSION

Paired instruction dependence offers a controlled case study of proxy compression: its unsigned switch rate is reliable and associated with learner behavior, yet mixes outcome-distinct responses. Correctness-resolved alternatives show larger exploratory held-out gains, while cross-seed agreement indicates recoverable corpus ordering.

7 CONCLUSION

  • The switch rate was substantially reliable and associated with learner behavior, but its unsigned form mixes both-correct and both-wrong responses.Correctness-resolved parameterizations yielded larger exploratory held-out gains under the tested predictor specification.Source: §7 · p. 9
  • Cross-seed agreement shows that corpus ordering remains recoverable.Source: §7 · p. 9
  • The findings motivate preserving outcome-relevant structure in counterfactual data diagnostics and validating them against the decisions they are intended to support.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

The supplement records study and analysis details, while the artifact provides materials including code, ordered split identifiers, and predictions.

REPRODUCIBILITY STATEMENT

ETHICS STATEMENT

The corpus study and closed-loop evaluation use simulation, while physical Go2 runs were supervised and safety-masked. These trials do not establish safety beyond recorded environments; deployment requires oversight, emergency stopping, and site-specific validation.

ETHICS STATEMENT

  • The corpus study and 768-episode closed-loop evaluation use simulation.Source: ETHICS STATEMENT · p. 9
  • The 48 physical Go2 runs used operator supervision and a depth-based safety mask, with no in-run intervention logged.These descriptive trials do not establish safety beyond the recorded environments.Source: ETHICS STATEMENT · p. 9
  • Deployment requires oversight, emergency stopping, and site-specific validation.Source: ETHICS STATEMENT · p. 9

AI USE STATEMENT

The paper frames paired instruction dependence as an information-theoretic measure of goal use under a balanced deterministic intervention, then evaluates it alongside correctness-aware prediction. Its prospective design separates 56 development corpora from 80 held-out corpora and tests whether the added audit feature improves prediction and selection.

AI USE STATEMENT

  • The authors report that they verified suggested code refinements and grammar changes and remain responsible for the manuscript content.Source: AI USE STATEMENT · p. 9

A INFORMATION-THEORETIC MOTIVATION

  • Under the balanced deterministic intervention, the empirical average of paired instruction dependence equals conditional mutual information and measures the ideal log-loss value of the goal.Outside that setting, switch rate and conditional mutual information need not coincide; the claim is about paired responses, not optimization or navigation success.Source: A INFORMATION-THEORETIC MOTIVATION · p. 13

B ENDPOINT AND PREDICTOR DETAILS

  • The local endpoint requires correct first decisions for both goals, distinguishing it from per-goal accuracy and from closed-loop target reach.This conjunction rejects constant-side policies that can attain deceptively high per-goal accuracy.Source: B ENDPOINT AND PREDICTOR DETAILS · p. 13
  • The primary M1 adds paired instruction dependence to M0’s per-goal correctness and corpus entropies, using the same ridge penalty to isolate the added feature.Test-corpus audit uncertainty is marginalized by resampling the 24 paired states; a sensitivity additionally resamples and refits on development audits.Source: B ENDPOINT AND PREDICTOR DETAILS · p. 14
  • +0.030 nats (95% corpus-bootstrap CI [−0.036, 0.090]) is the held-out log-score gain when checkpoint episode count is added to both predictors.The primary baseline excludes this simulator-specific assignment because it is not a measurement available for an arbitrary collected corpus.Source: B ENDPOINT AND PREDICTOR DETAILS · p. 14

C PROSPECTIVE CORPUS-POPULATION AND PRECISION DESIGN

D ROBUSTNESS OF THE PRIMARY CONCLUSION

The robustness and exploratory analyses qualify the primary conclusion: propagated log-score and selection effects remain uncertain, while response decomposition and seed comparisons reveal additional structure in held-out rankings. The closed-loop target-reach estimates are descriptive and do not establish a selector.

D ROBUSTNESS OF THE PRIMARY CONCLUSION

  • The reliability estimate treats the 24 audit states within a corpus as exchangeable and does not capture additional dependence or unmodeled state strata.Its variance decomposition uses estimated finite-state sampling variance Ad(1 − Ad)/(nA − 1); RA = 1 means that noise explains none of the observed between-corpus variance.Source: D ROBUSTNESS OF THE PRIMARY CONCLUSION · p. 15
  • The audit coefficient remains positive across perturbations, but propagated log-score and selection effects remain uncertain; the primary selection difference is −0.0266 and the exploratory resampling estimate is +0.0019.The two selection estimates use different procedures, with the latter resampling audit states, refitting M1, and reselecting corpora.Source: D ROBUSTNESS OF THE PRIMARY CONCLUSION · pp. 15–16

E EXPLORATORY STRUCTURAL AND DECISION DIAGNOSTICS

  • Exploratory tuning yields held-out gains of +.126 nats for ad and +.123 nats for bd, compared with +.046 for Ad under separate five-fold tuning.The ordering is post-hoc and does not isolate nonbinary responses from finite-sample or model-geometry effects; Figure 6 also reports a smaller, uncertain paired increment than marginal correctness contribution.Source: E EXPLORATORY STRUCTURAL AND DECISION… · p. 16, Table 8 · p. 16, Figure 6 · p. 17
  • Seed-to-seed correctness has Spearman ρ = .567, while after excluding 10- and 50-episode corpora M0 ranks better than M1 (ρ = .349 vs .172).The authors conclude that seed variation limits empirical resolution but does not account for the full M1 ranking deficit.Source: E EXPLORATORY STRUCTURAL AND DECISION… · p. 16
  • Across 16 rank-stratified closed-loop corpora, correlations with target reach are .368 for Ad, .361 for M0, and .423 for M1, but the sample does not establish a target-reach selector.Both predictors are described as conservative in held-out calibration, and a few severe log-score misses do not establish global undercoverage.Source: E EXPLORATORY STRUCTURAL AND DECISION… · p. 16, Table 9 · p. 17

F.1 DEVELOPMENT INTERFACE CHECKS

  • The interface check advanced Qwen3.5, which reached 1.00/.63 left/right correctness, while Gemma failed the .60 per-side criterion and SmolVLM2 was not advanced.SmolVLM2 achieved 31/432 paired decisions (.0718), versus Qwen2.5’s .6933 in the same development screen; these checks are not evidence for the primary predictive claim.Source: §F.1 · p. 17

G EMBODIED AND PHYSICAL RESULTS

In closed-loop evaluation, correct first commands did not always lead to task success. Physical-image testing also showed a right-goal deficit that was already present in matched simulation and was amplified by physical images.

G.1 PHYSICAL CLOSED-LOOP DEPLOYMENT

  • No strict success was verified in 48 physical runs because external target-zone measurement and independent adjudication were absent; this is not 48 observed failures.Seven strict failures were known and 41 runs were indeterminate; the 47/48 hardware first-choice rate is not directly comparable with the 82.9% simulation-subset mean.Source: §G.1 · p. 18, Figure 10 · p. 21

G EMBODIED AND PHYSICAL RESULTS

  • Controlled-shift performance remained substantial in every geometry/appearance cell, while audit-size prefixes did not improve selection monotonically.Source: Figure 7 · p. 19, Table 13 · p. 19

G.2 PHYSICAL-GO2 IMAGE STRATA

  • Matched simulation accuracies were .973/.677 for Qwen2.5 and .999/.749 for Qwen3.5; physical images amplified the existing right-goal deficit.This comparison is descriptive for the rank-stratified 16-adapter subset, not an estimate of prevalence across all 80 test corpora.Source: §G.2 · p. 19, Figure 12 · p. 22

H MEASURED COST

  • The paired query set costs 48 demonstrator queries and a median 4.18 GPU-min per corpus; pairing already-collected responses adds zero marginal queries and negligible arithmetic.The zero-marginal-cost condition depends on having correctness-labeled responses already; otherwise the full acquisition cost applies.Source: H MEASURED COST · p. 19, Table 16 · p. 22

I REPRODUCIBILITY DETAILS

  • The confirmatory test preserved corpus definitions, seeds, ordered partitions, and disjoint audit, training, and evaluation observations, with predictions and rules prespecified before test outcomes.Exploratory and descriptive analyses were separately labeled and did not alter the primary test.Source: I REPRODUCIBILITY DETAILS · p. 19

G EMBODIED AND PHYSICAL RESULTS

  • 637/768 first commands were correct versus 503/768 target-reaching successes, with 134 failures after correct commands and 131 after wrong commands.The paired corpus-bootstrap 95% interval for the first-command minus task-success gap was [10.4, 24.9] percentage points.Source: G EMBODIED AND PHYSICAL RESULTS · pp. 17–18, Figure 8 · p. 20

Related papers

Loading 2610.29646v1…