PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond End-Task Success: How to Audit Visual Experience Retrieval in Robotics

Eshika Pathak, Leela Krishna

arXiv:2609.26567v2cs.ROcs.CV

TL;DR

Selected-only success cannot show whether a retrieval rule chose well, because the score also reflects the quality of the experience library. The paper audits every experience in every query scene and finds that visual distance can predict pair success while failing to rank candidates within a scene; it recommends reporting selection distributions and best-fixed success.

  • Problem

    Selected-only evaluation conflates whether the retriever chose well with the quality of the stored library.

    Source: §I · p. 1

  • Method

    The audit executes every stored experience in every query scene, producing counterfactual outcomes for selected, unselected, random, and oracle choices.

    Source: §II · p. 2

  • Results

    At K=50, visual similarity ranks candidates within one scene no better than chance for four of five tested embeddings (AUROC 0.45–0.52).

    Source: §IX · p. 6

  • Takeaways & Limitations

    Ordinary studies can report the distribution of selected experiences and the success of the best single experience in hindsight.

    Source: §VIII · p. 5, §IX · p. 6

    The conclusions are limited to the evaluated scope and do not establish a general verdict beyond it.

    Source: §IX · p. 6

Abstract

from arXiv · show

Robots that store past experiences must select which one to reuse in a new scene. Most systems select by visual similarity, and most evaluations report only the success of the selected experience. That number does not show whether the selection was good: a rule can score well by repeatedly using one broadly transferable experience, or poorly because its preferred experience is weak. Since robots increasingly adapt by reuse rather than retraining, a score that describes the library rather than the rule misleads what the field builds next. We contribute an audit methodology: execute every stored experience in every query scene, over two manipulation tasks, three reuse mechanisms, and libraries of $K=3$, $10$, and $50$. Because every alternative's outcome is known, a score can be traced to per-scene selection or to library quality. The audited rules select by nearest-neighbor distance in five visual embeddings, from raw pixels to CLIP. (1) One fixed experience, chosen with hindsight, captures 30-58% of the gap between random selection and an oracle; per-scene selection competes for the remaining 0.07-0.15 in success rate. (2) At $K\ge10$, visual rules concentrate on one experience 1.5-3 times more than the oracle does, and their scores then follow that experience's quality. (3) Wherever a rule differs significantly from a shuffle that keeps its selection rates but pairs them with scenes at random, the rule is worse, for every learned image policy. (4) Visual distance predicts well whether a given pair will succeed (AUROC up to 0.96), yet ranks the candidates within one scene no better than chance for four of five embeddings at $K=50$ (AUROC 0.45-0.52). Exhaustive execution is usually infeasible, so the audit reduces to two cheap reports any study can give: the distribution of selected experiences, and the success of the best single experience in hindsight.

I. INTRODUCTION

The paper asks whether visual retrieval selects experiences that suit each query scene, since selected-only success conflates selection quality with library quality. It audits every alternative and finds that predicting pairwise transfer does not guarantee ranking candidates within a scene.

I. INTRODUCTION

  • Selected-only success conflates retrieval quality with library quality, because repeatedly choosing a strong or weak experience can inflate or depress the score.The reported score may describe the experience being selected rather than the quality of the rule.Source: §I · p. 1
  • The audit executes every stored experience in every query scene to measure the complete transfer matrix and separate appearance effects from layout effects.Appearance changes include lighting, color, texture, and camera pose, while scene layout remains independently varied.Source: §I · p. 1
  • Visual distance can predict whether an experience-scene pair succeeds yet rank candidates poorly within a scene, because most outcome variance is shared within each scene.A score can detect hard scenes without distinguishing which candidate is best for that scene.Source: §I · p. 1
  • The paper contributes an audit methodology rather than a new retrieval method or a general verdict on visual similarity.The audit measures whether a retriever chose well and can change conclusions drawn from ordinary evaluation.Source: §I · p. 1
  • The audit protocol, controls, and reporting diagnostics include the complete transfer matrix, library-composition comparisons, and selection distribution with a best-fixed reference.The controls include an oracle concentration reference, a best-fixed reference, frequency-matched shuffles, random sub-libraries, and quality-equalized libraries.Source: §I · pp. 1–2

II. AUDIT PROTOCOL

The audit builds a transfer matrix by executing each stored experience in each query scene, then evaluates visual and other selection rules against counterfactual outcomes. Its benchmark varies tasks, reuse mechanisms, scene appearance and layout, and library composition.

II. AUDIT PROTOCOL

  • Stack and Door scenes use 40 physical states across four layout levels and five appearance draws that preserve the simulator state.The setup uses a Panda arm under pose control, with object displacement, articulation, and obstacles varying layout.Source: §II · p. 2
  • Replay, Policy, and Policy+DR cover open-loop path retargeting and per-experience behavior-cloned image policies, with domain randomization affecting appearance and layout differently by task.Replay is camera-independent; on Stack, Policy+DR lowers appearance-caused failures from 26% to 2%, while on Door it also degrades layout performance.Source: §II · p. 2
  • 114,500 seeded executions plus 250 own-base replays provide five-seed success rates for every experience-scene pair, including selected, unselected, random, and oracle outcomes.The transfer matrix, rather than the execution count, is the unit of analysis.Source: §II · p. 2
  • The tested selection rules include nearest-neighbor retrieval in five visual embeddings, recency, structural scores, and a track-record reference that uses past outcome data.Visual embeddings range from raw 32 × 32 pixels to ImageNet ResNet-18, DINOv2, CLIP, and the Policy+DR visual encoder.Source: §II · p. 2
  • A frequency-matched shuffle preserves how often each experience is selected but randomly assigns those selections to scenes, so the discrimination gap measures added value from scene assignment.A positive gap means the real rule outperforms the mean shuffled rule; intervals are clustered by scene (n=40).Source: §II · p. 2
  • Predictions and analysis thresholds were recorded before data collection, and several predictions failed.The paper notes those failures where they change interpretation.Source: §II · p. 2

III. HOW MUCH CAN SELECTION ADD?

A hindsight-chosen best-fixed experience captures much of the available gain over random selection, leaving little room for per-scene selection. Outcome-informed track record is a reference for finding strong experiences, not a fair competitor to visual retrieval.

III. HOW MUCH CAN SELECTION ADD?

  • Best-fixed is the single highest-mean-success experience chosen after observing outcomes; it uses no per-scene information and is not deployable.It serves as a reference for how much selection could add.Source: §III · p. 2
  • Best-fixed captures 30–58% of the random-to-oracle gap in every stable setting (A = 0.30–0.58), and no tested rule is distinguishable from it.Intervals are wide, so this does not establish that the rules equal best-fixed.Source: §III · p. 2
  • The oracle exceeds best-fixed by only 0.07–0.15 in success rate across stable settings, and headroom does not grow with library size.At K=50, the headroom is 0.130.Source: §III · p. 2
  • Track record selects the best or second-best experience in every set and locks onto a strong one after about six scenes, but it uses outcome data unavailable to visual rules.At K=50, it trails best-fixed (A=0.26 versus 0.51), with an exploration cost from estimating 50 candidates using 40 scenes.Source: §III · pp. 2–3

IV. CONCENTRATION: THE SCORE FOLLOWS THE TOP

Visual rules often concentrate on one experience, and their scores can track that experience’s quality. Scene assignment adds mixed value: significant learned-policy cases are worse than frequency-matched shuffles, with appearance changes implicated in most cases.

IV. CONCENTRATION: THE SCORE FOLLOWS THE TOP

  • At K ≥10, visual rules are 1.5–3 times as concentrated as the oracle; at K=3, their excess concentration is small and often absent.The oracle’s top-pick share is 0.20 at K=10 and 0.10 at K=50, while visual rules reach 0.30–0.60 and 0.25–0.30, respectively.Source: §IV · p. 3
  • Across 91 cases, advantage signs match the top pick’s quality-margin signs in 69; raw pixels at K=10 are a counterexample.Their weak top pick is selected mainly on scenes where it works.Source: §IV · p. 3
  • In 42 of 65 cases, visual rules fall inside the frequency-matched shuffle envelope; this shows compatibility, not equivalence.The envelope is wide with 40 scenes, and its power to detect small gaps was not measured.Source: §IV · p. 3, Figure 1 · p. 3
  • Among significant cases, Replay has nine positive and two negative gaps, while all twelve learned-policy cases lie below the frequency-matched envelope.For those learned policies, random assignment with the same selection frequencies would have scored higher.Source: §IV · p. 3
  • In 9 of 12 significant learned-policy cases, the negative gap is larger on appearance-perturbed scenes than on layout-perturbed scenes.The three exceptions are fresh-scene Door Policy+DR cases, where smaller layout subsets have wide intervals.Source: §IV · p. 3

V. THE LIBRARY CHANGES WHAT THE SCORE MEANS

Retrieval scores depend on library composition: label-informed interventions reveal how experience quality can mask or expose scene discrimination. Local success also fails to establish transfer quality.

V. THE LIBRARY CHANGES WHAT THE SCORE MEANS

  • At K=10, equalizing experience quality makes four of five visual rules positive, while the policy encoder remains negative; equalized advantages are near zero at K=3.The same rules score 0.12–0.21 lower in unequal libraries; K=3 conclusions remain limited because A is unstable with a small denominator.Source: §V · pp. 3–4
  • Untuned CSLS and mutual proximity barely change results, with effects smaller than those of library equalization.Hubness corrections adjust similarity scores to reduce frequent-nearest-neighbor dominance.Source: §V · p. 4
  • Deleting the top pick flips two K=50 rules from negative to positive, locating the failure in pairing a concentrated rule with a weak experience.Deletion uses labels and is a diagnostic, not a retrieval method; re-ranking cannot fix this failure as a geometric artifact.Source: §V · p. 4
  • Among 50 Replay experiences, 49 succeed on their own base scene at least 0.8 (mean 0.92), yet own-base success does not predict cross-scene success (ρ=−0.04).Seven experiences are reliable at home but weak elsewhere, so acceptance on the collection scene does not measure transfer.Source: §V · p. 4

VI. PREDICTING TRANSFER IS NOT RANKING

Visual distance predicts whether an experience-scene pair will succeed, but ranks candidates within a scene much less reliably. The section attributes this gap to shared scene difficulty and argues that retrieval should be assessed by within-query ranking.

VI. PREDICTING TRANSFER IS NOT RANKING

  • For image policies, pooled AUROC reaches 0.85–0.96 under appearance perturbations and 0.70–0.91 under layout perturbations, but pooled prediction is not within-scene selection accuracy.The reported AUROC measures whether a pair succeeds, whereas selection ranks candidates for one scene.Source: §VI · p. 4
  • Within-query AUROC is lower than pooled AUROC in 62 of 65 eligible cases, with the gap widest at K=50.At K=50, pooled AUROC stays above 0.7 while within-query AUROC is near chance for every representation except ResNet.Source: Table IV · p. 4, §VI · p. 4
  • A hard scene lowers transfer for every candidate, so shared scene difficulty can predict outcomes without separating candidates within that scene.The section's design target is query-grouped ranking by transferability, not pooled similarity alone.Source: §VI · pp. 4–5
  • The scene-difficulty variance share is 0.71–0.81 for image policies, where all twelve significant gaps are negative; it is 0.42 and 0.45 in the two Stack Replay K=3 settings with the strongest positive results.The authors present this ordering as supporting evidence, not a confirmed relationship.Source: §VI · p. 5
  • ResNet is the only representation above chance in within-query AUROC at K=50 and the only rule positive at every Replay library size.CLIP and the policy encoder have within-query AUROC near 0.5 alongside K=50 advantages of −0.42 and −0.57, respectively.Source: §VI · p. 5

VII. LIBRARY SIZE CHANGES THE CONCLUSION

The policy encoder's discrimination gap changes sign across library sizes, while the authors caution that size trends and learned-policy results have limited scope.

VII. LIBRARY SIZE CHANGES THE CONCLUSION

  • The policy encoder's gap is −0.79 at K=3, +0.35 and +0.47 at K=10, and −0.28 at K=50; ResNet alone is positive at all three sizes.The predicted persistence of positive K=10 gaps at K=50 held for one rule of four.Source: §VII · p. 5
  • Nested-library size is confounded with recording quality because the expert rejection rate changes from 0.40 to 0.33 to 0.153 at K=3, 10, and 50, so the trend is not a scaling law.Excess concentration appears in 96–99% of 200 random K=10 compositions, but those sub-libraries share one pool and one query set.Source: §VII · p. 5
  • Two K=3 conclusions—that recency beat every visual rule and one encoder was negatively predictive in general—did not survive at K=10.All image-policy experiments use K=3, so those results are evidence about small libraries, not scaling.Source: §VII · p. 5

VIII. TWO REPORTS EVERY RETRIEVAL STUDY CAN GIVE

Because exhaustive execution is usually infeasible, the paper recommends two inexpensive reports that help distinguish per-scene selection skill from repeated use of a strong experience.

VIII. TWO REPORTS EVERY RETRIEVAL STUDY CAN GIVE

  • Report how often each experience is selected and, when exhaustive labels exist, compare its distribution with the oracle's rather than a uniform distribution.Heavy concentration warns that the score may describe a few experiences.Source: §VIII · p. 5
  • Report best-fixed success: the evaluation-scene success of always using the single best experience chosen in hindsight.This needs each item's mean success, not the full transfer matrix; on hardware, a subsample of scenes per item suffices for rough means.Source: §VIII · p. 5
  • In seven surveyed systems, the selection-distribution and best-fixed reports never appear together; two partial cases report retrieved-source composition instead.For systems retrieving sets of data, the analogous reports require translating the measures to retrieved items and a hindsight-chosen training subset.Source: §VIII · p. 5, Table V · p. 6
  • Future work should pair cross-scene quality estimation with query-conditional ranking and test both on independent libraries, larger policy libraries, more tasks, and hardware.Collection-time success is not an estimate of cross-scene quality.Source: §VIII · p. 5

IX. LIMITATIONS AND CONCLUSION

Within this benchmark, visual rules at K ≥10 concentrate on one experience more than the oracle, while changing library quality moves scores more than re-ranking does. The authors recommend reporting selection distributions and best-fixed success; the audit is limited to simulation and one benchmark.

IX. LIMITATIONS AND CONCLUSION

  • The audit covers one simulated benchmark; K=50 results use Stack under Replay, while learned-policy results use K=3.Forty scenes yield wide intervals; the frequency-matched envelope has no power analysis.Source: §IX · p. 6
  • At K ≥10, visual rules are 1.5–3 times as concentrated as the oracle, and changing the library moves scores more than re-ranking does.The paper describes the conclusion as methodological and within the scope of one benchmark.Source: §IX · p. 6
  • Ordinary studies can report the selection distribution and best-fixed success, adding frequency-matched and within-query analyses when counterfactual labels exist.The paper identifies these as low-cost reporting diagnostics.Source: §IX · p. 6
  • At K=50, visual similarity ranks candidates no better than chance for four of five embeddings (AUROC 0.45–0.52).Chance is 0.5.Source: §IX · p. 6

Related papers

Loading 2609.26567v2…