PaperlayerView exact revision on arXiv

Source-linked AI summary

VisCurate: Evaluating Behavioral Equivalence of Visual Skills

Md Farhan Ishraq, Md Farhan Ishmam, Reaz Hassan Joarder, Md Hasanul Kabir

arXiv:2610.30405v1cs.CV

TL;DR

Growing visual-skill libraries accumulate duplicates and defects, while names, descriptions, and code do not reliably reveal how skills transform images. VisCurate compares executed outputs on shared probes and maps behavioral relations to safe curation actions; its verifier reports zero false merges while recovering 51.2% of truly mergeable pairs.

  • Problem

    Names, descriptions, and source code can misrepresent whether visual skills behave alike, making library curation an equivalence problem.

    Source: §1 · p. 1

  • Method

    VisCurate runs candidate skill pairs on shared probes, classifies their output-grounded behavioral relations, and uses those relations to gate library edits.

    Source: §1 · p. 1

  • Results

    The verifier recovered 51.2% of truly mergeable pairs and made zero false merges, while text baselines accepted five of six possible false merges on hard negatives.

    Source: §5.2 · p. 6, §5.1 · p. 6

  • Takeaways & Limitations

    VisCurate provides an output-grounded verifier and benchmark for curating visual skills by behavior rather than description.

    Source: §7 · p. 8

    Output comparison detects domain-scoped bugs only when probes cover the affected domain, and cannot flag misleading descriptions or unused but correct skills.

    Source: §6 · p. 8

Abstract

from arXiv · show

Modern vision-based agents have shifted toward skill-based problem-solving, where a skill is an executable image-to-image operation, e.g., cropping, thresholding, or edge detection. As these skill libraries grow, they accumulate duplicated, mislabeled, and defective skills. The signals normally used to curate them (a skill's name, description, or source code) say little about how the skill visually changes the image. We present VisCurate, a framework and benchmark that curates visual skill libraries from observed execution rather than from text. VisCurate runs each candidate skill pair on a shared probe battery and assigns one of six behavioral relations, each tied to a library edit that is safe to make. The benchmark injects seven defect types into a clean library at controlled rates, producing 300 reproducible instances with ground-truth relations and repairs. While text baselines accept up to five of six false merges on hard negatives, our verifier accepts none and recovers 51.2% of the pairs that are truly mergeable. On two real, uncurated libraries, VisCurate finds that 92% of the skills a package shares with its downstream copy are redundant, while execution confirms only 10 of 17 published equivalence claims. The curation problem also remains open, as the strongest of the nine automated curators recovers only about half of the repairs. We emphasize that visual skill libraries should be curated by behavior, not by description.

1. Introduction

Visual-skill libraries accumulate duplicate, misleading, obsolete, and faulty skills, while names, descriptions, and code can misrepresent how skills change images. VisCurate instead evaluates behavior through shared probes and maps verified relations to library edits.

1. Introduction

  • A visual skill is a deterministic function that transforms an input image into an output image.The paper describes implementations using standard image-processing libraries including OpenCV, PIL, and NumPy.Source: §1 · p. 1
  • Text and code can mislead curation: similar descriptions may hide different outputs, while different implementations may behave identically.The paper notes 137,470 analyzed skills corresponding to approximately 56,043 distinct concepts [41].Source: §1 · p. 1
  • VisCurate executes candidate skill pairs on shared probes and assigns behavioral relations that correspond to safe library actions.Its output-grounded verifier and curation agent propose and validate library edits in a repeating pipeline.Source: §1 · p. 1, Figure 1 · p. 2
  • The benchmark injects seven defect types into a clean reference library at controlled rates, yielding 300 reproducible instances with ground-truth relations and repairs.The defects include behavioral faults, metadata misleads, duplicates, subsumptions, schema drift, domain-scoped bugs, and dead or identity skills.Source: §1 · p. 2
  • The framework is evaluated against text, embedding, and source-code baselines, against ground-truth graphs, and on two real uncurated libraries.The studies also measure scalability, curation-agent repair performance, and out-of-distribution certificate flips.Source: §1 · p. 2

2. Related Work

Prior visual-agent libraries are often fixed or append-only, while existing methods manage skill collections through text, code, or downstream utility. VisCurate compares visual operations by their executed image outputs and maps verified relations to library edits.

2.1. Skill Libraries for Visual Agents

  • Visual-agent work composes or retrieves executable skills, but the cited systems treat libraries as fixed or append-only.Other approaches find or manage skills through text, code, or downstream utility, but cannot determine from those signals whether visual skills apply the same transformation.Source: §2.1 · p. 2

2.2. Program Equivalence and Library Learning

  • Symbolic library-learning methods such as DreamCoder, LILO [9], and ReGAL [29] address equivalence where program structure or exact return values can reveal it.ReGAL is described as the closest method because it refactors programs while checking output preservation.Source: §2.2 · p. 2
  • VisCurate compares existing visual operations by image outputs, where similar code or descriptions can hide different behavior, and turns verified relations into concrete library edits.Source: §2.2 · p. 3

3. Methodology

VisCurate treats skills as deterministic image transformations and library curation as an agent acting under hidden defects, with edits gated by an output-grounded verifier. It compares behavior across shared probes and parameter settings, using a taxonomy and canonicalization safeguards to guide safe actions.

3.1. Problem Formulation

  • A visual skill is a named and described deterministic image transformation with a typed parameter schema; seeded-stochastic skills become deterministic when conditioned on their seed.The agent operates on a library whose injected defects are hidden, and structural edits take effect only when the verifier certifies a permitting relation.Source: §3.1 · p. 3
  • The curation environment is a partially observable process: the agent proposes library actions from observations, while the environment hides defects and applies edits only after verifier approval.Its episodes are scored against ideal repair actions derived from the corruption log, which the agent does not observe.Source: §3.1 · p. 3

3.2. The Equivalence Taxonomy

  • The taxonomy assigns each skill pair one of seven output-grounded relations, with a hierarchical stop-at-first classifier and complementary relations treated as advisory.Exact and perceptual relations permit merging; subsumption is directional, semantic-preserving permits parameterization, and uncertain pairs go to human review rather than an unwarranted merge.Source: §3.2 · pp. 3–4
  • Parameterized skills are compared over a matched parameter grid using the worst-case distance, which can expose divergences hidden at some settings.For example, Gaussian and box blurs can agree at small kernels but diverge at larger ones.Source: §3.2 · p. 4

3.3. Behavioral Signatures and Canonicalization

  • Behavioral signatures compare canonicalized outputs across a probe battery spanning images, channel layouts, bit depths, and degenerate cases.Canonicalization removes format and encoding noise; a shape check prevents merging outputs with different spatial extents, while seeded-stochastic, precision-sensitive, and platform-sensitive skills receive explicit handling.Source: §3.3 · p. 4

4. Experimental Setup

The experimental setup builds a known-relation base library, corrupts it reproducibly, and evaluates both curation agents and the verifier under controlled conditions. The verifier benchmark compares text, embedding, code, and output-grounded methods using safety-centered metrics.

4.1. Base Library

  • The 100-skill base library covers four image-processing families and has a designed relation graph of 944 ordered pairs, with unlisted pairs treated as distinct.The graph includes directional subsumptions, complementary pairs, and hard negatives such as Gaussian versus box blur and alternative resize methods.Source: §4.1 · p. 5

4.2. Corruption

  • Seven replayable defect types are injected into the base library across 300 reproducible instances, varying corruption rate, composition, seed, and single versus mixed mode.Each instance reconstructs the corrupted library, relation graph, and ideal repair action from a deterministic corruption plan.Source: §4.2 · p. 5

4.3. Curation Protocol

  • Curation agents see redacted skill summaries and may propose up to 200 actions, but edits require verifier certification and rejected actions still consume budget.The 2,700 episodes use the deterministic verifier without learned backends, allowing only EXACT, SUBSUMPTION, and DISTINCT to support edits.Source: §4.3 · p. 5

4.4. Equivalence Benchmark

  • The equivalence benchmark isolates verifier decisions on the 100-skill library, 944 designed pairs, and 177 probes, comparing eight text and code baselines at matched operating points.It prioritizes false-merge safety on truly distinct pairs, including a separate hard-negative slice; merge recall is measured on corrupted graphs because the clean graph has no true duplicates.Source: §4.4 · p. 5

5. Experiments and Results

Across designed and real-library tests, output-grounded verification avoided false merges while recovering only some mergeable pairs; automated curators likewise repaired only about half the faults. The experiments also expose trade-offs in relation coverage, model performance, and computational cost.

5.1. Text and Code do not predict Visual Behavior

  • Name matching and sentence embeddings each accepted five of six hard-negative false merges, while the verifier accepted none; source similarity can err in both directions on wrappers.On one corrupted instance, 25.7% of cross-original clone pairs were byte-identical despite wrapping different skills.Source: §5.1 · p. 6, Table 2 · p. 6

5.2. Redundancy recovered by the Verifier

  • 51.2% of truly mergeable pairs were recovered at precision 1.000, with no false merges across 300 corruption instances.The verifier’s recall stayed at zero false merges at every tested corruption rate.Source: Table 3 · p. 6, §5.2 · p. 6

5.1. Text and Code do not predict Visual Behavior

  • The verifier recovered SUBSUMPTION with F1 0.750, but SEMANTIC-PRESERVING precision was 0.037; the latter misses did not permit merging.Unlike the other methods, the verifier resolves the full behavioral relation taxonomy.Source: §5.1 · p. 6

5.3. Scaling Hard-Negatives

  • 0/69 engineered hard negatives were falsely merged, extending the verifier’s clean result beyond the six-pair initial slice.The six-pair slice alone could bound the false-merge rate only weakly.Source: §5.3 · p. 6, §5.1 · p. 6

5.4. Performance on Real Libraries

  • 23 of 25 shared pilgram filters were byte-identical on all 177 probes and certified EXACT, while execution certified only 10 of 17 documented cross-library equivalences as mergeable.The remaining two shared filters had diverged; the cross-library figure was 10 of 14 among fully alignable pairs.Source: §5.4 · p. 7

5.5. Performance of LLM Agent Curation

  • GPT-5.5 led the nine curators on recall (0.531), mean F1 (0.587), and Intrinsic Score (0.496), while attempting the fewest actions per episode.Qwen 3.5 27B was the only open model near it; Gemma 4 12B had precision 0.777 but repaired only a fifth of faults.Source: §5.5 · p. 8

5.6. Cost and Scalability

  • At k=10, screening retained 1.000 recall of mergeable pairs while discarding 95.1% of all pairs; candidate growth fit |C| = 2.38 n^1.13 against exhaustive pairing’s exponent 2.The fitted candidate-count curve had R2 = 0.998.Source: §5.6 · p. 8
  • Feature-only caching uses 5.4 GB at 10^4 skills versus 2.1 TB for raw output sets, a 395× saving; GPT-5.5 used 9,202 tokens per correct repair.A pair took 401 ms at the median, while the p95 of 64.5 s came almost entirely from subsumption searches.Source: §5.6 · p. 8

6. Discussion and Limitations

VisCurate separates behavioral comparison from the decision to edit a library, and only certified relations permit edits. Its output-based view has important boundaries: probe coverage, non-behavioral attributes, and malicious payloads remain concerns.

6. Discussion and Limitations

  • The verifier certifies whether skills behave alike, while the agent decides whether the library should change; edits require a supporting certified relation.The verifier made zero false merges in the evaluated cases, but its caution missed about half of mergeable pairs.Source: §6 · p. 8
  • Output comparison detects a domain-scoped bug only if probes cover that domain, and it cannot flag misleading descriptions or unused but correct skills.Skills with identical outputs may still differ in compute cost, dependency weight, and maintainability.Source: §6 · p. 8
  • A malicious skill can match a benign skill on every probe while carrying a harmful payload.The paper assigns side-effect containment to the sandbox, which confines execution and blocks unauditable code.Source: §6 · p. 8

7. Conclusion

VisCurate uses executed outputs to certify behavioral relations and translate them into safe library edits. Its verifier avoids false merges in the evaluated settings, but recovers only about half of mergeable pairs, and automated curation remains incomplete.

7. Conclusion

  • VisCurate provides an output-grounded framework and benchmark for curating executable visual skill libraries.Its verifier certifies behavioral relations, and a taxonomy maps those relations to safe edits.Source: §7 · p. 8
  • The verifier made zero false merges, while surface baselines produced up to five of six false merges and the strongest curator repaired only about half the faults.The paper concludes that visual skills should be curated by behavior rather than description.Source: §7 · p. 8
  • The authors identify improving curation-agent performance on visual skill libraries as a direction opened by this work.Source: §7 · p. 8

Supplementary Material

The supplementary table records a calibrated verifier operating point that was frozen before evaluation and reused in every study.

Supplementary Material

  • The output-grounded verifier’s calibrated operating point was frozen before evaluation and reused unchanged in every study.Source: Table A.1 · p. 11

A. Additional Methodology Details

This section details how VisCurate calibrates its verifier, screens candidate pairs, and gates curation edits, alongside the evaluation setup and methodological boundaries. It also defines the metrics used to assess curation agents and the baseline ladder.

A.1. Threshold Calibration

  • Thresholds are calibrated on 713 labeled pairs from eight operator families, maximizing precision on DISTINCT under an equivalence-recall floor, then frozen for evaluation.The calibration split is cluster-disjoint from evaluation data; 100 held-out pairs from further disjoint families check for leakage.Source: §A.1 · p. 11

A.2. Candidate Generation

  • Output-derived fingerprints cheaply screen pairs, while same-family pairs and engineered hard negatives are always retained for full hierarchical classification.Because fingerprints use output features rather than text, behaviorally identical skills with unrelated descriptions can still become candidates.Source: §A.2 · p. 11

A.3. Curation Under Output Checks

  • Structural edits require a verifier-certified relation and a fixed policy: EXACT and PERCEPTUAL permit merging, while UNCERTAIN routes to human review.SUBSUMPTION permits parameterization only in the direction that folds the specialization into the generalizer; unsupported edits are rejected.Source: §A.3 · p. 11

B.1. Inference and Serving

  • Curation agents are frozen language models; open-weight models run locally on A800 GPUs, while hosted models use provider APIs.The inference setup includes a 2,048-token reply cap for hosted curators and a shared server for up to 12 curation clients.Source: §B.1 · p. 11

B.2. Curation-Agent Metrics

  • Curation-agent metrics include mean precision, recall, and F1, alongside micro F1, action cost, compression, and predicted repairs.Intrinsic Score discounts mean F1 for budget spent on rejected, blocked, or invalid actions, and is zero when no non-END action is attempted.Source: §B.2 · p. 12

C.1. Baseline Ladder

  • The eight-rung baseline ladder ranges from name and text similarity through code and AST comparisons to LLM judges; Figure A.1 reports the verifier’s per-relation precision, recall, and F1 on G0.EXACT and PERCEPTUAL are omitted because they have no support in the clean reference graph; the verifier uses LPIPS, DINO, and CLIP backends.Source: §C.1 · p. 12, Figure A.1 · p. 13
  • On G0, merge false-discovery rate is 1.000 for any rung that merges, so the main text instead reports false-positive rates on DISTINCT pairs and hard-negative counts.A 0/6 hard-negative result bounds the false-merge rate only at 39.3% (one-sided 95%).Source: §C.1 · p. 12
  • Wrapper corruptions make code-similarity rungs unreliable on corrupted graphs, and the oracle currently enumerates only built-in skills, requiring an extension to score third-party implementations.The differential oracle and verifier execute the same skills, so their comparison is labeled approximate-versus-exhaustive rather than independently validated.Source: §C.1 · pp. 12–13

C.2. Verifier Self-Validation

Verifier self-validation finds zero false merges across corruption rates while recovering about half of truly mergeable pairs. The section also expands hard-negative checks and reports real-library results with an explicit oracle and adjudication scope.

C.2. Verifier Self-Validation

  • The repaired answer key materializes relations inherited through EXACT clones but excludes threshold-sensitive relations inherited through PERCEPTUAL clones.Across the 300 instances, 1,546 incidental relations were unrecorded; 740 of 14,506 labeled pairs were excluded rather than guessed.Source: §C.2 · p. 13
  • On the full Gρ replay (n=300 instances), false merges were zero at every one of ten corruption rates, and recall showed no monotone trend.The instance-clustered bootstrap interval for overall merge recall was [0.497, 0.528].Source: Table A.2 · p. 13, §C.2 · p. 13
  • The full 300-instance grid scores 13,658 pairs and yields merge recall 0.512, narrowing the pilot interval roughly threefold; UNCERTAIN appears twice rather than never.At scale, 52 EXACT pairs are missed: 32 are classified PERCEPTUAL, 14 SUBSUMPTION, and 6 SEMANTIC-PRESERVING.Source: §C.2 · pp. 13–14

C.3. Hard-Negative Expansion and Adjudication

  • On 69 engineered hard negatives, the verifier makes zero false merges, with an exact 95% confidence interval of [0.000, 0.052] and a one-sided upper bound of 4.25%.On the six original designed pairs, it returns DISTINCT for one and SEMANTIC-PRESERVING for five; neither relation permits merging.Source: §C.3 · p. 14
  • Twenty-six hard-negative pairs receive non-mergeable, non-DISTINCT labels, so blinded human adjudication is provided rather than replacing the disputed labels with automated judgments.The zero-false-merge result is unchanged whether the bound is measured against designed labels or adjudicators’ possible readings.Source: §C.3 · p. 14

C.4. Real-Library Studies

  • For the two divergent package filters, a random-probe smoke test flagged the same pair later found by the 177-probe run.The measured distances were LPIPS 0.2464 with DINO p90 0.0008 for perpetua, and LPIPS 0.0472 with DINO p90 0.0250 for aden.Source: §C.4 · p. 14
  • Of 17 pairs documented as directly equivalent in Corpus A, execution certifies 6 EXACT, 4 PERCEPTUAL, 3 COMPLEMENTARY, and 4 DISTINCT.Nineteen pairs allow an alternative defensible parameter alignment; divergences visible under only one alignment are treated as artifacts.Source: §C.4 · p. 14
  • A differential oracle labels all 182 Corpus A pairs using dense parameter sweeps and representation-derived tolerances, with 75 EQUIVALENT and 107 DISTINCT labels.It is independent ground truth for text and code baselines, but an exhaustive-versus-approximate check for the verifier because both execute the same skills.Source: §C.4 · p. 14

D. Curation-Agent Supplementary Results

These supplementary results examine agent behavior and cost, extend the verifier to neural and stochastic skills, and test whether accepted merges remain stable on out-of-distribution probes. They also report a hard-tail curator result and probe-screening trade-offs.

D. Curation-Agent Supplementary Results

E. Cost and Scalability Details

  • The cost analysis leaves token costs unreported for eight self-hosted main-table cells and treats cost per repair as undefined when a model makes zero correct repairs.For the instrumented verifier, the signature-cache hit rate is 78.2% (184,824/236,305), with median per-pair latency of 401 ms and p95 of 64.5 s.Source: E. Cost and Scalability Details · p. 15

F. Extension Studies

  • VisCurate’s extension studies cover neural and stochastic skills, out-of-distribution probe stability, and probe-battery ablation beyond its deterministic classical-operation core.The paper notes that modern agent ecosystems also use neural and stochastic tools.Source: F. Extension Studies · p. 15

F.1. Neural and Stochastic Skills

  • The neural-skill comparison uses per-probe permutation-null testing and explicitly does not transfer the deterministic path’s 0.99 precision calibration.Depth outputs receive per-image affine alignment; the study reports counts with exact bounds.Source: §F.1 · p. 15
  • For neural skills, the two depth models are distinguishable on all 40 probes, while SAM 2 large and small separate on 13 of 38 synthetic probes but not on 0 of 10 object-bearing probes.The default battery is not a valid segmentation domain: 137 of 177 probes are synthetic, and the instance-versus-semantic negative collapses on object-free inputs; non-rejection is not equivalence.Source: §F.1 · p. 15

F.2. Out-of-Distribution Certificate Flips

  • Exactly 1 of 74 real-library merges flips on out-of-distribution probes, a rate of 0.0135 (exact 95% CI [0.0003, 0.0730]).The flipped alb.gaussian blur versus cv2.GaussianBlur pair changes from PERCEPTUAL to SEMANTIC-PRESERVING; the study tests refutation only, not reverse flips.Source: §F.2 · pp. 16–17
  • At k=10, merge recall saturates while 95.1% of pairs are discarded; candidate screening grows near-linearly, and its proposed-pair share drops from 66.7% at n=10 to 8.2% at n=100.The screening curves are insensitive to radius from 0.05 to 2.0; the candidate-set fit is |C| = 2.38 n^1.13 (R2 = 0.998).Source: Figure A.3 · p. 17, Figure A.4 · p. 17

F.3. Probe-Battery Ablation

The ablation finds that probe diversity, rather than battery size, protects merge safety. The repository-audit protocol requires strong isolation for untrusted code and leaves such audits to deployments with that isolation.

F.3. Probe-Battery Ablation

  • Designed-relation accuracy stays flat across battery sizes, with overlapping intervals of roughly 0.33 to 0.58 across configurations at n=24.The study tests 21 configurations across battery sizes and selection strategies, with nested, data-independent subsets.Source: §F.3 · p. 17
  • Single-class “first N” selection produces a hard-negative false merge persisting to |B|=64, while stratified and random selection do not.The battery is ordered by probe class, so selecting the first probes can omit diversity.Source: §F.3 · pp. 17–18, Table A.8 · p. 18

G. Auditing Untrusted Repositories

  • Auditing untrusted skill packs requires a disposable isolated VM or container with no network, a read-only root, a non-root user, and no host credentials or volume mounts.Each pack runs in its own environment, and the VM is destroyed after the audit; the paper leaves this audit to deployments with the required isolation.Source: G. Auditing Untrusted Repositories · p. 18

Related papers

Loading 2610.30405v1…