PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Skill Discovery for Agentic Data Analysis

Zhisong Qiu, Kangqi Song, Shengwei Tang, Shuofei Qiao, Lei Liang, Huajun Chen, Shumin Deng

arXiv:2606.06416v1cs.AIcs.CLcs.LGcs.MA

TL;DR

Discovering reusable data-analysis skills from unlabeled exploration is difficult because analytical formats require different quality signals. DataCOPE uses verifier-guided contrastive skill discovery and improves mean scores by 9.71% on report-style tasks and 32.30% on reasoning-style tasks across four matched base models.

  • Problem

    Discovering generalizable data-analysis skills from unlabeled exploration is difficult because analytical formats require different quality signals for comparing trajectories.

  • Method

    DataCOPE iteratively coordinates trajectory generation, unsupervised verification, and contrastive distillation to discover reusable analytical procedures without ground-truth answers or success labels.

  • Results

    9.71% and 32.30%: DataCOPE improves mean scores on report-style and reasoning-style tasks, respectively, across four matched base models.

  • Takeaways & Limitations

    The discovered skills improve held-out generalization and transfer across different models.

Abstract

from arXiv · show

Inference-time skill augmentation provides a lightweight way to improve data-analytic agents by injecting reusable procedural knowledge without updating model parameters. However, discovering effective skills for data analysis remains challenging, as reliable supervision is expensive and success criteria vary across analytical formats. This raises the key question of how to discover reusable data-analysis skills from unlabeled exploration alone. We propose DataCOPE, an unsupervised verifier-guided skill discovery framework for data-analytic agents. DataCOPE derives verifier signals from the exploration trajectories and uses them to characterize relative quality or aggreement among trajectories. It iteratively coordinates a Data-Analytic Agent for trajectory generation, an Unsupervised Verifier for signal extraction, and a Skill Manager for contrastive skill distillation. For report-style analysis, we instantiate the verifier as an Adaptive Checklist Verifier that derives task-specific criteria, scores reports by verifiable coverage, and iteratively refines the checklist. For reasoning-style analysis, we instantiate it as an Answer Agreement Verifier that groups trajectories by answer agreement and uses self-consistency as an auxiliary signal. We evaluate DataCOPE on report-style analysis from Deep Data Research and reasoning-style analysis from DABStep. Across both settings, DataCOPE consistently improves held-out performance over baselines. Averaged across four model settings, DataCOPE improves the mean score by 9.71% and 32.30% on report-style and reasoning-style tasks respectively.

I. INTRODUCTION

DataCOPE addresses the difficulty of discovering reusable data-analysis skills without costly supervision by deriving verifier signals from unlabeled exploration trajectories. It supports both report-style and reasoning-style analysis and improves held-out performance across evaluated settings.

  • Motivation: Data-analysis skill discovery is difficult because reliable quality signals are costly or unavailable, while valid criteria differ between reasoning and open-ended analytical tasks.Reasoning tasks emphasize consistency with an expected answer, whereas open-ended reports emphasize completeness, evidence-supported claims, and analytical insight.
  • DataCOPE: DataCOPE performs unsupervised verifier-guided skill discovery by extracting relative quality or agreement signals from an agent’s own exploration trajectories.The framework uses these signals as contrastive evidence for skill discovery rather than treating them as direct correctness certificates.
  • Evaluation: 9.71% and 32.30% are the mean-score improvements DataCOPE achieves on report-style and reasoning-style tasks, respectively, averaged across four matched base models.The evaluation uses report-style analysis from Deep Data Research and reasoning-style analysis from DABStep, with consistent held-out gains over strong baselines.
  • DataCOPE: DataCOPE iteratively coordinates trajectory generation, unsupervised verification, and contrastive skill distillation without ground-truth answers or success labels.This coordination enables reusable procedural knowledge to improve agents without updating model parameters.
  • Verifier designs: The framework instantiates an Adaptive Checklist Verifier for report-style tasks and an Answer Agreement Verifier with self-consistency estimation for reasoning-style tasks.The checklist verifier refines task-specific criteria, while the agreement verifier groups trajectories by answer agreement.

II. PRELIMINARY · A. Data-Analytic Agents · B. LLM Agent Skills

The paper models data-analytic agents as policy-driven interactions with partially observed environments and defines skills as reusable procedural knowledge that conditions behavior without changing model parameters.

  • A. Data-Analytic Agents: Data-analysis tasks are formulated as task-conditioned POMDPs over user queries and associated data resources.Each task belongs to a task space U and is modeled as an interaction process.
  • A. Data-Analytic Agents: The environment state includes code interpreters, data files, intermediate variables, and other context, while actions include Python/SQL generation and final-answer submission.State transitions follow x_t+1 = T(x_t, a_t), but the agent observes o_t rather than directly accessing x_t.
  • A. Data-Analytic Agents: Following ReAct, the agent interleaves thought generation with environment actions at each interaction step.The historical trajectory records the task, thoughts, actions, and observations accumulated over time.
  • A. Data-Analytic Agents: Conditioned on interaction history, policy π_θ predicts successive thoughts and actions until termination, yielding a trajectory and final answer.The next-step pair is sampled as (z_t, a_t) ∼ π_θ(· | h_t).
  • B. LLM Agent Skills: A skill is a structured knowledge bundle that supplies reusable procedural guidance for solving specific tasks.The framework follows prior work in representing skills as structured artifacts.
  • B. LLM Agent Skills: The skill’s root Markdown document specifies applicability, solution strategies, and failure modes, while auxiliary resources support deterministic subtasks.The representation consists of a root document M and resource set R.
  • B. LLM Agent Skills: Injecting a skill conditions agent behavior without altering parameters θ.The skill-conditioned policy samples from π_θ(· | h_t, S) and produces answers conditioned on the task and skill.
  • B. LLM Agent Skills: DataCOPE iteratively combines unlabeled trajectory exploration, unsupervised verification, and contrastive skill distillation.Its overview describes trajectory grouping without gold answers or task-success labels, followed by reusable-procedure extraction and skill updates.

III. METHOD · A. Problem Definition

The problem is to discover a deployable, generalizable skill from unlabeled exploration trajectories without ground-truth answers, success labels, or human annotations. Discovery uses only exploration-time information, while held-out performance and ground-truth evaluation remain reserved for final assessment.

  • A. Problem Definition: DataCOPE targets unsupervised skill discovery for data-analytic agents.The goal is to improve performance on unseen data-analysis tasks without supervision.
  • A. Problem Definition: Discovery uses an unlabeled exploration set Dexplore, while Dtest is held out for final evaluation.During discovery, the agent accesses only task inputs, data resources, and interaction trajectories from Dexplore.
  • A. Problem Definition: The skill’s performance is defined for a task set D when skill S is injected into data-analytic agent πθ.The formal performance definition evaluates the resulting agent behavior over tasks in D.
  • A. Problem Definition: T is the final state or output produced on task u, whereas R⋆ is used only for offline ground-truth evaluation.Neither quantity provides supervision during the discovery phase.
  • A. Problem Definition: The discovery process E constructs a deployable skill bS from exploration data while keeping agent parameters θ fixed.Skill discovery changes the injected skill rather than updating the agent parameters.
  • A. Problem Definition: Discovery effectiveness is measured by how well the resulting skill generalizes to unseen tasks.The desired outcome is strong held-out performance after discovery.
  • A. Problem Definition: Ideally, the discovered skill approaches the best candidate skill in skill space S according to held-out performance.This defines the target quality of the discovery process.
  • A. Problem Definition: The optimality objective characterizes the desired outcome only; Dtest and R⋆ remain inaccessible during skill discovery.They are reserved for offline evaluation rather than used to construct the skill.

B. Overall Framework

DataCOPE discovers transferable skills through an iterative, unlabeled loop linking a Data-Analytic Agent, Unsupervised Verifier, and Skill Manager. The verifier organizes trajectories using non-privileged signals, enabling contrastive distillation of reusable analysis procedures.

  • Framework components: The framework comprises three components: Data-Analytic Agent πθ, Unsupervised Verifier ϕ, and Skill Manager ψω.Together, they form a closed-loop process over unlabeled exploration tasks.
  • Trajectory generation: At iteration r, the agent receives skill S(r), starts with S(0) = ∅, and generates ReAct-style exploratory trajectories in task-specific environments.The number of sampled trajectories per task is N.
  • Unsupervised verification: The verifier uses no ground-truth answers or success labels, instead extracting non-privileged signals about trajectory quality, uncertainty, agreement, or divergence.These signals provide indirect evidence for distinguishing reliable solution patterns from potentially flawed or incomplete ones.
  • Trajectory grouping: Verifier signals organize trajectories into behavioral groups, including high- and low-scoring reports, answer clusters, and differing self-consistency.Each group represents an unsupervised behavioral pattern.
  • Skill distillation: The Skill Manager contrasts grouped trajectories and available exploration data files to distill reusable procedural knowledge, which is injected back for iterative refinement.Verifier instantiations differ by task type: checklist verification and refinement for reports, versus answer clustering and self-consistency estimation for reasoning.

C. Adaptive Checklist Verifier

The Adaptive Checklist Verifier enables unsupervised evaluation of report-style analysis by generating task-specific criteria and iteratively refining them with report-generation skills. It alternates report-side skill evolution with checklist refinement to distill useful strategies while mitigating verifier overfitting.

  • Task-specific Checklist Generation: The Checklist Agent generates task-specific, checkable question–answer criteria for each analytical task without reference reports or ground-truth checklists.These criteria specify the insights or requirements that reports are expected to address.
  • Task-specific Checklist Generation: Checklist scores measure how extensively each report satisfies its criteria and provide an unsupervised quality signal rather than a ground-truth correctness label.The verifier assigns scores in [0, 1] for checklist-item satisfaction.
  • Report-Side Skill Evolution: The Skill Manager contrasts relatively positive and negative trajectory groups to retain strategies from high-scoring reports and suppress weaknesses from low-scoring reports.Groups are formed using average checklist scores across tasks, and report-side evolution continues until the average generated-checklist score decreases.
  • Contrastive Checklist Refinement: Contrastive checklist refinement addresses verifier overfitting by using high-scoring reports to identify omissions and low-scoring reports to expose effective weakness-detecting dimensions.The optimized checklist-generation skill is iteratively refined with reports fixed, then used in the next round of report-side skill evolution.

D. Answer Agreement Verifier

The Answer Agreement Verifier enables label-free evaluation for reasoning-style tasks by clustering final answers and using self-consistency to identify stable or divergent solution patterns. It organizes representative trajectories into answer-based groups for contrastive skill refinement while treating agreement as an auxiliary uncertainty signal rather than correctness.

  • Answer Agreement Verifier: The label-free verifier evaluates reasoning trajectories through answer-level clustering and self-consistency instead of predicting correctness.It uncovers stable solution patterns and divergent reasoning behaviors without gold labels during exploration.
  • Answer Clustering: Answer clustering partitions each task’s generated final answers using a type-specific equality metric, such as exact match.For task u at iteration r, the agent generates N trajectories whose answers are clustered into groups.
  • Self-Consistency Estimation: Self-consistency equals the relative size of a trajectory’s assigned answer cluster, measuring answer convergence but not guaranteeing correctness.The verifier therefore uses self-consistency only as an auxiliary uncertainty signal.
  • Agent-Side Skill Evolution: The verifier forms structured groups from answer clusters, represents each cluster with one concise trajectory, and compares representatives across clusters for skill refinement.Representatives are selected by prioritizing fewer interaction turns and fewer execution exceptions.
  • Agent-Side Skill Evolution: Filtering trajectories with saturated self-consistency focuses iterative refinement on less-certain cases that still require improvement.The Skill Manager applies this filtering before comparing representatives across the remaining clusters.

IV. EXPERIMENT · A. Experimental Settings · B. Main Results

DataCOPE is evaluated on report-based and reasoning-based data-analysis benchmarks across multiple base models, using exploration/test splits and comparisons against Skill Creator. It consistently improves performance across task formats and models, with larger gains on reasoning tasks and advantages attributable to unsupervised verifier signals.

  • A. Experimental Settings: The evaluation uses Deep Data Research for report-based analysis and DABStep for reasoning-based analysis, randomly partitioning each benchmark into Dexplore and Dtest at a 1:3 ratio.Metrics include sample-averaged and item-averaged accuracy for Deep Data Research.
  • A. Experimental Settings: DataCOPE is compared with Anthropic’s Skill Creator, implemented with Claude Code exploring agent trajectories under the same data-access privileges.The provided setup describes Skill Creator as the baseline for assessing the framework.
  • A. Experimental Settings: The evaluation spans Claude-Sonnet-4.6, Claude-Sonnet-4.5, GPT-5.2, DeepSeek-V4-Pro, and Qwen3.5-397B-A17B across different parameter scales and reasoning paradigms.GPT-5.2 uses medium reasoning mode; DeepSeek-V4-Pro and Qwen3.5-397B-A17B use non-reasoning mode.
  • B. Main Results: 47.39% to 57.10%: DataCOPE raises the mean Overall Avg. across four matched base models on reporting tasks.The reporting result is reported for the mean Overall Avg. metric.
  • B. Main Results: 29.14% to 61.44%: DataCOPE increases the mean score on reasoning tasks, with especially large gains on the hard DABStep split.The framework benefits both open-ended report generation and fixed-answer data reasoning.
  • B. Main Results: DataCOPE improves performance across Claude, GPT, DeepSeek, and Qwen model families, with Qwen3.5-397B showing the largest reporting gain and best reporting performance.The reasoning improvements are particularly strong on hard DABStep instances.
  • B. Main Results: 51.34% to 57.10% and 51.73% to 61.44%: DataCOPE exceeds Skill Creator on mean reporting Overall Avg. and mean reasoning all score, respectively.The results attribute DataCOPE’s advantage to unsupervised verifier signals rather than skill generation alone.

C. Analysis

Analysis shows that iterative skill refinement is useful but task-dependent and non-monotonic: it can improve reporting skills early, while reasoning refinement mainly increases consistency without improving test accuracy. Verifier components are critical, as removing adaptive checklist or answer-clustering mechanisms substantially degrades discovery performance.

  • Iterative Skill Refinement Is Effective but Not Monotonic: Reporting refinement yields stronger early improvements, but later iterations become less effective or invalid for 10-K and GLOBEM, while MIMIC continues benefiting.The analysis characterizes refinement gains as task-dependent rather than uniformly monotonic.
  • Iterative Skill Refinement Is Effective but Not Monotonic: Reasoning refinement substantially increases self-consistency on explore and test splits, while test accuracy remains nearly unchanged.Answer-level verification may reduce variance yet fail when the dominant answer cluster is incorrect.
  • Verifier Components are Critical for Skill Discovery: 67.12% falls to 53.32% when the Checklist Agent is removed for 10-K, showing that trajectory-level exploration alone provides insufficient feedback.Replacing task-specific checklists with generic ones further reduces the score to 52.21%, indicating that non-adaptive criteria can introduce noisy supervision.
  • Verifier Components are Critical for Skill Discovery: 62.82% falls to 53.85% when all trajectories are used without verifier-based selection on DABStep, while removing answer clustering drops the score to 47.93%.The analysis warns that relying solely on self-consistency can be harmful because multiple trajectories may converge on the same incorrect answer.

V. FURTHER ANALYSIS … C. Supervised Skill Discovery Analysis

Further analysis shows that DataCOPE performs best with sufficiently granular skills, transfers across data-analytic agents, and remains effective without labeled trajectories. It reaches 62.82% accuracy on DABStep with nine skills and demonstrates label-efficient performance across report-style and reasoning-style analysis.

  • A. Skill Granularity Analysis: Using all 9 discovered skills achieves the best DABStep performance at 62.82% accuracy.Using 2 or 3 skills performs worse than using a single skill, while increasing the number from 3 to 6 and 9 improves performance.
  • B. Data-Analytic Agent Analysis: DataCOPE consistently improves over the Skill Creator baseline across all four combinations of trajectory-generating and evaluation models.GPT-5.2 and DeepSeek-V4-Pro generate exploration trajectories, after which the skills are evaluated on different models.
  • B. Data-Analytic Agent Analysis: With GPT-5.2 generating trajectories, the discovered skill improves Qwen3.5-397B and GPT-5.2 by 5.33% and 7.69%, respectively.These improvements are reported relative to the Skill Creator baseline.
  • C. Supervised Skill Discovery Analysis: On the 10-K reporting benchmark, DataCOPE outperforms all supervised Skill Creator baseline variants.The compared settings add different numbers of randomly selected supervised trajectories to the Skill Creator method.
  • C. Supervised Skill Discovery Analysis: On DABStep, DataCOPE obtains 62.82% without any labeled trajectory and substantially outperforms the one trajectory supervised baseline.Its performance is comparable to the two and three trajectories supervised baselines.
  • C. Supervised Skill Discovery Analysis: When all exploration trajectories are supervised, the baseline reaches 72.19%, indicating that full supervision remains beneficial for fixed-answer reasoning tasks.DataCOPE nevertheless achieves competitive performance under zero annotation cost.
  • C. Supervised Skill Discovery Analysis: DataCOPE demonstrates strong label efficiency under zero annotation cost.This conclusion follows from its competitive DABStep performance without labeled trajectories.

D. Cost Analysis … VII. CONCLUSION

DataCOPE discovers reusable data-analysis skills from unlabeled exploration by coordinating trajectory generation, unsupervised verification, and contrastive skill distillation. The surrounding sections position this approach against predefined workflows and costly training, review related skill research, and report cost-effectiveness under fixed interaction budgets.

  • D. Cost Analysis: DataCOPE consistently reduces token consumption while improving task accuracy for Claude Code and ReAct agents under a 15-turn interaction cap.The controlled comparison uses a fixed budget of 15 interaction turns for both agent scaffolds.
  • A. Data-Analytic Agents.: Data analysis agents autonomously execute end-to-end tasks, with existing approaches broadly categorized into predefined workflows and other paradigms.Predefined workflows use general-purpose LLM reasoning and coding to navigate structured analytical pipelines, including visualization, insight, and report generation.
  • A. Data-Analytic Agents.: DataCOPE generates reusable skills that improve data-analysis capabilities without tying models to specific pipelines or requiring costly domain-specific training.The approach is distinct from rigid predefined workflows and resource-intensive model training.
  • B. LLM Agent Skills.: Prior work shows that modular, reusable skills distilled from scenarios or trajectories can improve agents’ ability to solve similar tasks across heterogeneous skill forms.Research includes skill induction and evolution from execution traces, failures, feedback, and task contexts.
  • B. LLM Agent Skills.: Related research extends skills to multimodal state evidence and studies skill-library organization, retrieval, routing, governance, and multi-skill orchestration.These directions advance reusable skill construction and deployment across diverse agent settings.
  • VII. CONCLUSION: DataCOPE is an unsupervised verifier-guided framework that discovers reusable data-analysis skills from unlabeled exploration trajectories.It coordinates a Data-Analytic Agent for exploration, an Unsupervised Verifier for signal extraction, and a Skill Manager for contrastive trajectory-group distillation; an Adaptive Checklist Verifier is instantiated for report-style tasks.
Loading 2606.06416v1…