PaperlayerView exact revision on arXiv

Source-linked AI summary

TeleTune: Evolving Agent Skills From Offline Telemetry

Justin Chih-Yao Chen, Elias Stengel-Eskin, Yan Chen, Pol Llado, Scott Counts, Mohit Bansal, Benjamin Van Durme, Harsh Jhamtani, Gaurav Verma

arXiv:2610.05437v2cs.AIcs.CL

TL;DR

User telemetry records software procedures but omits goals, cannot be replayed for evaluation, and may interleave tasks. TeleTune learns and validates a textual skill library from logged action-prediction errors and retrieves demonstrations for new tasks. It outperforms retrieval and workflow-induction baselines on WorkArena and Online-Mind2Web.

  • Problem

    User telemetry could provide procedural knowledge, but logs omit the goal behind each action, may be unreplayable, and can interleave tasks without marked boundaries.

    Source: §1 · pp. 1–2

  • Method

    TeleTune infers trajectory goals, proposes textual-library edits from logged action-prediction errors, keeps edits that improve held-out skill-guided progress, and retrieves demonstrations covering new-task subgoals.

    Source: §1 · p. 2

  • Results

    TeleTune achieves 77.1% average WorkArena success, 6.7% above AWM + Random-k, and 80.6% on Online-Mind2Web, 7.7% above Random-k.

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    Skill-guided progress tracks live success during optimization, and offline optimization on fixed logs costs 5 to 75 times fewer tokens than validating edits with live episodes.

    Source: §6 · p. 10, §4.2 · p. 10

    Optimization assumes pre-collected successful trajectories; training logs contain only states and actions, without task goals or replay access.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

Computer-use agents need to capture procedural knowledge of how people use software. User telemetry offers a scalable source of this knowledge. However, learning reusable skills from these logs requires addressing three challenges: (1) Goal Underspecification, since logs do not record the goal behind each action; (2) Non-Replayability, since past activity cannot be replayed to evaluate skill updates; and (3) Interleaved Trajectories, since logs may mix several tasks without marking their boundaries. To address these, we introduce TeleTune, a framework for learning a textual skill library from offline logs without recorded goals, cannot be replayed during optimization, and may interleave tasks. TeleTune uses action-prediction errors on logged trajectories to propose library edits and keep only those that improve held-out action-prediction accuracy, which we call skill-guided progress. The learned workflows also enable retrieval of demonstrations that cover the subgoals of a new task. At test time, the agent is provided with the learned library and the workflow-based retrieved demonstrations. Experiments on WorkArena and Online-Mind2Web show that TeleTune outperforms random retrieval, Agent Workflow Memory (AWM), and their combination. We find that the best baseline varies by setting, whereas TeleTune achieves average success rates of 77.1% and 80.6%, respectively, improving over the strongest baseline on each benchmark by 6.7% and 7.7%. Under the heaviest perturbation of the WorkArena training data,TeleTune keeps the highest average success rate at 68.5%, 6.3% above the strongest baseline. Our analyses show (1) skill optimization and workflow-based retrieval are complementary, (2) optimizing on fixed logs costs 5 to 75 times fewer tokens than validating the same edits with live episodes, (3) skill-guided progress tracks the live success rate.

1. Introduction

User telemetry captures software procedures but omits goals, cannot be replayed for evaluation, and may interleave tasks. TeleTune learns skills from offline logs and uses them to retrieve demonstrations for new tasks.

1. Introduction

  • Telemetry creates three learning challenges: goals are implicit, recorded activity cannot be replayed for fresh outcomes, and task boundaries may be unmarked.Prior approaches address parts of this setting, but none addresses all three challenges together.Source: §1 · pp. 1–2
  • TeleTune predicts logged actions using inferred goals and the current library, proposes edits from prediction errors, and keeps edits that improve held-out skill-guided progress.At test time, its learned workflows retrieve demonstrations that together cover a new task’s subgoals.Source: §1 · pp. 2–3
  • Evaluation uses WorkArena transfer at atomic L1 and compositional L2 levels, plus live interaction on new Online-Mind2Web tasks after training on annotator-recorded trajectories.WorkArena training pools are also perturbed at varying interruption rates to isolate task interleaving.Source: §1 · p. 3
  • TeleTune reaches 92.3% on WorkArena L1 (+12.0%), 61.8% on L2 (+1.4%), and 80.6% on Online-Mind2Web (+7.7%) versus the strongest baselines.Under the heaviest WorkArena training-data perturbation, it achieves 68.5% average success, 6.3% above the strongest baseline.Source: §1 · p. 3
  • Skill-guided progress tracks live success: on WorkArena L1, live success rises from 74.1% to 92.3% as the offline metric improves.The authors propose it as an offline proxy for evaluating skill updates.Source: §1 · p. 3

2. TeleTune: Telemetry-Informed Tuning for Agent Skills

TeleTune learns and validates a shared textual skill library from successful state-action logs, then retrieves complementary demonstrations matched to a new task’s subgoals.

2.1. Problem Formulation

  • Successful state-action logs have no goals or replay access; teacher-forced prediction accuracy on scoreable steps measures skill-guided progress.Train trajectories surface errors for edits, while dev trajectories determine whether to accept them.Source: §2.1 · p. 4

2.2. Skill Optimization

  • The library begins with induced workflows, then optimizes workflows and skills in separate stages, accepting proposed edits only when dev-set progress improves.Workflows are distilled across trajectories; each new skill is a short rule tied to an observed failure, and explicit edit operations keep changes readable.Source: §2.2 · pp. 4–5

2.3. Workflow-Based Retrieval

  • A planner maps task subgoals to workflows, and a greedy selector favors demonstrations covering still-uncovered subgoals over redundant neighbors.Common workflows are down-weighted so generic operations do not dominate retrieval.Source: §2.3 · p. 5

2.4. Inference

  • For each test task, retrieval builds demonstrations once; the agent then acts with the task goal, fixed library, and retrieved demonstrations in its prompt.The library supplies global knowledge across tasks, while retrieved demonstrations provide local evidence for the current task’s subgoals.Source: §2.4 · pp. 5–6

3. Experimental Setup

TeleTune is evaluated on WorkArena and Online-Mind2Web, with shared test tasks and step budgets across methods. The setup defines the datasets, backbone pairing, baselines, and evaluation protocol.

Dataset

  • WorkArena includes 270 training and 270 test instances on L1, and 216 training and 198 test instances on L2; Online-Mind2Web uses 640 training trajectories and 122 test tasks.WorkArena task families with empty traces, broken fixtures or controls, or L2 solutions that report infeasibility are pruned; Online-Mind2Web training and test website categories are aligned.Source: §3 · p. 6

Models

  • Each backbone learns its own library and handles goal inference, workflow induction, skill optimization, and test-time actions.GPT-5.4 is used for the main results, while Qwen3.6-35B-A3B-FP8 and Mistral-Medium-3.5-128B are used for cross-model analysis.Source: §3 · p. 6

Baselines

  • Baselines include Naive, Random-k with 8 goal-free demonstrations, AWM, and AWM + Random-k; all methods share the model, test tasks, and step budget.Workflow-based retrieval also uses 8 training demonstrations.Source: §3 · p. 6

Evaluation Protocol

  • Methods run on the same test tasks with three seeds, with limits of 100 WorkArena L1 steps, 200 L2 steps, and 200 Online-Mind2Web steps.Completed trajectories are judged by WebJudge using o4-mini; Online-Mind2Web reports discounted success excluding episodes blocked by CAPTCHA or bot-detection pages.Source: §3 · p. 6
  • Table 1 reports success rates averaged over three seeds and states that TeleTune achieves the best result in every column.It compares TeleTune with retrieval and workflow-induction baselines using GPT-5.4 as the agent.Source: Table 1 · p. 7

4. Results and Analysis

TeleTune outperforms the strongest baselines on average across WorkArena and Online-Mind2Web, and generalizes to other agent backbones. Analyses examine component complementarity, robustness to interleaved tasks, offline progress as a proxy for live success, and optimization and inference costs.

4.1. Main Results

  • TeleTune averages 77.1% on WorkArena, 6.7% above its strongest baseline, and 80.6% on Online-Mind2Web, 7.7% above Random-k.On WorkArena L1, its success rises from 80.3% to 92.3%.Source: §4.1 · p. 7
  • With Qwen-3.6, TeleTune reaches 75.6% on L1 and 50.0% on L2, beating AWM + Random-k by 11.9% and 5.1%; with Mistral-3.5, it is near the strongest baseline.Each backbone learns its own library; with Mistral-3.5, TeleTune leads AWM by 0.5% on L1 and trails Random-k by 1.1% on L2.Source: §4.1 · p. 7, Table 2 · p. 7

4.2. Ablations and Analysis

  • On WorkArena L1, dev-split skill-guided progress rises in the same order as live success across four checkpoints, suggesting it can serve as an offline proxy.During the skill stage, progress reaches 73.0% while live success rises from 74.1% to 83.1%, 88.7%, and 92.3%.Source: §4.2 · p. 8, Figure 3 · p. 8
  • Skill optimization and workflow-based retrieval are complementary: with the learned library, retrieval adds 8.6% on L1, 1.0% on L2, and 5.5% on Online-Mind2Web.On L1, full TeleTune is 27.2% above the variant with neither component, exceeding the sum of individual gains by 6.9%.Source: §4.2 · p. 8, Table 3 · p. 8
  • TeleTune has the highest WorkArena success at each tested interruption rate except L1 at q=1.0, where Random-k is 1.1% higher.At q=1.0, AWM’s L1 success falls by 26.4% to 49.3%; TeleTune retains entries only when they improve dev-split skill-guided progress.Source: §4.2 · pp. 8–9, Figure 4 · p. 9
  • Offline optimization costs 16 times fewer tokens than live validation on L1, 75 times fewer on L2, and 5 times fewer on Mind2Web.The comparison scores candidate libraries on fixed logs; live validation would require 4,416 episodes on L1 and resettable environments for dev tasks.Source: §4.2 · p. 9, Table 4 · p. 9
  • TeleTune’s L1 success comes with higher input use: 636.9K tokens per episode versus 343.4K–386.1K for baselines, mainly because the library is in every step’s prompt.Its L1 episode length is 31.4 steps, within the baseline range of 27.4–34.4; on L2 it uses 1.52M tokens versus 1.49M for the naive agent.Source: §4.2 · p. 9

5. Related Work

Prior work learns reusable agent knowledge from experience and procedural information from recorded behavior, but the cited approaches do not jointly address goal-free, non-replayable, interleaved logs. TeleTune differs by learning from recorded actions without task replay.

5. Related Work

  • Agent-learning work spans reusable skills and abstractions, natural-language guidance, retrieved trajectories, hindsight rewrites, reinforcement learning, and execution-feedback refinement.The cited examples include executable skills (Stengel-Eskin et al., 2024; Wang et al., 2023; Zheng et al., 2025), language guidance (Ni et al., 2026; Ouyang et al., 2026b; Shinn et al., 2023; Zhao et al., 2024), and other approaches.Source: §5 · p. 10
  • SkillOpt keeps bounded edits that improve held-out task performance but validates candidates by executing tasks, whereas TeleTune scores action predictions on fixed logs.TeleTune's skill-guided progress needs no live environment or new execution outcomes.Source: §5 · p. 10
  • AWM induces workflows from goal-labeled trajectories, NNetNav explores without predefined goals, and process mining finds routines in unsegmented logs; TeleTune combines goal-free learning with evaluation without replay.SkillClaw uses logs with task goals and reruns tasks in users' environments.Source: §5 · p. 10

6. Conclusion

TeleTune learns a textual skill library from offline logs without recorded goals, replay, or task boundaries, then uses learned workflows to retrieve demonstrations for new tasks. It outperforms the strongest baselines on WorkArena and Online-Mind2Web, and the results suggest logged activity can support skill learning and offline assessment.

6. Conclusion

  • TeleTune raises average success over the strongest baseline by 6.7% on WorkArena and 7.7% on Online-Mind2Web.It also leads under the heaviest task interleaving, while skill-guided progress rises alongside live success during WorkArena L1 optimization.Source: §6 · p. 10
  • TeleTune infers a goal for each trajectory, proposes library edits from action-prediction errors, and retains edits that improve held-out skill-guided progress.At test time, workflows retrieve demonstrations covering a new task's subgoals.Source: §6 · p. 10
  • The results suggest existing software-use logs can teach reusable skills and indicate whether an update helps without a live environment.Scoring edits on fixed logs costs 5 to 75 times fewer tokens than live validation.Source: §6 · p. 10

Appendix

The appendix documents the prompts used throughout TeleTune, from inferring goals and building and optimizing workflows to predicting actions from offline trajectories. It also describes prompts for guiding test agents and retrieving demonstrations through workflow-based subgoal matching, including constraints on goal descriptions, workflow generalization, and optimizer edits.

A.1. Goal-Inference Prompt

  • Goal inference reconstructs a specific desired outcome from each successful trajectory while preserving supported task details and avoiding UI-action recipes or sensitive information.WorkArena and Mind2Web prompts both request one concise goal sentence; uncertain details are omitted rather than used to generalize away clear constraints.Source: §A.1 · pp. 14–15

A.2. Workflow-Induction Prompt

  • Initial workflows capture recurring action patterns across at least two tasks, using typed placeholders for variable elements; Mind2Web workflows generalize across unrelated websites.The WorkArena prompt calls for workflows covering the task batch, while the Mind2Web prompt generalizes semantic action patterns across sites and avoids inventing controls.Source: §A.2 · pp. 15–16

A.3. Optimizer Prompt

  • The optimizer selects skills for generalizable action errors, workflows for missing task-family procedures, and traces when in-context examples do not cover failures.Skill rules must remain reusable and include a concrete training-case example; instance-specific values are forbidden in the general-rule portion.Source: §A.3 · p. 17
  • Stage-specific suffixes restrict legal edits, and Mind2Web skill guidance must rely on page evidence and action history rather than hidden task metadata.Workflow-only rounds direct workflow corrections, while skill rounds treat workflows as read-only and prohibit duplicating their procedures as skills.Source: §A.3 · p. 18

A.4. Predictor Prompt

  • The predictor prompts require one immediate next action grounded in the goal, action history, and—on Mind2Web—the current page observation.When the goal is blank, the Mind2Web prompt directs the predictor to infer the ongoing task from the action sequence and revise that inference as actions arrive.Source: §A.4 · p. 19

A.5. Agent Prompt

  • Test-agent prompts distinguish reusable skills and workflows from lower-level reference trajectories, which agents should adapt rather than follow verbatim.WorkArena receives retrieved demonstrations in one list without goals; Online-Mind2Web separates task-matched demonstrations labeled by covered subgoals from randomly sampled exploration trajectories.Source: §A.5 · p. 20

A.6. Retrieval Prompts

  • Workflow retrieval annotates training trajectories by the workflows they instantiate, then decomposes each test goal into ordered subgoals and maps them to library workflows.Both prompts use temperature 0; the WorkArena planner receives the full library, including skills, while other workflow inventories contain workflows.Source: §A.6 · p. 21

B. Workflow-Based Retrieval Details

Workflow-based retrieval annotates logged trajectories, maps new-task subgoals into the same procedural space, scores trajectory–subgoal matches, and greedily selects demonstrations that cover the subgoals.

Workflow Annotation

  • Each training trajectory is annotated with its instantiated workflows, with rarer workflows weighted more heavily in relevance scoring.A workflow shared by most training trajectories contributes little to a match, while a rare workflow contributes most.Source: B. Workflow-Based Retrieval Details · p. 21

Subgoal Planning

Relevance Scoring

  • Relevance combines weighted workflow overlap, semantic purpose, and a small preference for a compatible website; only sufficiently confident matches count as task-relevant.The semantic component distinguishes trajectories that instantiate the same broad workflow for different purposes.Source: B. Workflow-Based Retrieval Details · p. 22

Greedy Selection

  • Demonstration selection stops when subgoals are covered, no confident match remains, or the budget of 8 demonstrations is reached.Each selected trajectory removes the subgoals it supports from the uncovered set.Source: B. Workflow-Based Retrieval Details · p. 22

Workflow-Based Retrieval

  • Retrieval combines workflow-supported matches with deterministically sampled exploratory examples, falling back entirely to random retrieval when no workflow matches the task.On Mind2Web, matched examples are labeled with supported subgoals and exploratory examples are presented separately; on WorkArena, all demonstrations appear in one list.Source: B. Workflow-Based Retrieval Details · pp. 21–22

C. Hyperparameters

The reported settings are shared across WorkArena L1 and L2, Mind2Web, and backbones.

C. Hyperparameters

  • All runs use inferred goals, a two-stage schedule, validation on the full dev split without a curriculum, and shared settings across WorkArena levels, Mind2Web, and backbones.The Mistral-3.5 predictor and agent calls use temperature 0.7 and top-p 0.95.Source: C. Hyperparameters · p. 22

D. Data Processing and Statistics

The evaluation uses held-out WorkArena seeds from the same task templates and filters Online-Mind2Web test tasks to categories represented in the original training set.

WorkArena

  • WorkArena training uses successful oracle trajectories, with train and dev splits assigned by trajectory hash and test instances drawn from other seeds of the same task templates.The train and dev split is roughly even; the passage also notes that some WorkArena task families are removed.Source: D. Data Processing and Statistics · p. 22

Mind2Web

  • Online-Mind2Web supplies 300 live tasks without demonstrations, while the original Mind2Web training set contains 1,009 trajectories.Tasks are category-labeled from their instruction and website; 13 receive a none label.Source: D. Data Processing and Statistics · p. 22
  • Filtering Online-Mind2Web to categories present in the original training set removes all Info and Service tasks and leaves 122 test tasks: 29 easy, 54 medium, and 39 hard.The original training set has no Info or Service tasks.Source: D. Data Processing and Statistics · p. 22

Skill optimization

Skill optimization uses bounded edit and step budgets and accepts an edit only when mean action-prediction quality improves on the full dev split.

Skill optimization

  • Optimization stops a stage after 5 consecutive steps without an accepted edit or 25 steps, caps total steps at 100, and accepts edits only when mean Q strictly increases on the full dev split.Each step allows at most 3 edits; the library is limited to 100 skills, 40 workflows, and 600 characters per skill, and predictor sampling uses temperature 0.Source: Skill optimization · p. 23

Workflow-based retrieval

The retrieval setup selects workflow-relevant demonstrations using weighted relevance scores and applies specified test-time constraints. The accompanying data passages describe trajectory filtering, category balancing, and train/dev and test splits.

Workflow-based retrieval

  • Retrieval uses 8 demonstrations, ranked by 0.65 workflow, 0.25 semantic, and 0.10 site relevance weights.Workflow weights use IDF over training trajectories; the confidence threshold is 0.8.Source: Workflow-based retrieval · p. 23
  • The test agent receives text-only observations and has a 100-step budget on WorkArena L1 and 200 steps on other settings.Source: Workflow-based retrieval · p. 23
  • Train and dev use a trajectory-level hash split, while WorkArena test instances use other seeds of training task templates.Mind2Web test tasks are live Online-Mind2Web tasks.Source: Table 6 · p. 23
  • Filtering retains 966 of 1,009 training trajectories, then category balancing downsamples them to 640 without duplicating trajectories.The balancing procedure lowers the train/test category-distribution total-variation distance from 0.299 to 0.204.Source: Workflow-based retrieval · p. 23

E. Additional Results

The additional results report WorkArena success rates across increasing training-pool interruption rates, alongside per-episode token and step counts.

Noise Sweep

  • Table 7 reports WorkArena success rates as interruption rate q increases from 0 for clean training data to 1.0.The q=0 columns and Naive row are three-seed means; other entries are single-seed, and bold marks the best result per column.Source: Table 7 · p. 24, E. Additional Results · p. 24

Token and Step Counts

F. Qualitative Examples of the Learned Skills

The qualitative examples illustrate learned skills as concrete, reusable workflows for configuring forms and lists, narrowing and selecting results, completing commerce and booking tasks, and navigating web pages.

Mistral-3.5 Skill Library Learned on WorkArena L1

  • WorkArena L1 list skills run completed filters, use “Run filter” for sorting, and start knowledge searches by filling the Search field with the quoted query.The examples distinguish filter execution from sorting and specify the exact initial search action.Source: F. Qualitative Examples of the Learned… · p. 25
  • WorkArena L1 form skills click fields before filling them, follow the form’s field order, and prioritize required fields over optional ones.They also preserve the exact values specified in the goal rather than paraphrasing them.Source: F. Qualitative Examples of the Learned… · p. 25

Qwen-3.6 Skill Library Learned on WorkArena L2

  • WorkArena L2 skills use the task’s exact application name, open the first visible filtered result by its displayed ID, and continue to the next preview after each update.The examples warn against guessing application names or record IDs and against returning to the list between iterative updates.Source: F. Qualitative Examples of the Learned… · p. 27

GPT-5.4 Skill Library Learned on Mind2Web

  • Mind2Web skills open visible controls before entering values and favor the requested condition, mode, or parent section over shortcuts or guessed destinations.Examples include opening booking fields, satisfying item-condition constraints before secondary perks, and entering a section before choosing a subitem.Source: F. Qualitative Examples of the Learned… · pp. 28–29
  • Mind2Web workflows cover contact forms, location suggestions, section and category navigation, traveler setup, and launching a search after entering criteria.Date and text inputs are opened before use, and an existing date value can be selected to change a planner’s date.Source: F. Qualitative Examples of the Learned… · p. 29

F. Qualitative Examples of the Learned Skills

  • Commerce workflows encode adding items to a cart, choosing variants before purchase, or saving items to a named list or wishlist.The sequences include item selection, variant configuration where needed, and optional post-add or save confirmation actions.Source: F. Qualitative Examples of the Learned… · p. 30
  • Other learned skills configure dropdown-driven forms, initiate booking or reservation flows, and navigate to requested information or categories.Their steps include selecting options before continuing, confirming a reservation when required, and opening parent sections and target subsections.Source: F. Qualitative Examples of the Learned… · pp. 30–31
  • Learned workflows cover searching, filtering or sorting results, and opening a selected item or final result.They specify conditions such as a visible results list or available search box, and outcomes such as narrowed results or an opened detail page.Source: F. Qualitative Examples of the Learned… · pp. 30–31

Related papers

Loading 2610.05437v2…