PaperlayerView exact revision on arXiv

Source-linked AI summary

JEF-Hinter: Leveraging Offline Knowledge for Improving Web Agents Adaptation

Hadi Nekoei, Aman Jaiswal, Patrice Bechard, Oleh Shliazhko, Orlando Marquez Ayala, Mathieu Reymond, Massimo Caccia, Alexandre Drouin, Sarath Chandar, Alexandre Lacoste

arXiv:2510.04373v2cs.AI

TL;DR

LLM agents can struggle in unfamiliar domains, while fine-tuning and online interaction are costly or unavailable and raw demonstrations are difficult to transfer. JEF-Hinter distills offline traces into compact hints and retrieves relevant guidance at inference. The authors report improvements over strong baselines across benchmarks, with cross-benchmark generalization left unevaluated.

  • Problem

    Fine-tuning and online interaction can be costly or unavailable, while raw offline trajectories are long, noisy, and tightly tied to their source tasks.

    Source: §1 · p. 2

  • Method

    JEF-Hinter zooms into critical steps in offline traces, reflects them into context-aware hints, and retrieves relevant guidance for the base agent without fine-tuning.

    Source: §1 · p. 2

  • Results

    JEF-Hinter improves over strong baselines on MiniWoB++, WorkArena-L1, and WebArena-Lite, including gains in out-of-goal and out-of-task generalization.

    Source: §6 · p. 11

  • Takeaways & Limitations

    The results support reusing hints within an environment when tasks share similar interaction patterns.

    Source: §5.3 · p. 9

    Cross-benchmark generalization is not evaluated, and hint databases are environment-specific.

    Source: §1 · p. 2, §5.3 · p. 9

Abstract

from arXiv · show

Large language model (LLM) agents perform well in sequential decision-making tasks, but improving them on unfamiliar domains often requires costly online interactions or fine-tuning on large expert datasets. These strategies are impractical for closed-source models and expensive for open-source ones, with risks of catastrophic forgetting. Offline trajectories offer reusable knowledge, yet demonstration-based methods struggle because raw traces are long, noisy, and tied to specific tasks. We present Just-in-time Episodic Feedback Hinter (JEF-Hinter), an agentic system that distills offline traces into compact, context-aware hints. A zooming mechanism highlights decisive steps in long trajectories, capturing both strategies and pitfalls. Unlike prior methods, JEF-Hinter leverages both successful and failed trajectories, extracting guidance even when only failure data is available, while supporting parallelized hint generation and benchmark-independent prompting. At inference, a retriever selects relevant hints for the current state, providing targeted guidance with transparency and traceability. Experiments on MiniWoB++, WorkArena-L1, and WebArena-Lite show that JEF-Hinter consistently outperforms strong baselines, including human- and document-based hints.

1 Introduction

LLM agents can struggle in unfamiliar domains, while fine-tuning and online interaction are costly or unavailable and raw demonstrations transfer poorly. JEF-Hinter distills offline traces into context-aware hints and is reported to outperform strong baselines across three benchmarks.

1 Introduction

  • Fine-tuning can be costly, unstable, or unavailable for closed-source models, while reinforcement learning requires extensive online interaction and cannot be applied to closed-source models.Raw trajectory demonstrations are long, noisy, and tightly tied to source tasks; AutoGuide is limited to contrastive trace pairs and benchmark-specific prompting.Source: §1 · p. 2
  • JEF-Hinter distills offline trajectories into explicit, context-aware hints, using zooming and reflection to turn decisive steps into concise guidance.Hints capture effective strategies and common pitfalls, and can be generated from single, paired, or multiple traces.Source: §1 · p. 2
  • JEF-Hinter supports guidance from failed runs even when no successful run exists, and indexes hints with semantic keys for retrieval at inference.The system can use single traces, pairwise contrasts, or multi-trace aggregation, with step-level or goal-conditioned retrieval.Source: §1 · p. 2
  • Hint generation is performed offline and amortized across executions, leaving test-time overhead to lightweight retrieval and prompt augmentation.The authors report slightly higher cost than ReAct and greater efficiency than AutoGuide.Source: §1 · p. 2
  • Across MiniWoB++, WorkArena-L1, and WebArena-Lite, JEF-Hinter consistently outperforms strong baselines, including documentation retrieval and human-authored hints.The authors also report qualitative examples in which guidance steers actions toward the correct context and prevents repeated errors.Source: §1 · p. 2

2 Related Work

Prior work improves agents through prompting, search, or reuse of offline experience, but approaches differ in whether they rely on test-time computation, raw demonstrations, successful traces, or ongoing interaction. JEF-Hinter instead distills compact hints from heterogeneous offline traces and retrieves them just in time.

2 Related Work

  • Prompting and reflection methods adapt behavior online or across episodes, while search-based methods such as Tree-of-Thoughts use additional test-time computation.JEF-Hinter is presented as complementary to search, amortizing experience from offline trajectories instead.Source: §2 · p. 3
  • RAG demonstrations can be long, noisy, and task-bound, while AutoGuide distills guidelines from contrastive pairs and AWM induces workflows from successful trajectories.AutoManual relies on online interaction and iterative refinement.Source: §2 · p. 3
  • Memory approaches such as ACE [Zhang et al., 2026] and ReasoningBank store structured reasoning or procedural memories for repeated executions.JEF-Hinter differs by producing lightweight hints rather than replaying workflows or growing persistent memory structures.Source: §2 · p. 3
  • JEF-Hinter distills lightweight, context-aware hints from successful and failed offline trajectories without requiring contrastive pairs or online refinement.It zooms in on critical decision points, captures strategies and pitfalls, and indexes hints by semantic keys for fine-grained or goal-conditioned retrieval.Source: §2 · p. 3

3 Just-in-time Episodic Feedback Hinting

JEF-Hinter uses offline trajectories to create explicit hints without fine-tuning the base policy. It selects evidence flexibly, zooms into decisive observations, reflects them into hints, then retrieves relevant guidance during action.

3 Just-in-time Episodic Feedback Hinting

  • JEF-Hinter uses an LLM Hinter to transform offline trajectories and documents into explicit hints that augment the base policy without fine-tuning.Hint generation is offline, so the Hinter can be larger than the base agent while the resulting guidance remains lightweight at inference.Source: §3 · pp. 3–4

3.1 Data Collection

  • Evidence selection supports single-trace analysis, pairwise comparison, and multi-trace aggregation, drawing on trajectories from the base policy, other agents, or human demonstrations.Pairwise analysis can compare unequal rewards, or use equal-reward, fail-fail, or success-success pairs when needed.Source: §3.1 · p. 4

3.2 Hint Generation: Zoom & Reflect

  • Zooming selects critical steps and restricts observations to decisive windows while preserving the full sequence of reasoning, actions, and rewards.The observation window length ∆ controls retained context; the paper reports 80–95% context reduction while improving hint quality.Source: §3.2 · pp. 4–5
  • A semantic key summarizes the trajectory prefix, and the Hinter uses that context and the selected prompt to produce a reusable hint linked to its key.The hint can capture a beneficial action or a common error to avoid, and the linked hint-key pairs are stored in a database.Source: §3.2 · p. 5

3.3 Retrieve & Act

  • Step-level retrieval selects top hints for the current context but costs two model calls, whereas goal-conditioned retrieval selects hints once per episode and avoids repeated retrieval calls.In-task, cross-task, and hybrid source-task settings trade within-domain reliability against transfer; the figure overview also depicts goal- or context-conditioned retrieval.Source: Figure 2 · p. 4, §3.3 · p. 5

4 Experimental Setup

The evaluation spans three web benchmarks and tests hint transfer both across goals within source tasks and to tasks excluded from the hint database.

Benchmarks

  • MiniWoB++, WorkArena-L1, and WebArena-Lite span synthetic single-page tasks, enterprise workflows, and realistic multi-domain tasks, testing short- and long-horizon generalization.The benchmarks are described as spanning increasing complexity.Source: §4 · p. 6, Figure 3 · p. 6

Observation and action spaces

  • The evaluation uses AXTrees to reduce input size by about 10x, except MiniWoB++, where the smaller, more informative DOM is used.Actions are high-level UI primitives, including click, fill, select, scroll, and hover.Source: §4 · p. 6

Baselines

  • Comparisons include ReAct, AutoGuide†, JEF-Hinter without zoom, and JEF-Hinter with critical-step zooming.All methods use the ReAct framework; AutoGuide† was reimplemented because no public implementation was available.Source: §4 · p. 6

Offline datasets

  • Offline datasets use ReAct trajectories from held-out goals for MiniWoB++ and WorkArena-L1, plus parallel trace collection for WebArena-Lite.The collection uses five held-out goals per MiniWoB++ task and ten per WorkArena-L1 task.Source: §4 · p. 6
  • Both successful and failed trajectories are retained, allowing JEF-Hinter to generate hints from failures alone when successful traces are unavailable.AutoGuide requires successful-failed trace pairs; GPT-5 traces are also added in augmented datasets to ensure at least one successful trace per task.Source: §4 · p. 6

Evaluation protocol and generalization scope

  • Evaluation measures in-task transfer across different goals and out-of-task transfer when the source task is excluded from the hint database.The primary metric is average task success rate, reported separately for the two settings.Source: §4 · pp. 6–7

5 Empirical Study

The empirical study evaluates JEF-Hinter’s performance, hint-source alternatives, out-of-task transfer, and design choices across MiniWoB++, WorkArena-L1, and WebArena-Lite.

5.1 Does JEF-Hinter improve overall performance compared to baselines?

  • JEF-Hinter and its no-zoom variant outperform baselines across most tasks, and JEF-Hinter beats AutoGuide by using successful or failed trajectories.The comparison includes tasks where ReAct failed entirely, testing whether useful hints can be derived from failure-only data.Source: Figure 4 · p. 7, §5.1 · pp. 7–8
  • Zooming over two trajectories substantially improves results on MiniWoB++ and WorkArena-L1 compared with supplying a full single trace, with gains at no additional inference-time cost.A full-trace single input yields only marginal gains over ReAct; the comparison uses GPT-5 as both base agent and hinter.Source: §5.1 · p. 8, Table 1 · p. 8

5.2 How effective is JEF-Hinter compared to documentation and human hints?

  • Compared with documentation and human hints, JEF-Hinter automatically reuses offline traces; documentation offers modest gains, while human notes cover 16 goals and are hard to scale.Table 2 reports average reward, with standard errors of 0.01 on WorkArena-L1 and 0.03 on WebArena-Lite.Source: §5.2 · pp. 8–9, Table 2 · p. 8

5.3 Can JEF-Hinter generalize out-of-task?

  • WorkArena-L1 transfer beats ReAct and AutoGuide, while WebArena-Lite results are within the margin of noise; cross-benchmark transfer is untested because hints encode environment-specific structures.The evaluation removes the source task from the retrieval pool to test transfer beyond that task.Source: Figure 5 · p. 9, §5.3 · p. 9

5.4 Analysis & Discussion

  • Larger GPT-5 hinters generally produce higher-quality hints than GPT-5-mini, with the largest gains on complex, long-horizon tasks.The reported advantage is +5% on WorkArena-L1 and +2% on MiniWoB++.Source: §5.4 · p. 9, Figure 6 · p. 10
  • Case studies show targeted fixes: Ctrl/Cmd multi-selection in MiniWoB++, the Navigator’s All filter in WorkArena-L1, and clearing default constraints plus pagination in Shopping Admin.The Shopping Admin hint addresses aggregation across the full order history.Source: §5.4 · p. 10
  • 10% success without zoom versus 70% with zoom on WorkArena-L1 sort-hardware-list; zooming surfaced decisive steps from both successful and failed traces.Those steps captured the workflow from opening the filter panel through configuring and applying the sort.Source: §5.4 · p. 10

6 Conclusion

JEF-Hinter turns long offline trajectories into short, retrievable hints that help agents address failure modes without fine-tuning. Evaluations across three web-agent benchmarks show improvements over strong baselines, while the paper provides experiment traces and code for reproducibility.

6 Conclusion

  • JEF-Hinter zooms in on decisive steps in long traces, then distills reusable strategies and pitfalls into compact, transparent hints for inference.The conclusion frames this as data-centric adaptation using past trajectories, documents, and human instructions.Source: §6 · p. 11
  • Experiments on MiniWoB++, WorkArena-L1, and WebArena-Lite show improvements over strong baselines, including gains in out-of-goal and out-of-task generalization.Ablations examine retrieval design, hinter capacity, and failed-trajectory inclusion.Source: §6 · p. 11
  • The authors use AgentLab and BrowserGym, save package versions with experiment results, and provide code and experiment traces to support reproducibility.The supplementary materials include an anonymized codebase, with prompts, collection procedures, and case studies documented in the paper and appendices.Source: §6 · p. 11

A Documentation Search as Hints for LLM Agents

The paper investigates documentation search as a source of hints for LLM agents. It scrapes platform documentation and converts pages into structured Markdown for retrieval.

A Documentation Search as Hints for LLM Agents

  • Documentation search retrieves relevant knowledge directly from official platform resources as a hinting mechanism.The sources include ServiceNow for WorkArena-L1 and GitLab and shopping websites for WebArena.Source: A Documentation Search as Hints for… · p. 14
  • The documentation corpus is drawn from ServiceNow, GitLab, and shopping websites for the corresponding benchmarks.The source platforms are ServiceNow for WorkArena-L1 and GitLab and shopping websites for WebArena.Source: A Documentation Search as Hints for… · p. 14
  • Each webpage is cleaned into Markdown with structured metadata, including its title, summary, keywords, and breadcrumbs.Source: A Documentation Search as Hints for… · p. 14

Experimental Setup

This section describes documentation-search experiments and the procedures used to collect human hints and generate, compare, zoom, and retrieve trace-based guidance.

Experimental Setup

  • WorkArena-L1 experiments vary sparse BM25 versus dense retrieval, task-goal versus context-generated queries, and full pages versus hierarchical chunks.The setup uses GPT-5-mini; full-page retrieval returns three pages and chunked retrieval returns five section-level snippets, while Table 3 summarizes documentation corpus pages and chunks.Source: Experimental Setup · p. 14, Table 3 · p. 14
  • BM25 with task-goal queries and full-page retrieval performs on par with more complex documentation-search configurations.The authors report that this setting is faster and easier to implement; dense retrieval underperforms, possibly because embeddings are less attuned to domain-specific terminology.Source: Table 4 · p. 14, Experimental Setup · p. 14
  • Documentation coverage and usefulness vary: WebArena documentation is limited to GitLab and shopping tasks, while UI guidance is often tangential to complex interactions.The impersonation example improves when documentation clarifies legitimate task intent, whereas filtering and sorting degrade mainly because of skill errors.Source: Experimental Setup · p. 15

B Human Hint Collection

  • Human annotators select model-proposed actions or provide free-form hints, which prompt new action candidates until the task is completed.This collects intermediate human teaching signals alongside the final action sequence, including localized interaction strategies.Source: B Human Hint Collection · p. 15, Figure 7 · p. 16

C.1 Step selection

  • Step selection targets one or two decisive steps, such as branching decisions, avoidable mistakes, successful patterns, or success/failure points.Source: §C.1 · p. 16

C.2 Step-Sequence Hinting

  • Step-sequence hinting analyzes consecutive observations, actions, errors, and rewards to explain progress, the next action, recognition cues, and pitfalls.Source: §C.2 · p. 17

C.3 Hint Generation

  • Hint generation pairs detailed trace analysis with a short, actionable hint and a topic label for applicability.The prompt supports single- or multi-trace inputs and optional documents; examples identify using the Application Navigator instead of global search and explicitly clicking a form action rather than pressing Enter.Source: §C.3 · p. 17

C.4 Two-Trace Comparison (Desired vs. Undesired)

  • Two-trace comparison identifies the first divergence between successful and failed runs, then derives a general rule that follows the successful trace.Source: §C.4 · p. 19

D More results

Parallelizing hint generation distributes trajectories across workers and substantially reduces generation time. The reported speedup supports constructing larger hint databases for complex benchmarks.

D More results

  • The parallelized implementation distributes trajectories across multiple workers, addressing the scalability limit of sequential hint generation.Source: D More results · p. 21
  • The speedup makes it practical to build diverse, comprehensive hint databases for complex benchmarks without prohibitive computational overhead.Source: D More results · p. 21
  • Nearly 20× speedup over sequential hinting is achieved by the parallel implementation.The comparison is shown in Figure 8.Source: Figure 8 · p. 21, D More results · p. 21

E.1 Hint stats

The hint-database statistics are organized by benchmark and generation configuration. MiniWoB++ and WorkArena-L1 tables include method, base model, and hinter model; WebArena-Lite includes method and hinter model.

E.1 Hint stats

  • MiniWoB++ hint-database statistics are reported by method, base model, and hinter model.Source: Table 5 · p. 22
  • WorkArena-L1 hint-database statistics are reported by method, base model, and hinter model.Source: Table 6 · p. 22
  • WebArena-Lite hint-database statistics are reported by method and hinter model.Source: Table 7 · p. 22

E.2 Hint examples

The examples show hint-associated performance gains on individual tasks. The accompanying algorithms describe generating a hint database from trace evidence and retrieving hints at episode or step level before acting.

E.2 Hint examples

  • Performance on MiniWoB++ miniwob.click-scroll-list improves from 0.6 to 1 after applying a hint.Source: §E.2 · p. 23
  • Performance on WorkArena-L1 workarena.servicenow.all-menu improves from 0 to 1 after applying a hint.Source: §E.2 · p. 24
  • Performance on WebArena-Lite webarenalite.288 improves from 0 to 1 with GPT-5 as hinter and GPT-5-mini as base model.Source: Figure 11 · p. 24

F Hint generation and retrieval algorithm

  • Hint generation selects single, paired, or multi-trace evidence, optionally zooms into critical steps, and produces source-linked natural-language hints stored with retrieval contexts.Source: F Hint generation and retrieval… · p. 25
  • Retrieval supports goal-conditioned episode-level or contextual step-level modes, after which the agent executes actions and receives observations and rewards.Source: F Hint generation and retrieval… · p. 25

Related papers

Loading 2510.04373v2…