Source-linked AI summary
TIDE: Proactive Multi-Problem Discovery via Template-Guided Iteration
Soyeong Jeong, Jinheon Baek, Minki Kang, Sung Ju Hwang
TL;DR
Agents often wait for explicit requests even though multiple hidden problems may coexist in their accessible context. TIDE addresses this with reusable thought templates and iterative discovery, and consistently outperforms single-shot and multi-agent baselines across settings and backbones. The paper also identifies fixed templates and the bounded budget of iteration as directions and scope considerations for further work.
Problem
Agents typically act on explicit requests, while multiple hidden problems may coexist in context and remain undiscovered when users have not articulated them.
Method
TIDE combines reusable thought templates that organize contextual evidence with iterative discovery conditioned on cumulative findings, producing grounded problem candidates and concrete resolution actions.
Results
Across personal workspaces and software repositories, and across four LLM backbones, TIDE consistently outperforms single-shot and multi-agent baselines on retrieval, identification, and resolution.
Takeaways & Limitations
The findings support framing proactive assistance as an explicit multi-step discovery process over context rather than single-shot prediction from a user request.
Takeaways & Limitations
Templates are built once and fixed at inference, while iterative discovery uses a bounded budget; online template updates and further study of the trade-off remain open directions.
Abstract
from arXiv · showhide
Agents are widely deployed as assistants over documents, tools, and code. However, they typically act only on explicit user requests, which surface only the problems the user has noticed, while many other important problems coexist, hidden in plain sight, within the broader user context, with their total number unknown in advance. We frame this as the task of discovering multiple hidden problems from context, in which coexisting problems should be uncovered, grounded in supporting evidence, and paired with concrete actions. To this end, we introduce TIDE, a template-guided iterative framework with two complementary mechanisms. Specifically, motivated by the observation that single-pass prediction anchors on the most salient cases and yields generic claims, we propose iterative discovery, which surfaces a small batch of candidates per round while conditioning on what has already been found, so subsequent rounds extend coverage; and thought templates, reusable schemas distilled from previously solved cases that specify what contextual signals to attend to and how to connect them, anchoring each prediction in a recognizable problem class. We validate TIDE on two realistic settings, personal workspaces and software repositories, across four model backbones, showing substantial gains over single-shot and parallel multi-agent baselines on task coverage, identification, and resolution.
1 Introduction
TIDE reframes proactive assistance as discovering multiple hidden problems in user context, then grounding and addressing them rather than waiting for explicit requests. It combines iterative discovery with thought templates and improves coverage, identification, and resolution across two settings and four backbones.
- Reactive agents act only on explicit requests, assuming users already know what is wrong and what to ask.
- Important problems can remain hidden across documents, emails, and calendar entries, with multiple issues coexisting and their number unknown in advance.
- TIDE combines iterative discovery, reusable thought templates, evidence retrieval, and concrete actions to produce actionable plans for surfaced problems.
- Across personal workspaces and software repositories, TIDE consistently outperforms single-shot and parallel multi-agent baselines on task coverage, identification, and resolution.The evaluation spans four LLM backbones.
- Iterative discovery and thought templates contribute complementary gains, while templates transfer across model backbones.
2 Method
The method formulates hidden-problem discovery as predicting grounded, actionable problem candidates from context, addressing the coverage and fidelity failures of single-shot generation. TIDE combines reusable thought templates with iterative, cumulative-state-conditioned discovery.
- Task Formulation: The task is to approximate a latent set of hidden problems from documents, although none is an explicit request and the number of problems is unknown.
- Task Formulation: Each prediction describes a candidate problem, cites supporting documents, and proposes a concrete resolution action.
- Single-Shot Discovery: Single-shot discovery sacrifices coverage because salient problems overshadow subtler ones and sacrifices fidelity through generic or speculative claims.
- Thought Templates: Thought templates are reusable schemas distilled from solved cases, specifying a problem-class name, structural pattern, and ordered evidence flow.
- Thought Templates: Templates are constructed once from training cases and held fixed during inference.
- Iterative Discovery and Resolution: Iterative discovery generates small batches over multiple rounds while conditioning each round on the cumulative discoveries, pushing later rounds beyond already found cases.
- Iterative Discovery and Resolution: Each round couples problem identification with evidence retrieval and a proposed action, yielding plans that identify, ground, and address each surfaced problem.
3 Experimental Setup
TIDE is evaluated on personal workspaces and software repositories using multiple complementary metrics and several baselines across four long-context LLMs. The setup measures evidence retrieval, problem identification, and resolution, aggregating scores at the instance level.
- 3.1 Datasets: The evaluation covers personal workspaces and software repositories, extending existing data sources because no benchmark directly targets multi-problem discovery from context.Workspace instances combine user profiles with documents, emails, and calendar entries; repository instances contain multiple unresolved bugs and distractor functions.
- 3.1 Datasets: Repository evaluation includes 146 problems across 20 multi-bug test instances from 11 projects, with 2–41 problems and 6–646 candidate functions per instance.
- 3.2 Methods: The comparison includes SINGLE-AGENT, MULTI-AGENT, SELF-REFINE, REFLEXION, and TIDE under the same backbone LLM and context.
- 3.3 Evaluation Metrics: Each matched gold-prediction pair is scored for retrieval, identification, and resolution using evidence overlap and LLM-judge assessments against gold descriptions and actions.
- 3.3 Evaluation Metrics: Coverage averages matched scores over gold problems, while F1 combines coverage with the analogous average over predictions; both metrics are macro-averaged across instances.
- 3.4 Implementation Details: The study instantiates agents with four long-context LLMs and reports main results over three independent runs.
4 Results and Analyses
Across Workspace and Repository settings, TIDE consistently outperforms single-shot and parallel-agent baselines, while analyses attribute gains to iterative conditioning and reusable thought templates.
- Main Results: TIDE consistently achieves the best performance across retrieval, identification, and resolution under four LLM backbones.Its lead holds across both Workspace and Repository settings, with statistically significant gains in most comparisons.
- Multi-Problem Discovery: TIDE reaches four or more problems in most multi-problem Workspace instances, while baselines recover fewer problems.Across instances with increasing gold-problem counts, TIDE continues recovering most problems as baselines fall further behind.
- Iterative Discovery: TIDE keeps surfacing newly discovered problems across iterations, whereas MULTI-AGENT quickly shifts toward re-discovering salient problems.The methods begin similarly, but MULTI-AGENT’s newly discovered items drop sharply from the second step onward.
- Iterative Discovery: TIDE scales steeply with the per-instance call budget B, while MULTI-AGENT plateaus early and remains below TIDE even at B=10 versus TIDE at B=2.This comparison supports iterative conditioning rather than simply increasing parallel-agent count.
- Thought Templates: Template guidance adds retrieval coverage and a larger precision margin at every iteration compared with the no-template ablation.Templates are distilled from held-out cases that share no project or problem with test instances.
- Thought Templates: Raw few-shot demonstrations fall well short of TIDE across retrieval, identification, and resolution, indicating that abstraction into reusable reasoning patterns matters.The comparison replaces thought templates with demonstrations from the same training pool.
- Template Analysis: Templates transfer comparably across GPT and Gemini, and gains over SINGLE-AGENT grow as the template pool expands.The results suggest that additional templates supply recognizable problem classes rather than merely increasing example exposure.
5 Related Work
Related work studies task-oriented and proactive agents, as well as reusable reasoning templates, but TIDE addresses workflows containing multiple hidden problems rather than a localized user-specified task.
- Task-oriented LLM Agents: Task-oriented LLM agents are commonly evaluated in document, tool-use, web-interaction, and software-engineering environments with an already specified task.Typical goals come from user requests, issue descriptions, failing tests, or other localized instructions.
- Proactive Agents: Proactive-agent research moves beyond reactive interaction by anticipating needs or uncovering intent beyond literal requests.Examples include clarification questions and navigation of unarticulated knowledge gaps.
- Positioning: TIDE differs by framing proactive assistance as discovering multiple coexisting problems whose number is unknown in advance.This targets the broader problem structure present in real workflows.
- Templates for LLM Reasoning: Reusable reasoning templates externalize recurring reasoning patterns so they can be retrieved and applied across problems.This line of work complements, but does not replace, TIDE’s multi-problem discovery setting.
6 Conclusion
TIDE discovers multiple hidden problems through iterative discovery and thought templates, outperforming baselines across settings and backbones while recasting proactive assistance as multi-step context discovery.
- Conclusion: TIDE combines iterative discovery with thought templates to uncover multiple hidden problems from context.The framework targets both personal workspaces and software repositories.
- Conclusion: Across two realistic settings and four backbones, TIDE consistently outperforms single-shot and multi-agent baselines on retrieval, identification, and resolution.Iteration and templates provide complementary gains, and templates transfer across backbones.
- Conclusion: The findings recast proactive assistance as a multi-step discovery process that can surface issues users would not have thought to ask about.The conclusion presents this as a recipe for building proactive agents over context.
Limitations
The current design uses a fixed template library at inference and trades a bounded iterative budget for broader coverage, motivating online template updates and further study.
- Limitations: Templates are built once from solved cases and held fixed at inference, leaving online updates and automatically constructed cases as future directions.The paper reports that the fixed library already transfers across backbones.
- Limitations: Iterative discovery trades a small bounded budget for broader coverage, although the analyses find this trade-off favorable against matched-budget multi-agent baselines.Further investigation of the iterative paradigm remains an open direction.
Ethics Statement
TIDE surfaces hidden problems from working contexts, but its documents and learned templates may contain sensitive, biased, or undesirable content. The paper therefore recommends safeguards during both template construction and deployment.
- TIDE operates on real-world documents and templates distilled from previously solved cases, which may carry sensitive, biased, or undesirable content.
- The paper recommends content filtering, bias detection, and human-in-the-loop review at both template construction and deployment.
A Additional Implementation Details
The implementation controls decoding and template construction consistently across methods and runs. Reported results average over repeated runs to account for nondeterministic provider APIs.
- Decoding: All methods use the same inference code and backbone-specific decoding configuration, with provider-default sampling parameters unchanged.
- Decoding: Because provider APIs do not guarantee deterministic decoding, every Table 1 number averages three independent runs.
- Templates and Rounds: Template pools are distilled once offline and fixed across analyses from held-out training workspaces and multi-bug repository instances.
B Significance of the Gains
TIDE’s gains are evaluated with paired bootstrap comparisons against each baseline. Across both settings and all four backbones, most comparisons reach statistical significance.
- TIDE is paired with each baseline instance by instance, and margins are evaluated using one-sided p-values from a paired bootstrap.
- Most TIDE gains are significant at the 0.05 level across both settings and all four backbones.
C Reliability of the LLM Judge
The paper tests whether TIDE’s results depend on the judge model or template-source separation, and compares it with stronger multi-agent variants and resource usage. Results indicate robust evaluation and consistent performance advantages, while TIDE uses fewer calls at comparable cost in the reported workspace analysis.
- Judge Reliability: Swapping the LLM judge tests whether TIDE’s lead depends on a single evaluator, while human annotation checks agreement with human reading.
- Judge Reliability: Human annotators agree with the judge on most judgments, with substantial inter-annotator agreement of Fleiss κ = 0.80.
- Judge Reliability: Judge scores correlate with human labels at Spearman ρ = 0.92, and majority-label agreement reaches Cohen’s κ = 0.87.
- Template Separation: After removing templates sharing projects with test instances, the repository evaluation is rerun under exact project-level separation.
- Baseline Comparisons: TIDE consistently achieves the best retrieval, identification, and resolution performance across backbones and settings, whereas parallel agents tend to rediscover salient problems.
- Efficiency: Matching discovery calls leaves TIDE’s cumulative state and templates as per-call overhead, but early termination yields fewer calls and less total input at comparable cost.
G Qualitative Study
The qualitative cases show TIDE discovering and resolving the target workspace and repository problems that competing methods miss, using retrieved evidence and reusable templates to guide predictions.
- Workspace case: TIDE reaches the workspace’s VolunteerHub data-integrity issue in a later iteration, retrieves the gold documents, and escalates to the right manager with the relevant deadlines.The action includes the gating AccessHub ticket, vendor-deployment deadline, and presentation deadline.
- Workspace case: SINGLE-AGENT surfaces only an unrelated facility-rider procurement stall and retrieves none of the gold documents, making its identification, action, and addressee incorrect.Its action targets procurement intake rather than the VolunteerHub issue, AccessHub ticket, or metrics deadline.
- Repository case: TIDE identifies the mlxtend callable-metric bug, retrieves the relevant function, names the missing fallback branch, and adds the assignment matching the gold patch.The bug arises because the scoring function is bound only in string branches, so a callable metric reaches an UnboundLocalError.
- Repository case: SELF-REFINE returns a single unrelated bottleneck after repeated rewriting, misses the callable-metric bug, and proposes an edit that no gold patch touches.Its output concerns mcnemar_tables and adds an import for combinations.
- Template guidance: The repository prediction is guided by a template mined from sympy, transferring the same unbound-variable defect pattern across repositories.The template describes conditional initialization without an exhaustive fallback, which can cause UnboundLocalError when no branch matches.