PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Test-Time Adaptation through Human-AI Interaction

Zora Zhiruo Wang, Apurva Gandhi, Rulin Shao, Aspen Chen, Jonas Mueller, Zhiqi Liang, Jett Chen, Michael Ryan, Qianou Ma, Luxi He, Zhoujun Cheng, Andre He, Seungone Kim, Jiayi Geng, Mingqian Zheng, Weiwei Sun, Zheyuan Zhang, Xinran Zhao, Yike Wang, Abe Hou, Liwei Jiang, Pang Wei Koh, Diyi Yang, Graham Neubig, Daniel Fried

arXiv:2609.04141v1cs.AI

TL;DR

LLM agents often fall short of professionals’ personal standards, while recurring human-agent interaction signals can reveal criteria that users cannot fully specify upfront. TAHI uses those signals for context and weight adaptation and evolving rubric creation, improving individual task success, rubric failure detection, and cross-user performance.

  • Problem

    LLM agents produce broadly capable outputs that often do not meet professionals’ personal standards, while recurring criteria revealed through human-agent interaction remain underused.

  • Method

    TAHI streams human-agent interaction signals into context and weight adaptation while an evolving verifier crystallizes human activity into evaluation rubrics.

  • Results

    Agents improve solo task success by 4.5–20.9% within 20 sessions, evolved rubrics capture 16.0–22.3% more failures, and adapted agents gain up to 8.8% across users.

  • Takeaways & Limitations

    The results support continuously adapting agents from test-time interaction to capture both community expertise and individual expertise.

  • Takeaways & Limitations

    Offline adaptation can use outdated human feedback and therefore produce smaller gains than online adaptation, motivating future expertise consolidation.

Abstract

from arXiv · show

AI agents are trained on population-scale data to encode broad capabilities spanning those of many practitioners. Yet the artifacts they produce rarely meet the personal bar professionals need to stake their reputation on. On realistic, open-ended tasks where success criteria are heterogeneous and insufficiently documented, individual expertise lives precisely in the elevation and departure from the average. In practice, iterative human-agent interaction surfaces criteria that users cannot fully specify up front, yet apply repeatedly across tasks. We argue this cross-session interaction data is a rich, underused signal for closing the gap to individual expertise. In this work, we propose test-time adaptation through human-agent interaction (TAHI), which integrates these signals into agent context and weights, and crystallizes each user's training and evaluation criteria via an evolving rubric module. We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks. Our agents improve solo task success by 4.5-20.9% within only tens of tasks. Meanwhile, our evolving rubric module serves as a scalable annotation tool, creating evaluation rubrics that catch 16.0-22.3% more failures than those from LMs or humans alone. While agents are adapted towards individuals, we show these personalized agents also produce improvements in success of up to 8.8% that generalize across users.

1. Introduction

TAHI addresses the gap between broad agent capabilities and individual professional standards by using recurring human-agent interaction signals to adapt agents and evolve evaluation rubrics. Across writing and visual creation, the approach improves individual task success and produces more comprehensive rubrics while also capturing shared and personal expertise.

  • Motivation: Population-trained agents often fall short of professionals’ standards on realistic open-ended tasks with heterogeneous, incompletely documented criteria.Their outputs can become homogeneous or drift from users’ original opinions, requiring further editing.
  • Motivation: Recurring signals from multi-session human-agent interaction provide an underused learning signal for adapting agents to criteria users cannot fully specify upfront.These signals include continual suggestions and corrections across task sessions.
  • Approach: TAHI streams interaction signals into context and weight adaptation while an evolving verifier converts human activity into concrete evaluation criteria.The framework supports editable contexts, DPO-based LoRA weight updates, and an interface with multiple interaction channels.
  • Results: 4.5–12.9% and 4.5–20.9% improvements in solo task success rate are achieved with context and weight adaptation, respectively, within only 20 task sessions.On held-out tasks, the corresponding improvements are 3.2–6.6% and 4.8–5.2% on rubrics tied to the same humans.
  • Results: 16.0–22.3% more failures are captured by evolved rubrics than by rubrics produced by LLMs or humans alone.The comparison concerns imperfect agent solutions and reflects greater rubric comprehensiveness.
  • Results: Individually adapted agents capture both shared community expertise and personal expertise, with gains across users on shared and personal rubric items.Reported gains are 0.3–8.8% on shared items and 6.2–19.6% on personal items.

2. Efficient Test-Time Adaptation through Human-AI Interaction

The paper models human-agent interaction as an iterative, multi-action process and uses those signals to adapt agents across sequential tasks. TAHI combines inspectable context adaptation, persistent weight adaptation, interface-supported feedback, and evolving verification rubrics.

  • Agent Adaptation Methods: Context adaptation stores factual, preferential, and procedural knowledge, while weight adaptation trains model parameters to produce more human-guided outputs.Weight-based adaptation can potentially internalize implicit patterns that are difficult to articulate as text.
  • Human-Agent Interaction: Human-agent task sessions interleave agent and human action trajectories, with later human activity guiding subsequent agent actions.A session ends when the user takes no further action or does not trigger another agent round.
  • Human-Agent Interaction: The interface supports messaging, direct artifact editing, plan and rubric editing, verification, context editing, and triggering agent actions.These channels let users provide both verbal and nonverbal supervision signals.
  • Evolving Rubrics: Evolving verification rubrics convert human activity into gradable criteria and provide quality signals across task iterations.Initial rubrics are generated by an LM, then updated using later human activity; the resulting quality measure is intended to improve as iterations proceed.
  • Test-Time Adaptation: TAHI streams completed task sessions into adaptation, updating the agent after each task before it addresses the next one.The adaptive agent begins with an unadapted model and empty context, then uses each completed session to produce the next agent version.

3. Experiment: Adapting Agents to Human Expertise

The experiment adapts agents to individual expertise through context or weight updates, using human interaction across writing and data-visualization tasks. Both approaches improve solo performance and reveal trade-offs in generalization, efficiency, and which expertise themes agents absorb.

  • 3.1. Tasks, Setup, and Evaluation: The study collects 600 task interactions from 30 humans across paper-abstract writing and data-visualization scenarios, with context-adaptive, weight-adaptive, and non-adaptive groups.Each participant completes 20 tasks; the study also evaluates held-out tasks and recruits domain-relevant participants.
  • 3.2. Agents Can Efficiently Adapt to Human Expertise via Interaction: Within 20 sessions, context and weight adaptation improve solo success on writing tasks by 4.5% and 4.5%, and on data-visualization tasks by 12.9% and 20.9%.All paired tests are statistically significant, with t > 2.0.
  • 3.2. Agents Can Efficiently Adapt to Human Expertise via Interaction: On held-out tasks, context- and weight-adaptive agents improve success by 3.6% and 5.7% in writing, and 2.4% and 5.8% in data visualization.The reported improvements are on tasks unseen during data collection and adaptation.
  • 3.3. Benefiting from Human Interaction Signals Beyond Feedback: Adding non-message human actions yields 3.0% more gains in writing and 7.6% more gains in data visualization than using message actions alone.The comparison uses offline context adaptation to isolate the contribution of actions beyond text feedback.
  • 3.4. Weight Adaptation Has Greater Inference-Time Efficiency than Context Adaptation: Weight-adaptive agents use 62.6% and 88.3% fewer input tokens than context-adaptive agents on writing and data visualization, respectively.They also produce fewer output tokens while achieving better results, whereas context-adaptive agents produce longer responses.
  • 3.5. What Agents Learned, and Failed to Learn, from Humans: Agents absorb 70.4–73.7% of surface-form and terminology expertise but only 41.9–43.4% of context-specificity and problem-framing expertise.Data-visualization adaptation leaves 60.1% and 50.0% of color and visual-encoding expertise uncaptured, while 94.4% of HTML-convention requirements are fulfilled.

4. Eliciting Human Expertise with Scalable Rubric Evolvement

The evolving verifier uses human-agent interaction to produce more comprehensive rubrics for open-ended tasks, capturing both shared community standards and individual preferences. The section also explores how individualized expertise might be consolidated across users.

  • 4.1. Rubrics Derived from Human-Agent Interaction Capture More Failures than LLM-Only and Human-Only Rubrics: LLM-only and human-only rubrics each have limitations for evaluating open-ended tasks, motivating interaction-derived rubric evolution.LLM rubrics can miss expert-valued aspects, while human annotation is inconsistent and costly.
  • 4.1. Rubrics Derived from Human-Agent Interaction Capture More Failures than LLM-Only and Human-Only Rubrics: 16.0–22.3% more failures are captured by evolved rubrics than by rubrics produced by LLMs or humans alone.The comparison evaluates imperfect agent solutions using LLM-produced, human-written, and evolved rubric sets.
  • 4.1. Rubrics Derived from Human-Agent Interaction Capture More Failures than LLM-Only and Human-Only Rubrics: 98.3% and 88.7% are the average scores under LLM-only rubrics for writing and data visualization, compared with 82.3% and 66.4% under evolved rubrics.Lower evolved-rubric scores indicate that these rubrics identify failures missed by the LLM-only baselines.
  • 4.2. Rubrics Capture Shared Community Expertise and Individual Expertise: Different users produce different solutions and rubrics, reflecting the absence of a single universal notion of correctness in open-ended tasks.Community guidelines still provide a notion of relative correctness despite interpersonal variation.
  • 4.2. Rubrics Capture Shared Community Expertise and Individual Expertise: 67.9% and 70.7% of user rubrics fall under shared community guidelines for writing and data visualization, respectively.The remaining criteria represent personalized expertise and preferences, including differing priorities for implementation details or statistical rigor.
  • 4.3. Agent Learns Both Shared and Personalized Expertise: Agents adapted to individual experts improve on both personalized and shared criteria, with gains of up to 8.8% on shared criteria.This result indicates improvement beyond fitting only the preferences of the individual expert used for adaptation.
  • 4.4. Exploration: Deriving Community Expertise from Individual Adaptations: Individualized agents can be merged through shared context or weight consolidation, but effective expertise consolidation across users remains challenging.The explored approaches include merging interaction-derived contexts, pooling data for retraining, and averaging LoRA adapter parameters.
  • 4.4. Exploration: Deriving Community Expertise from Individual Adaptations: Offline adaptation may yield smaller gains because its human feedback can be outdated relative to the agent’s current state.The section reports that merging methods improve data visualization more consistently than writing, with variation in preserving shared and personal criteria.

5. Related Work

Prior work has developed test-time context and weight adaptation, human-signal learning, and scalable agent training, but open-ended tasks remain difficult to evaluate and support with high-quality data.

  • Test-Time Agent Adaptation: Test-time adaptation commonly updates agent context with procedural, factual, or preferential knowledge, while weight adaptation remains less explored.Context adaptation is favored for interpretability and implementation simplicity, whereas weight updates face data and inspection challenges.
  • Learning from Human Signals: Human-signal learning includes preference optimization, natural-language feedback, and imitation of demonstrations, but UI-level interaction differs substantially from these signals.The cited line of work primarily addresses instruction following or demonstration imitation rather than the full interaction setting considered here.
  • Evaluating Agents on “Non-Verifiable” Tasks: Open-ended creative and visual tasks lack execution-based verification, creating challenges for alternative evaluation methods and scalable training data.Existing agent training work targets common purposes such as software engineering and web navigation, while human-sourced data can be expensive to prepare at scale.

6. Conclusion

The paper introduces test-time adaptation through human-agent interaction using context and weight updates, together with an evolving verifier for open-ended tasks. Its experiments show adaptation within 20 interactions and capture of both community and individual expertise.

  • Conclusion: The framework performs test-time context and weight updates from human-agent interaction data, while its evolving verifier generates more comprehensive rubrics.The verifier automatically incorporates interaction signals during evaluation of realistic open-ended tasks.
  • Conclusion: Within 20 interactions, adapted agents improve individual task success, and both agents and rubrics capture shared community expertise alongside user-specific expertise.The conclusion presents these as the paper’s central empirical findings.

A. Data Collection

The data collection covers 20 interaction-data tasks and 30 held-out tasks, with task venues, topics, and source distributions documented separately.

  • Data Collection: 20 tasks provide human-agent interaction data, while 30 held-out tasks supply a varied distribution for testing generalization.The task venues, topics, and held-out-task sources are listed in Tables 5 and 6.

B. Exploration on Weight-Based Adaptation Approaches

The authors explored weight-adaptation approaches and selected DPO for the main experiments because it performed best.

  • DPO was selected for the main experiments after broad exploration of weight-adaptation approaches because of its superior performance.

B.1. Alternative Algorithms

The paper compares OPD and REINFORCE with DPO as alternative weight-adaptation algorithms. Preliminary experiments found both alternatives underperformed DPO, with REINFORCE rarely exploring high-scoring solutions.

  • OPD applies a token-level KL-divergence target from a teacher policy to the current agent policy.The paper uses on-policy self-distillation, where the teacher has access to later agent and human activities in the session.
  • REINFORCE uses scalar rewards from agent trajectories, optionally incorporating quantified human activities.The collective reward subtracts a human-activity penalty weighted by λ, and the baseline is the previous iteration's reward.
  • Both OPD and REINFORCE performed worse than DPO in preliminary experiments.
  • REINFORCE agents rarely explored high-scoring solutions under evolved human-feedback rubrics and repeatedly produced solutions scoring around 0.3–0.4 out of 1.The authors conjecture that limited exposure to expert-style solutions makes exploration and hill-climbing difficult without additional execution hints.

B.2. DPO Data Formulation

The authors compared three strategies for constructing pairwise DPO data and adopted first-last pairing after finding it performed best.

  • First-last pairing combines the first and last solution, enumerate pairs every possible earlier and later solution, and min-gap-k requires at least k iterations between paired solutions.
  • First-last pairing performed best and was adopted for offline and on-policy pair construction in the main approach.

C. Human Expertise Learned During Agent Adaptation

Tables 7 and 8 describe the expertise categories shown in Figure 5 for writing and data-visualization tasks.

  • Table 7 describes the expertise categories for writing tasks shown in Figure 5.
  • Table 8 describes the expertise categories for data-visualization tasks shown in Figure 5.

D. Rubrics Validation

The rubric-validation results support the quality of evolved rubrics for both detecting imperfect solutions and signaling task success. The appendix specifies prompt modules for inducing memory and skills, comparing and grading rubrics, and summarizing expertise across writing and visualization tasks.

  • D. Rubrics Validation: Evolved rubrics are as effective as LLM- and human-produced rubrics at revealing task success.The validation compares rubric-based evaluation of oracle solutions and finds all three rubric sources effective.
  • D. Rubrics Validation: LLM-only, human-only, and evolved rubrics all signal task success through scores around high scores on oracle solutions.Together with Figure 6, this supports the conclusion that failures on imperfect agent solutions reflect solution problems rather than evaluation-criteria problems.
  • D. Rubrics Validation: Writing-task expertise categories are documented as observed human expertise categories.Table 7 provides the writing-domain expertise taxonomy used in the validation material.
  • D. Rubrics Validation: Data-visualization expertise categories are documented separately as observed human expertise categories.Table 8 provides the corresponding visualization-domain taxonomy.
  • E. Prompts for LM-Supported Modules: The memory-induction prompt distills recurring user-specific verifier criteria into a joint general verifier list.It combines user-edited task-specific verifiers with fixed human-written visualization guidelines covering layout, labels, legends, annotations, spacing, axes, colors, validity, and renderability.
  • E. Prompts for LM-Supported Modules: Additional prompts merge durable cross-session memory, grade output files against verifier criteria, and summarize test-time task rubrics.The memory prompt preserves durable preferences while limiting duplication, the grading prompt returns criterion-level JSON results, and the rubric-summary prompt supports expertise extraction.
  • E. Prompts for LM-Supported Modules: The skill-induction prompt updates a generalized workflow from session logs while retaining only accurate and useful prior steps.It requires new session work, such as edits, fixes, chart changes, and verification, to be incorporated into the full updated skill.
  • E. Prompts for LM-Supported Modules: The shared-rubric prompt compares source and target verifier lists by deciding whether each source requirement is covered, paraphrased, or subsumed.Coverage is assigned per source rubric for the same abstract-writing task.
Loading 2609.04141v1…