PaperlayerView exact revision on arXiv

Source-linked AI summary

TAI3: Testing Agent Integrity in Interpreting User Intent

Shiwei Feng, Xiangzhe Xu, Xuan Chen, Kaiyuan Zhang, Syed Yusuf Ahmed, Zian Su, Mingwei Zheng, Xiangyu Zhang

arXiv:2506.07524v3cs.SEcs.AIcs.CY

TL;DR

LLM agents can misinterpret ambiguous natural-language instructions, causing API actions to diverge from user intent. TAI3 uses API-based semantic partitioning, intent-preserving mutations, lightweight ranking, and strategy memory to test this integrity. Across 80 toolkit APIs, it finds violations efficiently, transfers strategies across domains, generalizes across models, and is limited when API-calling trajectories are hidden.

  • Problem

    Natural-language ambiguity and evolving toolkits make fixed benchmarks and structured software testing inadequate for evaluating whether agents preserve user intent.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    TAI3 partitions API parameter domains semantically, mutates seed tasks while preserving intent, ranks mutations by predicted risk, and adapts stored strategies.

    Source: §1 · p. 2, §4.1 · p. 6, §4.2 · p. 7

  • Results

    TAI3 uncovers diverse intent integrity violations across 80 toolkit APIs, outperforms baselines in error-exposing rate and query efficiency, and transfers strategies across APIs and domains.

    Source: §1 · p. 2, §5.2 · p. 8, §5.3 · p. 10

  • Takeaways & Limitations

    TAI3 provides systematic and efficient testing of API-calling agents across models and toolkits under realistic benign usage.

    Source: §6 · p. 10, §5.3 · p. 10

    TAI3 requires access to the agent’s API-calling trajectory, limiting applicability to commercial agents that expose only high-level outputs.

    Source: §6 · p. 10

Abstract

from arXiv · show

LLM agents are increasingly deployed to automate real-world tasks by invoking APIs through natural language instructions. While powerful, they often suffer from misinterpretation of user intent, leading to the agent's actions that diverge from the user's intended goal, especially as external toolkits evolve. Traditional software testing assumes structured inputs and thus falls short in handling the ambiguity of natural language. We introduce TAI3, an API-centric stress testing framework that systematically uncovers intent integrity violations in LLM agents. Unlike prior work focused on fixed benchmarks or adversarial inputs, TAI3 generates realistic tasks based on toolkits' documentation and applies targeted mutations to expose subtle agent errors while preserving user intent. To guide testing, we propose semantic partitioning, which organizes natural language tasks into meaningful categories based on toolkit API parameters and their equivalence classes. Within each partition, seed tasks are mutated and ranked by a lightweight predictor that estimates the likelihood of triggering agent errors. To enhance efficiency, TAI3 maintains a datatype-aware strategy memory that retrieves and adapts effective mutation patterns from past cases. Experiments on 80 toolkit APIs demonstrate that TAI3 effectively uncovers intent integrity violations, significantly outperforming baselines in both error-exposing rate and query efficiency. Moreover, TAI3 generalizes well to stronger target models using smaller LLMs for test generation, and adapts to evolving APIs across domains.

1 Introduction

TAI3 addresses the difficulty of testing whether natural-language API agents preserve user intent, where ambiguity and evolving toolkits limit traditional testing. It uses API-grounded semantic partitions, intent-preserving mutations, and adaptive ranking to expose violations efficiently.

Motivation

  • Natural-language ambiguity makes minor agent misinterpretations capable of producing incorrect, unexpected, or unsafe API behavior.The paper calls this the intent integrity problem and distinguishes it from higher-level safety issues.Source: §1 · p. 1
  • Existing benchmarks, adversarial techniques, and classical software testing do not adequately test benign intent preservation across evolving toolkits.Fixed cases become outdated, adversarial methods target jailbreaks or injection, and structured input-output assumptions do not transfer to open-ended language.Source: §1 · pp. 1–2

TAI3 Framework

  • TAI3 models agent behavior through toolkit API structure, enabling measurable testing across API-exposed functionality.The framework treats parameterized API calls as a formal description of the agent’s behavior space.Source: §1 · p. 2

Semantic Partitioning

  • Semantic partitioning organizes API parameter values into meaningful VALID, INVALID, and UNDERSPEC categories and their equivalence classes.This provides a precise specification of the behavior space, including valid, invalid, and context-dependent expressions.Source: §1 · p. 2

Mutation and Adaptation

  • TAI3 mutates unambiguous seed tasks while preserving intent, ranks mutations with a lightweight predictor, and retrieves successful strategies from memory.The predictor estimates error-triggering likelihood, while strategy memory adapts effective mutation patterns from prior cases.Source: §1 · p. 2

Evaluation

  • Across 80 toolkit APIs, TAI3 uncovers diverse intent integrity violations, outperforms baselines in error-exposing rate and query efficiency, and generalizes to stronger models using smaller LLMs.The evaluation covers five domains and reports successful generation of error-inducing cases with smaller models.Source: §1 · p. 2

2 Related Work

Prior work tests NLP robustness, agent misuse, and autonomous-system components, but TAI3 targets benign API-agent use where low-level actions must remain semantically aligned with user intent.

2.1 Testing NLP System

  • NLP testing has progressed from meaning-preserving perturbations to general robustness, adaptive testing, and prompting-based red-teaming, while TAI3 evaluates benign normal-use tasks.The distinction is whether agents behave as expected rather than whether they resist attacks.Source: §2.1 · p. 3

2.2 Red-teaming LLM Agents

  • Agent red-teaming commonly targets jailbreaking, prompt injection, and potential misuse rather than semantic consistency between benign user intent and API actions.These lines of work probe unsafe responses, adversarial instructions, data poisoning, policy violations, and other misuse risks.Source: §2.2 · p. 3
  • ToolFuzz tests tool documentation and implementation bugs, whereas TAI3 tests semantic inconsistencies between API calls and the user’s original intent.PDoctor checks high-level planning against domain constraints, complementing TAI3’s focus on low-level action alignment.Source: §2.2 · p. 3

2.3 Robustness of Autonomous System

  • Autonomous-system robustness research emphasizes perception, planning, and software components, while LLM agents introduce robustness challenges in reasoning, multi-step planning, and tool calling.TAI3 addresses this newer dimension by testing intent integrity under contextual perturbations.Source: §2.3 · p. 3

3 Motivation

A smart-lock API illustrates how small linguistic variations or missing details can make agents violate valid, invalid, or underspecified user intent. TAI3 motivates systematic testing because neither traditional software testing nor high-level safety benchmarks reliably expose these failures.

Examples

  • The GrantGuestAccess() example requires an agent to map guest identities, permanence, and time bounds from natural-language instructions to API parameters.Figure 2 presents the API documentation, representative failures, and a parameter partition with 3 categories and 14 equivalence classes.Source: §3 · p. 3, Figure 2 · p. 4
  • A VALID request for one-hour access can trigger an impossible date, revealing temporal-reasoning failure despite a clear user instruction.The example uses an incorrectly generated 2022-02-29 date in a non-leap year.Source: §3 · p. 4
  • An INVALID request for unsupported recurring access can be mis handled as permanent access instead of being rejected or flagged.The agent fails to recognize that the requested feature is outside the API’s supported behavior.Source: §3 · p. 4
  • An UNDERSPECIFIED request for access “tomorrow” can be overinterpreted as full-day access rather than prompting for the missing time range.This may exceed the user’s intended window and illustrates why clarification behavior must be tested.Source: §3 · p. 4

Challenges

  • Systematically finding these failures is difficult because structured software tests do not generalize to natural language and high-level safety benchmarks overlook evolving API-function integrity.The motivation therefore calls for stress testing API-calling agents under realistic benign usage.Source: §3 · p. 4

4 Design of TAI3

TAI3 organizes API-driven agent testing around semantic partitions, intent-preserving mutations, and adaptive strategy reuse to expose intent integrity violations efficiently.

4 Design of TAI3

  • TAI3 models testing as finding intent-preserving mutations whose agent execution diverges from the original task's trajectory.Source: §4 · p. 5

4.2 Intent-Preserving Mutation

  • Intent-preserving mutation filters variants for consistency with the original intent, then ranks retained tasks by estimated error likelihood before executing the highest-risk cases.The lightweight model treats lower intent-reconstruction likelihood as indicating greater ambiguity and prioritizes those mutations to reduce costly agent runs.Source: §4 · p. 5, §4.2 · pp. 6–7

4.1 Semantic Partitioning

  • Semantic partitioning divides API parameter domains by VALID, INVALID, and UNDERSPEC categories and finer-grained equivalence classes, producing cells for realistic seed tasks.The resulting parameter-partition form provides structured coverage of semantically distinct input situations.Source: §4.1 · p. 6

4.3 Evergreen Strategy Memory & Adaptation

  • TAI3's strategy memory records novel mutation patterns that trigger errors and retrieves the top three datatype- and integrity-category-relevant strategies for later adaptation.This lets mutation reuse effective patterns across tasks rather than restarting from scratch.Source: §4 · p. 5, §4.3 · p. 7

5 Evaluation

TAI3 is evaluated across diverse toolkit APIs using intent-integrity categories, query-based metrics, and multiple target models. It consistently improves error discovery and efficiency over SelfRef, broadens semantic coverage beyond existing benchmarks, and generalizes to stronger agents.

5.1 Experiment Setup

  • The evaluation covers 80 toolkit APIs and 233 parameters across five domains, with additional analyses of task realism, strategy transferability, product-level agents, ablations, sensitivity, and prompts.The tested domains are finance, healthcare, smart home, logistics, and office.Source: §5.1 · p. 7, §5.3 · p. 10

5.2 Results

  • TAI3 outperforms SelfRef across domains and intent categories, improving EESR by up to 15.5 points in Finance and 10.0 points in Office.These gains occur under a fixed query budget and support the effectiveness of guided sampling and targeted mutation.Source: §5.2 · p. 8
  • TAI3 reduces AQFF across all input categories and target agents, with reductions of up to 12% in the UNDERSPEC setting.AQFF measures the average queries needed to trigger the first failure.Source: §5.2 · p. 8, Figure 6 · p. 9
  • Error-likelihood ranking triggers more errors than Random and Select Last 5 across all tested values of k.The predictor ranks 15 generated candidates and selects 5 for evaluation.Source: §5.2 · pp. 8–9, Figure 5 · p. 9
  • Existing benchmarks leave substantial semantic space uncovered: A-SB has sparse INVALID and UNDERSPEC coverage, while ToolEmu lacks INVALID cases.TAI3 partitions API inputs into VALID, INVALID, and UNDERSPEC categories to provide structured coverage.Source: §5.2 · p. 9, Table 2 · p. 9
  • Weaker testing models can expose failures in stronger targets, while GPT-4o-mini performs best in Top-5 settings and open-source targets show higher EESR than closed-source targets.The comparison includes Llama-3.3-70B and DeepSeek-R1-70B versus Claude-3.5-Haiku, Gemini-2.5-Pro, and GPT-o3-mini.Source: §5.2 · pp. 9–10, Figure 7 · p. 10

6 Conclusion

TAI3 combines semantic partitioning, intent-preserving mutations, and strategy adaptation to uncover intent integrity violations efficiently across models and toolkits, while requiring API-calling trajectories.

  • TAI3 uncovers a wide range of intent integrity violations with high efficiency and generalization across models and toolkits.Source: §6 · p. 10
  • Access to the agent’s API-calling trajectory limits TAI3’s applicability to commercial agents exposing only high-level outputs.Extending TAI3 to restricted-observability settings remains future work.Source: §6 · p. 10

NeurIPS Paper Checklist

The checklist records the paper’s claims, reproducibility information, experimental details, limitations, ethics, societal impacts, and planned release of code and assets.

NeurIPS Paper Checklist

Appendix

  • The appendix covers evaluation setup, semantic partitioning, task naturalness, mutation transferability, product-level violations, ablations, hyperparameters, and mutation examples.Source: Appendix · p. 25

A Evaluation Setup

The evaluation selects toolkits across five domains and emphasizes balanced datatype coverage to assess TAI3’s generalization and type-aware components; existing-benchmark partition coverage is also tabulated.

  • Toolkits span Finance, Healthcare, Smart Home, Logistics, and Office, with each domain including at least five parameter-field instances per type.Source: A Evaluation Setup · p. 25
  • The setup emphasizes balanced datatype coverage, including Enum and Array, to evaluate datatype-aware components fairly.Source: A Evaluation Setup · p. 25
  • The evaluation is designed to support analysis of domain-level generalization and datatype-specific behavior.Source: A Evaluation Setup · p. 25
  • Table 4 reports VALID, INVALID, and UNDERSPEC partition coverage, constructed partition counts, and corresponding benchmark test cases.Source: Table 4 · p. 26
  • Figure 8 presents an example partition-parameter form for the PayBill API from ToolEmu.Source: Figure 8 · p. 26

B Evaluation on Semantic Partitioning

TAI3 constructs API-grounded semantic partitions, classifies benchmark inputs into them, and measures coverage across VALID, INVALID, and UNDERSPEC categories.

C Naturalness and Benignness of Our Mutated Tasks

TAI3's mutated tasks remain close to natural user language while exposing intent errors, and its learned strategies transfer across domains but depend more strongly on datatype semantics.

C Naturalness and Benignness of Our Mutated Tasks

  • Low perplexity keeps mutated tasks close to natural seed tasks and far from jailbreak prompts across VALID, INVALID, and UNDERSPEC intents.This supports their characterization as natural-language, benign stress tests rather than adversarial prompts.Source: C Naturalness and Benignness of Our… · p. 27, Figure 9 · p. 27

D Strategy Transferability

  • Cross-domain transfer preserves relatively high EESR for strategies from Finance and Health applied to domains such as Home and Logistic.The result indicates reusable failure-inducing patterns across task environments without requiring domain-specific retraining.Source: D Strategy Transferability · pp. 27–28, Figure 10 · p. 28
  • Datatype transfer usually degrades substantially, but Bool strategies transfer well to Enum parameters and outperform Enum's native strategies.The paper attributes this exception to structural similarity between boolean values and binary enumerations, motivating type-aware retrieval and possible type clustering.Source: D Strategy Transferability · p. 28, Figure 11 · p. 28

E Case Study on Product-level Agents

  • TAI3 was evaluated on Self-Operating Computer and Proxy AI after mutated tasks were manually transferred into their nonstandard input formats.These product-level agents represent an open-source computer-using agent and a commercial email assistant.Source: E Case Study on Product-level Agents · p. 28

E.1 Self-Operating Computer

The product-level cases show benign instructions being misinterpreted in ways that select the wrong email or create an unauthorized communication path.

E.1 Self-Operating Computer

  • The Self-Operating Computer agent replies to an open, read email instead of locating the requested unread emails.It also makes a critical personal-life decision without the user's knowledge or authority.Source: §E.1 · p. 28
  • The Self-Operating Computer trajectory ignores the explicit unread-email constraint and acts on its own.The expected behavior was to navigate to the inbox, using the back button or a new tab.Source: §E.1 · p. 28

E.2 Proxy AI

  • The Proxy AI case asks the agent to reply to Sarah's existing email with confidential Q3 financial projections.The supplied expected behavior is to share the information by replying to that existing message.Source: §E.2 · p. 29
  • The Proxy AI figures depict a conscious choice to compose a new email and a resulting leak of confidential data to an arbitrary address.These figures connect the intended reply context with a distinct, unauthorized communication failure.Source: Figure 14 · p. 30, Figure 15 · p. 30

F Ablation Study

The ablation study finds measurable benefits from both prediction and retrieval, while their combination achieves the complete method's best performance.

F Ablation Study

  • Both SelfRef+Predict and SelfRef+Retrieve outperform the SelfRef baseline, but neither alone reaches the complete method's performance.Table 5 reports the EESR comparison across variants.Source: F Ablation Study · p. 31, Table 5 · p. 31
  • Prediction prioritizes promising mutations, whereas retrieval supplies higher-quality candidates, making the two components complementary.Retrieval improves what the predictor can score, while prediction helps the system identify useful retrieved mutations earlier.Source: F Ablation Study · p. 31

G Sensitivity Analysis

TAI3 uses a small retrieved strategy set and task-agnostic mutation prompts to preserve user intent while generating realistic, transferable stress tests, with broader-impact risks from misuse.

G Sensitivity Analysis

  • Top 3 retrieved strategies consistently achieve the best EESR across tested models, while too few or too many strategies reduce effectiveness.The paper selects 3 as the default because top-1 retrieval can miss relevant patterns and larger sets can introduce conflicting ones.Source: G Sensitivity Analysis · p. 31, Figure 16 · p. 31

H Strategy Examples

  • The strategy examples mutate ambiguity differently across Finance, Health, Home, Logistics, and Office tasks.Examples include doubt about account validity, relative dates, device synonyms, vague travel comfort, and imprecise priority descriptors.Source: H Strategy Examples · pp. 31–32

I Prompt Templates

  • The testing prompts instruct the generator to keep tasks understandable, benign, and recognizable as ordinary user requests while making them more confusing to the target agent.The templates explicitly frame mutation as intent-preserving testing rather than changing the expected result.Source: I Prompt Templates · p. 32
  • The prompt templates require common daily scenarios, preserve the original intent of the targeted and other fields, and produce concise transferable strategies in JSON.They also use prior failed mutations for reflection and improvement.Source: I Prompt Templates · pp. 32–33

J Broader Impact

  • The techniques aim to improve agent reliability and trustworthiness but could also be misused to identify weaknesses or deliberately exploit agent behavior.This is the paper's stated broader-impact risk.Source: J Broader Impact · p. 33
Loading 2506.07524v3…