PaperlayerView exact revision on arXiv

Source-linked AI summary

Robustness Analysis of Agentic AI to Inconsistent and Incomplete Tool Responses

Jiachen Xu, Torben Bach Pedersen, Zhongming Yao, Xiaoyu Zhang, Yushuai Li

arXiv:2608.22676v1cs.AI

TL;DR

The paper asks whether agents can distinguish failed tool returns from well-formed falsehoods when those returns first enter context. It uses controlled retail-domain injections and log-probability measurements at single decision points. The results show asymmetric recognition: incompleteness appears in the schema channel and redirects actions toward state rereading, while inconsistency is visible only through particular trajectory conflicts.

  • Problem

    The study asks whether agents can distinguish failed returns from schema-valid falsehoods at the tool-return boundary, where the two faults require different remedies.

  • Method

    The study injects controlled faults into a retail ReAct trajectory and measures return likelihood under schema-only and whole-prefix conditionings plus legal-action distributions.

  • Results

    Incomplete returns are improbable under the schema alone and shift mass toward state-rereading tools, while inconsistent returns leave that channel unchanged and register selectively against prior context.

  • Takeaways & Limitations

    Recognition is asymmetric: each fault is legible in some channel, but no single channel identifies all conditions.

  • Takeaways & Limitations

    The qualitative results come from teacher-forced single decision points in one domain, with one model, one injection site, and template rather than adversarial noise.

Abstract

from arXiv · show

Robustness to a bad tool return means answering it in the way that return calls for, which depends on how the tool went wrong. A tool that has failed and a tool that returns a well-formed falsehood are different problems with different remedies. We ask whether the two already differ at the moment the return arrives. This is a qualitative pilot study: we score single decision points rather than running agents to completion. We inject controlled faults into a retail customer-service domain and read two channels off the model's log-probabilities: the likelihood of the returned content under the tool schema alone and under the whole trajectory, and its distribution over the legal actions, read for both shape and where the mass sits. An incomplete return is legible in every case, being improbable under the schema alone in a range no other condition enters, and it moves the mass toward the tools that re-read state wherever there is room to move. An inconsistent return leaves the schema channel untouched and registers in the likelihood comparison on the field whose true value the context already carries verbatim, not on the one whose contradiction runs through the domain policy. The action distribution gives each condition a distinct signature, but orders them by how far the return bears on the next action rather than by fault family. Recognition is therefore asymmetric: each condition is legible in some channel, and no channel is legible on all of them.

I. INTRODUCTION

The paper distinguishes incomplete tool returns from well-formed but false returns and asks whether agents can tell them apart at the return boundary. It finds that different log-probability and action-distribution channels reveal different aspects of the fault.

  • Diagnostic gap: Completed-episode entropy can indicate that something went wrong but cannot identify which remedy the return requires.The paper therefore studies the tool-return boundary rather than relying on a single trajectory-level curve.
  • Fault distinction: Incomplete returns omit evidence, whereas inconsistent returns are schema-valid but conflict with user intent or earlier tool results.The remedies differ: retry or graceful degradation for incompleteness, and independent cross-checking for inconsistency.
  • Diagnostic channels: The returned string is scored under the tool schema alone and under the whole trajectory to distinguish faults anomalous relative to different information sources.Schema-only improbability characterizes execution failure, while trajectory-relative improbability can expose contradiction with prior context.
  • Study design: Controlled substitutions in a retail customer-service setting compare an environment error string with schema-valid payloads falsifying two fields.The two inconsistent fields differ in whether their true values are already present verbatim in the prefix or require domain-policy reasoning.
  • Main findings: Incomplete returns are schema-improbable and shift probability toward state rereading, while inconsistent returns leave the schema channel unchanged and register only for verbatim context collisions.Action distributions distinguish all three conditions but order them by relevance to the next action rather than fault family.

II. MEASUREMENT DESIGN

The study evaluates tool-return robustness at a single decision point in a 16-tool ReAct-style retail environment. Reference trajectories and shared prefixes make injected conditions comparable.

  • Setting: The benchmark contains 114 retail tasks over 16 tools, with the agent choosing among tool calls and user replies at each decision point.Each tool return is appended to context before the next decision.
  • Setting: The injection is positioned immediately before a decision point, after a prefix containing policy, user turns, and earlier calls with returns.The notation uses i for the decision point, τ<i for the prefix, and u for the user's stated intent.
  • Measurement: The measurement is confined to the single decision point following the injection, making the action distribution a direct probability vector rather than an empirical frequency.The quantities are computed from log-probabilities under fixed prefixes.
  • Control: Prefixes are replayed from reference action sequences against the real database so all arms share a byte-identical prefix at the injection point.Each injected row is compared with a clean row differing only in the returned content.

B. Quantities

The paper combines conditional surprisal comparisons with legal-action policy measurements. These quantities separately assess return likelihood, action-distribution shape, and where action probability is concentrated.

  • Conditional surprisals: A returned string is scored per token under the schema-only and whole-prefix conditionings, and their difference G measures the trajectory's added likelihood contribution.S0 reads the string against the schema alone, S1 against the whole prefix, and G = S1 − S0.
  • Action entropy: Action entropy is computed over 17 legal actions: 16 tools plus replying to the user.The probability vector comes from the next-token distribution, tool-name continuation, and reply remainder.
  • Action location: Entropy measures the shape of the legal-action distribution, while the leading action and grouped probabilities record where its mass is located.The groups include two state-rereading tools, seven write actions, and replying.
  • Paired contrast: Every quantity is reported relative to the paired clean row from the same task and step.For any quantity X, the contrast is the injected value minus the true value, with the pre-injection term canceling for entropy.
  • Interpretation: A syntactically valid tampered field is expected to be unremarkable under S0 but anomalous under S1, whereas an execution failure is improbable under the schema alone.This asymmetry determines how the two likelihood channels should be interpreted.

C. Injection design

The injection design isolates incomplete and inconsistent returns at a fixed mid-trajectory tool site while preserving the surrounding database and trajectory context. Independent arm eligibility and replay controls define the study's comparison set.

  • Injection site: The injection site is the first get_order_details call at position two or later, ensuring a mid-trajectory, information-dense payload.Fixing the site by tool keeps field vocabulary constant while allowing its index to vary.
  • Experimental arms: Four arms comprise clean C, failed-return F, and inconsistent Ns and Nm conditions, with the latter falsifying different payload fields.F replaces the payload with the environment's error string, while C preserves the true return.
  • Validity control: Schema-only contamination is controlled by restricting Ns to legal status values and retaining S0 as a likelihood measure rather than a validator.This keeps schema-valid but jointly incoherent records visible without thresholding them away.
  • Eligibility: 53 of 114 tasks qualify for the paired set, while the error arms enter independently wherever their target fields exist.Consequently, the 189 scored decision points are not 53 × 4.
  • Propagation control: Only the payload at k = 0 is substituted; later returns are byte-identical across arms, so later likelihood differences reduce exactly to trajectory-conditioned differences.The database remains untouched throughout the replay.

A. The likelihood gap

The likelihood gap cleanly separates failed returns from inconsistent payloads at the schema level, while within the inconsistent family it responds chiefly to literal collisions with context already containing the true value.

  • Schema-level separation: ∆S0 is positive for all 53 failed-return rows, entering a region no other arm reaches.The separation remains when every arm is scored using only its first token, ruling out return-length as the explanation.
  • Schema-level separation: The inconsistent arms leave the schema axis unchanged, distinguishing them from failed returns.
  • Schema-level separation: The failed-return arm also moves ∆G upward on 45 of 53 rows because the error string is improbable under both conditionings.
  • Within-family sensitivity: Within inconsistent returns, Nm moves ∆G upward on 30 of 31 rows when the true identifier is verbatim in the prefix, but on all six rows when it is absent.Ns remains flat when its contradiction must be inferred through domain policy, including 6 of 14 policy-mediated rows.
  • Within-family sensitivity: The gap therefore detects token-level collisions more reliably than claims that conflict only with domain rules.

B. Entropy

Entropy measures how concentrated the legal-action distribution becomes, while separate leader and location readings show how each return changes the next decision.

  • Entropy shape: The clean median entropy is about 0.005 nats versus a maximum of log 17 ≈2.83, leaving 31 of 53 rows below 0.01 nats.Action-distribution counts therefore use the 22 rows whose clean entropy clears the 0.01-nat floor.
  • Entropy shape: Failed returns sharpen and redirect the distribution together: among 22 rows with room to move, entropy ends below 0.01 nats on 15 and the leading action changes on 15.
  • Entropy shape: Ns can sharply increase uncertainty, reaching ∆H = +1.20 in one row, while its entropy falls below the floor on 8 of 20 movable rows.
  • Entropy shape: Nm barely changes entropy, with |∆H| at most 0.18 across its 37 rows.
  • Interpretation: The arms are ordered by the return’s effect on the next action: absence invites state rereading, rewritten status changes permitted actions, and rewritten identifiers alter later values.

C. Where the probability mass goes

The probability mass on replying provides a directional readout that distinguishes incomplete returns from inconsistent ones when the clean distribution leaves room to move.

  • Reply probability: The clean π(V) distribution is bimodal, with 35 of 53 rows above 0.99 and 14 below 0.01.The paired plot preserves each row’s clean starting level so the ceiling remains visible.
  • Reply probability: On the 18 tasks with room to move, failed returns raise π(V) on every task, from below one half to essentially one.Across all 53 rows, π(V) rises on 34, falls on 2, and ends at or above 0.99 on 46.
  • Reply probability: Inconsistent returns have no common π(V) direction: Ns raises it on 17 rows and lowers it on 14, while Nm raises it on 15 and lowers it on 2.
  • Interpretation: Only incomplete returns consistently move probability toward tools that reread state, and they do so wherever the clean distribution permits movement.

D. What the return leaves behind

After the injection, likelihood and action channels separate the effects of incomplete and inconsistent returns. The incomplete return leaves a persistent likelihood trace and redirects action mass toward state rereading, while inconsistent returns remain flat in likelihood and differ in action consequences.

  • Likelihood trace: 51 of 51 tasks show a likelihood increase for the incomplete return one call after injection, with a median of +0.085 nats per token.The interquartile range is [+0.058, +0.138], and the sign test gives p < 10^-15.
  • Likelihood trace: The incomplete-return likelihood trace fades to +0.028 on 32 of 41 tasks at k = 2 and +0.008 on 14 of 19 at k = 3, while inconsistent arms stay flat.The same shape remains when restricting to the 19 tasks whose reference sequence reaches every step.
  • Action distribution: The action distribution’s shape remains stable, with median ∆H within 0.0011 nats of zero for every arm and step.The separation appears in where probability mass moves, not in distributional entropy.
  • Action distribution: The incomplete return redirects leading actions toward state-rereading tools, while the inconsistent status and identifier returns produce different movements despite similar leading-action effects.The incomplete arm raises π(V), whereas the inconsistent status arm moves toward write actions and the identifier arm produces no π(V) shift.

IV. DISCUSSION

The discussion interprets each diagnostic channel as sensitive to a different aspect of a return and bounds the pilot’s conclusions. The likelihood comparison detects some context contradictions, while action distributions reflect implied changes to the option set rather than fault provenance.

  • Channel interpretation: G detects values that the prefix already writes out, but not contradictions reached only through domain policy.The action distribution instead responds to the state implied by the return and is indifferent to why the return is wrong.
  • Channel interpretation: The action distribution orders conditions by how much the returned content changes the next-action option set, not by whether the fault is incomplete or inconsistent.Figure 2 compares paired likelihood gaps, leading-action changes, and π(V) direction against each task’s clean row.
  • Un tested cases: The proposed behavior for irrelevant and ambiguous returns remains a prediction rather than a result.The paper predicts that irrelevant returns may resemble inconsistent ones in likelihood, whereas ambiguous returns may disperse action probability.
  • Interpretive control: Replay of the reference action sequence isolates degradation from polluted context but removes the agent’s subsequent action drift and task-completion effects.This makes k ≥ 1 traces attributable to the prefix while limiting their interpretation as rollout outcomes.
  • Limitations: The study scores teacher-forced decision points rather than completed agent runs, and uses one domain, one model, one injection site, and template noise.Arm counts are exact within arms but only indicative across arms because target fields determine entry.

V. CONCLUSION

The study finds that failed and well-formed false returns are distinguishable at the return boundary, but not through a single universal signal. Likelihood and action channels expose different aspects of the fault, with action changes ordered by consequence for the next decision.

  • Conclusion: The incomplete return is schema-improbable, whereas an inconsistent return surfaces only when its value collides with information already written in the prefix.Thus, the likelihood channel distinguishes the families asymmetrically.
  • Conclusion: The action distribution registers all three conditions but orders them by how strongly the return bears on the next action.The failed call redirects mass toward rereading state, falsified status creates uncertainty, and falsified identifier moves neither.
  • Conclusion: At later calls, identical scored return bytes preserve the asymmetry: only the failed call remains visible in likelihood, while falsified status leads to a state-changing action.A crossed design is proposed to test whether this ordering reflects consequence rather than fault provenance.
Loading 2608.22676v1…