PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounding LLM Agents in Business Processes: Process-Aware Knowledge Graphs for Reliable Multi-Step Service Invocation

Saiful Khan, Afrah Farea

arXiv:2610.37242v1cs.AIcs.SE

TL;DR

LLM agents calling enterprise APIs must choose endpoints while respecting process order and valid payload keys and codes, but endpoint-focused retrieval omits these constraints. The paper combines OData metadata and BPMN processes in a typed property graph and uses dense retrieval with process-chain expansion. It reports improved Recall@3 on process-chain queries and fewer invalid calls with typed context.

  • Problem

    Endpoint-focused service discovery omits process order and schema constraints needed for multi-step API calls with valid keys and code values.

    Source: §1 · p. 1

  • Method

    The method links OData metadata and BPMN processes in a typed property graph, expands dense retrieval seeds along process edges, and supplies the retrieved typed context to a ReAct agent.

    Source: §1 · p. 2

  • Results

    +5.7 points in Recall@3 on process-chain queries (p = 0.037, n = 74); the method stays close to dense retrieval elsewhere, and typed context yields invalid calls in 3 of 101 runs.

    Source: §7 · p. 12

  • Takeaways & Limitations

    The results support using process and schema constraints in retrieval for reliable multi-step API calls in structured enterprise ecosystems.

    Source: §7 · p. 12, §6.2 · p. 12

    The mock validator checks calls against the same schema and ValueHelp nodes used to build the graph, so the invalid-call results are not measured against an independent target.

    Source: §6.2 · p. 11

Abstract

from arXiv · show

Large language model (LLM) agents that call enterprise APIs must select the correct endpoints, follow process order, use valid payload keys, and choose coded values from predefined lists. However, current retrieval-augmented service discovery treats endpoints independently and therefore misses process and schema constraints. We address this limitation by integrating Open Data Protocol (OData) metadata and Business Process Model and Notation (BPMN) reference processes in a typed property graph. A hybrid retriever identifies candidate endpoints using dense embeddings and follows process edges to retrieve subsequent endpoints. We evaluate the method on 42 enterprise services with process-chain ground truth. For queries with low lexical overlap between the query and downstream entities, it improves Recall@3 by 5.7 points (n=74), with consistent gains across embedding models and cut-offs. It matches dense retrieval on other queries and provides no improvement on OpenAPI services without a process layer, indicating that the gains arise from process structure. We also evaluate an LLM agent on 81 multi-step queries. It completes 38.7% of tasks, compared with 10.3% for a chunk-based baseline. In an elicitation probe, providing valid code lists eliminates invented codes but leaves invalid keys in 46% of write runs. Full typed context reduces invalid calls to 3% (3 of 101 runs), with no invented codes. These results show that exposing process and schema constraints during retrieval improves the reliability of agent-driven API calls.

1 Introduction

Reliable multi-step API use requires both process-order and payload constraints, which endpoint-focused retrieval does not capture. The paper combines OData metadata and BPMN processes in a typed graph to retrieve endpoints with context for execution.

1 Introduction

  • Reliable multi-step automation requires process order and valid payload keys and codes, beyond selecting the right endpoint.A call can fail if it invents a code, omits a required reference, or uses the wrong sequence.Source: §1 · p. 1
  • Prior retrieval-augmented discovery treats endpoints as isolated units and omits process, structural, and execution context.The Discovery Agent ranks endpoints but stops at retrieval and has no process-chain ground truth.Source: §1 · p. 2
  • The method links OData service metadata with BPMN task order in a typed property graph, then expands dense retrieval seeds along process edges.Retrieved endpoints include schema context for an execution agent issuing validated GET and POST calls through a mock HTTP layer.Source: §1 · p. 2
  • The paper introduces a graph schema and ingestion pipeline, a process-aware hybrid retriever, and an execution-capable ReAct agent using value-help and cardinality context.The approach is evaluated with a benchmark of 42 enterprise services, 14 BPMN reference processes, and 288 queries with two-annotator process-chain ground truth.Source: §1 · p. 2

2 Background

The background introduces OData service structure, BPMN workflow order, dense retrieval, and LLM agents. OData describes data structure, while BPMN describes task sequence; the method links these complementary descriptions.

OData services and BPMN processes

  • OData services expose machine-readable metadata describing entity collections, record fields, references, and fixed code lists.BPMN models describe task order but not the data each task uses; the paper links activities to the EntitySets they read or write.Source: §2 · p. 3
  • An EntitySet is a collection of records, an EntityType defines record structure, and a NavigationProperty links collections with cardinality.A client reads with GET and creates records with POST; payloads carry property values.Source: §2 · p. 3

Dense retrieval and LLM agents

  • Dense retrieval embeds a query and returns stored items with the closest vectors, but cannot find an undescribed next process step.The paper uses FAISS for dense search.Source: §2 · p. 3
  • An LLM agent iterates between reasoning and tool calls until the task is done, a pattern called ReAct.In this setting, an invalid call invents an EntitySet, key, or code absent from the service description.Source: §2 · p. 3

3 Related Work

Prior work addresses service retrieval, composition, tool use, and business-process integration, but the paper positions process-sequence edges as a retrieval signal for multi-step, payload-aware API execution.

3.1 Retrieval-augmented service discovery

  • The retrieval gain is attributed to typed process-sequence edges, not generic graph structure over service metadata.Graphs built only from extracted entities do not provide those edges.Source: §3.1 · pp. 3–4

3.2 Classical composition and knowledge graphs for services

  • Classical service-composition approaches include manually modeled semantic frameworks and AI planning, while this method derives workflow edges from BPMN for LLM retrieval context.The method reduces but does not remove manual work: the BPMN layer and a small activity-to-EntitySet mapping are written by hand.Source: §3.2 · p. 4

3.3 LLM tool use and agent frameworks

  • Broad API tool-use systems generally do not model structural rules such as fixed value lists or navigation cardinality that determine payload validity.This system supplies those rules with each retrieved endpoint and enforces them in a mock execution layer.Source: §3.3 · p. 4

3.1 Retrieval-augmented service discovery

  • Process-aware retrieval extends endpoint-ranking approaches by using process-chain expansion, a setting the authors say earlier service-discovery work had not evaluated.Pesl et al. introduce SOCBench-D and a Discovery Agent that does not execute calls; CRAFT and RestGPT combine retrieval with single-call execution.Source: §3.1 · p. 3, §3.4 · p. 5

3.4 Business-process and API integration

  • Prior links between business-process models and executable services have focused mostly on modeling, whereas this system uses activity-to-endpoint mapping directly for retrieval.The authors present their benchmark as a repeatable setting for evaluating process-chain expansion in retrieval-augmented service discovery.Source: §3.4 · p. 5

4 Approach

The system combines OData metadata and BPMN process order in a typed graph, then uses dense retrieval and bounded process expansion to supply an agent with endpoint and schema context. Across retrieval and mock-execution evaluations, gains are concentrated in process-chain retrieval and typed-context execution, while OpenAPI services without a process layer show no significant retrieval gain.

4.1 System Overview

  • Offline ingestion indexes EntitySet summaries; online retrieval seeds bounded process expansion from dense matches, and a ReAct agent uses retrieved context to issue GET and POST calls.The graph stores navigation cardinality and valid codes so the agent can shape payloads and choose from value-help lists.Source: §4.1 · p. 5

4.2 Offline: Ingestion Pipeline

  • The graph links OData entity sets, properties, navigation cardinalities, and value-help codes with BPMN activities and sequence order.The independent audit found 1.000 coverage and exactness for the listed source structures, but did not cover the hand-written activity-to-EntitySet mapping edges.Source: §4.2 · p. 5, §5.3 · p. 8

4.3 Online: Retrieval and Execution

  • The online retriever preserves dense-match order while adding downstream entities; the agent receives required keys, navigation cardinalities, value-help lists, and process-chain details.A mock HTTP layer validates calls, and missing keys or invalid codes can be rejected for retry.Source: §4.3 · pp. 5–6, Figure 2 · p. 6

5.1 Datasets

  • OData-Proc combines four public OData corpora and contains 288 queries, split into 150 training and 138 test queries; all 74 process-chain probes are in the test split.Execution experiments used 81 test queries of four multi-step types targeting SAP; a mock HTTP layer validated payloads against OData metadata without a live backend.Source: §5.1 · pp. 6–7

5.4 RQ1: Retrieval quality

  • +5.7 points: Hybrid improved Recall@3 over Dense on the 74-query process-chain probe (0.732 vs. 0.675), while staying close to Dense on non-probe queries (0.901 vs. 0.917).Across the full test split, Hybrid gained +2.3 points at k = 3 (0.810 vs. 0.787); the probe result had p = 0.037.Source: §5.4 · p. 8, Table 3 · p. 9

5.5 RQ2: Ablation

  • Removing either process-edge type brought probe Recall@3 back to the Dense level; both entity-to-activity mappings and sequence edges are needed to expand to later activities.Full Hybrid beat mapping-only by +5.9 Recall@3 points on the probe (p = 0.028).Source: §5.5 · pp. 8–9

5.6 RQ3: End-to-end agent

  • Our agent completed tasks at a rate of 0.387 versus 0.103 for ChunkRAG (p < 0.001); the gain held for navigation chains (0.517 vs. 0.100) and process-chain goals (0.316 vs. 0.103).In our method’s POST runs, 3 of 101 were invalid and it invented no codes. ChunkRAG+VH invented no codes in 128 POST runs, but 59 runs still contained invalid calls (0.46).Source: §5.6 · p. 9, §5.7 · p. 10

5.8 RQ5: External generalization

  • Hybrid and Tokens did not differ significantly at any k on 110 SOCBench-D OpenAPI queries without a business-process layer (all p ≥ 0.07).At k = 10, Hybrid was slightly worse (∆ = −0.041, p = 0.071).Source: §5.8 · p. 11, Table 6 · p. 11

6 Discussion

The discussion estimates the effort to author process-to-entity mappings and tests LLM-assisted proposals. It also identifies circularity in the evaluation and limits the method’s demonstrated transfer to ecosystems with a process layer.

6.1 Mapping Effort and Scalability

  • The 45 hand-written USES ENTITY SET edges are the main practical bottleneck, although authoring scales with process count rather than catalogue size.The 42 services, 132 EntitySets, and 8,481 properties were loaded without manual work; the mapping requires no ontology expertise and can be reviewed in full.Source: §6.1 · p. 11
  • 205 person-minutes produced 12 reviewed mapping edges, and the median rate implies about 840 person-minutes for all 41 mapped activities.Every rewritten edge matched the released mapping; the sample covered 12 of the 45 edges.Source: §6.1 · p. 11
  • LLM-proposed mapping edges reached precision 0.79, recall 0.93, and F1 0.86 against the 45 hand-written edges, but domain review remains necessary.The proposals replace writing 45 edges from scratch with checking 53; three missed edges show why review is needed.Source: §6.1 · p. 11

6.2 Threats to Validity

  • The evaluation is partly circular because the mock validator uses the graph’s schema and value-help nodes, and the annotators share label-generation inputs.The provenance split, fidelity audit, and symmetric value-help baseline reduce but do not remove these concerns; the second annotator was an LLM using the same machine-suggested labels.Source: §6.2 · p. 11
  • The retrieval gain depends on process edges, and the study has not demonstrated transfer beyond SAP’s unusually well-documented reference processes.The authors state that the method targets ecosystems with a process layer, native or written; chunk-RAG remains a good fit for free-form APIs.Source: §6.2 · pp. 11–12

7 Conclusion

The paper combines process and schema context for multi-step API calls and reports retrieval and agent gains, while noting low completion and evaluation limitations.

7 Conclusion

  • On process-chain queries, Recall@3 improves by +5.7 points, while multi-step task completion is 0.387 versus 0.103 for the baseline.The method stays close to dense retrieval elsewhere; its invalid-call rate is 3 of 101 POST runs versus 100 of 127 for the baseline, but deployment readiness remains limited.Source: §7 · p. 12

Supplementary Information

The supplementary sections provide secondary analyses and details referred to in the main paper, with a stated convention for p-values.

Supplementary Information

S1 Tool Surface Per Condition

The two agent conditions share the mock GET/POST layer and validator but differ in retrieval outputs; the supplementary material also explains why raw metadata without retrieval was not tested.

S1 Tool Surface Per Condition

  • Both conditions use the same mock GET/POST layer and validator, but grounding context arrives in retrieval results while ChunkRAG returns summaries only.Neither condition has a separate schema or value-help tool.Source: S1 Tool Surface Per Condition · p. 15
  • ChunkRAG+VH adds value-help lists as plain text to ChunkRAG’s retrieval result, changing nothing else.Source: S1 Tool Surface Per Condition · p. 15
  • Raw tool use without retrieval was not reported because streaming the metadata XML into a tool-use prompt timed out in all runs.The authors state that this would mix format problems with system performance.Source: S1 Tool Surface Per Condition · p. 15

S2 Benchmark Construction and Annotation Protocol

The benchmark grew through independent review of automatically generated candidates, then resolved disagreements and rebuilt process-chain ground truth. Its main test split was fixed, with earlier agent analyses retained as a smaller subset.

S2 Benchmark Construction and Annotation Protocol

  • Two annotators independently reviewed 251 automatically generated candidates, each accompanied by machine-suggested labels and a derivation note.Annotator A was a human author and Annotator B was Claude; their per-candidate decisions are released.Source: S2 Benchmark Construction and… · p. 15
  • The annotators agreed on keeping or rejecting 250 of 251 candidates (99.6%), while label agreement on 415 pairs among 249 jointly accepted candidates was κ = 1.000.Both annotators saw machine-suggested labels, so the latter figure measures agreement on corrections rather than independent labeling.Source: S2 Benchmark Construction and… · p. 15
  • The merge added 208 queries to the main benchmark and 41 to the elicitation probe; process-chain ground truth was rebuilt from the graph.One keep/reject disagreement was dropped as unanswerable, and two candidates were dropped overall.Source: S2 Benchmark Construction and… · p. 15
  • The benchmark was split deterministically into 150 training and 138 test queries; some secondary agent analyses used an earlier 26-query multi-step subset created before the benchmark grew.Source: S2 Benchmark Construction and… · p. 16

S3 Graph-Construction Fidelity Audit

The audit reports full coverage and exactness for checked graph classes; follow-on analyses show that process expansion adds gold entities, while context completeness and gains remain sensitive to retrieval.

S3 Graph-Construction Fidelity Audit

  • Every audited structural class reached full coverage and exactness, with ComplexType definitions explicitly out of scope.The audit treats normalized formatting differences as matches and counts distinct elements after duplicate rows are merged; it does not cover hand-written USES ENTITY SET edges.Source: S3 Graph-Construction Fidelity Audit · p. 16, Table S2 · p. 16

S4.1 Where the gain comes from

  • At k = 3, process expansion added 32 of 304 gold entities but pushed 16 dense golds out; at k = 10, its net gain fell to nine.At k = 3, 28 additions were in the process-chain probe and four were in multi-step workflows; expansion added nothing for query types without a process signal.Source: Table S3 · p. 17, S4.1 Where the gain comes from · p. 17

S4.2 Completeness of the context passed to the agent

  • Top-5 discovery context included all 3 required payload keys and the 1 value-help list, but only 75/85 navigation cardinalities and 67/79 process dependencies.At least one entity was missing in 34 of 35 failing pairs; when both entities were retrieved, keys and cardinality were complete.Source: S4.2 Completeness of the context… · p. 17

S4.3 Embedding sensitivity

  • With all-mpnet-base-v2, Hybrid beat Dense on the probe at every k, including Recall@3 of 0.515 vs. 0.507, but none of the differences was significant.The paired probe test gave p = 0.54; weaker seeds left more queries out of reach of graph expansion.Source: S4.3 Embedding sensitivity · p. 17

S5 Additional Agent Analyses (RQ3)

Additional analyses use an earlier 26-query subset to examine agent behavior, seed checking, execution under oracle retrieval, and sensitivity to other LLMs. They distinguish retrieval failures from execution limits and serving instability.

S5 Additional Agent Analyses (RQ3)

  • The additional analyses used an earlier 26-query multi-step subset, before the benchmark grew; seed verification, oracle retrieval, and LLM sensitivity checks were limited to it.The main paper reports RQ3 on the 81 multi-step test queries.Source: S5 Additional Agent Analyses (RQ3) · p. 17

S5.1 Results on the subset and why they differ

  • On the earlier subset, ChunkRAG completed more tasks than the agent (0.295 vs. 0.192), a nonsignificant difference, while the agent made fewer invalid calls (4/78 vs. 10/78).The subset preceded benchmark growth; retrieval misses caused most failures for both methods, and process context helped with correct seeds but hurt when seeds were wrong.Source: S5.1 Results on the subset and why… · pp. 17–18, Table S4 · p. 18

S5.2 Checking seeds before expansion

  • Seed verification eliminated invalid POST calls on the subset (0 vs. 3 for the unverified agent), but completion rose only slightly (0.205 vs. 0.192).Verification could reject a wrong seed but could not add a missing one; completion was 0.421 when discovery returned all gold entities and 0.000 when it did not.Source: S5.2 Checking seeds before expansion · p. 18

S5.3 Separating execution from retrieval (oracle retrieval)

  • With oracle retrieval, completion remained low for both agents (0.256 for ours and 0.359 for ChunkRAG), while ours issued 3 schema-invalid writes and invented no codes, versus 15 writes and 6 invented codes for ChunkRAG.The results show execution limits remained under perfect retrieval; only the typed prompt produced correct 1:N array nesting, in 6 of 29 posting runs versus 0 of 22.Source: S5.3 Separating execution from… · pp. 18–19

S5.4 Sensitivity LLMs and serving

  • The non-primary LLM checks were mixed: gemma4:31b pointed in the same direction as the primary model, while qwen3.6 mostly hit CPU-serving latency limits.The authors base conclusions on qwen3-coder:30b, which had 0 timeouts in 642 runs.Source: S5.4 Sensitivity LLMs and serving · p. 19

S6 Mapping Authoring

The mapping workflow documents how activities are linked to service entity sets, and tests LLM-proposed edges against hand-written mappings. Separate per-sector results on SOCBench-D are mixed.

S6.1 Authoring procedure

  • A timed rewrite of 12 of the 45 mapping edges took 205 person-minutes, and every rewritten edge matched the released mapping.The sample covered ten mapped activities and had a median of 20.5 minutes per activity.Source: S6.1 Authoring procedure · p. 19

S6.2 LLM-proposed mapping

  • LLM-proposed mappings reached precision 0.79, recall 0.93, and F1 0.86 against 45 hand-written edges, recovering 42 edges while leaving domain review necessary.The proposals yielded 11 false positives and missed three edges; the workflow asks reviewers to check 53 proposed edges instead of writing 45 from scratch.Source: S6.2 LLM-proposed mapping · p. 19

S7 Per-Sector Results on SOCBench-D (RQ5)

Related papers

Loading 2610.37242v1…