PaperlayerView exact revision on arXiv

Source-linked AI summary

Right-Sizing LLM-Agent Decomposition in VAT Determination: A Pilot Controlled Sweep

Pedro Santos

arXiv:2608.23395v1cs.MAcs.AIcs.SE

TL;DR

The paper asks whether complex VAT determination benefits from many narrow agents or one strong tool-using agent when the activity surface is controlled. It runs a preregistered pilot varying worker assignments across four orchestrated configurations and a no-orchestrator baseline, finding directional intermediate leadership but no supported intermediate-optimum claim. The results motivate a bounded midpoint-partition heuristic while showing that content faults remain problematic and generalization is limited.

  • Problem

    The study addresses limited controlled evidence about agent-scope granularity because prior comparisons often change orchestration, models, prompts, tools, and tasks together.

  • Method

    The paper runs a 4,400-run pilot on oracle-labeled VAT determination, holding the activity surface fixed while varying worker assignments, matched-token controls, and failure injections.

  • Results

    The intermediate configurations lead at 0.830 accuracy, but the pre-stated fine-endpoint bar is missed; matched-token and injection results qualify the apparent advantage.

  • Takeaways & Limitations

    The pilot supports testing one dependency-layer midpoint boundary as a bounded VAT-workflow heuristic, while indicating that localized fallback protects availability more than content integrity.

  • Takeaways & Limitations

    The evidence is pilot-scale and bounded to one synthetic VAT-style activity surface, one model operating point, and synthetic failure conditions rather than general enterprise agents.

Abstract

from arXiv · show

Recent LLM-agent systems make conflicting design bets: decompose work across many narrow agents, or use one strong tool-using agent. This pilot studies that choice on bounded cross-border VAT determination with reverse charge, where every case has an oracle label and each intermediate decision is independently scoreable. We hold the activity surface fixed (subtasks, tools, I/O schemas, validation checks, orchestrator, base model, and merge policy) and vary only the assignment of subtasks to workers across four orchestrated configurations, from one wide worker to five narrow ones, against S0, a tuned no-orchestrator single agent, with a deterministic rule engine as oracle. The program spans 4,400 runs: a 40-case, five-repeat main sweep, matched-token arms separating prompt-budget from agent-count effects, and three failure-injection arms, all judged against pre-registered falsification criteria. The two intermediate configurations lead on accuracy (0.830, against endpoints at 0.720 and 0.770) but miss the pre-stated bar against the fine endpoint, so the intermediate-optimum hypothesis remains unsupported at pilot scale. The single agent does not Pareto-dominate the orchestrated set. The matched-token criterion fires: the budget-matched single agent lands 6.5 points below the leader, but the interval includes zero, so any advantage is consistent with a prompt-budget explanation. Under injection, availability faults are absorbed at every granularity, with wide-scope restart over-recovering its baseline by +0.160, while one schema-conforming hallucinated record degrades every configuration and inverts the ordering, hitting fragmented configurations hardest. The contribution is a bounded, preregistered pilot heuristic for right-sizing decomposition (place one partition boundary at the dependency-layer midpoint), released with oracle, dataset, harness, raw traces, and analysis pipeline.

1 Introduction

The paper isolates agent-scope granularity in bounded VAT determination by fixing the activity surface and varying only worker assignments. A pilot sweep combines controlled comparisons, matched-token tests, and failure injections, but its intermediate-optimum hypothesis remains unsupported at pilot scale.

  • The unresolved empirical question is what changes when only agent scope changes, because prior comparisons often vary orchestration, models, prompts, tools, and tasks simultaneously.
  • The VAT testbed provides oracle-labeled cases and independently scoreable decisions for classification, jurisdiction, rate, exemption, and reverse-charge logic.The workflow combines parallelizable lookups and classifications with sequential dependency in final reverse-charge reasoning.
  • The sweep holds subtasks, tools, schemas, validation checks, model family, orchestrator, and merge policy fixed while varying worker assignments across C1–C4 against no-orchestrator baseline S0.C1 is an orchestrated one-worker control, not the no-orchestrator baseline.
  • 4,400 valid runs cover a main sweep, matched-token comparisons, and timeout, hallucinated-output, and shared-tool-outage injections evaluated against preregistered falsification criteria.The matched-token arm tests whether apparent multi-agent gains may instead reflect prompt budget.
  • The two intermediate configurations reach 0.830 accuracy versus 0.720 for the wide endpoint and 0.770 for the fine endpoint, but the pre-stated bar against the fine endpoint is not met.S0 does not Pareto-dominate the orchestrated set, while the matched-token criterion fires with a confidence interval including zero.
  • The pilot is a bounded candidate heuristic for one VAT-style workload, one model family, four orchestrated granularities, and synthetic failure injections rather than a general enterprise-agent design rule.

2 Related Work

Related work frames agent granularity as an unresolved design choice involving orchestration, role scope, call scaling, prompt-budget confounds, and fault handling. This paper positions its contribution as a controlled VAT pilot that varies only worker scope while adding matched-token and fallback comparisons.

  • Agent-proliferation debate: Prior systems provide evidence for both orchestrated specialist workers and strong single-agent designs, leaving the relative value of decomposition unsettled.
  • Decomposition and role granularity: Classical and learned multi-agent research treats role scope and role discovery as design variables that affect coordination and performance.
  • Compound-system scaling and prompt-budget confounds: Compound-system studies show that additional calls can improve performance and then degrade it, motivating tests for bounded optima rather than assuming more agents are better.
  • Compound-system scaling and prompt-budget confounds: Multi-agent gains may reflect total prompt budget rather than agent count, so controlled matched-token comparisons are needed.
  • Tax/legal-critical agents and fault handling: Tax-agent research includes legal-critical validation and fault-tolerance concerns, but prior systems evaluate individual architectures rather than sweeping granularity on a bounded oracle-labeled VAT workflow.
  • The paper claims a first controlled VAT-style pilot that fixes the activity surface while distinguishing S0 from the one-worker orchestrated condition C1.It also includes matched-token control, localized fallback, and released artifacts.

3 VAT Determination as a Testbed

The study uses a synthetic, bounded EU-style VAT workflow because it supports oracle labels, auditable intermediate decisions, and a fully specified activity surface. The harness evaluates final accuracy alongside step-level, citation, consistency, and error-localization measures, within clear scope limits.

  • VAT determination is chosen as a methodological testbed because bounded rules make labels unambiguous while structured dependencies expose coordination trade-offs.
  • Scope of the testbed: The testbed excludes full jurisdictional VAT law, real invoice parsing, production semantics, and legal-compliance claims, so findings support only a bounded VAT-style heuristic.
  • The activity surface A fixes subtasks, input and output schemas, validation checks, tools, dependency order, and read-only reference materials.
  • The five subtasks are classification, jurisdiction determination, rate lookup, exemption checking, and reverse-charge decision.
  • The dependency structure permits direct classification and registration checks while making jurisdiction depend on classification and customer status, followed by downstream rate, exemption, and reverse-charge decisions.
  • 40 synthetic cases span four scenario families, and the deterministic rule engine supplies final and intermediate oracle labels rather than serving as a competing agentic baseline.
  • The harness scores final and intermediate accuracy, citation completeness, consistency, and the location of errors entering the VAT trace.

4 Configurations

C1–C4 share the same activity surface and orchestrator but partition the five subtasks from one wide worker to five narrow workers; S0 remains a separate no-orchestrator comparator. The configurations therefore isolate worker scope within orchestration while preserving tools, schemas, validation, dependencies, and repair controls.

  • Experimental controls: Across conditions, the model, tools, schemas, validation checks, references, dependencies, timeouts, retries, concurrency, and logging remain fixed.
  • Experimental controls: Within C1–C4, the only experimental variable is assignment of subtasks to workers; S0 differs by lacking orchestration ledgers and dependency-ordered dispatch.
  • Agent definition: An agent is defined by role, role-scoped prompt, tool permissions, visible state, and interaction protocol, with prompt wording alone insufficient to distinguish agents.
  • Orchestration: The orchestrator tracks task dependencies and progress, dispatches independent subtasks in parallel when permitted, and retries invalid, timed-out, or incomplete outputs within a fixed budget.
  • Experimental layout: The dashed midpoint in Figure 1 marks the dependency-layer boundary used by the merge policy and matched-token reference.
  • Worker partitions: C1 assigns all five subtasks to one orchestrated worker, while C2 separates classification and jurisdiction from downstream rate, exemption, and reverse-charge reasoning.
  • Worker partitions: C3 isolates classification and jurisdiction while keeping downstream synthesis co-resident, testing whether early isolation reduces propagation without full fragmentation.
  • Worker partitions: C4 assigns exactly one subtask per worker with restricted visible state and task-relevant tools, making it the finest decomposition.

5 A Chain-Error Model

The model treats trace accuracy as a product of required subtask correctness, while recognizing that dependencies, handoffs, and repair can violate independence. It frames intermediate granularity as a possible accuracy and cost-quality optimum rather than a guaranteed monotonic improvement.

  • Chain-error baseline: A(c) is modeled as the probability that all d required subtask decisions are correct under configuration c.Under independent per-step correctness and no retry repair, trace accuracy is represented by the product of marginal step-correctness probabilities.
  • Chain-error baseline: All five required subtasks make improvements to any step affect trace-level accuracy multiplicatively.
  • Cross-step dependence: Independence is only a reference point because wrong upstream decisions, shared state, and handoffs can create correlated errors.The pilot does not estimate a single correlation parameter; observed accuracy may deviate from the independence baseline in either direction.
  • Retry repair: Validation and retry improve effective step accuracy only when failures are detectable and repairable within the available budget.Silent semantic errors that satisfy the schema and cite plausible rules may remain unrepaired.
  • Retry repair: Orchestrated configurations repair exposed subtasks locally, whereas S0 retries the complete trace, creating a structural asymmetry in repair.The asymmetry comes from validation-driven localization in C1–C4 versus whole-trace repair in S0.
  • Granularity expectations: Intermediate decompositions may outperform either extreme when local focus and repair gains exceed coordination, handoff, and fragmented-context costs.The expected cost-quality frontier can exclude configurations that add calls, retries, or handoffs without corresponding accuracy gains.

6 Methodology

The methodology fixes the activity surface and evaluates worker assignment across orchestrated configurations against a no-orchestrator single-agent baseline. It measures accuracy, cost, latency, matched-token effects, and localized recovery under paired failure injections using preregistered pilot-scale analysis.

  • Metrics: The study measures accuracy, cost, and latency for each condition, case, and repeat.
  • Metrics: Final-answer accuracy requires the emitted VAT determination to match oracle labels across jurisdiction, rate, exemption, reverse charge, liability, and amount or non-charging reason.Step accuracy, trace consistency, and earliest failing subtask support intermediate and error-propagation analyses.
  • Metrics: Token cost sums prompt and completion tokens across model calls, including retries, while dollar cost is derived from a pinned price sheet.
  • Baselines and oracle: S0 is the tuned no-orchestrator ReAct/function-calling single-agent comparator for the combined orchestrator, ledger, validation, and subtask-repair effect.The deterministic rule engine supplies oracle labels and intermediate targets but is not a competing agentic baseline.
  • Matched-token control: The matched-token control retunes S0 to within ±10% of C2's mean per-case token cost using a development split, then compares evaluation performance without retuning.A post-selection sensitivity analysis similarly matches S0 to the highest-accuracy orchestrated configuration but is not confirmatory.
  • Failure injection: Failure injections use paired case identities and deterministic target subtasks across conditions, testing timeout recovery and schema-conforming hallucinated outputs.Hallucinated records can evade retry when validation passes, leaving downstream contradiction detection as the recovery route.
  • Statistical analysis: The 40-case, five-repeat design is explicitly pilot-scale, with paired confidence intervals and effect sizes interpreted as inconclusive when intervals are wide.

7 Results

The main sweep found a non-monotonic accuracy pattern, with intermediate configurations highest but without sufficient evidence to support the intermediate-optimum hypothesis. Cost–quality and matched-token analyses further show trade-offs between orchestration, accuracy, and prompt budget, while error localization shifts with granularity.

  • Study coverage: 4,400 valid runs covered the main sweep, two matched-token arms, and three failure-injection arms under pre-registered falsification criteria.The main sweep comprised five conditions × 40 cases × five repeats; all results came from the released analysis pipeline.
  • Main sweep: 0.830 accuracy tied across the two intermediate configurations, versus 0.720 for the wide endpoint and 0.770 for the fine endpoint.The C1–C4 accuracy sequence was non-monotonic: 0.720, 0.830, 0.830, 0.770.
  • Main sweep: Neither intermediate configuration materially outperformed the fine endpoint, so the intermediate-optimum hypothesis remained unsupported for this workload.C2 vs. C4 was +0.060 with CI [−0.005, +0.125], while C3 vs. C4 was +0.060 with CI [−0.020, +0.135].
  • Cost–quality trade-offs: S0 did not weakly Pareto-dominate the orchestrated configurations; the descriptive non-dominated set was {C1, S0, C3}.C3 weakly dominated C2 and dominated C4 on the token-cost, accuracy plane.
  • Cost–quality trade-offs: 0.075 accuracy gain from S0 to C3 cost +14% tokens and +42% median latency along the non-dominated set.Median latency rose with worker count, from ≈12 s for S0 and C1 to 18.4 s for C4.
  • Error localization: 34 earliest failures occurred in each intermediate configuration, while wide scopes concentrated failures at jurisdiction and the finest scope at reverse-charge synthesis.Under C4, RCH accounted for 39 of 46 earliest failures; under C2, JUR-earliest failures were 0.
  • Matched-token comparison: 6.5 points separated the budget-matched single agent from C2, but the confidence interval included zero, leaving a prompt-budget explanation consistent with the result.The contrast was −0.065 with CI [−0.165, +0.025] and permutation p = 0.265.

8 Localized Fallback under Injected Failures (RQ4)

Availability failures were generally absorbed across configurations, while a single schema-conforming hallucinated record degraded every condition and most severely harmed fragmented designs.

  • Experimental setup: 3,000 runs covered timeout, hallucination, and shared-tool-outage arms, with each condition evaluated on 40 cases and five repeats.Each injection mode applied alone and was paired by case identity and target subtask with the un-injected main sweep.
  • Availability perturbations: 0.975–1.000 substitution success under shared-tool outage accompanied recovery in every configuration, with all paired accuracy deltas straddling zero.The affected subset contained eight cases, and token penalties ranged from +363 to +1,138.
  • Availability perturbations: 0.965–1.000 substitution success under forced timeout produced absorption in S0, C2, C3, and C4, while repair cost tracked affected-worker scope.C2 and C3 paid +4,433 to +4,611 tokens, C4 paid −87, and S0 paid −1,442.
  • Availability perturbations: +0.160 was C1’s timeout accuracy delta, the only availability cell excluding zero, with 0.880 accuracy and a +21,793-token cost.The repair consumed roughly two additional full-pipeline attempts, approximately 2.9× the un-injected C1 cost.
  • Content perturbation: −0.485 was the hallucination delta for the main-sweep leaders C2 and C3, and hallucination inverted the ordering with S0 at 0.560 accuracy.All five deltas were negative; fragmented configurations fell to 0.310–0.345 accuracy, while C1 reached 0.405.
  • Content perturbation: 0.885–0.925 substitution success still left most hallucinated traces wrong because schema validation could not detect oracle-incorrect but schema-conforming records.Orchestrated validated-trace accuracy was 0.335–0.458, and downstream contradiction was required for recovery.

9 Discussion

The pilot’s intermediate-accuracy peak is directional rather than established, while injection results distinguish availability recovery from content propagation and motivate a narrowly scoped midpoint heuristic.

  • Granularity interpretation: 0.830 accuracy at both intermediate configurations realizes the expected peak shape but misses the pre-registered material bar against the fine endpoint.The paper therefore treats the non-monotonicity as directional rather than established.
  • Granularity interpretation: Jurisdiction determination concentrates earliest failures in wide scopes, whereas reverse-charge synthesis concentrates them in the finest scope.Intermediate configurations minimize total errors by placing one partition boundary between these two error masses.
  • Repair mechanism: +0.160 above baseline makes C1’s timeout overshoot a validation-filtered full-chain redraw effect, requiring wide scope and costing approximately 2.9× tokens.Half-pipeline and single-subtask redraws bought approximately nothing, and S0 already uses whole-trace retry as its normal mechanism.
  • Injection mechanism: −0.460 to −0.485 content deltas show that hand-off seams absorb availability faults but propagate schema-conforming content faults, especially in fragmented configurations.Orchestrated conditions replaced poisoned records in approximately 78% of cases, yet finished at only 0.310–0.405 accuracy.
  • Matched-token interpretation: +0.005 to +0.010 accuracy from +8–11% tuned prompt text indicates that prompt budget was not S0’s binding constraint, although the pre-registered criterion withholds an agent-count conclusion at n = 40.S0’s modal earliest failure remained jurisdiction determination, the subtask isolated by successful partitions.
  • Scoped heuristic: One dependency-layer midpoint boundary is the candidate heuristic for this VAT workload, separating preparation from synthesis without treating the heuristic as a general rule.The recommendation is scoped to the stated rule set, model, and harness; approximately 90 cases are proposed before any clause graduates from heuristic to finding.

10 Threats to Validity

This pilot’s validity is bounded by its synthetic VAT setting, pinned homogeneous model, fixed merge policy, matched-token design, failure injections, and pilot-scale sample. These constraints limit generalization and make several comparisons exploratory or sensitivity analyses rather than broad claims.

  • External validity: The study covers one bounded VAT activity surface and is not a general design rule for enterprise agents.The harness also excludes full jurisdictional VAT rules, special-sector schemes, real invoice parsing, master-data defects, production posting, and statutory reporting.
  • External validity: Synthetic cases are generated under the bounded rule set and share an origin with the oracle, which can inflate measured accuracy through rule-engine surface alignment.Manual spot-checking verifies oracle conformance within the harness but does not establish production realism.
  • Construct validity: The main sweep uses one pinned homogeneous model family, so the granularity–performance relationship may shift with heterogeneous models or different capability tiers.The results support replication on the same versions but do not establish stability across providers, generations, or operating points.
  • Construct validity: The observed granularity pattern may partly depend on the single pre-specified dependency-layer merge policy; alternative clustering policies were deferred.Workers are distinguished by permissions, state, schema scope, and protocol, but deep functional specialization is not proven.
  • Construct validity: Matched-token results remain a sensitivity analysis because architectures differ in context segmentation, ledger state, validation localization, retry unit, and handoff structure.The budget-band check used one repeat over eight development cases, and both targets resolved to the same tuning rung.
  • Statistical validity: S0 is a disclosed strong comparator rather than an exhaustively optimized single-agent upper bound, and its eight-case tuning split lacks exempt-outcome examples.The exempt output shape accounts for roughly 20% of evaluation-set line determinations, although the schema specifies it.
  • Construct validity: Failure results characterize controlled timeout, hallucination, and one correlated shared-tool outage rather than general deployment robustness.The S0-versus-orchestrated hallucination contrast also partly reflects different injection timing conventions, while C1–C4 share that convention.
  • Statistical validity: Forty cases with five repeats per condition constitute a pilot-scale sample, so non-significant headline comparisons are inconclusive and tail latency is not reported.Per-scenario-family comparisons use ten cases each and are descriptive only; non-headline comparisons are exploratory after limited multiplicity correction.

11 Conclusion

The conclusion presents a preregistered pilot of agent-scope granularity on fixed synthetic VAT workflows and releases the experimental artifact. Intermediate configurations led accuracy, but the stated intermediate-optimum claim remains unsupported at pilot scale.

  • Conclusion: 4,400 valid runs cover a fixed-surface pilot with four orchestrated partitions, a tuned no-orchestrator baseline, matched-token controls, and three failure-injection arms.The study uses 40 oracle-labeled cases with five stochastic repeats and preregistered falsification criteria.
  • Conclusion: The matched-token criterion fired, but the point estimate favoring C2 remains consistent with a prompt-budget explanation because the interval includes zero.The comparison addresses prompt-budget concerns without eliminating structural differences between single-agent and orchestrated systems.
  • Conclusion: Availability faults were absorbed at every granularity, while one schema-conforming content fault degraded every configuration and inverted the main-sweep ordering.Wide-scope restart over-recovered its baseline by +0.160, whereas fragmented configurations were hit hardest by the content fault.
  • Conclusion: The paper offers one bounded heuristic for VAT-style workflows: place a single partition boundary at the dependency-layer midpoint.The artifact releases the oracle, dataset, harness, prompts, injection seeds, raw traces, and analysis pipeline.

B Per-scenario-family descriptives

Scenario-family descriptives show family-specific strengths rather than a monotonic granularity advantage. The intermediate configurations are the only conditions above 0.70 in every reported family.

  • Per-scenario-family descriptives: Family means are aggregated from five repeats to case level, with families defined deterministically from case inputs.Mixed cases are multi-category; single-category cases are split by domestic versus cross-border transaction type.
  • Per-scenario-family descriptives: C1 leads domestic cases at 0.96 but falls to 0.70 on intra-community B2B, where jurisdiction and reverse-charge synthesis dominate.Each family contains ten cases, so these comparisons are descriptive only.
  • Per-scenario-family descriptives: C4 reaches 0.62 on intra-community B2B, while the intermediate configurations are the only conditions above 0.70 in every family.The family texture matches the paper’s error-localization analysis.

C Failure-injection configuration

The failure-injection configuration freezes seeded targets across conditions and repeats, combining isolated subtask faults with shared-tool outages. Hallucinated records are schema-conforming but contradict oracle labels and use unsupported citations.

  • Failure-injection configuration: The injection plan freezes one shared target subtask τ per case across conditions and repeats.Assignments are stored in data/injection_plan.json and consumed by the harness.
  • Failure-injection configuration: The design includes timeout, hallucinated-output, and shared-tool-outage conditions.The shared-tool outage affects one seeded case per half-family block.
  • Failure-injection configuration: Hallucinated records are generated deterministically as schema-conforming decisions that contradict oracle labels and cite unsupported rules.The released generator constructs the full 40-record set in the plan file.
  • Failure-injection configuration: The example injected record targets RCH and combines reverse-charge fields with a domestic jurisdiction path.The record’s citation key is RC.B2B.INTRA_EU while the shown support includes GEN_SERVICE and domestic.

D Pinned model, decoding, and price sheet

The study pins the model, decoding, timeout, retry, pricing, oracle, dataset, and seed settings. Dollar cost is derived from token counts using a versioned price sheet.

  • The pinned setup uses claude-haiku-4-5-20251001 at temperature 0.2, with 4096 maximum tokens per call and 120-second timeouts.
  • The study grants three retries per subtask for C1–C4 and three whole-trace repairs for S0.
  • The pinned price sheet charges $1.00 / $5.00 per 1M input/output tokens, with price changes recorded as new file versions.
  • The evaluation uses oracle commit e2d2bdd22, dataset SHA-256 3dc683ec4186..., generator seed 42, and 40 evaluation plus 8 development cases.
Loading 2608.23395v1…