Source-linked AI summary
Do Enterprise Systems Need Learned World Models? The Importance of Context to Infer Dynamics
Jishnu Sethumadhavan Nair, Patrice Bechard, Rishabh Maheshwary, Surajit Dasgupta, Sravan Ramachandran, Aakash Bhagat, Shruthan Radhakrishna, Pulkit Pattnaik, Johan Obando-Ceron, Shiva Krishna Reddy Malay, Sagar Davasam, Seganrasan Subramanian, Vipul Mittal, Sridhar Krishna Nemala, Christopher Pal, Srinivas Sunkara, Sai Rajeswar
TL;DR
Enterprise dynamics vary across tenant configurations and evolve over time, raising whether agents must learn rules that can be read at inference time. The paper introduces runtime discovery agents and CascadeBench, finding that discovery remains more robust under deployment shift than purely internalized models. It concludes that learned priors should be combined with runtime retrieval when transition logic is readable.
Problem
Tenant-specific, evolving configurations make historical-transition world models potentially brittle, leaving whether readable runtime rules should replace sole reliance on internalized dynamics unresolved.
Method
The paper introduces enterprise discovery agents that retrieve relevant live configuration and reasons over it, and evaluates them with CascadeBench and real audit-log transitions.
Results
Discovery agents remain more robust under configuration shift, while offline-trained world models perform well in-distribution but degrade on held-out configurations.
Takeaways & Limitations
When transition logic is readable from the live system, agents should combine learned priors with runtime retrieval rather than rely solely on internalized dynamics.
Takeaways & Limitations
The evaluation is single-platform, focuses quantitatively on Tier 1 and Tier 2 transitions, and compares methods using a small set of open-weight models.
Abstract
from arXiv · showhide
World models enable agents to anticipate the effects of their actions by internalizing environment dynamics. In enterprise systems, however, these dynamics are often defined by tenant-specific business logic that varies across deployments and evolves over time, making models trained on historical transitions brittle under deployment shift. We ask a question the world-models literature has not addressed: when the rules can be read at inference time, does an agent still need to learn them? We argue, and demonstrate empirically, that in settings where transition dynamics are configurable and readable, runtime discovery complements offline training by grounding predictions in the active system instance. We propose enterprise discovery agents, which recover relevant transition dynamics at runtime by reading the system's configuration rather than relying solely on internalized representations. We introduce CascadeBench, a reasoning-focused benchmark for enterprise cascade prediction that adopts the evaluation methodology of World of Workflows on diverse synthetic environments, and use it together with deployment-shift evaluation to show that offline-trained world models can perform well in-distribution but degrade as dynamics change, whereas discovery-based agents are more robust under shift by grounding their predictions in the current instance. Our findings suggest that, in configurable enterprise environments, agents should not rely solely on fixed internalized dynamics, but should incorporate mechanisms for discovering relevant transition logic at runtime.
1 Introduction
Enterprise dynamics vary across tenant deployments and evolve over time, challenging world models trained on historical transitions. The paper evaluates runtime discovery as a complement to learned dynamics using CascadeBench and deployment-shift evaluation.
- Enterprise configurations make the same action produce different effects across deployments and can make historical-transition models brittle under deployment shift.
- Enterprise discovery agents recover transition logic at runtime by querying state, inspecting workflow definitions, or issuing targeted probe actions.
- CascadeBench evaluates next-state prediction across multi-tenant environments and varying configuration complexity tiers.
- Offline-trained world models perform well in-distribution but degrade as dynamics change, whereas discovery agents remain more robust by grounding predictions in the active deployment.
- The paper concludes that enterprise agents should combine learned priors with runtime discovery rather than rely only on fixed internalized dynamics.
2 Related Work
Prior world-model research generally assumes that environment dynamics should be internalized into learned simulators. This paper studies the complementary case where enterprise system behavior is externally accessible and introduces CascadeBench to test adaptation across deployments.
- World models support planning by learning latent environment dynamics for simulated rollouts and policy optimization.
- Language-agent world models approximate responses from structured interfaces such as the web, code environments, and tool APIs.
- The paper examines settings where system behavior is accessible at inference time through interfaces, logs, or configuration files.
- Tool-augmented agents use external APIs and structured interfaces to ground reasoning in real system responses.
- CascadeBench addresses the gap left by fixed-configuration workflow benchmarks by testing transition prediction across varying deployments.
3 Enterprise Dynamics
Enterprise transition dynamics depend on tenant-specific, evolving configuration that is explicit and readable. The paper models cascades as configuration-dependent state transitions and distinguishes three levels of prediction complexity.
- An enterprise platform’s state spans interconnected tables, while customer-specific configuration determines how actions affect that state.
- The instance configuration includes business rules, workflow definitions, approval policies, SLA definitions, and access control lists.
- Unlike standard world-model settings, enterprise configuration changes over time and is stored in inspectable records with defined conditions and actions.
- A single field update can trigger multi-step cascades across tables, SLA timers, and notifications, with depth and branching determined by interacting rules.
- The paper separates transition complexity into Tier 1 schema-determined effects, Tier 2 rule-composed cascades, and Tier 3 execution-inferred behavior.
4 Enterprise Gym
Enterprise Gym constructs live, diverse environments and collects ground-truth cascades from actual platform execution. CascadeBench then evaluates transition prediction under controlled configuration shift while making relevant context available.
- Enterprise Gym: Each world is deployed to a live platform instance, so actions produce actual database states rather than simulated transitions.
- Enterprise Gym: The environment catalog contains 1,596 business-rule patterns spanning 6 industries and 11 operational domains.
- Data Collection: Candidate tool calls run in isolated sandboxes, audit logs recover causal changes, and retained samples record state transitions and cascade paths.
- Data Collection: The corpus contains 27,243 verified transition samples across 64 worlds, with world-level splits requiring generalization to unseen deployment regimes.
- Benchmarking: CascadeBench compares predicted field-level changes with audit-log ground truth while isolating reasoning over provided rules from memorization and retrieval noise.
5 Approaches
The paper compares prompting, fine-tuning, and runtime discovery for predicting structured enterprise state transitions. Discovery retrieves task-relevant configuration at inference time, while learned models internalize dynamics from supervised transition data.
- Approaches: All approaches predict structured field-level diffs from the current state and proposed action.The target is the same across prompting, learned world models, and discovery agents.
- Approaches: The prompted baseline infers action effects from supplied context without fine-tuning or runtime access to instance-specific configuration.Depending on the evaluation setting, prompts may contain only state and action information or additional context.
- Approaches: The learned world model internalizes dynamics by fine-tuning on state-action-next-state tuples collected from Enterprise Gym.Training targets the minimal field-level difference between consecutive states and tests transfer across different industries, organizations, and rule sets.
- Approaches: The discovery agent predicts without executing actions or updating parameters by querying the live instance configuration and reasoning over retrieved information.Its retrieve-then-reason strategy selects a task-relevant subset of the deployed configuration before prediction.
- Approaches: Discovery retrieval is adaptive and supports sequential multi-step rollouts by appending each predicted state before predicting the next.This lets the agent retrieve relevant rules, schemas, records, or SLA definitions as cascade complexity increases.
6 Experiments
The experiments evaluate transition prediction across CascadeBench and World of Workflows using prompting, fine-tuning, and runtime discovery. Fine-tuning is strong in-distribution but loses much of its advantage under shift, while discovery improves robustness and multi-step prediction by accessing active transition logic.
- 6 Experiments: The evaluation uses proprietary frontier models, LoRA-finetuned open-weight models, and zero-shot prompted baselines across CascadeBench and World of Workflows.Methods are scored using IoU and IoU(T+F), which measures affected table-field pairs as well as stricter transition agreement.
- 6 Experiments: Without business rules, prompted models reach 9–16 IoU(T+F) on CascadeBench, while SFT adds roughly 2–3 IoU points there and about 10 points on World of Workflows.With business rules, SFT is not uniformly beneficial: Qwen-3.5-27B reaches 50.9 IoU, while other models gain little or regress.
- 6 Experiments: In-distribution IoU rises to 91.6 for Gemma-4-31B and 82.0 for Qwen-3.6-27B after fine-tuning, but both fall to roughly 40–41 IoU on CascadeBench under shift.Fine-tuned models remain stronger than prompted baselines, but most of their in-distribution edge is lost on unseen schemas and configurations.
- 6 Experiments: Discovery improves state-prediction IoU over matched prompted baselines for every evaluated model and rollout horizon from k=1 through k=5.At k=1, Opus 4.6 rises from 0.40 to 0.45 and Sonnet 4.6 from 0.32 to 0.44; gains reach roughly 0.10 IoU in some settings.
- 6 Experiments: Discovery is most beneficial when transition logic exceeds the schema, recovering much of the performance gap on Tier 2 cascades and Tier 3 conflicts.The comparison focuses on cases where missing rule context makes prompting inadequate.
7 Discussion
CascadeBench performance depends strongly on access to business rules, especially for cascades and execution-dependent effects. Runtime discovery improves prediction across rollout depths by repeatedly grounding forecasts in the active deployment.
- Effect of Business Rules: 38–51 IoU with business rules falls to 7–12 without them across frontier, base, and SFT models.The collapse is consistent across model sizes and families, indicating that business rules carry the dynamics measured by CascadeBench.
- Tier-Stratified Prediction: Prompting without rules handles simple schema-determined effects but fails on harder tiers involving hidden workflows, rule cascades, and execution-dependent effects.Runtime discovery recovers much of the information available in the oracle rule-in-context setup.
- Depth Analysis: Discovery Agents outperform matched prompted baselines from k=1 through k=5, although performance generally decreases as rollout depth grows.Longer horizons require deeper cascade prediction and create more opportunities for error accumulation.
- Depth Analysis: Repeated retrieval of live records, active rules, and reference identifiers refreshes deployment context during multi-step prediction.This contrasts with baselines that rely primarily on initial context and prior outputs.
8 Conclusion
The paper finds that enterprise transition prediction is vulnerable to deployment-specific configuration changes. It concludes that readable live rules favor combining learned priors with runtime retrieval rather than relying solely on internalized dynamics.
- Conclusion: Offline-trained world models perform well in-distribution but degrade on held-out configurations, while discovery agents remain more robust under shift.Discovery agents retrieve relevant rules at inference time and avoid some error compounding seen in purely internalized models.
- Conclusion: When transition logic is readable from the live system, agents should combine learned priors with runtime retrieval and reasoning.The paper identifies learning when, what, and how to retrieve as a next step.
9 Limitations
The evaluation is bounded by assumptions about readable business rules, platform coverage, transition tiers, model selection, and tool-use reliability.
- Access and Deployment Assumptions: The discovery agent assumes business rules are readable on the live instance; access controls reduce it to the prompted baseline.This constrains applicability where runtime rule access is unavailable.
- Evaluation Scope: The evaluation is single-platform on ServiceNow, with quantitative results focused on Tier 1 and Tier 2 transitions.Tier 3 results are reported separately, and its stratification is restricted to audit-log-detectable multi-rule conflicts.
- Evaluation Scope: Broader execution-order dynamics, including asynchronous interleaving, race conditions, and internal scheduling, remain out of scope.These dynamics are present in the Enterprise Gym corpus but are not separately stratified because attribution requires resolving execution-order semantics.
- Model and Tooling Scope: Discovery-agent comparisons rely on a small set of open-weight models where LoRA finetuning is feasible.On weaker open-weight models, discovery performance is bounded by tool-use capability.
A Limitations
The evaluation is bounded by inspectability, tool-use reliability, platform coverage, and the limited scope of same-model comparisons. These constraints make the relative value of discovery versus training deployment-dependent.
- Inspectability: Runtime discovery assumes that relevant business rules and supporting tables are readable on the live instance.Access controls can prevent discovery and reduce it to the prompted baseline.
- Tool use: Tool-use reliability limits discovery, with open-weight 27–31B models underperforming LoRA variants in some conditions.The paper therefore reports that frontier APIs favor discovery, whereas constrained open-weight deployments favor finetuning.
- Platform scope: Experiments evaluate only ServiceNow, while other enterprise platforms differ in rule formalisms, cascade semantics, and inspectability guarantees.Transfer to other platforms is expected but not directly demonstrated.
- Tier 3 scope: Tier 3 evaluation covers audit-log-detectable multi-rule conflicts but excludes broader execution-order behaviors such as asynchronous interleaving and race conditions.Extending Tier 3 to these cases is left for follow-up work.
- Comparison scope: The same-model discovery-versus-training comparison covers only Qwen-3.5/3.6-27B and Gemma-4-31B, a small open-weight model set.The comparison is restricted to models where LoRA finetuning is feasible.
- Observed robustness: Across all evaluated models and horizons k = 1, . . . , 5, discovery improves over matched prompted baselines, with the largest gains at intermediate horizons.The passage attributes this pattern to retrieval tracking the evolving state as static-context errors compound.
C Tier-Stratified Results
CascadeBench stratifies enterprise dynamics from schema-determined effects to rule-composed cascades and execution-inferred conflicts. Discovery matches the oracle on the first two tiers but both methods remain bounded on Tier 3.
- Tier definitions: CascadeBench measures three tiers: schema-only effects (T1), rule-composed cascades (T2), and execution-inferred dynamics (T3).The benchmark operationalizes increasing transition complexity across these tiers.
- Direct prompting: T1 Direct IoU remains 0.56–0.60 across all eight models, while T2 and T3 fall to 0.00 under Direct prompting.Schema following suffices for T1, whereas business rules are load-bearing for T2 and T3.
- Discovery performance: Discovery Agent mean IoU decreases from T1 (0.648) to T2 (0.635) and T3 (0.524).The Oracle condition is flatter: T1 0.634, T2 0.635, and T3 0.569.
- Oracle comparison: Discovery matches the rule-oracle on T1 and T2 without preloaded ground-truth rules, with mean DA−Oracle deltas of +0.014, +0.001, and −0.046 across T1–T3.Both methods plateau on T3 because execution-order resolution is not exposed in the configuration they read.
- Rule context: Open-weight Qwen-3.5-27B, Qwen-3.6-27B, and Gemma-4-31B reach ALL IoU ≥0.66 when rules are supplied in the prompt.In this regime, the passage identifies rule content rather than model scale as the determining factor.
- Benchmark construction: CascadeBench is generated through schema creation, business-rule cascade construction, and live execution with audit capture and validation.The construction pipeline deploys rules, executes triggering actions, and retains only fully validated executable examples.
H CascadeBench: Failure Mode Analysis
Even with complete context, CascadeBench exposes reasoning failures in record creation, deep-cascade coverage, and multi-record effects. These failures indicate that retrieval alone does not solve multi-step cascade composition.
- Failure analysis: Oracle evaluation identifies three recurring failures despite providing schema, supporting data, and all relevant business rules.The analysis treats these as failures under complete context rather than missing-information errors.
- P1: Creation blindness: Insert/creation blindness causes models to omit new records or predict only 1–3 of their 7–12 auditable field changes.Creation-phase audits are recalled at 24–27%, roughly half the 36–47% recall for update audits.
- P2: Cascade fade-out: Cascade fade-out reduces recall from 75–85% for BR1–BR2 to 4–11% for rules at execution order ≥400.Entire tables produced by deep-cascade rules can disappear from predictions.
- P3: Single-record assumption: The single-record assumption makes models predict at most one affected record per table, missing parallel updates and sibling inserts.This affects cascades using result-set iteration or multiple inserts.
- Implication: The failure modes persist when rules are present, so closing the gap to perfect prediction requires training models to compose multi-step rule executions.The paper distinguishes this reasoning bottleneck from retrieval failure.
- Representative trajectory: One trajectory produced 22 predictions with 15 correct, yielding 68% precision and 34% recall against 44 ground-truth audits.The model accurately traced early rules but missed 17 changes from later rules.