Source-linked AI summary
Agent Gym: A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback
Pouya Ghiasnezhad Omran, Michael Zimmermann, Duncan Cambridge, Ashmita Kapoor, Tanya Dixit
TL;DR
Deployed LLM agents remain behaviorally fixed while business rules and edge cases evolve, and existing approaches do not provide structured post-deployment correction without source-code changes. Agent Gym wraps existing agents in a continuous evaluation-and-evolution loop using declarative governance, runtime investigation and correction, and human-in-the-loop learning. An open-source invoice-processing implementation demonstrates that the framework is fully operational, while promotion of corrections into permanent agent logic remains future work.
Problem
Deployed agents face evolving business rules, edge cases, and production failures, while existing approaches lack structured continuous correction without modifying source code.
Method
Agent Gym wraps existing agents in a domain-agnostic loop combining declarative constitution artifacts, runtime evaluation and correction, and human-in-the-loop rule learning.
Results
An open-source invoice-processing reference implementation demonstrates that the complete framework is fully operational and ready for adoption.
Takeaways & Limitations
Domain experts can contribute knowledge through traceable, reversible corrections while the acting agent remains independently versioned and unchanged.
Takeaways & Limitations
The current learning loop modifies only the ALF rule base; automatic promotion of systematic corrections into permanent agent logic is future work.
Abstract
from arXiv · showhide
Large Language Model (LLM) agents deployed in production environments face a fundamental tension: the agent's behavior is frozen at deployment time, while the business rules and edge cases it must handle continue to evolve. Existing approaches address agent construction and one-time evaluation but provide no structured mechanism for continuous post-deployment behavioral correction without modifying the agent's source code. Most of the approaches offered in the market, require intense collection of logs and traces, and re-examining the agent design by the engineering team, a process which is heavy, long and negates the economical value of agentic transformation. We introduce Agent Gym, a modular, domain-agnostic framework that wraps any existing LLM-based agent in a continuous evaluation-and-evolution loop. The framework provides six composable capabilities --- Act, Evaluate, Investigate, Correct, Learn, and Observe --- organized across three architectural zones: a constitution layer that codifies domain knowledge in configuration artifacts, a runtime inference pipeline that chains acting, investigation, and adaptive correction, and a learning loop that enables subject matter experts to discover and validate new correction rules through natural language interaction. The key technical contributions include a hybrid deterministic-LLM correction engine with 21 condition operators and three-tier actions, a three-layer investigation architecture for ground-truth-free compliance validation, and a programmatic safety loop that guarantees rule correctness before human approval. We further introduce the Spec-to-Note Gap, an autoencoder-inspired view of agentic system transparency. An open-source reference implementation for invoice processing demonstrates that the framework is fully operational and ready for adoption.
1. Introduction
Production LLM agents face frozen behavior amid evolving business rules, edge cases, and compliance requirements. Agent Gym addresses this gap with a modular framework for continuous evaluation and correction without changing agent source code.
- The static agent dilemma arises because deployed agent behavior is frozen while business environments and business rules continue to change.
- Production failures include unseen edge cases, systematic rule misinterpretations, and rule evolution that agents cannot absorb without code modification.
- SME-to-correction workflows pass through engineers who interpret issues, modify code or prompts, redeploy, and request verification, making the process slow and error-prone.
- Traditional evaluation relies on labeled ground truth, which may be unavailable for novel cases or insufficient to capture nuanced business rules.
- Agent Gym wraps existing agents in a domain-agnostic continuous evaluation-and-evolution framework without requiring source-code modification.
- Its contributions include a 21-operator hybrid correction engine, ground-truth-free investigation, a safety loop before SME approval, and constitution-based governance artifacts.
6. The Spec-to-Note Gap,
The paper presents the Spec-to-Note Gap as an autoencoder-inspired perspective on agentic-system transparency. An open-source invoice-processing implementation demonstrates that the framework is operational and ready for adoption.
- The Spec-to-Note Gap generalizes round-trip correctness from code to whole-system agentic transparency.
- The open-source invoice-processing reference implementation demonstrates the complete framework is operational and ready for adoption.
2. Related Work
Related work primarily addresses agent construction, static evaluation, self-improvement, constitutional alignment, human feedback, rule management, and documentation. Agent Gym positions itself as a complementary operational layer for continuous correction and governance.
- LLM Agent Frameworks: Agent-construction frameworks build capable agents but lack a structured mechanism for post-deployment behavioral evolution; Agent Gym wraps them in a continuous correction loop.
- Agent Evaluation: Static benchmarks and LLM-as-judge methods measure agent quality at a point in time, whereas Agent Gym combines deterministic and LLM-based evaluation for production correction.
- Self-Improving Agents: Self-Refine and Reflexion modify the agent itself, while Agent Gym applies corrections in a separate downstream layer with independent version control.
- Constitutional AI and Alignment: Agent Gym extends the constitutional metaphor to runtime validation and correction through a reconstructed rules book rather than model training.
- Human-in-the-Loop Learning: Human feedback produces structured correction rules applied at inference time without retraining, making corrections immediate, editable, and reversible.
- Business Rule Management: Unlike deterministic BRMS engines alone, Agent Gym combines rule-based processing with LLM-driven actions for context- and semantics-dependent corrections.
- Model Documentation: The Spec-to-Note Gap proposes automated system-level documentation connected to round-trip correctness and existing documentation standards.
3. The Agent Gym Framework
Agent Gym wraps an existing agent in a three-zone architecture that separates constitutional domain knowledge, runtime validation and correction, and human-guided evolution. Its black-box, configuration-driven design combines deterministic checks with controlled LLM investigation and correction.
- Design Principles: Agent Gym treats the acting agent as a black box, observing input-output behavior without constraining its internal architecture.The framework requires only JSON artifacts as output and supports monolithic pipelines, multi-agent systems, and single LLM calls.
- Design Principles: Corrections operate in a separate downstream layer, preserving the agent’s original output while enabling auditability and independent rollback.
- Design Principles: Domain knowledge resides in declarative configuration artifacts, while humans review, approve, and validate proposed correction rules before production use.The constitution comprises master data and a human-readable rules book; permanent constitutional or agent-logic changes require multistakeholder approval.
- Architecture Overview: The architecture spans three zones: a constitutional layer, a sequential runtime pipeline, and a learning loop for SME-driven rule discovery.The runtime stages are acting, investigation against the constitution, and targeted ALF correction; the learning agent discovers new rules through a safety loop.
- Architecture Overview: Six composable capabilities organize the framework: Act, Evaluate, Investigate, Correct, Learn, and Observe.Observe is represented by operational dashboards planned for a future release.
- Investigation Agent: Investigation validates decisions against the rules book without ground-truth data through deterministic checks, cached LLM rule discovery, and conservative triple-checking.Deterministic checks verify sources and required validation steps; discovered rules are cached using a SHA-256 hash, and only violations confirmed in three runs are reported.
- ALF — Adaptive Learning Framework: ALF separates deterministic detection from contextual correction, applying matched rules downstream through a hybrid correction engine.Detection uses 21 condition operators with AND logic and no LLM involvement; correction can leverage LLMs when contextual understanding is required.
3.5. Learning and Evolution Loop
The Learning and Evolution Loop lets SMEs turn natural-language feedback into validated ALF correction rules while preserving deterministic checks, approval control, and auditability. It also integrates evaluation against ground truth when reference data exists.
- Learning Agent: The Learning Agent lets SMEs review agent outputs, describe desired corrections in natural language, and collaboratively discover ALF rules.Its modules include case loading, rule discovery, impact assessment, and rule writing.
- ALF Correction: ALF evaluates conditions deterministically with 21 operators, then collects matching rules, groups actions by tier, and executes them in fixed order.Scope-based mutual exclusion prevents conflicts, and the pipeline limits correction to at most two LLM calls per case.
- Programmatic Safety Loop: The programmatic safety loop validates candidate rules, verifies target coverage, detects collateral matches, and auto-tightens conditions before SME approval.The loop is enforced in code and may broaden failed target matches or add narrowing constraints when unintended matches occur.
- Governance and Lifecycle: No rule is applied without explicit human approval, while enabled flags, backups, conflict checks, and metadata support controlled rule lifecycle management.The framework also preserves audit history and enables rollback or deactivation without deletion.
- Future Evolution: A periodic review path is anticipated for promoting consistently triggered corrections into agent logic, but promotion automation, review queues, per-rule tracking, and sunset policies remain unimplemented.These missing capabilities are identified as future work for keeping the correction rule base lean.
- Evaluation: The Evaluation Engine combines schema-driven deterministic comparison with optional LLM-as-judge assessment when ground truth is available.Numeric fields use configured financial tolerances, while normalized text fields use exact matching.
4. Constitution-Based Agent Creation
Agent Gym uses a constitution of declarative artifacts both to govern an existing agent and to specify the behavior of new agents. This separates domain knowledge from framework code and supports refinement through operational experience.
- Constitution as Specification: The rules book and master data serve simultaneously as governance instruments for deployed agents and specifications for creating new agents.Together they describe validation rules, schemas, taxonomies, thresholds, and pipeline configurations.
- Bootstrap and Refinement: A constitution can be bootstrapped from an existing agent or authored first by domain experts and then used to guide agent construction.The latter reverses the conventional code-first flow by treating desired processing logic as the initial artifact.
- Co-evolution: As correction rules expose systematic errors, the constitution records those patterns and evolves from initial design intent toward accumulated operational experience.This creates a structured basis for later agent re-engineering when code-level changes become warranted.
- Onboarding New Agents: Onboarding a new agent requires the acting agent and a constitution, while the ALF, investigation, and learning components remain domain-independent.Adapting to new domains therefore centers on generating or changing constitutional artifacts rather than modifying framework code.
5. The Spec-to-Note Gap: An Autoencoder View of Agentic System Transparency
The Spec-to-Note Gap treats system transparency as a natural-language round trip from specification through implementation to an auditor-generated note. Comparing the specification and note can expose behaviors that ordinary evaluation misses.
- Motivation: Agentic systems are difficult to document manually because they combine planners, tools, retrievers, guardrails, and multiple agents whose behavior changes over time.The proposed pattern addresses documentation of the resulting system as a whole.
- Transformation Chain: The pattern encodes a natural-language specification into implementation and decodes the implementation into a transparency note using an LLM auditor.The auditor inspects code, prompts, tools, evaluation results, and sample execution traces.
- Autoencoder Analogy: The specification is the input, the implemented system is the latent representation, and the transparency note is the reconstruction in this autoencoder analogy.The analogy frames system behavior as a natural-language round trip.
- Diagnostic Gap: The gap between specification and transparency note functions as a reconstruction loss that surfaces missing capabilities, silent scope creep, and unmeasured behaviors.The pattern extends round-trip correctness from function-level consistency to whole-system granularity.
- Human Review: A structured transparency note gives SMEs a review interface for flagging wrong assumptions, missed populations, and regulatory edge cases, with comments becoming system tickets.The note is intended to communicate behavior in domain-readable language rather than expose evaluation harnesses.
- Open Questions: Open questions include auditor faithfulness, regeneration timing, optimization use of the gap, and adversarial auditing as a release gate.These questions define unresolved directions for operationalizing the pattern.
6. Reference Implementation
The open-source invoice-processing implementation instantiates Agent Gym as a self-contained dual-mode system with inference and SME-guided learning. It implements the framework’s investigation, correction, safety, and evaluation components and demonstrates domain adaptability and operational readiness.
- Scope and Platform: The reference implementation demonstrates the complete Agent Gym framework for invoice processing using Google ADK and Gemini models.The implementation is described as open source and built around the framework’s full capability set.
- System Packaging: A unified dual-mode LlmAgent with 18 registered function tools supports both document-processing inference and SME-guided rule discovery.The package is self-contained, including data, configurations, test cases, rules, and evaluation artifacts.
- Acting Pipeline: The acting agent uses a nine-stage sequential pipeline covering classification, extraction, progressive compliance validation, transformation, output generation, and audit logging.Each stage emits numbered JSON artifacts, and early exit can skip remaining validation after rejection.
- Investigation: The investigation agent implements cached rule discovery, conservative triple-check validation, batched deterministic rule groups, and section-filtered rules-book context.Its Layer 3 thresholds are 90% for violations and 70% for ambiguous cases, with ambiguity treated as compliant.
- Correction Engine: The ALF engine implements all 21 condition operators and uses dynamic field references, Collect-Plan-Execute aggregation, and scope-based mutual exclusion.These implementation details instantiate the deterministic correction architecture described earlier.
- Learning Mode: Learning mode implements candidate-rule discovery, deterministic impact assessment with the target case included, and autotightening against collateral matches.Autotightening can use vendor name, amount range, rejection template text, and service category.
- Testing and Evaluation: The implementation includes tests for operators, action executors, schema validation, conflict detection, and impact assessment, plus two-layer quality evaluation.Evaluation combines configurable-tolerance deterministic field comparison with optional LLM-as-judge assessment.
- Demonstrated Properties: The demonstrated properties include domain adaptability, operational readiness through web or command-line use, and self-containment with bundled cases, documents, ground truth, and rules.Replacing master-data YAML and the acting pipeline adapts downstream components automatically.
7. Discussion and Future Work
Agent Gym separates reusable framework infrastructure from domain-specific configuration, while identifying automation, scaling, lifecycle, and validation boundaries for future work.
- The framework separates acting, correction, and learning components, enabling independent versioning, testing, and evolution across business processes.
- The current implementation validates the framework in a single domain, so multi-domain validation remains necessary to establish domain agnosticism.
- Bootstrap is currently manual, while an LLM-assisted bootstrap agent is proposed to reduce constitutional-artifact authoring effort for new domains.
- Investigation-agent validation costs scale with case and rule-group counts, although caching, batching, section filtering, and early exits reduce costs in practice.
- Future work includes proactive rule suggestion, multi-run trend analysis, and lifecycle management for inactive or consistently firing rules.
- Consistently firing correction rules should be promoted into the acting agent’s permanent logic and then retired from ALF, but this loop is not yet automated.
8. Conclusion
Agent Gym wraps existing LLM agents in a continuous human-guided evaluation and correction loop without source-code modification. Its reference implementation demonstrates operational deployment, while its architecture combines rule-based correction, ground-truth-free investigation, governance artifacts, and transparency tooling.
- Agent Gym addresses the static agent dilemma through continuous observation, correction, and learning without modifying the acting agent’s source code.
- Its hybrid deterministic-LLM correction engine separates reliable detection from flexible correction, while investigation validates compliance without ground truth.
- A programmatic safety loop checks proposed rules before human approval, and constitutional artifacts serve as both human documentation and machine specifications.
- The Spec-to-Note Gap provides an autoencoder-inspired approach to automated agentic-system transparency.
- An open-source reference implementation demonstrates that Agent Gym is fully operational rather than merely theoretical.
Declaration on Generative AI
The authors used LLM-based tools during preparation for drafting, language checks, LaTeX formatting, and bibliography management, then reviewed and edited the content.
- LLM-based tools assisted with drafting, grammar and spelling, LaTeX formatting, and bibliography management.
- The authors state that they reviewed and edited the content and take full responsibility for the publication.