PaperlayerView exact revision on arXiv

Source-linked AI summary

An Auditable Symbolic-RAG-Generative AI Architecture for Goal-Oriented Conversation Orchestration

Ramon Gonzalez, Antonio Diaz

arXiv:2609.07152v1cs.AI

TL;DR

Goal-oriented conversational systems must answer factual questions, interpret visitor information, and advance business objectives without becoming rigid questionnaires. The paper proposes GRACE, a Symbolic-RAG-Generative architecture that separates grounded answering from visitor-evidence state updates and selects constrained follow-ups. Across two domains and languages, it reports strong symbolic-state performance, including 84.9% exact state-transition accuracy and 100% monotonicity.

  • Problem

    Goal-oriented conversational systems must combine factual answering, visitor understanding, and business progression without reducing interaction to rigid questionnaires.

  • Method

    GRACE compiles business intent into immutable objectives and uses separate RAG answering, visitor-evidence state updates, and visitor-utility-constrained follow-up selection.

  • Results

    Across both domains and languages, GRACE achieved 84.9% exact state-transition accuracy, 91.6% evidence precision, 89.6% evidence recall, 100% monotonicity, and 94.1% terminal-state accuracy.

  • Takeaways & Limitations

    The evaluation supports integrating explicit symbolic control with generative dialogue while preserving company-grounded answers and visitor-only business-state progression.

Abstract

from arXiv · show

Goal-oriented conversational systems must answer factual questions, understand visitor-provided information, and advance business objectives without becoming rigid questionnaires. This paper proposes a Symbolic-RAG-Generative architecture centered on the Goal-oriented Retrieval-Augmented Conversation Engine (GRACE). An instruction-constrained Business Goal Compiler transforms business intent into an immutable objective set, normalized priority vector, canonical questions, and initial state vector. At runtime, GRACE receives the complete conversation history, latest visitor message, current state, and grounded answer generated by a separate RAG component. It updates completion only from visitor-authored evidence and selects one contextually modulated follow-up. The core policy maximizes expected business progress subject to a minimum visitor-utility constraint. We formalize the state, monotonic transitions, source separation, question modulation, and constrained policy; present the reference architecture; and define an evaluation comprising 24 English real-estate and 10 Spanish professional-cleaning conversations, totaling 119 protocol-defined visitor turns. Across both domains, GRACE achieves 84.9% exact state-transition accuracy, 91.6% evidence precision, 89.6% evidence recall, 100% monotonicity, and 94.1% terminal-state accuracy. The evaluation establishes compelling symbolic-state performance across standard, multi-goal, RAG-detour, validation, refusal, and robustness scenarios.

I. Introduction

GRACE combines generative understanding and grounded answering with an explicit, auditable goal-state envelope for advancing business objectives without reducing conversation to fixed questions.

  • Design position: GRACE targets a bounded middle ground between rigid slot-filling, retrieval-only assistants, and flexible but behaviorally variable LLM agents.It preserves explicit business-state control while allowing generative understanding and natural phrasing.
  • Core contributions: The architecture separates business-goal compilation from runtime orchestration and maintains an immutable, monotonic symbolic completion vector.The contributions also include canonical questions, priorities, and auditable transition records.
  • Relation to dialogue state tracking: GRACE retains explicit-state principles from dialogue state tracking but changes state semantics from task constraints to sufficient visitor-authored evidence for approved objectives.It adds immutable priorities, monotonic completion, refusal handling, and evidence witnesses for transitions.
  • Provenance: GRACE separates visitor-authored evidence from company-grounded answers, preventing retrieved facts from being treated as information learned about the visitor.Grounded company facts may condition follow-up wording, but they cannot complete visitor objectives.
  • Policy: The system applies an explicit visitor-utility floor before optimizing business progress and can select one approved question or NONE.This constrains orchestration while retaining flexibility for out-of-scope or interrupted interactions.

III. Problem Formulation

The formulation defines a compiled objective specification, complete conversation history, context-modulated canonical questions, and the information supplied to runtime orchestration.

  • A. Objective Set, State, and Conversation History: The Business Goal Compiler converts a manager’s business intention into a stable symbolic specification before runtime orchestration begins.Its outputs define the objective set and the conversation information consumed by GRACE.
  • A. Objective Set, State, and Conversation History: Each objective q_i includes a stable identifier, canonical question, completion criterion, and weight.These fields provide the operational structure for objective tracking and question selection.
  • A. Objective Set, State, and Conversation History: CurrentState represents the state used by the orchestration process.The formulation introduces it as a formal runtime component alongside the compiled objectives.
  • A. Objective Set, State, and Conversation History: The history H_t is the complete ordered conversation before the current decision, including visitor messages and AI-visible responses.This makes history broader than visitor-authored text alone.
  • A. Objective Set, State, and Conversation History: An AI response combines the grounded RAG answer with the question or NONE returned by GRACE.The history therefore records both sides of the visible system response.
  • A. Objective Set, State, and Conversation History: The canonical question defines what must be learned, while its wording is modulated by the grounded answer and conversation context.For example, a generic budget question can be reframed around a retrieved property price.

B. Retrieval Grounding and Provenance

The retrieval component supplies company-grounded answers independently of symbolic state, while GRACE preserves a strict provenance boundary: only visitor evidence can complete objectives.

  • Retrieval interface: The RAG component produces a company-grounded answer from the current visitor message and passages retrieved from a frozen company corpus.It does not use dialogue history or symbolic state.
  • Retrieval interface: The retrieval interface ranks corpus passages using multilingual embedding similarity and a configured retrieval depth.The similarity score ranks candidate passages, with larger values indicating greater semantic similarity.
  • Retrieval grounding: A grounded company claim must be supported by at least one retrieved passage.Equations (7)–(8) specify this interface rather than introducing a new retrieval algorithm.
  • Provenance: Retrieved passages and RAG answers belong to company evidence and may determine factual answers or modulate follow-ups, but cannot complete visitor objectives.This keeps factual synthesis separate from business-state progression.
  • Source-constrained transition: A completion event requires an unfinished coordinate and visitor-authored evidence satisfying that objective’s published criterion.The accepted event is therefore tied to evidence in the conversation history and current visitor turn.
  • Source-constrained transition: The binary transition is monotonic: once valid completion is recorded, the completion flag cannot return from 1 to 0.The model addresses evidence accumulation, while changed, revoked, or uncertain preferences remain future work.

D. Visitor-Constrained Business Policy

GRACE applies a transparent visitor-utility constraint before selecting the highest-priority unfinished objective, while an offline Compiler validates and publishes stable symbolic goals.

  • Business policy: Among visitor-admissible questions, GRACE selects the highest-weight unsatisfied goal and resolves equal priorities using the Compiler’s approved order.The policy deliberately uses transparent deterministic business utility rather than learned elicitation probabilities.
  • Visitor utility: GRACE assigns visitor utility using normalized relevance, burden, repetition, and privacy components with an explicit reference threshold of ε=0.The coefficients are policy choices rather than learned quantities, and the reference threshold admits actions whose relevance offsets their penalties.
  • Constrained selection: Equation (14) first excludes actions below the visitor-utility floor, then selects the feasible action with maximum business utility.The no-question action remains feasible at the reference threshold, ensuring at least one admissible action.
  • Offline compilation: The offline Business Goal Compiler converts manager intent into goals, priorities, evidence criteria, and an initial state before deterministic validation and human approval.Published output requires unique identifiers, complete fields, normalized weights, and permitted data criteria.

B. GRACE Runtime

GRACE separates grounded company answers from visitor evidence at runtime, validates symbolic state transitions, and emits one modulated question or NONE alongside the response.

  • Runtime flow: The RAG component produces a grounded answer before GRACE combines it with history, the visitor message, goals, weights, and current state.The grounded answer may modulate a follow-up but cannot provide completion provenance.
  • Source separation: The reference deployment keeps retrieval and grounded synthesis on a separate Pinecone-connected RAG service while orchestration receives the completed result independently.This separation supports load balancing and prevents retrieval latency or corpus access from being hidden inside symbolic state updates.
  • Validation: Validation checks schema conformance, immutable goal identifiers and weights, legal monotonic transitions, visitor support, and applicable contact syntax.If validation fails, the accepted state remains unchanged.
  • Turn processing: GRACE identifies visitor-only evidence, validates it, forms the new state, generates faithful question modulations, and applies the constrained policy.The turn procedure persists the new state and transition record, then appends a question only for an ASK action.
  • Evaluation setup: The evaluation used 34 scripted conversations and 119 protocol-defined visitor turns across English real-estate and Spanish professional-cleaning demonstrations.The Compiler, GRACE, and RAG synthesizer used Mistral Small 4 with temperature 0.2 and a 4,096-token maximum output length.

B. Performance Metrics

The evaluation distinguishes exact turn-level state updates from terminal outcomes and separately measures evidence support and monotonicity.

  • State-transition accuracy: State-transition accuracy requires every state coordinate to be correct at the exact turn where visitor evidence appears.A correct terminal vector cannot conceal a delayed or misplaced update.
  • Terminal accuracy: Terminal-state accuracy measures whether each conversation preserves the complete evidence-derived outcome, unlike transition accuracy’s turn-level requirement.The two metrics therefore distinguish eventual correctness from correct transition timing.
  • Monotonicity: Monotonicity compliance tests the binary-state invariant separately because an unsupported transition can still be monotonic.This isolates transition direction from evidentiary correctness.
  • Evidence metrics: Evidence precision penalizes completions that are unsupported or temporally misattributed, measuring whether recorded events are justified at the correct turn.Completion events are conversation–turn–objective tuples, so one visitor turn may contain several events.
  • Evidence metrics: Evidence recall measures how much completion evidence supplied by visitors is captured at the correct turn by the state update.The reference set is derived exclusively from visitor-authored evidence.

VI. Demonstration Configurations

The demonstrations instantiate the frozen architecture in two languages and domains with deliberately concentrated conversational challenges, using company-authored knowledge and independently compiled objectives.

  • Configuration control: The two demonstrations use frozen model, decoding, goal, corpus, retrieval, index, and validator configurations within each demo.Each company was simulated experimentally and supplied plain-language business intent plus a company-authored FAQ collection.
  • Real-estate configuration: The real-estate configuration compiles budget, location, requirements, and contact into four ordered objectives from the manager’s intention.Its company-authored corpus covers properties, locations, prices, amenities, financing, viewing procedures, reservations, and verified contact channels.
  • Demonstration design: The evaluation covers 24 English real-estate conversations and 10 Spanish professional-cleaning conversations organized into focused challenge families.Families test reordered or dense evidence, RAG interruptions, validation and correction, refusal and re-engagement, negation, contradiction, social language, and language switching.
  • Cleaning configuration: The cleaning configuration targets client or facility type, location, and contact details for personalized advice or quotations before human intervention.Its FAQ corpus covers service areas, communities and offices, cleaning frequency, specialist treatments, products, quality control, incidents, credentials, and contact channels.
  • Evaluation units: The English real-estate condition contains 86 evaluated turns, while the Spanish cleaning condition contains 33.Recorded state vectors were interpreted according to the objective order defined by each Compiler output.

A. English Real-Estate Demonstration

The English real-estate demonstration tests GRACE on exact state tracking and evidence attribution across challenging conversation families, including RAG detours and multi-goal turns. It shows strong robustness and monotonicity, while exposing occasional criterion-aligned attribution errors.

  • Results: 84.9% of 86 real-estate turns produced the exact four-coordinate reference state (STA).All 15 robustness turns and 19 of 20 RAG-detour turns matched their reference states.
  • Robustness: RAG questions, social language, negation, and language switching generally did not contaminate budget, location, requirements, or contact coordinates.These conditions were explicitly included among the demonstration’s robustness and RAG-detour cases.
  • Results: Evidence attribution reached 91.2% precision and 88.3% recall across completion events rather than visitor turns.GRACE matched 83 of 91 recorded completions to the correct objective and turn, also matching 83 of 94 reference completions.
  • Results: 100% monotonicity was maintained, and 22 of 24 conversations ended in the correct state (TSA = 91.7%).The system never reversed a completed coordinate despite 13 incorrect intermediate vectors.
  • Error analysis: Criterion-aligned attribution errors remained visible when one message supported multiple objectives or when wording admitted competing interpretations.The Spanish demonstration reports related failures in reverse ordering, explicit location correction, and language switching, including an “office cleaning” ambiguity.
  • Evaluation design: The evaluation covers standard, multi-goal, RAG-detour, validation, refusal, and robustness conversation families.These families vary evidence order and density, introduce malformed or revised values, and test refusal, contradictory data, and multilingual input.

VIII. Discussion

The discussion presents GRACE as an auditable architecture that integrates grounded answering with symbolic business-goal progression, while exposing evaluation strengths and remaining scope boundaries. Results are strong across domains and languages, but turn-level attribution and broader comparative evaluation remain open.

  • Cross-domain findings: 84.9% STA, 91.6% EP, 89.6% ER, 100% MC, and 94.1% TSA were obtained across both domains and languages.The results support coexistence of RAG answers with visitor-only state representation and operation across English real estate and Spanish professional cleaning.
  • Evaluation boundary: Corner-case scripts provide strong behavioral coverage but do not estimate normal customer use.They concentrate multi-goal evidence, interruptions, invalid contacts, corrections, refusals, and language switching while avoiding real-prospect personal data.
  • Error analysis: Some valid evidence is committed one turn late, so history-based recovery improves terminal accuracy without guaranteeing exact turn-level attribution.The authors recommend evaluating all currently supported objectives before persisting the new state while retaining history-based recovery.
  • Architecture and contribution: GRACE combines company-grounded answering, visible symbolic state, unsolicited visitor evidence, and constrained objective selection without reducing interaction to fixed questions.The architecture separates knowledge answering from business-state progression and applies a visitor-utility constraint.
  • Evaluation: Across 34 conversations and 119 visitor turns, GRACE achieved 91.6% evidence precision, 100% monotonicity, and 94.1% terminal-state accuracy.The evaluation covered two domains, two languages, independently compiled objective sets, and corner cases including RAG interruptions, corrections, refusals, and language switching.
  • Future work: Future work should compare GRACE with closed-question, RAG-only, and general-agent alternatives using common goals and corpora and broader behavioral and human-centered metrics.Proposed measures include state accuracy, grounding, completion, burden, latency, cost, complexity, variance, naturalness, and usefulness.
  • Future work: The state model should expand beyond binary completion and be tested across more domains, languages, objective sets, model families, and naturally occurring conversations.The proposed extensions include uncertain, revised, revoked, and contradictory preferences plus controlled prompt-ablation experiments.

Appendix A English Real-Estate Demonstration: Complete Conversation Traces

The English real-estate appendix reports 24 complete conversation traces in chronological order, covering canonical, reordered, multi-goal, factual-detour, validation, refusal, correction, and language-switching scenarios.

  • Trace coverage: Twenty-four English real-estate conversations are reported chronologically, linking each visitor message and visible response to recorded and evidence-derived reference states.The appendix spans Tables VII–XXX.
  • RAG and factual detours: The traces include factual interruptions and supported or unsupported availability, schools, legal-process, and other knowledge detours.These cases test whether factual questions coexist with business-goal state progression.
  • Validation and correction: Validation and correction scenarios cover invalid email, invalid telephone, ambiguous budgets, explicit location changes, and corrected requirements.These cases examine state updates when visitor-provided information is malformed, ambiguous, or revised.
  • Robustness and refusal: The appendix also covers contact refusal, explicit stopping, re-engagement, budget-specific refusal, temporary uncertainty, irrelevant social turns, language switching, and contradictory contact information.These traces test refusal handling, recovery, robustness, and multilingual interaction.

Appendix B Spanish Professional-Cleaning Demonstration: Complete Conversation Traces

The Spanish professional-cleaning appendix reports 10 complete conversation traces in chronological order, covering progression, RAG interruptions, validation, correction, refusal, and language switching.

  • Trace coverage: Ten Spanish professional-cleaning conversations are reported chronologically, linking each visitor message and visible response to recorded and evidence-derived reference states.The appendix spans Tables XXXI–XL.
  • Progression patterns: The demonstrations include canonical, reverse, and all-objectives-together progression patterns.These cases test whether the same compiled-goal representation handles different disclosure orders and dense evidence.
  • RAG interactions: The traces include frequency, service-area, and specialist-service RAG interruptions or questions.These cases examine factual detours within Spanish professional-cleaning conversations.
  • Validation and correction: Validation and correction cases cover invalid email and location correction.These scenarios test recovery from malformed contact information and revised visitor evidence.
  • Refusal and language: The appendix includes refusal with voluntary re-engagement and English-Spanish language switching.These cases test refusal handling and multilingual robustness.
Loading 2609.07152v1…