MACS: A Hybrid Multi-Agent Framework for Reliable Conversational E-Commerce Recommendation

Juli Huang; Hannah Clay; Sajjad Beygi; Thomas Sarda; Negin Golrezaei; Amin Saberi

Source-linked AI summary

MACS: A Hybrid Multi-Agent Framework for Reliable Conversational E-Commerce Recommendation

Juli Huang, Hannah Clay, Sajjad Beygi, Thomas Sarda, Negin Golrezaei, Amin Saberi

arXiv:2608.14068v1Aug 14, 2026cs.IRcs.AI

TL;DR

Fixed-catalog conversational recommendation requires reliable constraint satisfaction and preference tracking without unsupported product claims. MACS separates language interaction from deterministic catalog enforcement and persistent preference management, achieving the strongest multi-turn macro Pass@5 at 72% versus 56% and 52% for catalog-bound baselines.

Problem
Fixed-catalog conversational recommenders must satisfy accumulated user constraints, retrieve verified inventory, and generate grounded responses without web search, but this setting remains underexplored.
Method
MACS combines an LLM-based shopping agent for language interaction with a merchant agent that deterministically retrieves and filters products from the fixed catalog while preserving session preferences.
Results
72% macro Pass@5 versus 56% for GPT+Catalog and 52% for Gemini+Catalog, with the strongest reliability-oriented performance across evaluated multi-turn scenarios.
Takeaways & Limitations
Architectural separation between language interaction and correctness-critical enforcement is effective for improving fixed-catalog recommendation reliability without sacrificing response quality.
Takeaways & Limitations
Evaluations cover only a single consumer-electronics domain, so generalization to other merchant catalogs or recommendation domains is not established.

Abstract

from arXiv · show

Conversational recommendation for e-commerce is increasingly mediated by large language models (LLMs), yet many real-world deployments operate under a stricter requirement: recommendations must be drawn only from a merchant's fixed catalog, without web search or unsupported product claims. In this setting, the main challenge is reliability under hard constraints: the system must satisfy user requirements, remain grounded in available inventory, and preserve preferences across multiple conversational turns. We present MACS (Multi-Agent Commerce System), a hybrid multi-agent framework for reliable conversational recommendation in fixed-catalog settings. MACS uses LLMs for language-facing tasks such as interpreting user requests, eliciting preferences, and generating responses, while correctness-critical operations, including product retrieval, hard-constraint filtering, brand exclusion, and progressive relaxation, are executed deterministically by the merchant agent. A session-persistent preference layer tracks constraints across turns, enabling consistent handling of budget overwrites and exclusion reversals. On a 140-query single-turn benchmark, MACS achieves the highest pass rate (87.1%) and perfect brand compliance (1.000). On a 10-scenario multi-turn benchmark, MACS achieves the strongest macro Pass@5 (72% vs. 56% GPT+Catalog / 52% Gemini+Catalog) with zero constraint drift. The advantage is sharpest on exclusion reversal (100% vs. 20% / 0%) and constraint accumulation (100% vs. 60% / 40%). Mean judged response quality is similar across systems (0.751 vs. 0.736). These results suggest that hybrid architectures combining deterministic constraint enforcement with session-persistent preference tracking provide stronger reliability-oriented performance than catalog-bound prompt-only baselines in the fixed-catalog merchant setting.

1 Introduction

MACS addresses reliable conversational recommendation in fixed-catalog shopping by combining LLM-based interaction with deterministic merchant-side enforcement. Its contributions include a cooperating-agent architecture, session-persistent preference management, and an evaluation framework separating constraint correctness from response quality.

Motivation: Fixed-catalog shopping requires recommendations from verified merchant inventory without web search while respecting user constraints.This setting gives merchants control over product data.
Architecture: MACS separates language-facing shopping-agent tasks from deterministic merchant-agent retrieval and hard-constraint enforcement.The merchant agent retrieves products exclusively from the fixed catalog.
Session state: Session-persistent preference state preserves constraints across turns and supports budget overwrites and exclusion reversals.This enables consistent multi-turn recommendation behavior.
Evaluation framework: MACS introduces an evaluation methodology that separates deterministic constraint correctness from response quality.The framework evaluates these dimensions separately.

2 Related Work

MACS connects conversational recommendation, knowledge-graph retrieval, and LLM reliability under hard constraints, while targeting their combination in fixed-catalog commerce settings. Its design separates LLM-based preference elicitation from deterministic constraint enforcement and uses inspectable graph queries for retrieval.

MACS addresses the combination of deterministic constraint enforcement, session-persistent preference state, and fixed-catalog grounding absent from prior work in these areas.
Conversational Recommender Systems: Conversational recommender systems elicit preferences through multi-turn interaction using bandit-based questioning, reinforcement-learning dialogue policies, and LLM-integrated preference and response pipelines.
Knowledge Graphs and LLM Agents for Shopping: MACS uses knowledge graphs operationally for substitutes and compatibility through explicit Cypher queries, making its retrieval path directly inspectable.The substitutes relation covers products serving the same purpose at a similar price point.
LLM Reliability and Preference Elicitation: LLM faithfulness under hard constraints remains an open problem: chain-of-thought prompting reduces constraint violations but does not eliminate them in multi-turn settings.Relevant constraints include brand exclusions, budget ceilings, and specification minimums.
LLM Reliability and Preference Elicitation: MACS decouples preference elicitation from constraint enforcement, keeping elicitation as an LLM task while delegating enforcement to a deterministic predicate.

3 Problem Statement and System Overview

MACS frames fixed-catalog conversational recommendation as constrained multi-turn retrieval with grounded responses, persistent preferences, and deterministic enforcement of hard requirements. Its shopping and merchant agents separate language-facing interaction from catalog-critical operations through a structured API.

Problem definition: MACS retrieves products R_t⊆C satisfying all accumulated constraints at turn t and generates factually grounded responses from a structured merchant catalog.The same session also supports add-to-cart, checkout, return-policy, shipping, and warranty queries.
Reliability requirements: The system requires catalog grounding, deterministic constraint correctness, and cross-turn persistence until constraints are explicitly revised.Every returned product must exist at a verified price, hard requirements cannot be inferred, and active constraints cannot be silently dropped.
System overview: MACS separates an LLM-based shopping agent for preference elicitation, intent routing, and response generation from a merchant agent for catalog operations and session state.The merchant agent uses SQL constraint filtering, knowledge graph traversal, and session-persistent state.
System overview: The shopping agent never retrieves products directly; requests pass through the merchant agent’s structured API, preventing the LLM from hallucinating catalog contents.This interface enforces the separation between language-facing tasks and correctness-critical catalog access.

4 The Shopping Agent

The shopping agent transforms each user turn into structured intent and preferences, then executes catalog-grounded actions while preserving and revising constraints across turns. It returns recommendations satisfying current constraints with explanations and refinement options.

Catalog Execution: Correctness-critical operations remain deterministic, including product retrieval, hard-constraint enforcement, disallowed-brand or attribute exclusion, and progressive relaxation of lower-priority constraints.The shopping agent requests catalog results from the merchant agent before generating its response.
Turn Processing: Each turn combines query rewriting, intent routing, structured extraction, and catalog-grounded execution for searches, refinements, comparisons, cart actions, checkout, and FAQs.Query rewriting adds structured hints, deterministic rules intercept common intents, and ambiguous requests use an LLM router.
Session State: Slot values persist across turns unless explicitly revised, while ordinal references resolve against the latest recommendation context and updated constraints can replace earlier preferences.Brand exclusions are enforced at the SQL level and through post-retrieval title filtering.
Recommendation Output: The output presents products satisfying current constraints, explains their selection, and offers follow-up options for refinement, comparison, and additional catalog browsing.This keeps the response aligned with the active preference state while supporting continued interaction.

5 The Merchant Agent

The merchant agent performs correctness-critical commerce operations deterministically over the verified catalog, including constraint enforcement, inventory validation, relationship queries, and progressive relaxation. Its data architecture combines relational storage, a knowledge graph, and caching to support grounded, responsive recommendations.

Constraint Enforcement: Hard constraints are enforced deterministically through SQL filtering, brand exclusion, progressive relaxation, and knowledge-graph traversal over the merchant’s catalog.Every returned product exists at a verified price, and complex structural queries are resolved without relying on unsupported catalog claims.
Data Architecture: 21,000+ catalog items and 1,490+ laptops are stored relationally with 12 structured specification fields, supporting SQL-based filtering and exclusion.The product store is authoritative for catalog data, including price, brand, specifications, reviews, and ratings.
Data Architecture: 2,400 nodes and 8,500 edges support Similar_To and Compatible_With queries resolved through Cypher traversal in ∼17 ms without an LLM call.The knowledge graph falls back to SQL-based search when unavailable, avoiding a hard dependency on the graph layer.
Data Architecture: ∼12× latency reduction is achieved on cache hits, reducing reads from 446 ms cold to 36 ms across 𝑛= 10 runs.The cache and session layer stores search results, product summaries, price data, and LLM-generated narratives with 30 secs-30 min TTLs.
Constraint Enforcement: When fewer than three results satisfy hard constraints, optional specifications are relaxed by importance while price ceilings and brand exclusions remain fixed.The system discloses relaxation, always returns real purchasable products, and never produces empty results or fabricated alternatives.

6 Evaluation

MACS is evaluated with deterministic constraint checks and LLM-as-judge response scoring across single-turn and multi-turn catalog-grounded recommendation tasks. It outperforms catalog-bound baselines on pass-rate reliability, state management, and disclosure while maintaining comparable judged response quality.

Evaluation protocol: Single-turn evaluation allows one clarifying question before a catalog-grounded recommendation and separates deterministic constraint correctness from G-Eval response quality.Hard constraints are verified with SQL before the judge is called.
Single-turn results: 87.1% pass rate: MACS led GPT+Catalog at 72.1% and Gemini+Catalog at 68.6%, with brand compliance 1.000 and filter compliance 0.970.The comparison covers 140 single-turn queries.
Expanded benchmark: 85.3% pass rate: MACS achieved avg. 0.681 across 225 independent queries, including 94.1% for the multi-constraint group.Targeted fixes raised follow-up service QA from 0%→100% and gaming-specific queries from 25%→100%.
Multi-turn results: 72% macro Pass@5: MACS exceeded GPT+Catalog at 56% and Gemini+Catalog at 52%, while judged response quality remained similar at 0.751, 0.736, and within one SD.MACS achieved 100% Pass@5 on exclusion reversal and constraint accumulation, versus 20%/0% and 60%/40% for the baselines.
State-management results: 100% Pass@5: MACS uniquely dominated exclusion reversal and constraint accumulation, scenarios requiring mid-conversation updates through session-persisted slot dictionaries.MACS also led dense constraint at 100% versus 80%/80%, while intent pivot remained 0% across all systems.
Reliability and disclosure: 0.925 disclosure score: MACS explicitly disclosed constraint relaxation on 6 of 8 catalog-impossible queries, exceeding GPT+Catalog and Gemini+Catalog, both below 0.6.Grounding scored 1.000 for all systems; drift was 0.000 for MACS and GPT+Catalog versus 0.014 for Gemini+Catalog.

7 Conclusion

MACS is presented as a hybrid multi-agent framework that improves reliability in fixed-catalog conversational recommendation, achieving the strongest macro Pass@5 while preserving response quality. The evaluation is limited to consumer electronics and shared candidate pools, motivating broader benchmarks, human validation, and independent-retrieval comparisons.

Conclusion: 72% macro Pass@5, compared with 56% for GPT+Catalog and 52% for Gemini+Catalog, is MACS’s strongest reported score.The conclusion describes MACS as consistently outperforming catalog-bound prompt baselines on reliability-oriented metrics, especially when constraints persist across turns.
Conclusion: MACS’s results support separating language interaction from correctness-critical enforcement as a practical way to improve reliability without sacrificing response quality.The conclusion frames this architectural separation as effective specifically for fixed-catalog conversational commerce.
Limitations: The evaluation covers only laptops and accessories, so its results do not establish generalization to other merchant catalogs or recommendation domains.The baselines receive MACS-prepared candidate sets, isolating constraint-aware generation over a shared candidate pool rather than full end-to-end retrieval quality.
Future work: Future work includes standardized external benchmarks, independent human validation of G-Eval scores, independent-retrieval comparisons, broader domains, and expanded component ablations.The paper specifically names the retail subset of 𝜏-Bench as one external benchmark.

Appendix A Multi-Turn Scenario Descriptions

Appendix A describes 10 multi-turn scenarios, each containing 3–5 scripted turns and testing a specific conversational state-management property. All scenarios focus on recommendation except S4 and S5.

Scenario design: The benchmark includes 10 multi-turn scenarios with 3–5 scripted turns each, designed to test conversational state management.All scenarios are recommendation-focused except S4 and S5.

Appendix B Query Construction Details

The benchmark comprises 225 queries spanning five intent classes and four buyer personas, including specified and underspecified requests across 40 interaction groups. A 140-query three-way-comparison subset preserves the same distribution.

Query composition: 225 queries span five intent classes: direct purchase, multi-constraint filtering, vague preference elicitation, service inquiries, and comparison.Queries were authored across four buyer personas.
Query composition: 138 specified (≥1 hard constraint) and 87 underspecified queries are organized across 40 interaction groups.The specified category contains at least one hard constraint.
Query composition: The 140-query three-way-comparison subset follows the same distribution as the full benchmark.It preserves the benchmark’s intent-class, persona, and query-specification distribution.