Source-linked AI summary
Autoresearch for Marketplace Catalogs: From Legacy Forms to AI-Native Matching
Kartik Ravisankar, Hojat Abdolanezhad, Daniel Capo, Sang Su Lee, Shishir Dash, Vijay Anand Raghavan
TL;DR
Service marketplaces need explicit provider-preference catalogs when AI-native matching replaces structured request forms, but legacy schemas encode those preferences implicitly and across occupations. The paper reconstructs catalogs with independent occupation-level propose-evaluate-keep loops, marketplace-specific judging and critic penalties, and parity mapping for audit. In production across 132 occupations, the system reports high legacy-preference coverage while retaining human sign-off, with unresolved trust gaps from shared model-family biases and silent critic disagreement.
Problem
AI-native matching requires an explicit provider-preference catalog, while legacy marketplaces encode that structure implicitly across fragmented occupation-specific Q&A schemas.
Method
The system independently reconstructs each occupation’s catalog through propose-evaluate-keep refinement, six-rubric judging, seven-persona weighted penalties, and parity mapping to legacy Q&A.
Results
73.3% of legacy answers map directly to regenerated tags, 20.8% are acceptable intake-only non-tags, and 6.0% are regrettable misses across 132 production occupations.
Takeaways & Limitations
Parity mapping provides a coverage signal and human-reviewable artifact supporting migration from legacy forms to provider-preference tags.
Takeaways & Limitations
The editor and critics share a model family, and critic disagreement is collapsed into an aggregate score, leaving two identified trust gaps.
Abstract
from arXiv · showhide
Two-sided service marketplaces are moving from deterministic request-form intake to AI-native probabilistic matching, enabled by large language models (LLMs) that infer intent, preferences, and latent constraints from natural language. Relying on inferred intent rather than fixed-form fields forces these platforms to regenerate the provider-side preference taxonomy underwriting matching, search, and pricing: attributes interpretable to service providers while remaining a useful signal for marketplace decisions. We present an autoresearch loop that generates this taxonomy, one occupation at a time, and has been deployed in production at a major U.S. consumer services marketplace since April 2026, spanning 132 occupations. Instead of one global hierarchy, the loop treats each occupation as an independent generation problem and runs iterative propose-evaluate-keep refinement cycles. Each candidate tag set is scored by a recalibrated six-rubric LLM-as-judge framework, and a 7-critic panel of distinct personas contributes weighted penalties to an adjusted score, with no hard vetoes. A separate parity-mapping stage maps legacy request-form Q&A pairs back to the generated taxonomy, yielding both a coverage signal and an interface for human quality assurance; it does so by first inferring the provider attribute each legacy question was meant to measure, rather than translating questions to tags literally.
1 Introduction
The paper frames AI-native marketplace matching as requiring an explicit provider-preference catalog reconstructed from implicit, occupation-specific legacy Q&A schemas. It proposes independent occupation-level autoresearch loops, marketplace-grounded evaluation, and parity mapping, reporting high reconstruction fidelity at production scale.
- Motivation: Legacy Q&A forms embed an implicit, fragmented catalog, whereas AI-native matching requires an explicit shared representation for consumers, providers, and marketplace systems.The catalog supports matching, search, and pricing while translating consumer intent and provider preferences.
- Problem formulation: The task is catalog reconstruction: infer latent provider attributes from legacy Q&A pairs and reorganize them into an explicit representation for AI-native interactions.The paper treats each service domain as a smaller autoresearch problem because domains have distinct vocabularies and matching semantics.
- Contributions: The generated unit is a provider-facing preference tag that providers toggle in production and downstream marketplace systems use.This distinguishes the work from occupation-hierarchy, text-label taxonomy, corpus-taxonomy, and product-schema construction.
- Contributions: Independent per-occupation autoresearch loops avoid a single global hierarchy and limit semantic interference between superficially similar but operationally distinct attributes.Each occupation runs its own propose-evaluate-keep loop.
- Evaluation and validation: A recalibrated multi-rubric LLM judge and seven-persona critic panel apply marketplace-specific evaluation, while parity mapping links generated tags back to legacy Q&A for human review.The mapping measures coverage of operational distinctions encoded in the legacy system.
- Results: 73.3% of legacy answers map directly to regenerated tags, 20.8% are acceptable intake-only non-tags, and 6.0% are regrettable misses.Pooled preference coverage is 92.5%, with a 100% per-occupation median and 97.8% mean; 132 occupations were live in production.
2 Background
The marketplace connects consumers with local service professionals whose targeting preferences determine lead eligibility and purchasing. Its redesign replaces rigid binary targeting with a reconstructed catalog containing canonical eligibility tags and specialty preference tags, while exposing deployment gaps that affect filtering.
- Marketplace setting: The marketplace serves home, wellness, and event categories, with pros specifying targeting preferences that determine which consumer leads they can receive and purchase.The production redesign covers plumbing, house cleaning, photography, landscaping, and roughly 130 other occupations.
- Taxonomy: The legacy taxonomy consists of occupation and category hierarchies with category-specific request-form Q&A schemas.The reconstructed catalog is organized around two tag classes.
- Tag classes: Canonical tags determine strict-match eligibility for leads in a category.A provider must possess the category’s associated canonical tag for the filter to consider that provider eligible.
- Tag classes: Specialty preference tags let providers refine which leads they want within categories where they are already eligible.They operate after the canonical eligibility distinction.
- Production findings: 40.66% of filter evaluations failed because the request’s required canonical tag was absent from the provider profile.A filter evaluation checks one candidate pro against one incoming consumer request.
3 Related Work
The paper positions its contribution at the intersection of taxonomy construction, schema generation, autoresearch, prompt optimization, and LLM evaluation. Its distinguishing focus is reconstructing provider-preference catalogs within occupations for two-sided service marketplaces rather than building global hierarchies or product schemas.
- Research context: The work combines automated taxonomy construction, agentic schema generation, autoresearch, prompt optimization, and LLM-based evaluation.
- Hierarchical taxonomy construction: Unlike CLIMB, which generates occupations within a global hierarchy, this work reconstructs provider-preference tags independently within each occupation.The objective is reconstruction from legacy semantics rather than hierarchy generation.
- Schema and catalog generation: Unlike AttributeForge’s product attributes for e-commerce, this system reconstructs provider-preference catalogs for service marketplaces where matching depends on capabilities and screening preferences.
- Autoresearch and prompt optimization: The propose-evaluate-keep loop treats prompts as optimizable artifacts, but searches against marketplace catalog objectives rather than a single fixed benchmark metric.The loop draws on Self-Refine, Constitutional AI, multi-agent debate, GEPA, DSPy, and MIPROv2.
- LLM evaluation: The six-rubric evaluation framework adapts LLM-as-judge and critique-based evaluation to marketplace-specific failure modes that generic taxonomy metrics can overlook.
4 System Design
The system reconstructs one occupation’s catalog through repeated generation, rubric-based evaluation, critic penalties, and targeted prompt edits, then maps the best catalog back to legacy Q&A for auditable human review. Its scoring combines tag quality with set-level coverage and proliferation controls.
- 4.1 Per-Occupation Autoresearch Loop: Catalog reconstruction is formulated as an iterative propose-evaluate-keep process run independently for each service occupation.The current production deployment seeds 132 occupation-specific jobs from a common baseline prompt.
- 4.1 Per-Occupation Autoresearch Loop: Each occupation’s legacy RF data and current prompt produce a candidate tag set that receives a six-rubric judge score and weighted critic penalties.The critic panel runs in parallel before the gate decides whether an edit survives.
- 4.5 Editor Agent: The editor makes one targeted prompt change at a time, selecting the weakest unaddressed dimension from six rubric means and synthetic parity coverage.A two-iteration recency constraint prevents fixation on one persistent weakness.
- 4.2 Evaluation: The judge scores tags on screening versus intake, legibility, preference variance, cross-category consistency, and other marketplace-specific criteria on a /15 per-tag composite.Recalibration lowered the legibility score for bare-noun tags such as “Appliances” because they can be uninformative to working pros.
- 4.2 Evaluation: The generator and judge use one model family, while critics, editor, and parity mapper use another to reduce shared-bias reinforcement.The paper describes this cross-family arrangement as a production-scale defense against correlated biases.
- 4.3 Set-level scoring: The set-level penalty subtracts for tag proliferation and coverage gaps that per-tag rubric scores cannot detect.This prevents individually strong tags from forming a bloated or incomplete catalog.
- 4.6 Parity mapping: After termination, parity mapping creates a many-to-many relation between best-catalog tags and legacy Q&A, including uncovered items, and emits a human-reviewable JSON mapping and Google Sheet.The mapping supports auditable sign-off before deployment.
5 Deployment
The production system reconstructs occupation-specific catalogs in parallel, retains detailed audit artifacts, and requires human review before deployment. It has scaled to 132 occupations, with new occupations onboarded through minimal configuration.
- Catalog Generation: Each occupation is processed independently and in parallel, producing a best tag set, parity mapping, and residual sets.The loop persists iteration logs, score snapshots, rationales, and parity outcomes for inspection during and after each run.
- Human QA at Scale: Every generated catalog undergoes human sign-off through shared per-occupation review tabs before deployment.Reviewers can inspect parity cases, edit tag names, resolve comments, and record Keep, Update, or Delete verdicts.
- Scale and Coverage: 132 occupations are live, and a single configuration entry is sufficient to onboard another occupation.Automotive Detailing completed generation, autoresearch, parity mapping, and human QA within five hours on one day.
6 Evaluation and Operational Lessons
Evaluation separates catalog quality from deployment outcomes, combining judge scores, critic penalties, parity and operational analyses. The results show strong catalog coverage and early convergence, while exposing deployment gaps, evaluator disagreement, and remaining taxonomy drift.
- Catalog Quality vs. Deployment Outcomes: The catalog-quality score and deployment outcomes are measured separately because deployment failures do not necessarily indicate catalog defects.A 40.66% missing-canonical-tag rate reflects profile seeding, while critic penalties do not measure request-time tag emission.
- Catalog-Quality Metrics: Across 132 occupations, per-tag E composites concentrate mostly between 11 and 14, with low outliers concentrated on Canonical Coverage.The outliers occur where legacy categories lack an obvious single-tag canonical mapping.
- Catalog-Quality Metrics: The critic panel assigns the largest aggregate penalty mass to Pro, while Adversarial penalties are sparse and heavy-tailed.The Reasoning bonus appears roughly when the editor’s latest change improves its rationale.
- Cross-Occupation Drift: Across 14 occupations and 189 tags, all provider-facing tag strings are occupation-unique, while 12 structural prefixes show partial standardization.Capability and Work type appear in 9/14 occupations; residual singleton prefixes and case variants remain targets for later refinement.
- Evaluation Limits: The external evaluator validation remains planned, so agreement between the production judge and an independently developed evaluator has not yet been reported.The planned evaluation would compare verdict agreement, Cohen’s kappa on six shared dimensions, and qualitative disagreement tags.
- Ablations: Removing the Parity critic left the adjusted-score trajectory and final best tag set unchanged in the observed Accounting ablation.The Parity critic scored 7/10 in both kept iterations and therefore contributed no penalty.
- Deployment-Side Metrics: Deprecated tags generated roughly 486,000 filter events because request-time reverse mapping ignored the catalog’s active-status flag.The autoresearch loop did not surface this catalog-hygiene issue directly.
- Deployment-Side Metrics: 40.66% of MVP filter evaluations failed because canonical tags were missing from provider profiles, making deployment-side seeding the dominant operational gap.The catalog tag existed and enrichment emitted it, but the provider profile had not been seeded with that tag.
7 Discussion
The paper defends a compositional production architecture for reconstructing marketplace catalogs, combining per-occupation independence, a seven-persona critic panel, and separate parity mapping. It also identifies trust gaps in shared-model evaluation and silent critic disagreement, with concrete mitigations planned.
- Architecture: The system integrates per-occupation generation, a seven-persona critic panel, and parity mapping rather than proposing a single new algorithm.The authors frame the contribution as an architecture whose integration and production scale are the primary additions.
- Deployment: 132 occupations run in production with on-demand onboarding, exposing operational lessons that a global hierarchical build could miss.The production deployment is part of the paper’s claimed contribution and scale evidence.
- Trustworthiness: Shared-model evaluation leaves a residual trust gap: the editor and seven critics share the Claude Sonnet 4.6 family, so aligned systematic bias may escape detection.The proposed mitigation is adding a critic from a third model family.
- Trustworthiness: Aggregate critic penalties hide disagreement, so a high-variance verdict can look identical to a low-variance verdict.The paper proposes exposing per-critic penalties and disagreement summaries to the editor.
8 Conclusion
The paper concludes with a per-occupation autoresearch system for marketplace catalog generation, deployed across 132 occupations. Its reported contribution is the integration of independent optimization, critic penalties, parity-based migration, and production lessons.
- Conclusion: The system generates provider-preference catalogs per occupation and is deployed at a 132-occupation consumer services marketplace.The conclusion identifies the optimization unit, parallelism, critic panel, parity migration, and MEMORY_ONLY typing primitive as compositional contributions.
- Conclusion: Production deployment surfaced three deprecated tags driving 486K filter events, an operational hygiene finding a global hierarchy could have masked.The paper distinguishes deployment results and integration from the algorithmic ancestry it acknowledges.