HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application

• Benchmarks assess open-domain and domain-specific knowledge, but current agent benchmarks largely overlook rule application.Rule application is especially challenging when rules are vague or have logical dependencies such as exceptions and cross-category relationships.Source: §1 · p. 2
• HSCodeComp evaluates agents on complete 10-digit product-code prediction using hierarchical rules and noisy, realistic product descriptions.It contains 632 expert-annotated products from large-scale e-commerce platforms.Source: §1 · p. 2
• 46.8% accuracy for SmolAgent with GPT-5 versus 95.0% for human experts shows the benchmark remains difficult for current systems.The paper also reports that test-time scaling fails to improve HSCodeComp performance.Source: §1 · pp. 2–3
• Analysis finds that current agents lack capabilities needed for hierarchical rule application, and test-time scaling does not improve their performance.The paper says these observations underscore the challenge and the need for more effective agent designs.Source: §1 · p. 3

2.1. Previous Works in HSCode Prediction

• Prior HSCode prediction uses text classification, pretrained BERT models, or LLM prompting, while existing benchmarks risk data leakage and are often unreleased.HSCodeComp instead uses noisy product descriptions collected from large-scale online shopping platforms.Source: §2.1 · p. 3

2.2. Benchmarking Level 1 Knowledge Utilization

• Open-domain benchmarks test agents on long-form web content, web environments, and tool use for deep search and reasoning.Examples include WebArena, WebShop, ALFWorld, GAIA, BrowseComp, WebWalkerQA, and BrowseComp-ZH.Source: §2.2 · p. 3

2.3. Benchmarking Level 2 Knowledge Utilization

• Domain-specific benchmarks test structured knowledge, but the reviewed landscape lacks evaluation of Level 3 abstract rule-based knowledge.HSCodeComp is introduced as a realistic, expert-level e-commerce benchmark for that gap.Source: §2.3 · p. 3

• The task assigns a unique valid 10-digit HSCode to a noisy but realistic product description.The code is a global standard for classifying traded goods and supports tariffs, customs, and trade governance.Source: §3 · p. 4
• A product input can include its title, attributes, platform category, image, price, currency, and webpage URL.Attributes may include properties such as material and package size.Source: §3 · p. 4
• Prediction draws on hierarchical tariff rules, human-written decision rules, and official customs-rulings databases.Tariff rules can contain implicit logic, exceptions, and vague linguistic constraints; rulings databases document historical classification decisions.Source: §3 · p. 4
• The output is a valid 10-digit HSCode, with chapter, heading, and subheading represented by the first 2, 4, and 6 digits and country-specific codes in the final four.The complete code must follow a valid path in the official HS taxonomy.Source: §3 · p. 5

4.1. Benchmark Construction

• Products come from a large global e-commerce platform, and category balancing plus semantic redundancy filtering reduces topical skew and dominance by common, easy products.The source profiles retain noisy information to reflect real-world task challenges.Source: §4.1 · p. 5
• Two specialists gather product information, extract features, search customs rulings, apply hierarchical decision rules when needed, and verify codes; an additional expert validates annotations.The annotation pipeline is shown in Figure 3.Source: §4.1 · p. 5, Figure 3 · p. 6
• 632 products span 32 first-level e-commerce categories and 27 HS chapters; a separate senior-expert review of 10% found 2% disagreement.Disagreements between the two annotators trigger senior review, and instances are excluded if neither annotation is valid.Source: §4.1 · p. 5

4.2. Evaluation Metric

• The primary metric is 10-digit HSCode accuracy, measuring exact matches between normalized predictions and human-annotated codes.Accuracies at 2-, 4-, 6-, and 8-digit levels are also reported to compare performance across granularities.Source: §4.2 · p. 6

5.1. Experimental Setup

• The evaluation covers 14 foundation models, six open-source agent frameworks, and three commercial systems manually tested on 49 representative examples.The open-source frameworks use GPT-5 as the default backbone; the tested models include VLMs with product images.Source: §5.1 · pp. 6–7

5.2. Main Results

• 46.83% 10-digit accuracy is achieved by SmolAgent (GPT-5 VLM), compared with 95% for experienced human experts.Table 1 reports the baseline results.Source: §5.2 · p. 7, Table 1 · p. 7
• On the same HSCodeComp subset, open-source agents outperform closed-source agents.The reported case studies identify premature decisions and information misprocessing in closed-source agents.Source: Table 2 · p. 7, §5.2 · p. 8
• Adding human-written decision rules decreases accuracy for SmolAgent and WebSailor, while Aworld achieves only marginal gains.The authors consequently remove the decision rules from the default agent setup.Source: §5.2 · p. 8, Table 3 · p. 8
• Most baselines improve when given product images, which reveal material and surface features absent from text descriptions but relevant to classification.These visual attributes align with the predefined rules.Source: §5.2 · p. 8
• 42.72% to 42.09% 10-digit accuracy follows adding webpage visits to SmolAgent (GPT-5 LLM); the authors report that page content can overwhelm key information.They retain search-engine snippets and remove the webpage-visit tool from the default open-source setup.Source: §5.2 · p. 8

6.1. Overthinking Decrease Open-source Agent Performance

• Reducing WebSailor's pre-tool reasoning depth improves accuracy, with No-Think nearly matching SmolAgent.The authors attribute performance variation among open-source agents primarily to prompt-defined reasoning depth.Source: §6.1 · p. 9, Table 5 · p. 9

6.2. Effect of Backbones in Open-source Agent Systems

• GPT-5 is the best of four tested backbones for SmolAgents, so the authors select it as the default for open-source agents.The tested alternatives are Gemini-2.5-pro, Claude-4-Sonnet, and Qwen-MAX.Source: §6.2 · p. 9, Table 6 · p. 9

6.3. Failure Modes of Closed-Source and Open-Source Agents

• Qualitative failures include premature decisions, information misprocessing, unnecessary self-correction, reasoning hallucination, wrong rule application, and lack of domain knowledge.The examples include revising correct codes incorrectly and confusing silicone products with rubber rather than plastic.Source: §6.3 · pp. 9–10
• Agents reduce outdated, hallucination, and other errors relative to LLMs, leaving “Error but Valid” as their predominant error type.Figure 4 presents distributions of four coarse-grained failure categories for LLM/VLM-only systems and agents.Source: §6.3 · p. 10, Figure 4 · p. 10

6.4. Per-category Performance Analysis

• Figure 5 shows category distributions for Challenging Product Distribution (CID) and Average Performance Distribution (APD).The category distribution is shown with blue bars on the left in both panels.Source: Figure 5 · p. 10

6.5. Test-time scaling Cannot Improve Performance Effectively

• Increasing majority-voting trials yields negligible improvement, while self-reflection lowers SmolAgent accuracy from 42.72% to 42.57%.The tested voting settings use K = {1, 2, 4, 8, 16}.Source: §6.5 · p. 11

7. Conclusion

• HSCodeComp is introduced as a realistic, expert-level e-commerce benchmark for multi-hop reasoning with hierarchical tariff rules.The authors identify a gap in evaluating deep-search agents on hierarchical rule application.Source: §7 · p. 11
• 46.8% agent accuracy versus 95.0% for human experts marks a substantial performance gap on the benchmark.The authors say they will release HSCodeComp to support future research.Source: §7 · p. 11

8. Ethics Statement

• The dataset is fully anonymized and contains no personally identifiable information.The authors state that the research follows data-privacy guidelines.Source: §8 · p. 11
• Human annotators received over 34.6 USD per hour, compared with an average 3.13 USD hourly wage on Amazon Mechanical Turk.The authors describe the remuneration as fair and competitive for the expertise and effort required.Source: §8 · p. 11
• The compensation structure was designed to recognize the expertise and effort required for the task.The authors say it was intended to provide a fair and competitive wage.Source: §8 · p. 11

9. Reproducibility statement

• HSCodeComp’s code and dataset will be publicly released, but product URLs and images are omitted for security and compliance reasons.Source: §9 · p. 11

B. Dataset distribution

• The product-category and HSCode-chapter distributions closely mirror real-world product distributions, supporting the benchmark’s relevance to practical trade scenarios.Source: Figure 7 · p. 16, B. Dataset distribution · p. 16

C. Semantic Distribution of Hierarchical Tariff Rules

• Chapter titles and notes show semantic overlap: chapters often lie closer to neighboring sections’ centroids, and the section centroids form overlapping clusters.This pattern reflects shared vocabulary and concepts across adjacent sections, including distinctions between raw materials and finished goods.Source: C. Semantic Distribution of… · p. 16, Figure 8 · p. 17

D. More Detailed Experiments on Open-source Agents with Different Backbone LLMs

• GPT-5 performs consistently better than Gemini 2.5 Pro when used as the backbone for the four tested open-source agent systems.Source: D. More Detailed Experiments on… · p. 17, Table 7 · p. 18

E. Improvement Gain from Agents

• 56 corrected samples in HSCodeComp account for much of SmolAgent (GPT-5)’s gain, primarily through fewer outdated and hallucination failures.The accompanying analysis also reports a significant reduction in the rate of those failures.Source: E. Improvement Gain from Agents · p. 17, Figure 9 · p. 18, Figure 10 · p. 19

F. Implementing details and Knowledge Forms

• Agents can access CROSS, hierarchical tariff rules, and other resources, but hierarchical decision rules and webpage visits are excluded for open-source evaluation because they did not improve performance.The described tariff rules contain implicit logical relationships and vague descriptions; CROSS also includes revoked precedents and plain-text email rulings.Source: F. Implementing details and Knowledge… · pp. 19, 21, Figure 11 · p. 20, Figure 12 · p. 20

G. Case study of Failure Modes

• Six failure modes include premature decisions, information misprocessing, unnecessary self-correction, reasoning hallucination, wrong rule application, and lack of domain knowledge.The authors report these limitations across open-source and closed-source systems.Source: G. Case study of Failure Modes · p. 22
• In the sticker case, the predicted code follows the printed-design classification, while the true code is under plastics and self-adhesive articles.The discussion distinguishes decorative printed pictures from labels where printing is incidental to labeling.Source: G. Case study of Failure Modes · p. 24
• A glove example attributes a flawed analysis to prioritizing the “Nitrile Gloves” title over the explicit PVC material attribute.Source: G. Case study of Failure Modes · pp. 26–27

H. Case Study of Overthinking

• The overthinking case studies highlight decreased performance for open-source agents and comparatively lower performance for state-of-the-art closed-source agents.Source: H. Case Study of Overthinking · p. 34

H.1. Open-source Agent Cases

• For the polyester men’s T-shirt, WebSailor selects the women’s code while Smolagents selects the men’s code.The cited comparison gives 6109901050 for women’s and 6109901007 for men’s.Source: §H.1 · p. 35

I. Case Study of Multi-modal Information in Agent Systems

• The multimodal case study compares SmolAgent with and without product-image processing and presents four representative cases with evidence and reasons.Source: §I · p. 37, Table 10 · p. 38