PaperlayerView exact revision on arXiv

Source-linked AI summary

ATLAS: Benchmarking and Adapting LLMs for Global Trade via Harmonized Tariff Code Classification

Pritish Yuvraj, Siva Devarakonda

arXiv:2509.18400v1cs.AI

TL;DR

HTS classification is an important but little-studied bottleneck in global trade. The paper builds a benchmark from CROSS rulings and fine-tunes LLaMA-3.3-70B as ATLAS; ATLAS outperforms leading models on tariff-code accuracy while costing less to run.

  • Problem

    HTS classification has received little attention from machine learning researchers despite its role as a bottleneck in global trade.

    Source: §1 · p. 2

  • Method

    The paper releases a CROSS-based HTS benchmark and fine-tunes LLaMA-3.3-70B on ruling-derived product descriptions, HTS codes, and reasoning traces.

    Source: §2.4 · p. 4, §3.1 · p. 4

  • Results

    40% fully correct 10-digit classifications and 57.5% correct 6-digit classifications, with ATLAS outperforming GPT-5-Thinking by 15 points and Gemini-2.5-Pro-Thinking by 27.5 points.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The released benchmark and model position HTS classification as a new community task for LLM evaluation and future research.

    Source: §1.1 · p. 2, §2.4 · p. 4

    Best 10-digit accuracy remains 40%, leaving substantial room for advances in reasoning, retrieval, and alignment.

    Source: §5 · p. 8

Abstract

from arXiv · show

Accurate classification of products under the Harmonized Tariff Schedule (HTS) is a critical bottleneck in global trade, yet it has received little attention from the machine learning community. Misclassification can halt shipments entirely, with major postal operators suspending deliveries to the U.S. due to incomplete customs documentation. We introduce the first benchmark for HTS code classification, derived from the U.S. Customs Rulings Online Search System (CROSS). Evaluating leading LLMs, we find that our fine-tuned Atlas model (LLaMA-3.3-70B) achieves 40 percent fully correct 10-digit classifications and 57.5 percent correct 6-digit classifications, improvements of 15 points over GPT-5-Thinking and 27.5 points over Gemini-2.5-Pro-Thinking. Beyond accuracy, Atlas is roughly five times cheaper than GPT-5-Thinking and eight times cheaper than Gemini-2.5-Pro-Thinking, and can be self-hosted to guarantee data privacy in high-stakes trade and compliance workflows. While Atlas sets a strong baseline, the benchmark remains highly challenging, with only 40 percent 10-digit accuracy. By releasing both dataset and model, we aim to position HTS classification as a new community benchmark task and invite future work in retrieval, reasoning, and alignment.

1 INTRODUCTION

HTS classification is a global trade bottleneck: codes are hierarchical, and accurate six-digit and ten-digit assignments serve worldwide interoperability and U.S. customs compliance, respectively. The paper introduces a CROSS-based benchmark and ATLAS, an adapted LLaMA model, as contributions to this underexplored task.

1.1 CONTRIBUTIONS

  • The first open-source HTS classification benchmark is built from U.S. Customs CROSS rulings, with training, validation, and test splits.Source: §1.1 · p. 2
  • ATLAS reaches 40% 10-digit and 57.5% 6-digit accuracy, outperforming GPT-5-Thinking (25%) and Gemini-2.5-Pro-Thinking (13.5%).The model is a fine-tuned LLaMA-3.3-70B released as open source.Source: §1.1 · p. 2
  • ATLAS is up to 8× cheaper per inference and supports self-hosting so sensitive trade data can remain in secure environments.Source: §1.1 · p. 2
  • The contributions position tariff-code classification as a new LLM evaluation task tied to global commerce and trade compliance.Source: §1.1 · p. 2

2 DATASETS

The paper turns authoritative but lengthy and unstructured CROSS customs rulings into a machine-learning dataset for HTS classification. The dataset captures disputed or clarified codes, and its hierarchical labels and real-world stakes make the task challenging.

2 DATASETS

  • CROSS contains legally binding CBP decisions on HTS codes, but the rulings are lengthy, unstructured, and scattered across thousands of HTML pages.Source: §2 · p. 2

2.1 DATA COLLECTION

  • The collected dataset contains 18,731 rulings covering 2,992 unique HTS codes across product categories.Source: §2.1 · p. 3
  • CROSS omits codes not disputed or clarified; frequent rulings may indicate high-demand or ambiguous categories, while absent codes may be stable or rarely used.Source: §2.1 · p. 3

2.2 DATA TRANSFORMATION INTO LLM-TRAINABLE FORMAT

  • GPT-4o-mini transforms each verbose ruling into a structured prompt–response example, including a product description, reasoning path, and HTS code.Source: §2.2 · p. 3

2.3 DATASET SPLITS

  • The split assigns 18,254 rulings to training and holds out 200 each for validation and final testing.Source: §2.3 · p. 3

2.4 DISCUSSION

  • Rulings can hinge on subtle product distinctions, correctness is hierarchical at six and ten digits, and errors carry trade and compliance consequences.The benchmark is intended to establish tariff classification as a high-impact LLM evaluation task.Source: §2.4 · pp. 3–4

3 MODEL TRAINING

ATLAS adapts LLaMA-3.3-70B to CROSS-derived product descriptions paired with HTS codes and reasoning traces, then examines training stability and outlines untested directions for improving the baseline.

3 MODEL TRAINING

  • LLaMA-3.3-70B was selected as a dense baseline because budget constraints precluded tuning multiple frontier models and dense models are simpler to fine-tune and deploy.The authors also cite predictable memory use and reduced sensitivity to expert-routing heuristics as benefits for reproducible downstream research.Source: §3 · p. 4

3.1 SUPERVISED FINE-TUNING OBJECTIVE

  • SFT pairs ruling-derived product descriptions with the correct HTS code and a reasoning trace, minimizing token-level negative log-likelihood on ground-truth outputs.The objective assigns conditional probabilities to target tokens given the input and preceding target tokens.Source: §3.1 · p. 4

3.2 TRAINING SETUP AND STABILITY

  • Fine-tuning ran for 5 epochs, approximately 1,400 steps, on 16 A100-80GB GPUs, using bf16 precision, gradient accumulation, and a simulated batch size of 64.Training used AdamW with a cosine learning-rate schedule initialized at 1×10−7 and fully sharded data parallelism.Source: §3.2 · p. 4
  • Loss falls sharply over the first 200 steps and stabilizes near convergence, with no validation overfitting or catastrophic loss spikes reported.The authors report stable gradient norms and describe this fine-tuning as feasible under modest compute budgets when optimization is tuned for stability.Source: §3.2 · p. 4, Figure 1 · p. 5

3.3 ABLATIONS AND FUTURE WORK

  • The study evaluates only LLaMA-3.3-70B and identifies smaller models, retrieval, contrastive or hybrid objectives, and preference optimization as directions for future work.The authors frame these as potential avenues for clarifying tradeoffs or improving long-tail accuracy and decision boundaries; substantial room for methodological innovation remains.Source: §3.3 · p. 5

4 RESULTS AND EVALUATION

On a 200-ruling test set, Atlas leads on exact 10-digit and harmonized 6-digit classification, with lower estimated inference costs than proprietary models.

4 RESULTS AND EVALUATION

  • On a held-out test set of 200 CROSS rulings, the task evaluates 10-digit codes using exact-match, 6-digit-match, and average digit-level accuracy metrics.A fully correct result requires all 10 digits to match; the first six digits capture the harmonized portion.Source: §4 · p. 5

4.1 FULLY CORRECT CLASSIFICATIONS

  • 40% of Atlas classifications exactly match the 10-digit HTS US code, versus 25% for GPT-5-Thinking; open-source baselines without domain adaptation score at most 3%.Table 2 covers fully correct classifications on the 200-sample test set.Source: §4.1 · pp. 5–6, Table 2 · p. 6

4.2 PARTIALLY CORRECT CLASSIFICATION

  • 57.5% vs. 55.5%: Atlas exceeds GPT-5-Thinking on 6-digit accuracy for the globally harmonized portion of HTS codes.The passage identifies this as transfer to the globally harmonized layer.Source: §4.2 · p. 6, Table 3 · p. 6

4.3 AVERAGE DIGIT-LEVEL ACCURACY

  • 6.3 digits: Atlas correctly predicts that many digits per code on average, compared with roughly 3–5 for general-purpose models.This metric captures partial progress even when the complete code is not correct.Source: §4.3 · p. 6, Table 4 · p. 6

4.4 VISUAL COMPARISON

  • Figure 2 shows Atlas leading at both evaluation levels, with its advantage especially pronounced on the 10-digit U.S.-specific task.Source: §4.4 · p. 6, Figure 2 · p. 7

4.5 COST EFFICIENCY OF INFERENCE

  • For cost comparisons, the estimate covers 1,000 product descriptions with about 1k input and 200 output tokens; open-model costs assume on-demand A100 hosting.Locally hosted open-source models can cost a fraction of proprietary APIs at scale.Source: §4.5 · p. 7, Table 5 · p. 7

4.6 DISCUSSION

  • 40% vs. 25% fully correct 10-digit classifications, with Atlas nearly 5× cheaper than GPT-5-Thinking and almost 8× cheaper than Gemini-2.5-Pro-Thinking.The discussion also reports strong performance on the globally harmonized 6-digit regime.Source: §4.1 · pp. 5–6, Table 2 · p. 6, §4.6 · p. 7
  • The discussion connects Atlas’s 6-digit performance to the globally harmonized regime, extending its relevance beyond U.S.-specific tariff codes.Source: §4.6 · p. 7

5 SUMMARY AND FUTURE DIRECTIONS

The conclusion presents Atlas and its CROSS-based benchmark as a new evaluation task for tariff classification, while emphasizing that substantial accuracy challenges remain.

Benchmark and release

  • The paper releases Atlas and benchmark splits to support reproducibility and frames HTS classification as a new LLM evaluation task.Source: §5 · pp. 7–8

Performance

  • 40% fully correct 10-digit classifications and 57.5% at 6 digits: Atlas exceeds GPT-5-Thinking by 15 points and Gemini-2.5-Pro-Thinking by 27.5 points.Source: §5 · p. 8

Cost and deployment efficiency

  • Atlas is nearly 5× cheaper than GPT-5 and 8× cheaper than Gemini, and supports self-hosted deployment for sensitive trade and supply-chain applications.Figure 3 compares estimated costs per 1,000 inferences and describes self-hosted Atlas as substantially cheaper than proprietary APIs.Source: §5 · p. 8, Figure 3 · p. 8

Open benchmark challenge

  • Despite these gains, best 10-digit accuracy remains only 40%, motivating further work on reasoning, retrieval, and alignment.Source: §5 · p. 8

Future directions

  • Future directions include broader ruling coverage, smaller Atlas variants for resource-constrained deployment, and enhanced reasoning or retrieval-augmented methods.Source: §5 · p. 8

Related papers

Loading 2509.18400v1…