PaperlayerView exact revision on arXiv

Source-linked AI summary

Performance Trade-offs of Optimizing Small Language Models for E-Commerce

Josip Tomo Licardo, Nikola Tankovic

arXiv:2510.21970v1cs.AIcs.CL

TL;DR

Commercial LLMs can be costly and resource-intensive to deploy for e-commerce, motivating study of smaller open-weight alternatives. The paper fine-tunes and quantizes a 1B Llama 3.2 model for multilingual intent recognition, finding that it matches GPT-4.1 accuracy while optimization effects vary by hardware.

  • Problem

    Commercial LLM deployment faces high computational and operational costs, while the practical viability of small models for e-commerce intent recognition remains under investigation.

    Source: §1 · p. 2

  • Method

    The paper fine-tunes a 1B Llama 3.2 model using QLoRA and synthetic data, then evaluates post-training quantization and operational performance across GPU and CPU settings.

    Source: §1 · p. 2, §3.4.2 · p. 5

  • Results

    99% exact match accuracy on structured intent recognition places the 1B Llama 3.2 model on par with GPT-4.1; 4-bit GPTQ causes an 82% inference slowdown on an older GPU.

    Source: §6 · p. 12

  • Takeaways & Limitations

    Optimization choices should account for target hardware because a model’s operational efficiency in one deployment context may not transfer to another.

    Source: §6 · p. 13

    The study uses only synthetically generated data and tests one older-generation GPU and one type of consumer CPU.

    Source: §5.4 · p. 12

Abstract

from arXiv · show

Large Language Models (LLMs) offer state-of-the-art performance in natural language understanding and generation tasks. However, the deployment of leading commercial models for specialized tasks, such as e-commerce, is often hindered by high computational costs, latency, and operational expenses. This paper investigates the viability of smaller, open-weight models as a resource-efficient alternative. We present a methodology for optimizing a one-billion-parameter Llama 3.2 model for multilingual e-commerce intent recognition. The model was fine-tuned using Quantized Low-Rank Adaptation (QLoRA) on a synthetically generated dataset designed to mimic real-world user queries. Subsequently, we applied post-training quantization techniques, creating GPU-optimized (GPTQ) and CPU-optimized (GGUF) versions. Our results demonstrate that the specialized 1B model achieves 99% accuracy, matching the performance of the significantly larger GPT-4.1 model. A detailed performance analysis revealed critical, hardware-dependent trade-offs: while 4-bit GPTQ reduced VRAM usage by 41%, it paradoxically slowed inference by 82% on an older GPU architecture (NVIDIA T4) due to dequantization overhead. Conversely, GGUF formats on a CPU achieved a speedup of up to 18x in inference throughput and a reduction of over 90% in RAM consumption compared to the FP16 baseline. We conclude that small, properly optimized open-weight models are not just a viable but a more suitable alternative for domain-specific applications, offering state-of-the-art accuracy at a fraction of the computational cost.

1 Introduction

E-commerce intent recognition matters for conversational shopping, but deploying powerful commercial LLMs can be costly and resource-intensive. This paper investigates whether a hardware-aware, fine-tuned 1B model can match GPT-4.1 while offering operational benefits.

1 Introduction

  • Commercial LLM deployment entails computational and API costs, vendor lock-in, data privacy and security concerns, and environmental costs that hinder adoption, particularly for small and medium-sized enterprises.Source: §1 · p. 2
  • Domain-specialized smaller open-weight models have rivaled or exceeded GPT-4 on narrow tasks, motivating a diverse ecosystem of specialized agents.Source: §1 · p. 2
  • The study combines QLoRA fine-tuning with post-training quantization to test a small model’s accuracy parity with GPT-4.1 and profile operational trade-offs.Its contributions include a metaprompting-based synthetic dataset and a hardware-aware analysis of speed, memory, and energy.Source: §1 · p. 2
  • A fine-tuned 1-billion-parameter Llama 3.2 model reaches 99% accuracy on multilingual e-commerce intent recognition, matching GPT-4.1.The study also analyzes inference speed, memory consumption, and energy efficiency across GPU and CPU environments, and reports a hardware-aware performance analysis.Source: §1 · p. 2

2 Background and Related Work

This section reviews model specialization, post-training quantization, and synthetic data for structured outputs. It situates the paper’s multilingual e-commerce dataset and optimization experiments within those research areas.

2.1 From Generalists to Specialists: The Power of Fine-Tuning

  • Fine-tuning has enabled specialized smaller models to rival or outperform GPT-4 on narrow tasks, while PEFT methods reduce the resources required to adapt them.LoRA freezes pretrained weights and adds trainable low-rank matrices; QLoRA quantizes the base model to 4-bit precision during fine-tuning.Source: §2.1 · p. 3

2.2 Post-Training Quantization for Efficient Inference

  • Post-training quantization methods such as GPTQ and AWQ target lower memory use, latency, and energy, but their benefits depend on model size, task, and hardware.Quantization can also affect accuracy and capabilities, supporting hardware-aware evaluation.Source: §2.2 · p. 3

2.3 Synthetic Data and Structured Output Generation

  • Synthetic-data generation methods build training sets from seed instructions or through multi-model pipelines, with generator quality affecting the resulting data.Source: §2.3 · p. 3
  • The paper uses an advanced LLM to generate a structured, noisy, multilingual dataset tailored to an e-commerce task, building on research into structured outputs and format constraints.Prior work reports that strict format restrictions can impair reasoning, while targeted fine-tuning can produce robust, domain-aware structured outputs.Source: §2.3 · pp. 3–4

3 Methodology

The methodology is a multistage pipeline for evaluating optimized small language models. It covers task definition, data generation, model selection and fine-tuning, quantization, and evaluation of accuracy and operational performance.

3 Methodology

  • The experimental pipeline includes task definition, synthetic dataset generation, model selection, and fine-tuning.Source: §3 · p. 4
  • Post-training quantization follows fine-tuning in the multistage experimental pipeline.Source: §3 · p. 4
  • The pipeline evaluates both accuracy and operational performance.Source: §3 · p. 4

3.1 Task Definition and Dataset Generation

The task is to extract structured shopping-cart intent from free-form queries. The authors construct a synthetic multilingual dataset with varied phrasing and injected noise for model training and validation.

3.1 Task Definition and Dataset Generation

  • The model converts free-form shopping-cart requests into JSON fields for action, canonical product name, and integer quantity.Source: §3.1 · p. 4
  • The synthetic examples span English, Croatian, and Spanish, with add/remove actions, catalog products, weighted quantities, and varied request templates.Source: §3.1 · p. 4
  • The dataset adds typos, slang, greetings, emojis, unrelated brand names, and code-switching to increase stylistic diversity and robustness.Source: §3.1 · p. 4
  • The pipeline produced a dataset of 3,000 examples, with a data point pairing a user request about 12 lip balms with JSON specifying action remove, product Lip Balm, and quantity 12.Source: §3.1 · p. 4
  • The dataset was published on the Hugging Face Hub and divided into 90% training and 10% validation sets.Source: §3.1 · p. 4

3.2 Models and Baselines

The study centers on Llama 3.2 1B, with other small open-weight models included for comparison and GPT-4.1 models used as commercial accuracy baselines.

3.2 Models and Baselines

  • Llama 3.2 1B was the primary model selected for fine-tuning and quantization.The study also evaluated Gemma 3 1B, Gemma 2 2B, Qwen 2.5 1.5B, and Qwen 2.5 3B.Source: §3.2 · p. 4
  • Gemma 3 1B, Gemma 2 2B, Qwen 2.5 1.5B, and Qwen 2.5 3B formed the representative open-weight comparison set.Source: §3.2 · p. 4
  • GPT-4.1 series models provided the state-of-the-art commercial accuracy benchmark and were evaluated with few-shot prompting.Several examples were included in prompts to guide response formatting.Source: §3.2 · p. 4

3.3 Experimental Pipeline

The pipeline first specializes Llama 3.2 1B with QLoRA, then merges the adapter and prepares GPU- and CPU-oriented quantized variants for deployment.

3.3 Experimental Pipeline

  • The experimental pipeline specializes the base model through fine-tuning, then optimizes it for deployment through quantization.Source: §3.3 · p. 4

3.3.1 Fine-tuning

  • QLoRA fine-tunes a 4-bit NF4 Llama 3.2 1B base using rank-8 adapters, training only on the JSON completion for five epochs.The adapter targets all linear projections in self-attention and feed-forward blocks; training used batch size 8 and learning rate 2e-5.Source: §3.3.1 · p. 5

3.3.2 Post-Training Quantization

  • The trained LoRA adapter is merged with the full-precision FP16 base to create the unified model used for subsequent quantization.Source: §3.3.2 · p. 5
  • GPTQ produces a 4-bit GPU version using 300 random training samples for calibration, while GGUF produces 3-, 4-, and 5-bit CPU variants.The GGUF variants are Q3_K_M, Q4_K_M, and Q5_K_M.Source: §3.3.2 · p. 5

3.4 Evaluation Framework

Evaluation uses a separate unseen test set, a strict exact-match criterion for structured outputs, and operational profiling on specified GPU and CPU hardware.

3.4 Evaluation Framework

  • Accuracy was evaluated on 100 unseen examples structured like the training data.Source: §3.4 · p. 5

3.4.1 Accuracy Metric

  • Exact Match Accuracy requires valid JSON and exact agreement on action, product, and quantity.The metric treats partial correctness as insufficient because it can cause functional errors in production.Source: §3.4.1 · p. 5

3.4.2 Performance Metrics and Hardware

  • Operational profiling measured tokens per second, memory, and GPU energy on an NVIDIA T4 and CPU performance on an AMD Ryzen 7 5800HS.GPU comparisons covered FP16 and GPTQ; CPU comparisons covered GGUF variants.Source: §3.4.2 · p. 5

4 Results

The results show that fine-tuning brings the 1B model to commercial-baseline accuracy, while quantization's efficiency benefits vary sharply by hardware and bit depth.

4.1 Model Accuracy Comparison

  • 0.99 accuracy after QLoRA fine-tuning, a 21% improvement, places Llama 3.2 1B on par with GPT-4.1-nano and near the top commercial models.Table 1 compares exact-match accuracy across models; Figure 1 visualizes overall performance.Source: Table 1 · p. 6, §4.1 · p. 6, Figure 1 · p. 7

4.2 The Impact of Quantization on Accuracy

  • The 4-bit GPTQ and 5-bit GGUF Q5_K_M variants retain the FP16 model's 0.99 accuracy, while Q4_K_M falls to 0.89 and Q3_K_M to 0.60.The paper describes the sharper loss at aggressive quantization as a “quantization cliff”.Source: §4.2 · p. 6

4.3 GPU Performance Profile (FP16 vs. GPTQ)

  • On the NVIDIA T4, 4-bit GPTQ cuts total VRAM from 3.27 GB to 1.93 GB (41%) but slows inference by 82.2% and raises energy per token by 489.3%.Load time falls 93.4%, from 16.95 s to 1.12 s; the authors suggest higher-precision conversion during inference adds overhead.Source: Figure 2 · p. 7, §4.3 · pp. 7–8, Figure 3 · p. 8

4.4 CPU Performance Profile (GGUF)

  • On the AMD CPU, Q4_K_M reaches 47.9 tokens/second, an 18× improvement over FP16, while GGUF reduces RAM from 14.39 GB to 1.15–1.51 GB.GGUF variants also load significantly faster than FP16.Source: Figure 5 · p. 9, §4.4 · p. 9, Figure 6 · p. 9

4.5 Accuracy vs. Efficiency Trade-offs on CPU

  • The CPU Pareto trade-off favors Q5_K_M for 0.99 accuracy at around 42 tokens/s and Q4_K_M for peak speed near 48 tokens/s at 0.89 accuracy.Q3_K_M is suboptimal because it adds no significant memory or speed advantage over the other quantized versions while suffering a severe accuracy penalty.Source: Figure 8 · p. 10, §4.5 · p. 10, Figure 9 · p. 11

5 Discussion

The discussion argues that a specialized 1B Llama 3.2 can match GPT-4.1 on the task, while optimization benefits depend on the hardware-software pairing.

5.1 Small Models Can Achieve State-of-the-Art Accuracy

  • A 1B Llama 3.2 matched GPT-4.1 at 99% accuracy after specialized fine-tuning, supporting domain specialization over assuming larger models always win.The discussion frames this as evidence that a small model can develop strong competency within its narrow domain.Source: §5.1 · p. 11

5.2 Quantization is Not a Free Lunch: The Hardware-Software Synergy

  • On NVIDIA T4, 4-bit GPTQ cut VRAM 41% but slowed inference 82%; CPU GGUF exceeded 18× speedup, with llama.cpp leveraging CPU vector instructions for low-bit multiplication.The authors suggest dequantization overhead explains the T4 slowdown; CPU vector instructions support efficient low-bit integer matrix multiplication.Source: §5.2 · pp. 11–12

5.3 Practical Recommendations and the Pareto Frontier

  • For CPU deployment, Q5_K_M preserves 99% accuracy; Q4_K_M prioritizes throughput with a slight accuracy dip, while 3-bit GGUF is Pareto-dominated.The recommended variant depends on whether accuracy or speed is the priority.Source: §5.3 · p. 12

5.4 Limitations of the Study

  • Evidence is bounded by synthetic-only data, one NVIDIA T4 and consumer CPU, a single intent task, and binary Exact Match Accuracy.The authors note that synthetic queries may not capture real-world complexity and that results may not transfer to more complex tasks.Source: §5.4 · p. 12

6 Conclusion

The conclusion reports that a fine-tuned 1B Llama 3.2 matches GPT-4.1 on structured intent recognition, while optimization outcomes depend on deployment hardware. It advocates hardware-aware specialized models and identifies real-world data, newer GPUs, and broader tasks as future research needs.

6 Conclusion

  • A 1-billion-parameter Llama 3.2 reached 99% exact match accuracy on structured intent recognition, matching GPT-4.1 after parameter-efficient fine-tuning.The paper presents this as confirmation of its primary hypothesis.Source: §6 · p. 12
  • On an older GPU, 4-bit GPTQ caused an 82% inference slowdown; CPU GGUF achieved over 18× throughput improvement and approximately 90% lower RAM than the FP16 baseline.The conclusion emphasizes that operational benefits depend on the deployment environment.Source: §6 · p. 12
  • The authors recommend choosing optimization strategies in light of target hardware and describe specialized models as potentially more cost-effective, private, and computationally efficient.They frame applied AI as a possible ecosystem of smaller, specialized, hardware-aware models rather than only ever-larger generalists.Source: §6 · p. 13
  • Future work should validate performance on real-world anonymized queries, repeat GPU analysis on modern architectures, and extend evaluation to additional e-commerce tasks and optimization methods.Suggested extensions include product recommendation, order-status inquiries, AWQ, and DoRA.Source: §6 · p. 13
Loading 2510.21970v1…