PaperlayerView exact revision on arXiv

Source-linked AI summary

When Do LLMs Actually Help? Evaluating LLMs as Data Quality Annotators

Praphulla Lal Shrestha

arXiv:2608.18158v1cs.CLcs.AI

TL;DR

Reliable evidence on when LLMs help with data-quality annotation remains limited beyond one-time accuracy. This study compares an LLM with rule-based baselines across two e-commerce tasks and finds its value strongly task dependent.

  • Problem

    Evaluations emphasize predictive accuracy more than whether LLM annotations remain reliable across repeated executions.

  • Method

    The study evaluates GPT-4o-mini against rule-based baselines on entity matching and brand mislabeling, using zero-shot and few-shot prompts plus repeated-run consistency tests.

  • Results

    LLMs offered little advantage for entity matching but outperformed rules for brand mislabeling, while repeated predictions were highly consistent.

  • Takeaways & Limitations

    LLM usefulness depends on task characteristics: lexical matching favors rules, whereas brand-labeling decisions benefit from contextual and semantic reasoning.

  • Takeaways & Limitations

    Results may not generalize beyond one model, easy or synthetic benchmarks, untested brand-labeling consistency, and generic prompts.

Abstract

from arXiv · show

LLMs have been increasingly used to catch data quality issues automatically, but we know very little about how consistent these judgments actually are. This study tests an LLM on two e-commerce data quality tasks, entity matching and brand mislabeling, against rule based baselines and human verified ground truth, under both zero-shot and few-shot prompting. On entity matching while using the Abt Buy benchmark (2,194 labeled pairs), a simple rule based baseline (F1=0.950) performed about as well as LLM zero shot prompting (F1=0.948). Moreover, a few-shot prompt revision that looked effective on a small validation sample reduced full-scale performance to F1=0.914. This showed that small sample prompt evaluation can be misleading. On brand mislabeling detection, using 500 Amazon product listings with synthetically injected labeling errors, the LLM clearly outperformed a naive rule based baseline (F1=0.833 vs 0.721), because it could draw on background knowledge of brand product relationships that a simple rule could not access. Testing consistency across repeated runs (200 pairs, 5 runs at temperature 0.7) showed the model agreeing with itself 99.7% of the time on average, with 99% of pairs giving identical answers across all 5 runs. Using majority voting across these runs only improved F1 by 0.005, at 5 times the inference cost. These results suggest that the value of using an LLM over traditional methods depends heavily on the task. LLMs offer little advantage when strong lexical signals already exist, but a clear advantage when the task requires background knowledge, all while remaining highly consistent across repeated queries.

I. INTRODUCTION

This study evaluates GPT-4o-mini as a reliable data quality annotator across entity matching and brand mislabeling detection in e-commerce data. It finds that LLM value depends strongly on task characteristics: limited advantage where lexical rules suffice, but substantial benefit where contextual and semantic reasoning is needed.

  • Motivation: Data quality problems commonly arise in webscraped and e-commerce data, including duplicate records, incorrect brand assignments, inconsistent metadata, and missing values.These datasets support search, recommendation, analytics, and business decision-making, increasing the importance of maintaining their quality.
  • Motivation: LLMs are increasingly explored as flexible annotators because they can understand natural language and generalize beyond explicit rules.Applications include entity matching, data labeling, record validation, and anomaly detection.
  • Reliability: Accuracy alone is insufficient for evaluating data quality annotators because inconsistent repeated outputs can reduce confidence in automated decisions.The study addresses the reliability of annotations, an aspect receiving comparatively little attention in prior evaluations.
  • Study Scope: The study tests GPT-4o-mini on entity matching and brand mislabeling detection using e-commerce datasets and compares it with rule-based baselines.Entity matching uses the public Abt-Buy benchmark, while brand mislabeling uses a synthetically corrupted Amazon product dataset.
  • Results: Zero-shot prompting offers limited advantage over a simple Jaccard similarity baseline for entity matching, whereas the LLM substantially outperforms a rule-based approach for brand mislabeling detection.The contrast suggests contextual and semantic reasoning is valuable when explicit rules are insufficient.

II. RELATED WORK

Related work covers classical entity resolution, product-attribute validation, LLM annotation, repeated-run reliability, and cost-efficient prompting. These studies provide benchmarks for interpretable matching, attribute-specific validation, annotator agreement, and the costs of added prompting complexity.

  • Classical entity resolution: Classical entity resolution spans deterministic, probabilistic, and clustering-based methods, with deterministic matching serving as an interpretable baseline that degrades as attribute noise increases.This field predates LLM-based approaches by decades.
  • Product attribute and brand data quality: Naive keyword checking can fail when a value appears textually but belongs to the wrong attribute, producing rule-based baseline precision of 0.564.Wang et al. address catalog attribute validation with MetaBridge, which requires training and per-category unlabeled data.
  • LLMs as data annotators: GPT-4 achieved 68.4% versus 38.6% for MTurk non-expert crowdworkers in financial relation extraction, but no evaluated model achieved perfect self-agreement across two identical runs.Prompt choice affected consistency more than temperature did.
  • Reliability and consistency under repeated sampling: Human expert agreement was 33.7% in cognitive distortion detection, while majority labels remained stable across 5 runs for over 84% of examples.That study treats cross-run consistency as a reliability proxy, unlike the present study’s objectively verified ground truth.
  • Cost-efficient LLM pipelines and small model evaluation: A decomposed relevance pipeline improved GPT-4o accuracy by 9.7%, whereas self-reflection prompting underperformed simpler prompts across six LLMs and five prompting strategies.These findings contrast with added complexity yielding marginal or negative returns in the present study.

III. METHODOLOGY · A. Datasets · B. Baselines

The study evaluates entity matching and brand mislabeling using benchmark-derived datasets with human-verified or synthetic ground truth, alongside task-specific rule-based baselines. Entity matching uses Jaccard word overlap, while brand mislabeling uses a manufacturer-substring heuristic.

  • A. Datasets: Entity matching uses Abt.csv and Buy.csv, containing 1,081 and 1,092 products, respectively.The study also uses a human-verified perfect mapping file identifying true matches between the catalogs.
  • A. Datasets: 2,194 labeled entity-matching pairs form a class-balanced dataset of mapping-derived positives and randomly sampled nonmatching negatives.Positive pairs come directly from the verified mapping file.
  • A. Datasets: Brand mislabeling uses 500 Amazon.csv rows from the Amazon-Google Products benchmark, each with a populated manufacturer field.The dataset was sampled down from the benchmark for this task.
  • A. Datasets: Synthetic brand-mislabeling ground truth swaps manufacturers for approximately half the rows and leaves the remainder untouched.Swapped manufacturers are real manufacturers drawn from elsewhere in the dataset.
  • B. Baselines: Entity matching uses Jaccard word-overlap similarity on normalized product names, with matches determined by a threshold.Normalization lowercases names and strips punctuation.
  • B. Baselines: 0.2 was the threshold with the best entity-matching F1 among tested thresholds and became the reported baseline.The baseline applies the threshold to normalized product-name similarity.
  • B. Baselines: Brand mislabeling uses a naive substring baseline that treats a product as correctly labeled when its manufacturer appears in the product title.Products whose manufacturer name does not appear in the title are flagged as mislabeled.

C. LLM Setup

The study uses GPT-4o-mini with zero-shot prompting and temperature 0 for reproducible main results, while separately testing few-shot guidance and repeated-run consistency for entity matching.

  • Experimental setup: GPT-4o-mini is used through the OpenAI API, with zero-shot prompting as the primary method and temperature 0 for all main results.These settings apply to both entity matching and brand mislabeling tasks.
  • Experimental setup: The entity-matching few-shot variant emphasizes shared model or SKU codes and includes two worked examples.This variant is tested in addition to the primary zero-shot approach.
  • Experimental setup: Consistency testing for entity matching uses five repeated runs at temperature 0.7 on a 200-pair stratified subsample.The consistency analysis is restricted to the entity-matching task.

D. Evaluation · IV. RESULTS · A. Entity Matching (Abt-Buy, 2,194 Pairs)

On the Abt-Buy benchmark, entity-matching methods were evaluated against ground truth using standard quality metrics and repeated-run consistency measures. Although a targeted prompt revision appeared helpful on a small validation sample, it reduced full-scale performance by overgeneralizing code dependence.

  • D. Evaluation: Precision, recall, F1, and accuracy were measured against ground truth, with repeated runs additionally assessed by per-pair agreement and majority-vote accuracy.These metrics defined evaluation for the entity-matching methods and consistency testing.
  • A. Entity Matching (Abt-Buy, 2,194 Pairs): The Abt-Buy comparison evaluated three entity-matching methods on 2,194 labeled pairs using precision, recall, F1, and accuracy.The benchmark table reports the metric comparison across the three methods.
  • A. Entity Matching (Abt-Buy, 2,194 Pairs): Precision, recall, F1, and accuracy were compared across the three entity-matching methods on the Abt-Buy benchmark.The figure presents the same metric comparison visually.
  • A. Entity Matching (Abt-Buy, 2,194 Pairs): 2,022 of 2,194 pairs were correctly classified by both the rule-based method and zero-shot LLM, while the rule-based method won 67 cases and the LLM won 64.Both methods were wrong on 41 pairs; LLM wins mainly involved low lexical overlap but clear semantic equivalence.
  • A. Entity Matching (Abt-Buy, 2,194 Pairs): A prompt revision fixing 28 of 67 prior errors while breaking 1 of 50 controls looked effective on the small validation sample.The validation sample combined the 67 previously wrong cases with 50 control cases and instructed the model to prioritize SKU or code matches.
  • A. Entity Matching (Abt-Buy, 2,194 Pairs): At full scale, the revised prompt reduced F1 from 0.948 to 0.914 and recall from 0.902 to 0.846.The intervention appears to have overgeneralized, making the model more code dependent across the broader dataset than the small validation sample suggested.

B. Brand Mislabeling (Amazon, 500 Listings)

On 500 Amazon listings, the LLM was correct more often than the rule-based method, primarily because the rule flagged products whenever manufacturer names were absent from titles. The error analysis also showed substantial agreement, but the LLM corrected many rule-based mistakes.

  • Error analysis: 146 listings favored the LLM: it was correct while the rule-based method was wrong.Both methods were correct on 272 listings, while both were wrong on 43.
  • Error analysis: 39 listings favored the rule-based method: it was correct while the LLM was wrong.The remaining comparison category reported in the error analysis was that both methods were wrong on 43 listings.
  • Method behavior: The rule-based method’s high recall but low precision came from flagging products whose manufacturer name did not literally appear in the title.This literal title-matching rule produced flags even when the product was not mislabeled.

C. Consistency and Reliability (200-Pair Sample, 5 Runs, Temperature = 0.7)

The LLM was highly self-consistent across five repeated runs at temperature 0.7, while majority voting produced only a small F1 improvement over a single deterministic run.

  • Consistency: 0.997 average per-pair agreement was observed across five repeated runs at temperature 0.7 on a stratified 200-pair sample.The model remained highly self-consistent under non-zero temperature.
  • Consistency: 99.0% of pairs produced identical answers across all five runs.
  • Majority voting: F1=0.974 for five-run majority voting exceeded F1=0.969 for a single run at temperature 0 on the same 200-pair sample.The resulting improvement was small despite aggregating five runs.

V. DISCUSSION

LLM annotation value is strongly task dependent: it adds little over lexical rules for entity matching but offers greater promise for brand mislabeling. Prompt changes can look beneficial on small samples yet hurt full-scale performance, while repeated runs remain highly consistent.

  • Task dependence: F1=0.950 for the Jaccard baseline essentially matched F1=0.948 for LLM zero-shot entity matching.Abt-Buy product names have high lexical overlap, leaving little room for semantic understanding to add value.
  • Task dependence: F1=0.721 for the rule-based brand-mislabeling baseline left more room for LLM-based annotation to help.The baseline also had precision 0.564.
  • Prompt validation: F1 fell from 0.948 to 0.914 when an exact-SKU/model-code prompt revised on a small sample was applied to all 2,194 entity-matching pairs.Recall also fell from 0.902 to 0.846, showing that small, hand-picked validation samples can overstate prompt benefits.
  • Reliability: 99.7% average self-agreement and identical answers for 99% of pairs across five runs indicated high run-to-run consistency.Majority voting improved F1 only from 0.969 to 0.974 while costing five times more.

VI. LIMITATIONS

The study’s limitations concern dataset realism and generalizability: Abt-Buy is comparatively easy, brand-mislabeling errors were synthetically generated, and only one LLM was evaluated. Generic prompts may also underestimate performance achievable with category-specific prompting.

  • Dataset limitations: Abt-Buy’s structurally consistent product names make it comparatively easy, so results may not generalize to noisier real-world catalogs.The benchmark’s catalog structure may not reflect the variability of real-world data.
  • Dataset limitations: Synthetic manufacturer swaps create clearly implausible brand mismatches rather than the subtler labeling errors found in practice.This construction limits how directly the brand-mislabeling results transfer to naturally occurring errors.
  • Methodological limitations: Only GPT-4o-mini was evaluated, while generic prompts were not tailored to product categories or attribute types.Recent category-specific prompt generation work reportedly achieves substantially higher performance, suggesting the reported brand-mislabeling results may be a lower bound.

VII. CONCLUSION

The study finds that the value of LLM-based data quality annotation depends on the task, with zero-shot prompting performing on par with a simple word-overlap baseline. It therefore recommends per-task evaluation and caution when generalizing from small-scale prompt tests.

  • Task-dependent value: LLM annotation offered no uniform advantage over traditional rule-based methods because its value depended on the nature of the data quality task.The study evaluated GPT-4o-mini on entity matching and brand mislabeling detection.
  • Task-dependent value: Zero-shot LLM prompting performed on par with a simple word-overlap baseline.This comparison is reported among the study’s main results.
  • Practical implications: Practitioners should evaluate LLM-based annotation per task and avoid generalizing conclusions from small-scale prompt testing.The conclusion frames both recommendations as implications of the combined findings.
  • Future work: Future work should examine additional model families, test consistency on brand mislabeling, and study harder, more realistic mislabeling scenarios.The proposed scenarios should better reflect subtle brand and category errors.
Loading 2608.18158v1…