PaperlayerView exact revision on arXiv

Source-linked AI summary

Where A Small Language Model Helps in Invoice Categorisation, Understood Through Embedding Geometry

Emma Ceccherini, Daniel Lawson, Anjulika Salhan

arXiv:2608.18033v1stat.MLcs.LG

TL;DR

Invoice categorisation is a nuanced accounting task with consequences for reporting and compliance. This paper analyses small-language-model embedding geometry and fine-tunes SBERT, achieving 0.96 accuracy and stronger performance on challenging categories and new clients.

  • Problem

    Assigning invoices to GL codes requires specialised judgement, yet evidence on effective in-house small-language-model approaches remains limited.

  • Method

    The paper analyses SBERT and DeBERTa embedding geometry and fine-tunes SBERT on professionally labelled OCR invoice text from seven clients.

  • Results

    0.96 accuracy: fine-tuned SBERT outperformed zero-shot LLM and vendor-identity baselines, particularly for smaller categories and newly onboarded clients.

  • Takeaways & Limitations

    Pre-trained embedding geometry is associated with classification performance, while structured field labelling does not improve fine-tuned SLM performance and can be omitted.

  • Takeaways & Limitations

    The study uses one modest confidential dataset from a UK accountancy firm, so results may not transfer directly across firms, domains, or countries.

Abstract

from arXiv · show

Categorising invoices into the correct General Ledger (GL) code underpins financial reporting and tax compliance. This is a skilled accounting judgement rather than a routine task: the correct category depends subtly on the nature of the purchasing business, the vendor and the invoice text. Whilst AI is increasingly being adopted across industries to automate tasks, including invoice categorisation, implementations built on in-house small language models (SLMs) can simultaneously reduce cost and improve data security, confidentiality, and interpretability. We investigate this approach by first analysing the pre-trained embedding geometry of a small sentence transformer (SBERT) and classic SLM (DeBERTa). The sentence-embedding space of this financial corpus is globally anisotropic but composed of locally isotropic clusters, extending prior token-level findings to sentence embeddings in a financial setting, and these clusters are strongly correlated with the vendor identity. SBERT fine-tuned on a single GPU reaches 0.96 accuracy on invoice classification, above both a zero-shot LLM and a vendor identity baseline, increasing performance for smaller, challenging categories and new clients. For this important generalisation problem, SBERT reaches 0.9 F1 with roughly 100 client-specific invoices, showing that an in-house SLM implementation is promising. Combining these results with geometric analysis shows that pre-trained embedding geometry is associated with classification performance and reveals a counterintuitive finding that a structured input that would help a human reader does not improve the SLM performance.

1 Introduction

Invoice categorisation is a specialised accounting task essential to financial reporting and tax compliance. The paper evaluates in-house small language models as a cost-effective, interpretable alternative and uses embedding geometry to explain their classification performance and input-representation limits.

  • Motivation: Every business invoice must be assigned the correct General Ledger (GL) code, making categorisation central to financial reporting and tax compliance.The task requires specialised accounting skills, and reducing classification errors can improve financial management while reducing compliance costs and penalty risk.
  • Motivation: Zero-shot small LLMs are not well suited to this specialised invoice-classification task, despite increasing AI adoption for automation.The introduction contrasts off-the-shelf zero-shot models with in-house SLMs for this financial-domain application.
  • Motivation: In-house SLMs offer data security, confidentiality, lower costs, and improved interpretability for regulated accounting applications.Encoder-only models such as BERT provide interpretability tools that are unavailable for closed external LLMs and may support audit, regulation, and public trust.
  • Contributions: 0.96 accuracy: fine-tuned SBERT, with ∼22M parameters and a single GPU, outperforms a vendor-only baseline in smaller and challenging categories.This result supports SLMs as a potentially reliable and cost-effective route for deployment-level invoice classification.
  • Contributions: Embedding geometry provides insight into financial-invoice classification performance, robustness, and input representation, extending geometric analysis to this corpus.A more costly structured input that helps human readers does not improve SLM performance because it increases embedding-space similarity.

2 Methods

The study uses confidential OCR-derived invoice text, manually labelled into GL categories, and evaluates three input formats on selected well-populated categories. It compares sentence-level SBERT embeddings with DeBERTa token embeddings using alternative pooling strategies.

  • Dataset: The confidential dataset contains real invoices from seven UK accountancy-firm clients, with OCR output manually labelled by professional accountants.The invoices span various business domains and are associated with correct accounting categories.
  • Dataset: 2,423 invoices span 42 categories, but experiments use the 12 categories with more than 40 samples, covering 2,168 invoices.The dataset averages 75 tokens per invoice and is severely class-imbalanced; Purchases is the largest category with 705 invoices.
  • Input construction: Three OCR-derived inputs vary from vendor name, date, and price only to full line items, with the third adding explicit field labels.The structured variant labels Vendor Name, Invoice Date, Line Items, and Price.
  • Embedding models: SBERT uses native pooling for 384-dimensional sentence embeddings, while DeBERTa produces 768-dimensional token embeddings pooled using either [CLS] or mean pooling.SBERT has ≈22M parameters; DeBERTa has ≈184M parameters.

3 Geometric analysis

Pre-trained invoice embeddings are globally anisotropic but organized into locally isotropic clusters, with SBERT’s geometry more semantically structured than DeBERTa’s. SBERT clusters align more strongly with vendor identity than accounting labels, while DeBERTa-CLS collapses under richer inputs and largely captures embedding norm.

  • Effective dimensionality: SBERT has higher effective dimensionality across all input types, with inputs B and C exceeding A and differing little from each other.This reflects SBERT’s semantically rich embeddings and the additional textual content in B and C.
  • Global anisotropy: DeBERTa embeddings occupy a narrow cone, whereas SBERT embeddings are more homogeneously distributed but remain anisotropic with cosine similarity 0.3-0.5.Both DeBERTa pooling strategies produce extremely high average cosine similarity.
  • Clustering structure: NMI between vendor identity and SBERT cluster membership is > 0.8 across input types, compared with 0.5-0.6 for true labels.This indicates that pre-trained SBERT embeddings align more strongly with vendor identity than accounting categories.
  • Clustering structure: 61% of label uncertainty is accounted for by SBERT-B clustering, which combines the lowest average cosine similarity with the highest effective dimensionality among SBERT variants.These geometric properties later align with the classification results.
  • Local isotropy: Sentence embeddings form a globally anisotropic manifold composed of locally isotropic clusters, extending token-level local-isotropy findings to invoice sentence embeddings.Centring by true labels or K-means clusters recovers isotropy to a similar degree, consistent with the near one-to-one relationship between vendors and accounting labels.
  • DeBERTa collapse: For DeBERTa-CLS, richer inputs reduce the best cluster count from 52 to 2, effective dimensionality from 10 to 1, and cosine similarity to > 0.99.The collapse is linked to embedding norm and likely arises because repeated averaging dilutes semantic signal; mean pooling is less extreme but still underperforms SBERT.

4 Classification

Fine-tuning substantially improves invoice classification, with SBERT outperforming DeBERTa and the zero-shot LLM while structured input does not help. SBERT also generalises more robustly to unseen clients, reaching F1 0.9 with approximately 100 client-specific invoices.

  • Fine-tuned classification: SBERT on input B outperforms DeBERTa, while structured input C does not improve language-model performance.The structured input increases within-corpus similarity, despite helping a human learner.
  • Fine-tuned classification: Fine-tuning substantially improves performance over head-only baselines, closing the gap and surpassing the vendor baseline.
  • Fine-tuned classification: SBERT achieves per-class F1 0.71 for Director’s loan account with 2 layers plus business-name prefix, versus 0.53 with 6 layers without the prefix.DLA and Purchases can have identical invoice text, with their distinction depending on who paid and whether the purchase furthers the business.
  • Fine-tuned classification: The zero-shot Qwen3-4B baseline underperforms SBERT, especially after fine-tuning, while DeBERTa fine-tunes successfully only at depth 2.DeBERTa’s weaker fine-tuning is attributed most likely to its parameter count relative to the sample size.
  • Unseen-client generalisation: At ε = 0 for a fully unseen client, SBERT achieves F1 10% higher than DeBERTa, and the vendor baseline severely underperforms both models.The vendor baseline depends entirely on overlap between the new client’s suppliers and suppliers seen during training.
  • Unseen-client generalisation: To achieve an F1 of 0.9 or higher, SBERT needs, on average, 100 client-specific data points, compared with 174 for DeBERTa.Both models require a warm-up period before reliable performance and continue improving beyond it.

5 Conclusion

The study finds that fine-tuned sentence embeddings make small language models useful for invoice categorisation, particularly for smaller categories and newly onboarded clients. It also shows that invoice embeddings have globally anisotropic geometry with locally isotropic clusters, while structured field labelling adds no benefit after fine-tuning.

  • Conclusion: Invoice sentence embeddings are globally anisotropic but contain locally isotropic clusters, extending previous token-level findings to sentence embeddings in this financial corpus.The experiment uses a small dataset covering only a subset of existing categories.
  • Conclusion: Fine-tuning makes the SLM especially useful for smaller categories and newly onboarded clients, beyond the respectable performance achieved using vendor identity alone.Using full invoice content suggests that line-item text is at least partially mapped to labels through semantic meaning.
  • Conclusion: Sentence embeddings produce more robust representations that generalise better than token embeddings, supporting a relationship between embedding geometry, performance, and robustness.Both models achieved similar high accuracy under naive comparison.
  • Conclusion: Fine-tuned performance on structured Input C is comparable to Input B, so omitting field labels from free-text OCR can reduce human effort, cost, and complexity.Input B requires only a single human-in-the-loop at the end, which is irreducible for regulatory compliance.
  • Conclusion: Future research should develop uncertainty-aware models that identify confidently assigned data with interpretable reasons and focus expert attention where it is most needed.The proposed direction aims to support expert review rather than eliminate it.

Limitations

The study’s conclusions are limited by a modest, single-firm confidential dataset and by geometric analysis involving only two model families. Results may not transfer directly beyond this setting, while broader model comparisons and hyperparameter searches remain necessary.

  • Dataset scope: The dataset contains 2,168 invoices across 12 categories from seven clients at one UK accountancy firm, limiting transferability beyond this setting.The authors describe the sample as modest and caution that results may not transfer directly to firms with different accounting conventions.
  • Model comparison: The geometric analysis and its relationship to classification performance compare only two model families.The authors recommend analysing more model types and conducting additional experiments to strengthen the explanatory claim.
  • Training configuration: The study uses one standard training-parameter set for SBERT and DeBERTa rather than an exhaustive hyperparameter search.This choice supports a consistent comparison but leaves the hyperparameter space incompletely explored.

Ethical considerations

The study uses confidential, commercially sensitive data under data-handling agreements and does not release it, while removing identifying information from visualisations and code. It rejects fully automated use, stresses human accounting oversight, and limits the material to academic and educational purposes.

  • Data privacy: The confidential dataset may contain identifying information about businesses and suppliers and will not be released.The data is used under the firm’s data-handling agreements.
  • Data privacy: Vendor identities and legends are removed from visualisations to protect data privacy.
  • Human oversight: The authors reject a fully automated pipeline and stress human-in-the-loop oversight by professionals with accounting skills.
  • Scope and liability: The material is intended only for academic and educational purposes, with the authors and institutions disclaiming liability for outcomes.
  • Code release: The code is partially released with identifying information removed, while some code is withheld to protect the firm’s intellectual property.

A Appendix

Qwen3-4B processed 434 test invoices in approximately 5 minutes.

  • 434 test invoices were processed by Qwen3-4B in approximately 5 minutes.

A.1 Experimental details Frozen transformers.

Frozen-transformer experiments train linear heads on sentence embeddings using stratified data splits and weighted cross-entropy. Additional comparisons use a constrained zero-shot Qwen3-4B baseline and controlled partial-layer fine-tuning.

  • Frozen transformers: The encoder is frozen while a linear classification head is trained on sentence embeddings using inverse-class-frequency-weighted cross-entropy.Training uses AdamW, a 5e−5 learning rate, batch size 16, and up to 60 epochs with early stopping on validation weighted F1.
  • Frozen transformers: The dataset uses a stratified 60/20/20 split for training, validation, and testing.
  • Frozen transformers: Qwen3-4B provides a zero-shot baseline under the in-house deployment resource constraint, while API-hosted LLMs are excluded for confidentiality.Invoices are classified independently with greedy decoding and thinking mode disabled.
  • Frozen transformers: Fine-tuning trains the last 2, 4, or 6 layers alongside the classification head, with and without a business-name prefix.Transformer layers use 1e−5 learning rate, the classification head uses 5e−5, and 10% linear warm-up is added.

A.2 Supplementary Figures

The supplementary figures show the dataset’s category distribution, vendor-related structure in pre-trained SBERT representations, angular similarity patterns, and classification errors for a fine-tuned model.

  • Dataset composition: 42 accounting categories are shown across the full invoice dataset, with only categories before the dashed line included in the analysis.The figure distinguishes the full category distribution from the subset used for analysis.
  • Embedding geometry: Pre-trained SBERT representations visualised through their first three principal components separate points by vendor identity for the 20 most popular vendors.Points are coloured by vendor, with less-popular vendors shown in grey; the legend is omitted for privacy.
  • Embedding geometry: Except for Accountancy Fee, within- and between-class cosine similarities overlap across the displayed classes, indicating that angular direction matters for successful classification.The comparison includes the all-class average, Purchases, Accountancy Fee, and Light, Heat & Power using frozen SBERT on input B.
  • Classification analysis: A confusion matrix reports predictions from SBERT fine-tuned with 2 layers using a business name/nature prefix.The supplementary figure presents the model’s class-level confusion pattern under this input configuration.

A.3 Supplementary Tables

The supplementary tables define the dataset’s accounting categories and document similarity analyses, classification reports, baselines, and fine-tuning configurations. They also specify reporting conventions, evaluation splits, and input-configuration comparisons.

  • Dataset definitions: Table A1 defines the accounting categories, noting the provider’s custom categorisation, proprietary software, and largest-value assignment for multi-category invoices.UK nominal ledger codes are conventions of different accounting software rather than a standardised system.
  • Embedding geometry: Tables A2 and A3 report average cosine similarities within and between true-label classes and embedding clusters.The similarity summaries use true labels for the class comparison.
  • Classification reports: Tables A4–A8 provide mean±s.d. classification reports over 5 splits for SBERT, DeBERTa CLS, DeBERTa mean, the LLM baseline, and the vendor baseline.The reports include precision, recall, and F1, while the vendor-baseline table is likewise reported over 5 splits.
  • Fine-tuning configurations: Tables A9 and A10 compare accuracy, macro F1, and weighted F1 across fine-tuning configurations and unfrozen final layers for input types A and C.Both tables compare inputs with and without a business-nature prefix, using a single split.
  • Selected fine-tuning reports: Tables A11–A13 report mean±s.d. precision, recall, and F1 over 5 splits for selected SBERT and DeBERTa fine-tuning configurations.The configurations include SBERT with 2 unfrozen layers plus a business-name prefix, SBERT with 6 unfrozen layers, and DeBERTa with 2 unfrozen layers plus a business-name prefix.
Loading 2608.18033v1…