Where A Small Language Model Helps in Invoice Categorisation, Understood Through Embedding Geometry

Motivation

• Off-the-shelf zero-shot small LLMs are not well suited to this specialized task, while in-house SLMs offer security, confidentiality, lower costs, and interpretability advantages (Das et al., 2025; Wang et al., 2025; Belcak et al., 2025; Anwar et al., 2024; Luo and Specia, 2024).The paper notes that encoder-only SLM tools may support audit, regulation, and public trust (Rogers et al., 2020; Clark et al., 2019).Source: §1 · p. 1

Contributions

• Fine-tuned on a single GPU, SBERT (∼22M parameters) reaches 0.96 accuracy on invoice classification.The paper presents this result as evidence that SLMs may offer a reliable, cost-effective deployment route.Source: §1 · p. 2
• SBERT contributes over a vendor-only baseline in smaller, challenging categories; on unseen clients, that baseline collapses and SBERT improves macro F1 over DeBERTa.The paper frames performance on unseen clients as a direct measure of client onboarding.Source: §1 · p. 2
• Invoice sentence embeddings are globally anisotropic but locally isotropic in clusters strongly associated with vendor identity.The authors present this as extending prior token-level findings to sentence embeddings in a financial setting.Source: §1 · p. 2
• SBERT’s pre-trained space has higher effective dimensionality and lower average cosine similarity than DeBERTa’s, whose CLS sentence embedding collapses as input grows richer.The paper links these geometric differences to SBERT’s increased performance over DeBERTa.Source: §1 · p. 2
• A more costly structured input that helps human readers does not improve SLM performance, which the paper attributes to increased embedding-space similarity.The authors present this finding as directly relevant to practitioners choosing input representations.Source: §1 · p. 2

Related work

• Prior GL-code invoice classifiers used static word embeddings or deep learning, but the authors know of no sentence-based SLM paired with thorough geometric investigation.The paper also identifies its use of geometric tools to interpret invoice-classification performance as, to the authors’ knowledge, the first on a financial invoice corpus (Ethayarajh, 2019; Gao et al., 2019).Source: §1 · p. 2

2.1 Dataset

• The confidential dataset contains real invoices from seven clients, with OCR text as input and categories manually labelled by professional accountants.The clients span various business domains.Source: §2.1 · p. 2
• The full dataset has 2,423 invoices across 42 categories; experiments restrict analysis to 12 categories with more than 40 samples (2,168 invoices).The restriction targets categories with enough labelled data for reliable training and evaluation, together accounting for most invoice volume.Source: §2.1 · pp. 2–3
• Three inputs vary from vendor, date, and price only, to OCR text with line items, and that same text with explicit field labels.The labelled format names Vendor Name, Invoice Date, Line Items, and Price.Source: §2.1 · p. 3

2.2 Experimental set up

• SBERT (≈22M parameters, 384 dimensions) is compared with DeBERTa (≈184M, 768 dimensions), whose token embeddings are pooled using [CLS] or mean.SBERT embeds each invoice using native pooling from its last layer; DeBERTa pooling uses the last layer.Source: §2.2 · p. 3

3.1 Effective dimensionality

• SBERT has higher effective dimensionality across input types, with B/C exceeding A and little difference between B and C; DeBERTa has low effective dimensionality, especially with CLS pooling.For DeBERTa, the first principal component correlates more than 90% with embedding norm across input types.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 Anisotropy

• DeBERTa pooling strategies have extremely high average cosine similarity, whereas SBERT embeddings are more homogeneously distributed but remain anisotropic (cosine similarity 0.3-0.5).Source: §3.2 · p. 4, Table 2 · p. 4

3.3 Clustering

• SBERT clusters align more with vendor identity (NMI > 0.8) than true labels (0.5-0.6); input A is most vendor-aligned, while B/C are slightly more label-aligned and SBERT-B explains 61% of label uncertainty.Compared with true-label partitions, clustering produces higher within-cluster similarity but similar between-cluster similarity, supporting sharper separation by vendor than by accounting category.Source: §3.3 · p. 5

3.4 Local isotropy

• Invoice sentence embeddings are globally anisotropic but almost perfectly isotropic after centring by class or cluster; either partition removes anisotropy to a similar degree.The authors attribute this to a near-perfect many-to-one mapping between vendors and accounting labels, which expose nearly the same variation directions.Source: §3.4 · p. 5, Table 4 · p. 5

3.5 CLS collapse

• DeBERTa-CLS collapses on inputs B/C: clusters drop from 52 to 2, effective dimension from 10 to 1, and cosine similarity exceeds 0.99; its structure tracks embedding norm, not semantics (Liang et al., 2021).The authors attribute this to CLS’s near-uniform attention and repeated averaging as inputs grow; mean pooling shows a milder version, while the CLS classifier predicts only larger classes.Source: §3.5 · pp. 5–6

4.1 Frozen transformers and vendor baseline

• SBERT input B substantially surpasses DeBERTa mean and CLS and is best across models; structured input C does not help, and SBERT reaches around 0.8 F1 on all but one category.DeBERTa CLS achieves approximately 0% accuracy on some smaller categories, consistent with the embedding collapse described in Section 3.5.Source: §4.1 · p. 6

4.2 Fine Tuning

• Fine-tuned SBERT reaches DLA per-class F1 of 0.53 (6 layers no b.n.) or 0.71 (2 layers + b.n.), with the latter 27 percentage points above the vendor baseline.SBERT also improves by 10% over the vendor baseline on Stationery & Postage; rich invoice text matters for the long tail of hard invoices.Source: §4.2 · p. 6

4.1 Frozen transformers and vendor baseline

• The vendor-to-label baseline outperforms frozen SBERT, DeBERTa, and the LLM on every input type; a linear head needs more samples to learn the many-to-one vendor map.Hundreds of vendors are grouped into 12 classes, whereas a single memorised vendor can let the baseline classify a small category.Source: §4.1 · p. 6, Table 5 · p. 7

4.2 Fine Tuning

• Fine-tuning substantially improves performance over head-only baselines, surpasses the vendor baseline, and SBERT outperforms zero-shot Qwen3-4B by +0.55 M F1 after fine-tuning.Table 6 reports input B results across fine-tuning configurations and numbers of unfrozen layers, with and without a business-nature prefix.Source: §4.2 · p. 6, Table 6 · p. 7
• Fine-tuning unravels SBERT’s X-shaped manifold into a simplex-like geometry and separates DLA from Purchases, consistent with the DLA F1 improvement.The authors describe simplex geometry as potentially supporting uncertainty calibration, which they leave for future work.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3 Generalise to new clients

• To reach F1 of 0.9 or higher, SBERT needs an average of 100 client-specific data points versus DeBERTa’s 174; SBERT’s higher-dimensional, less anisotropic space supports faster separation.Both models need a warm-up period, then continue improving with more client data and converge toward their fine-tuned baselines.Source: §4.3 · pp. 7–8
• At ϵ = 0, SBERT achieves an F1 10% higher than DeBERTa on a fully unseen client, while the vendor baseline severely underperforms.The vendor baseline depends on supplier overlap with training data, whereas line-item text carries semantic signal that transfers to unseen vendors.Source: §4.3 · p. 7, Figure 4 · p. 8

5 Conclusion

• Although the experiment uses a small dataset covering only a subset of categories, its results suggest promise for in-house deployment where cost, confidentiality, and interpretability matter.Source: §5 · p. 8
• Invoice sentence embeddings are globally anisotropic but contain locally isotropic clusters, extending prior token-level findings; their robustness relative to token embeddings supports a link between geometry and performance.The models had similar high accuracy under naive comparison, while sentence embeddings generalised better than token embeddings.Source: §5 · p. 8
• Fine-tuning made the SLM useful beyond respectable vendor-identity performance, especially for smaller categories and newly onboarded clients, suggesting invoice line-item text maps to labels through meaning.The conclusion contrasts vendor identity alone with using full invoice content.Source: §5 · p. 8
• Structured Input C performed comparably to Input B after fine-tuning, so omitting field labels from OCR text can remove a human-in-the-loop step and reduce cost and complexity.Input B retains one final human review, which the conclusion describes as irreducible for regulatory compliance.Source: §5 · pp. 8–9
• Future research could develop an uncertainty-aware model that gives interpretable reasons for confident assignments and directs expert attention to uncertain cases.Source: §5 · p. 9

Limitations

• The study uses 2,168 invoices across 12 categories from seven clients at one UK accountancy firm, so results may not transfer directly to different conventions, domains, or countries.The authors describe the sample as modest and the findings as evidence of viability in this setting.Source: Limitations · p. 9
• The geometric analysis compares two model families, and further model types and experiments are needed; the study also uses standard training parameters without an exhaustive hyperparameter search.Source: Limitations · p. 9

Ethical considerations

• The confidential, commercially sensitive data may identify businesses and suppliers, was used under firm datahandling agreements, and will not be released.Vendor identities and legends are removed from visualized representations to protect privacy.Source: Ethical considerations · p. 9
• The authors reject a fully automated pipeline and emphasize keeping professionally skilled accountants in the loop.The material is for academic and educational purposes only, and the authors and institutions disclaim liability for outcomes from its use.Source: Ethical considerations · p. 9
• The authors partially release code after removing identifying information and retaining some code to protect the firm’s intellectual property.Source: Ethical considerations · p. 9

• Qwen3-4B took approximately 5 minutes to process 434 test invoices.Source: A Appendix · p. 12

A.1 Experimental details

• Frozen-encoder classifiers use mean pooling or DeBERTa’s CLS head, with a weighted cross-entropy objective and a stratified 60/20/20 train, validation, and test split.Training uses AdamW at 5e−5, batch size 16, up to 60 epochs, and early stopping on validation weighted F1.Source: §A.1 · p. 12
• The zero-shot baseline uses Qwen3-4B (Qwen Team, 2025), with firm-provided category definitions, greedy decoding, and thinking mode disabled.API-hosted LLMs were excluded for confidentiality; a deterministic parser maps outputs to canonical labels, and unparseable outputs count as errors.Source: §A.1 · p. 12
• Fine-tuning trains the last 2, 4, or 6 layers with a classification head, comparing configurations with and without a business-name prefix.Transformer layers use a 1e−5 learning rate and the head 5e−5; joint AdamW optimization includes 10% linear warm-up to help DeBERTa avoid early divergence.Source: §A.1 · p. 12
• Fine-tuning ran on an NVIDIA RTX PRO 500 Black with 6 GB VRAM; SBERT (2 layers + b.n.) took about 10 minutes and DeBERTa (2 layers + b.n.) approximately 15 minutes.The reported runs were 46 epochs for SBERT and 31 epochs for DeBERTa.Source: §A.1 · p. 12

A.2 Supplementary Figures

• Figure A1 shows invoice distribution across 42 accounting categories, with categories before the dashed line included in the analysis.Source: Figure A1 · p. 13
• Figure A2 visualizes the first three principal components of pretrained SBERT representations for input B, coloring points by the 20 most popular vendors and showing others in grey.The legend is removed for data privacy.Source: Figure A2 · p. 14
• Figure A3 shows overlapping within- and between-class cosine similarities for all-class average, Purchases, and Light, Heat & Power; Accountancy Fee is the exception.The caption states that the direction of angular variation in embedding space is important for successful classification.Source: Figure A3 · p. 15
• Figure A4 presents the confusion matrix for SBERT fine-tuned with 2 layers and a business name/nature prefix.Source: Figure A4 · p. 16

A.3 Supplementary Tables

• The dataset uses the UK firm's custom accounting categories, and multi-category invoices are assigned to the category with the largest total value.The passage notes there is no standardised UK nominal ledger and that code numbers vary by accounting software.Source: Table A1 · p. 17
• Tables A2 and A3 report average cosine similarity within and between true-label classes and clusters, respectively.Source: Table A2 · p. 18, Table A3 · p. 19
• Tables A4–A8 provide five-split mean±s.d. classification reports for SBERT, two DeBERTa representations, the LLM baseline, and the vendor baseline.The reports cover precision, recall, and F1; the supplied passages do not show their values.Source: Table A4 · p. 19, Table A5 · p. 19, Table A6 · p. 20
• Tables A9 and A10 compare accuracy, macro F1, and weighted F1 across fine-tuning configurations and unfrozen-layer counts for input types A and C, with and without a business-nature prefix.These comparisons are for a single split, and each table marks its best performance in bold.Source: Table A9 · p. 21, Table A10 · p. 21
• Tables A11–A13 provide five-split mean±s.d. classification reports for SBERT with two or six unfrozen layers and DeBERTa with two unfrozen layers, using a business-name prefix where specified.The reports cover precision, recall, and F1; the supplied passages do not show their values.Source: Table A11 · p. 22, Table A12 · p. 22, Table A13 · p. 22