Source-linked AI summary
Text Data Analysis and Classification Methods - Insights from Customer Letters in Life Insurance
Andreas Groll, Marie Punsmann, Leonid Zeldin
TL;DR
Life insurers possess valuable customer-letter data that is rarely analyzed, while cancellation reasons are not systematically recorded. This paper extracts and classifies letters using statistical and machine-learning approaches, finding comparable suitability across three low-feature models, with the SVM attaining a Youden index of 0.894. The approach is exploratory and does not fully optimize issues such as OCR errors or scalability to much larger datasets.
Problem
Customer letters are a largely unexplored insurance data source, and cancellation letters and their reasons need to be identified for better analysis of customer behavior.
Method
The paper extracts letter text and applies classification models to identify cancellations and classify their reasons using one-versus-rest binary models.
Results
The SVM achieved the highest Youden index at 0.894, while the three models were approximately equally suitable overall.
Takeaways & Limitations
Extracted cancellation information can be added to insurer databases and the approach extended to other letters and business-event reasons.
Takeaways & Limitations
Computational efficiency was not further optimized and may become relevant for significantly larger datasets.
Abstract
from arXiv · showhide
The business of life insurance companies is characterized by long-term contracts. For this reason, data describing customers is of immense value. A portion of the data provided to the customer is rarely or not at all analyzed. This includes customer letters of any kind. This work focuses on classifying customer letters as cancellations and identifying the respective reason, if available. The outlined approach can also be applied to other business transactions and reasons. We discuss data acquisition and preparation, present alternatives, and explain the reasons for the chosen approach. A successful implementation of such a tool can lead to a better understanding of customer cancellation behavior by the insurer, enabling more targeted actions in certain situations.
1 Introduction
The paper proposes analyzing customer letters to identify cancellations and their stated reasons, addressing a largely unexplored source of potentially valuable insurance data. It outlines extraction, classification, and integration of these findings into insurer databases.
- Customer letters are a largely underestimated and unexplored source of potentially high-value insurance data.They are typically read, addressed, and archived, with only occasional manual or rule-based classification.
- The work develops a process combining image and text recognition with machine-learning classification of customer letters.The methods are intended to categorize letters while allowing classification rules to self-learn to some extent.
- The study specifically separates cancellation letters from other letters and analyzes the reasons customers provide for cancellation.This focuses the broader text-analysis process on customer cancellation behavior.
- Extracted information can be added to databases and extended to other customer letters and business-event reasons.The paper connects this integration with better understanding of customer actions, personalized services, communication, and potential new-business identification.
- The manuscript covers data acquisition, privacy, technical requirements, preparation, text extraction, procedures, results, and future steps.Data labeling is described as the most labor-intensive preparation step.
2 Motivation
The motivation is to use customer letters from a life insurer to classify cancellations and their reasons, because long-term contracts create few customer interactions and existing transaction labels are flawed. The work is exploratory and tests several information-retrieval and classification approaches without directly addressing algorithmic bias.
- Long-term life insurance contracts involve limited customer interactions, making rare contacts valuable for understanding the insurer’s portfolio.
- The study examines letters concerning cancellation, premium pause, premium waiver, increase, and policyholder change.Possible reasons are created for each business event, followed by classification into those reasons.
- Content analysis revealed that the insurer’s existing business-event classification was flawed, so the study focuses on cancellation-labeled letters.
- The first goal is to identify cancellation documents from all documents and, where possible, output cancellation likelihoods despite imperfect initial labels.
- The second goal is to estimate probabilities for four common cancellation reasons plus an “other reason/no reason” category.
- The study does not directly address discrimination or algorithmic bias because it classifies predefined categories without assigning value judgments.
- The work is exploratory, testing statistical and machine-learning methods for information retrieval rather than constructing a perfectly adapted model.
3 Data
The data section describes acquiring and legally governing life-insurance letters, manually labeling cancellation reasons, and cleaning the corpus for supervised classification. After processing, the dataset remains highly imbalanced, with most cancellations lacking a stated reason.
- The study uses 14,575 customer letters from a major German life insurance company, approximately covering five years of correspondence.
- The source files are protected, non-searchable PDFs ranging from single-page letters to documents exceeding 40 pages.Text extraction is therefore required before processing.
- The study avoids storing personal data, which also prevents graphical analyses such as word clouds.
- Data access involved coordination with the company’s data protection officer and responsible data entity before the analyses were conducted.
- Supervised classification required manually examining cancellation letters because archived business transactions did not record letter content or cancellation reasons.The cancellation focus reflects the relevance of cancellations for lapse-rate forecasting.
- Preprocessing rechecked transaction labels, excluded unrelated emails and attachments, and removed duplicate documents.
- Reasons occurring fewer than 10 times were grouped into “other reasons” and merged with “no reason” for analysis.
- After cleaning, 12,092 documents remained, including 1,257 cancellations, but only 83 had one of the considered reasons.The imbalance was addressed through oversampling for most analyses.
4 Extracting Text
Text extraction converts protected, non-searchable PDFs into machine-readable text through image conversion and OCR. The workflow uses Tesseract via Python, concatenates document pages, and accepts extraction quality without further optimization because handwriting is limited and difficult to decipher.
- Protected PDFs are converted to unprotected files, then to PIL images before text recognition because Python cannot directly read them.
- Optical Character Recognition converts text in images into machine-readable text for subsequent processing.OCR is the final and most computationally intensive procedural step.
- The workflow uses the open-source Tesseract tool through its Python implementation, pytesseract.The choice reflects availability, Python compatibility, and reported performance in related studies.
- Tesseract identifies text regions and handles irregularly aligned or spaced text through baseline detection, word-gap analysis, and adaptive segmentation.
- Tesseract combines dictionary, numerical, and case-pattern comparisons with static and adaptive classifiers to improve recognized words across passes.
- Pages are concatenated for later processing, so the resulting text no longer records page counts or page breaks.
- Machine-written text is recognized more effectively than handwriting, but handwriting is uncommon and often difficult or impossible to decipher.The extracted-data quality is therefore considered acceptable without further optimization.
5 Methods in Text Data Analysis and Classification
The paper preprocesses customer-letter text into token-based representations, selects informative features, and applies interpretable and machine-learning classifiers. Class imbalance is addressed through oversampling and threshold selection using the Youden Index.
- Text preprocessing: Preprocessing tokenizes texts, optionally removes stop words, and applies stemming to group words sharing a stem.Tokenization creates word-based token vectors; n-grams represent consecutive words.
- Text representation: The Document-Term Matrix summarizes token frequencies for each document without preserving token order.The entry h_jω records the frequency of token ω in document j.
- Feature selection: Feature selection uses the Gini index to identify tokens with stronger associations to specific classes, including a normalized version for imbalanced datasets.Higher Gini values indicate stronger discriminatory power.
- Machine-learning classifiers: Random Forests and Support Vector Machines classify Document-Term Matrix data using ensemble tree voting and separating hyperplanes, respectively.Random Forests reduce correlation among trees through bootstrap samples and random feature subsets, while SVMs can use kernels for nonlinear relationships.
- Word Search: Word Search classifies documents by matching manually selected tokens associated with predefined categories.The approach prioritizes interpretability and computational efficiency but depends on domain knowledge and may miss context-dependent meaning.
- Imbalanced classification: Random resampling enlarges the smaller class, while the Youden Index balances sensitivity and specificity when selecting the model threshold.The index is defined as sensitivity plus specificity minus one.
6 Results
The analysis proceeds in two stages: first identifying cancellation documents, then assigning cancellation reasons only among documents classified as cancellations.
- Cancellation classification: The first stage classifies documents into “Cancellation” and “No Cancellation.”
- Reason classification: The second stage assigns cancellation reasons to the subset of documents classified as “Cancellation.”Only cancellation documents are considered in this stage.
6.1 Locating Cancellations
The study compares Word Search, Random Forest, and Support Vector Machine approaches for identifying cancellations, using prepared letter text and selected tokens. On test data, all three models perform similarly overall, with no clear best model.
- Classification methods: The classification pipeline applies Word Search, Random Forest, and Support Vector Machine models to distinguish “Cancellation” from “No Cancellation.”Random Forest training uses oversampling because the cancellation class is smaller.
- Data preparation: The texts are cleaned, tokenized, stemmed, stripped of stop words and numbers, then represented with monogram and bigram document-term matrices.Tokens are filtered by occurrence and Gini index before a joint training matrix is constructed; test data use only training-selected tokens.
- Word Search: Word Search selects cancellation-related tokens manually from word clouds and frequency analysis, retaining 12 tokens comprising 8 monograms and 4 bigrams.Only tokens occurring more frequently in cancellations than non-cancellations are retained.
- Word Search results: 0.920 sensitivity and 0.966 specificity were achieved by the final Word Search model on unseen test data.The final model followed refinements that raised training specificity to 0.965 and the Youden Index to 0.890.
- Model comparison: 0.894 was the largest test-data Youden Index among the three models, while Random Forest achieved the best specificity and SVM the best sensitivity.The three Youden indices differed by at most 0.07, so no systematic performance difference was detected.
- Model comparison: All three models are approximately equally suitable and not computationally intensive because they use few features.The smaller Random Forest model outperformed its large counterpart, while the SVM small model also outperformed its large model.
6.2 Cancellation Reasons
The study classifies cancellation letters into four stated reasons plus an “other reason/no reason” class using one-vs-rest models. Results indicate weak reason classification overall, with Word Search selected as the final model because it performs best in three of four reason comparisons.
- Reason classes: Cancellation reasons are grouped into “change of profession,” “financial reasons,” “retirement,” “death,” and a fifth “other reason/no reason” class.The fifth class combines cancellations with another or unspecified reasons because it does not reveal the actual reason.
- Data constraints: The analysis is constrained by scarce examples, including only eleven training and five test documents for the least common reason, “death.”The authors caution that such small counts require careful handling and should not be removed casually.
- Classification procedure: The reason-classification procedure creates four separate binary models, each estimating whether a cancellation fits one specified reason.A document can therefore be assessed against each of the four named reasons rather than classified directly into one five-way model.
- Financial reasons: Financial-reason classification performs poorly on test data: Word Search identifies two of eight cancellations, with sensitivity 0.25 and specificity 0.992.The training results are substantially stronger, with sensitivity 0.722 and specificity 0.988, indicating a large train–test gap.
- Model comparison: The best model varies by cancellation reason: Word Search leads for financial reasons and job change, SVM for retirement, while all models tie for death.The source reports that Word Search is least effective for retirement and that job-change test classifications are very poor.
- Model selection: Word Search is selected as the final reason-classification model because it performs best for three of the four reasons, although no model classifies particularly well overall.Some undetected documents were poorly read or contained no recognized words, preventing assignment to a reason.
7 Outlook
The study applies text processing and classification to customer letters, distinguishing cancellations and assigning possible reasons. The outlook emphasizes deployment with new letters while noting data-quality, sample-size, and scope limitations.
- Data and preparation: The study uses 14,575 scanned letters and archived PDF emails, requiring privacy compliance and technical processing to extract usable text.Preparation includes duplicate removal, word and bigram segmentation, stemming, and stop-word processing.
- Classification pipeline: Letters are classified first as “Cancellation” or “Non-cancellation,” after which cancellations are evaluated against four separate binary reason models.A negative result for all reason checks is treated as no stated reason.
- Results: Reason classification is uniformly less accurate than the initial cancellation classification, and the available number of reason-labeled letters is insufficient for reliable recognition.The results are exemplified with “Financial reasons,” while Word Search performs best for three of four reasons.
- Data quality: Incorrectly read words can distort the analysis, while Levenshtein-distance cleaning is not adopted because genuinely different words may differ by one character.The authors identify deeper data cleaning as a possible future contribution.
- Computation: Each method and analysis runs in under one hour on the available dataset, so computational optimization was not prioritized.Efficiency may become more relevant for substantially larger datasets.
- Deployment: The long-term plan is to apply trained models or newly selected Word Search tokens to new customer letters, using Word Search as the final model.New texts require reduction to word stems and removal of numbers; Word Search does not provide probabilities.
- Future scope: A broader analysis of complete correspondence sequences could reveal customer behavior over time and potentially improve model performance.The current study uses only letters from the last few years that were specifically requested and made available.
A.1 Additions to the Problem Statement and Methods
The appendix identifies the available customer-letter data and the Python packages used to implement the analysis.
- Data overview: Table 11 provides an overview of available customer letters and their triggering reasons based on the original classification.
- Implementation: Table 12 lists the packages used in Python for the analysis.
A.2 Additions to the Classification
The appendix reports sensitivity and specificity results for Word Search, Random Forest, and SVM across cancellation classification and four cancellation reasons. The listed results show substantial variation between models, datasets, and reasons.
- All-document classification: The large Random Forest model reaches sensitivity 0.496 and specificity 0.503 on the test data.Its training-data results are sensitivity 1 and specificity 0.913.
- All-document classification: The small Random Forest model reaches sensitivity 0.912 and specificity 0.976 on the test data.On training data, it reaches sensitivity 0.924 and specificity 0.973.
- All-document classification: The large SVM model reaches sensitivity 0.143 and specificity 0.863 on the test data.Its training-data results are sensitivity 1 and specificity 0.913.
- All-document classification: The small SVM model reaches sensitivity 0.926 and specificity 0.968 on the test data.On training data, it reaches sensitivity 0.908 and specificity 0.968.
- Cancellation classification: For cancellation classification, Word Search reaches sensitivity 0.25 and specificity 0.992 on the test data.The corresponding training-data values are sensitivity 0.722 and specificity 0.988.
- Cancellation classification: Random Forest reaches sensitivity 0.25 and specificity 0.970 on cancellation test data, while SVM reaches sensitivity 0.125 and specificity 1.
- Reason classification: The appendix includes result tables for the Job Change, Retirement, and Death cancellation reasons, with displayed examples including SVM sensitivity 0.4 and specificity 1 for Job Change.Other displayed examples include SVM sensitivity 0.6 and specificity 0.997 for Retirement.