Source-linked AI summary
Data Science Approaches to Evaluating Honours Candidates
Francesca von Braun-Bates, Sunreeta Sen, Indraayudh Talukdar, Anirban Lahiri
TL;DR
The paper addresses how public sentiment toward individuals can be estimated from fragmented OSINT for transparent Honours decision support. It develops a modular pipeline combining web collection, filtering, co-reference resolution, and sentiment analysis, and finds that MINOS most clearly separates positive, ambiguous, and negative reputational cases. The approach supports scalable, auditable assessment while retaining human decision-making.
Problem
The Honours system needs fair and transparent ways to assess public conduct from fragmented online information, because recipients must maintain high standards.
Method
The paper chains automated web search, text processing, relevance filtering, co-reference resolution, and AFINN, VADER, and MINOS sentiment analysis into an auditable pipeline.
Results
MINOS gives the clearest separation between positive, ambiguous, and negative cases among public figures with known reputational outcomes.
Takeaways & Limitations
The workflow can provide committees with a thorough, impartial, reproducible intelligence picture while supporting scalable and human-in-the-loop Honours assessment.
Takeaways & Limitations
Internet data accessibility limits coverage, and online articles may introduce bias, so final award decisions require human review.
Abstract
from arXiv · showhide
We present a modular data-science pipeline for estimating public sentiment towards individuals from fragmented, unstructured open-source intelligence (OSINT). The method chains web search, text extraction, relevance filtering, tokenisation, co-reference resolution, and sentiment analysis to convert heterogeneous web material into auditable person-level sentiment distributions. We compare AFINN and VADER with MINOS, a domain-informed sentiment algorithm designed to detect language associated with reputational risk, misconduct, and positive public contribution. Applied to public figures with known reputational outcomes, MINOS gives the clearest separation between positive, ambiguous, and negative cases. The results show that chained NLP and OSINT methods can support transparent, reproducible, human-in-the-loop sentiment assessment for high-stakes decision support. We demonstrate the approach on the UK Honours system, where individuals are required to display high standards of public conduct to maintain an Honour.
1. Introduction
The paper applies data science to make UK Honours assessment more thorough and transparent by combining NLP techniques across broad OSINT sources. Its novelty lies in integrating established methods for this high-stakes, whole-problem context and introducing MINOS alongside existing sentiment algorithms.
- The paper presents the first known application of data science to the UK Honours system.
- Combining NLP techniques creates an open-source intelligence picture for measuring public opinion of Honours recipients.
- The pipeline uses broad sources, co-reference resolution, multiple sentiment algorithms, and subjects varying in field and celebrity status.
- MINOS is introduced as a novel algorithm for identifying positive and negative personality traits alongside AFINN and VADER.
- The individual techniques are established; the novelty is their use together to solve a complete problem in a challenging context.
- The paper covers Honours-system background, subject definition, methodology, sentiment results, and future challenges.
2. The Honours System
The UK Honours system recognises outstanding public contributions but requires recipients to maintain high standards of conduct. Forfeiture provides the governance context in which robust, timely evidence about potential misconduct is needed.
- Honours recognise outstanding contributions across charity, education, science, arts, business, and other areas of UK public life.
- Recipients must maintain high standards of conduct after receiving an Honour or risk forfeiture, including possible posthumous revocation.
- Nominations pass through public or Civil Service submission, sift committees, vetting, and final royal approval.
- Vetting checks factual accuracy, criminal records, and probity with professional bodies and government departments.
- The independent Forfeiture Committee lacks investigative powers and relies on evidence from official investigations and court proceedings.
- Automated data-driven methods could support proactive misconduct detection and timely interventions in this high-risk environment.
3. Selecting Subjects of Interest
The evaluation tests the method on three groups with contrasting reputational outcomes: infamous, awarded, and forfeited subjects. The expected pattern is positive sentiment for awarded people, near-zero sentiment for forfeited people, and negative sentiment for infamous people.
- The study uses two control groups alongside the forfeiture group to test whether the approach separates contrasting reputational outcomes.
- The infamous group contains people whose behaviour contrasts directly with enhancing public good and whose crimes are publicly documented.
- The awarded group comprises Honour recipients who have retained their awards, suggesting continued behaviour supporting the public good.
- The forfeited group comprises people who forfeited an Honour or were posthumously stripped of it by the Committee.
- Twenty subjects were selected per group, with awarded and forfeited subjects drawn from London Gazette announcements.
- A successful method is expected to produce highly positive awarded scores, near-zero forfeited scores, and highly negative infamous scores.
4. Methodology
The methodology builds an auditable pipeline from automated web search and relevance filtering through co-reference resolution and sentiment analysis. It produces person-level sentiment summaries and evidence for human evaluation, while retaining algorithm-specific comparability limits.
- 4.1. Data Collection, Processing and Filtering.: The pipeline automates Google searches, extracts plain text, and preserves source URLs to support traceability and auditability.
- 4.1. Data Collection, Processing and Filtering.: Articles shorter than 500 characters or lacking the subject’s best-known full name are discarded to reduce irrelevant material.
- 4.1. Data Collection, Processing and Filtering.: The resulting dataset provides a focused base for sentiment analysis, reducing noise and increasing the accuracy of later analytical steps.
- 4.2. Co-reference Resolution.: Co-reference resolution links pronouns and abbreviated references to candidates, improving attribution of relevant content.
- 4.3. Sentiment Analysis.: AFINN assigns word-level scores from -5 to +5, whereas VADER handles intensifiers and idioms but is challenged by formal content.
- 4.3. Sentiment Analysis.: MINOS uses expert-augmented positive and negative lexicons with strong emphasis on misconduct and criminality.
- 4.4. Posterior Marginalisation.: Sentiment distributions are integrated into a centroid representing average sentiment for each subject of interest.
- 4.4. Posterior Marginalisation.: Centroids can be compared within an algorithm but not across algorithms because their priors differ and may be unbounded.
5. Results, Analysis and Discussion
The results show that relevance filtering and co-reference-resolved sentiment analysis distinguish awarded, forfeited, and infamous groups, while posterior shapes reveal information lost by centroid summaries. MINOS most consistently matches the expected sentiment patterns and provides the clearest group separation.
- Filtering: Article file sizes span seven orders of magnitude, with the most meaningful content usually falling between 1 kB and 1 MB.Short files were largely spurious browser messages, while files up to 10^7 characters were dominated by unrelated noise.
- Centroid comparisons: After co-reference resolution, the analysis retained only sentences explicitly referencing each individual and compared centroid distributions across awarded, forfeited, and infamous groups.Figure 3 uses box plots whose boxes show inter-quartile ranges and whose colours identify infamous, forfeited, and awarded groups.
- Centroid comparisons: Across algorithms, awarded individuals had clearly positive centroids, infamous individuals clearly negative centroids, and forfeited individuals spanned zero.Outliers made individual assignment from centroids difficult, and only MINOS separated the infamous inter-quartile range from the other two groups without overlap.
- Selected posteriors: For Kumar Bhattacharyya, all sentiment algorithms produced positive centroids, but only MINOS showed a consistently positive posterior with few negative sentences.The other three algorithms exhibited long negative tails that were not expected for an awarded subject.
- Selected posteriors: For infamous individuals, centroids were negative across algorithms, although VADER compound also produced extremely positive results exceeding 80% positive.Three distributions had longer negative than positive tails and desirable properties throughout the posterior as well as in the centroid.
- Selected posteriors: For forfeited individuals, the expected mixture of positive and negative sentiment was obscured by algorithm-specific distributions, including VADER’s near-zero cancellation and positive skew.VADER compound captured positive actions but failed to adequately convey the publicised offences associated with the forfeiture.
- Overall comparison: Across subject groups, MINOS was most consistent with prior knowledge and was identified as the most promising algorithm for wider application.The authors also note that centroid summaries can integrate over important detail in posterior distributions.
6. Conclusions and Future Work
The paper combines several NLP techniques to create a scalable intelligence picture for human assessment, while recognising data-access limitations and potential internet-source bias. It concludes that automated analysis can improve Honours-system efficiency and transparency without replacing human decision-making.
- Method: Web-scraping, tokenisation, co-reference resolution, and sentiment analysis are combined to maximise information available to a human sifter.The pipeline collects attributable open-source information, divides long articles into sense units, identifies relevant references to subjects of interest, and produces posterior sentiment distributions.
- Method: Bayesian inference creates a summary score for each subject of interest while making prior assumptions explicit.The score summarises the available evidence rather than concealing the assumptions used in inference.
- Future Work: The methodology is presented as a robust, scalable solution despite inherent limitations regarding data accessibility.Future work includes improving access to news and subscription-based platforms and refining the MINOS model and vocabulary.
- Conclusions: Parallel automated analysis can scale to approximately 200 000 living Honours recipients, unlike the current manual vetting approach.The authors connect this scale with the possibility of dynamically assessing forfeiture risk as recipients’ information picture changes.
- Conclusions: The approach is intended to provide committees with an impartial, thorough intelligence picture rather than replace human decision-making.The authors identify human involvement as essential because internet-based articles may introduce bias into results.