Source-linked AI summary
In Search of Credible News
Momchil Hardalov, Ivan Koychev, Preslav Nakov
TL;DR
Online misinformation makes it difficult to distinguish real news from fake news, and research has largely neglected languages beyond English. The paper builds Bulgarian news datasets and applies a language-independent, feature-rich classifier; across three test datasets, accuracy is 75–99%, above majority-class baselines.
Problem
Online misinformation makes it increasingly difficult to distinguish real news from fake news, while credibility research has largely focused on English rather than other languages.
Source: §1 · pp. 1–2Method
Results
Across all three test datasets, accuracy was 75–99%, well above the majority-class baseline.
Source: §4.3 · p. 8Takeaways & Limitations
Strong relative performance across three test datasets from different sources suggests the model learns to distinguish credible from fake news rather than merely classifying topics, sources, or author style.
Source: §4.3 · p. 8The datasets and evaluation use four Bulgarian news sources, with one training set and three separate balanced test sets.
Source: §4.1 · p. 5
Abstract
from arXiv · showhide
We study the problem of finding fake online news. This is an important problem as news of questionable credibility have recently been proliferating in social media at an alarming scale. As this is an understudied problem, especially for languages other than English, we first collect and release to the research community three new balanced credible vs. fake news datasets derived from four online sources. We then propose a language-independent approach for automatically distinguishing credible from fake news, based on a rich feature set. In particular, we use linguistic (n-gram), credibility-related (capitalization, punctuation, pronoun use, sentiment polarity), and semantic (embeddings and DBPedia data) features. Our experiments on three different testsets show that our model can distinguish credible from fake news with very high accuracy.
1 Introduction
Online platforms have made it harder to distinguish real news from misinformation, while research has focused largely on English. The paper creates a Bulgarian news dataset and reports high accuracy distinguishing credible from fake news.
1 Introduction
- Online information spreads through social media and news sites, where unreliable, unverified, and manipulative content makes real news harder to distinguish.The problem can extend beyond cyberspace when fake online news enters mainstream media.Source: §1 · pp. 1–2
- Most credibility and veracity research has focused on English, leaving other languages largely neglected; the paper studies Bulgarian with a language-independent approach.The task includes fake news designed to sound funny, so it can also be viewed as humor detection.Source: §1 · p. 2
- With no suitable publicly available dataset, the authors collected credible and fake news and report that their model achieved very high accuracy distinguishing them.Credible items came from trusted online sources; fake items were written to amuse or sometimes confuse readers lacking subject knowledge.Source: §1 · p. 2
2 Related Work
Related work examines credibility and rumors on social media, personal blogs, online news portals, and fake-news detection methods. The paper also distinguishes its news-focused task from prior humor-detection work.
2 Related Work
- Castillo & al. study false information about newsworthy events in tweets using reputation, writing-style, and time-based features.Zubiaga & al. report that trusted users can spread rumors without raising suspicions about the news or its source [19].Source: §2 · p. 2
- Johnson & al. examine how blog users act during newsworthy events and how biased users try to influence others.Researchers disagree about how much people trust personal blogs.Source: §2 · p. 2
- Online news portals also face credibility concerns, including pressure to publish breaking news before facts and sources can be double-checked..Source: §2 · p. 2
- A review compares fake-news detection approaches using linguistic analysis, discourse, linked data, and social-network features.Unlike prior humor work mixing genres, this paper uses news for both credible and fake examples and does not assume humor explains the latter.Source: §2 · p. 3
3 Method
The method combines linguistic, credibility-related, and semantic features in a language-independent approach to distinguish credible from fake news. It preprocesses texts, selects features, and classifies them with regularized logistic regression.
3 Method
- The approach combines linguistic n-grams, credibility features, and semantic embeddings and DBPedia data to distinguish credible from fake news.The method is presented as language-independent.Source: §3 · p. 3
Linguistic (n-gram) Features
- Tokenization and stop-word removal precede linguistic feature generation, which uses unigram and bigram presence, tf-idf weighting, and vocabulary richness.Stop words are described as common functional words considered not particularly useful for this task.Source: Linguistic (n-gram) Features · p. 3
Credibility Features
- Credibility features include article length, capitalization, pronoun fractions, URLs, punctuation, hashtags, quotation marks, and sentiment polarity.Sentiment features use positive and negative word proportions and summed sentiment scores; only a selected subset of the listed features was ultimately used.Source: Credibility Features · pp. 3–4
3.2 Classification
- Document semantics are represented by averaging word2vec vectors trained on long Bulgarian DBPedia abstracts across non-stop-word tokens.Logistic regression with L-BFGS and elastic-net regularization handles the overlapping feature set; hyperparameters are tuned on training data.Source: Semantic (Embedding and DBPedia)… · p. 4, §3.2 · p. 4
4 Experiments and Evaluation
The experiments use four Bulgarian news sources to construct balanced test sets and evaluate a language-independent feature-based classifier. Accuracy is 75–99% across the test sets, with performance varying by feature combination and source.
4.1 Data
- Four Bulgarian sources supplied diverse credible and fake news, and the authors designed the data to reduce topic, author, and source confounds.Dnevnik and Ne!Novinite formed the train/test split; bTV versus bTV Duplex and Dnevnik versus Bazikileaks were additional, near-perfectly balanced test sets.Source: §4.1 · pp. 4–5
4.2 Feature Selection
- LVQ ranked credibility features on training data, after which feature combinations were selected by cross-validation; Table 1 presents the ranking.Source: §4.2 · pp. 5–6, Table 1 · p. 6
- The selected credibility features included negative-word fraction, all-uppercase and initial-uppercase word fractions, and lowercase-only word fraction.Source: §4.2 · p. 6
- The selected set also included plural-pronoun fraction, exclamation-mark counts, and double-quote counts.Source: §4.2 · p. 6
4.3 Results
- Semantic-containing combinations performed best overall; semantic features led on Dnevnik–Ne!Novinite but transferred less well to other sets, while linguistic features underperformed on bTV.On bTV, “Credibility + Semantic” was the best-performing combination.Source: Table 2 · p. 7, §4.3 · pp. 7–8
- 75–99% accuracy across three datasets exceeded majority-class baselines; Dnevnik–Ne!Novinite reached 99.36%, Dnevnik–Bazikileaks 85.53%, and bTV 75.91%.The bTV test set was hardest because both classes came from sources absent from training; Dnevnik–Bazikileaks retained credible examples from a training source.Source: §4.3 · p. 8
5 Conclusion and Future Work
The paper presents a language-independent, feature-rich approach for distinguishing credible from fake news, with high accuracy across three test sets from four sources. It also outlines feature and modeling directions for future work and reports public release of the code and datasets.
5 Conclusion and Future Work
- The approach combines linguistic n-grams, credibility-related cues, and semantic embeddings and DBPedia data; experiments across three test sets and four sources achieved accuracy well above a majority-class baseline.Source: §5 · p. 8
- Future work includes linked-data and discourse features, features from humor- and rumor-related tasks, and deep learning to eliminate feature engineering.Source: §5 · p. 8
- The authors made their source code and datasets publicly available for research purposes.Source: §5 · p. 8