Source-linked AI summary
Distinguishing articles in questionable and non-questionable journals using quantitative indicators associated with quality
Dimity Stephen
TL;DR
Evidence about the quality of articles in questionable journals remains limited, motivating comparisons with non-questionable journals. The study compares quantitative indicators across psychology articles from questionable, WoS-indexed, and mid-tier journals. QJ articles diverge from peer-reviewed disciplinary standards on several indicators, while concerns also appear in non-QJ journals.
Problem
Evidence about QJ article quality remains insufficient to fully justify existing concerns, and manual review is resource-intensive.
Source: §1 · p. 1Method
The study compares quantitative quality indicators across psychology articles from QJs, WoS-indexed journals, and mid-tier non-QJ journals.
Source: §3.1 · pp. 5–6Results
QJ articles diverge from peer-reviewed psychology standards on indicators associated with editorial and review processes, while mid-tier and WoS journals also show ethics-reporting and statistical-interpretation concerns.
Source: §5.2 · p. 14Takeaways & Limitations
The findings are an initial examination, and further qualitative work and attention to journal classification and publishing pressures are needed for a comprehensive understanding of QJ article quality.
Source: §5.2 · p. 14Journal-policy differences were not accounted for, so their effects on the results could not be estimated.
Source: §5.1 · p. 13
Abstract
from arXiv · showhide
This study investigates the viability of distinguishing articles in questionable journals (QJs) from those in non-QJs on the basis of quantitative indicators typically associated with quality. Subsequently, I examine what can be deduced about the quality of articles in QJs based on the differences observed. I contrast the length of abstracts and full-texts, prevalence of spelling errors, text readability, number of references and citations, the size and internationality of the author team, the documentation of ethics and informed consent statements, and the presence erroneous decisions based on statistical errors in 1,714 articles from 31 QJs, 1,691 articles from 16 journals indexed in Web of Science (WoS), and 1,900 articles from 45 mid-tier journals, all in the field of psychology. The results suggest that QJ articles do diverge from the disciplinary standards set by peer-reviewed journals in psychology on quantitative indicators of quality that tend to reflect the effect of peer review and editorial processes. However, mid-tier and WoS journals are also affected by potential quality concerns, such as under-reporting of ethics and informed consent processes and the presence of errors in interpreting statistics. Further research is required to develop a comprehensive understanding of the quality of articles in QJs.
1. Introduction
Evidence about article quality in questionable journals (QJs) remains limited, despite substantial concern about their potential effects. This study compares quantitative indicators across QJs and two non-QJ journal groups to assess whether their articles differ and what those differences suggest about quality.
1. Introduction
- Research on QJ article quality remains limited, leaving concerns insufficiently justified and motivating comparisons with peer-reviewed journals.The author notes that empirical studies of QJ practices, content, and quality make up only 19% of the literature on the topic.Source: §1 · p. 1
1.1. Defining and measuring article quality
- Research quality includes originality, plausibility, and value, while article-level indicators span theory, methodology, communication, impact, and collaboration.The study uses quantitative indicators as proxies, although citations capture only one aspect of article quality.Source: §1.1 · p. 2
- Communication characteristics and collaboration have ambiguous links to quality, with associations varying across indicators and disciplines.Readability may reflect poor writing or complex ideas, while author counts show different associations with peer-assessed quality across fields.Source: §1.1 · p. 3
1.2. Previous studies of the quality of articles in QJs
- Prior comparisons generally found poorer reporting or quality ratings in QJs, but small samples and mixed findings leave substantial questions unresolved.Examples include lower reporting quality in QJ studies and ratings ranging from poor to excellent among articles in questionable nursing journals.Source: §1.2 · pp. 4–5
1.3. Aims of the current study
- The study compares psychology articles from QJs, WoS-indexed journals, and mid-tier non-QJ journals using nine measurable indicators.The indicators cover references, text length and readability, spelling, citations, author teams, ethics and consent documentation, and statistical errors.Source: §1.3 · p. 5
3. Methods
The study assembles English-language psychology articles from QJs, WoS journals, and mid-tier journals, then extracts and compares quantitative indicators. Its procedures include sample screening, text cleaning, statistical group comparisons, and indicator-specific measurements.
3.1. Sample selection and collection
- Figure 1 summarizes the data sources and inclusion criteria for the comparison of QJ, WoS, and mid-tier psychology journals.The samples were restricted to articles from 2010–2020 with available full texts, with non-QJ samples limited to open-access journals.Source: §3 · p. 5, §3.1 · pp. 5–6, Figure 1 · p. 6
3.2. Data extraction and cleaning
- Articles were screened for research content and English-language titles, then PDFs were converted to XML for extracting text and references.Titles and abstracts were manually checked and corrected, while full texts received no additional cleaning because of resource constraints.Source: §3.1 · p. 6, §3.2 · p. 7
3.3. Analysis of quantitative indicators
- One-way ANOVAs compared groups, followed by Tukey’s HSD pairwise tests when differences were significant.The analyses were conducted in R.Source: §3.3 · p. 7
3.3.1. Length of abstract and full-text
- Abstract and full-text length was measured by word count, with reference lists excluded from full-text counts.End matter, including acknowledgements and ethics or funding statements, was included.Source: §3.3.1 · p. 7
3.3.2. Percentage of articles with misspellings
- Misspellings were identified in titles and abstracts, with scientific terms excluded and PDF-conversion errors corrected in those sections.The study compared group proportions using chi-square tests and pairwise proportion comparisons where differences emerged.Source: §3.3.2 · p. 7
3.3.3. Readability
- Readability was measured with the Flesch Reading Ease score, which uses syllable, word, and sentence counts.The passage describes scores of 60–70 as corresponding to a US eighth-grade reading level; academic articles typically score below 50.Source: §3.3.3 · p. 7
3.3.5. Size and internationality of authorship teams
The study compares authorship-team size and internationality, ethics and consent documentation, and statistically consequential p-value misreporting across journal groups using quantitative tests. It also notes that the ethics and p-value searches may underestimate the prevalence of the documented practices and errors.
3.3.5. Size and internationality of authorship teams
- Author counts and the number of countries represented per article were compared across groups using Kruskal-Wallis tests followed by Wilcox pairwise tests.The number of authors was identified using Semantic Scholar author IDs, and author countries were extracted with JATSdecoder.Source: §3.3.5 · p. 8
3.3.6. Ethics approval and informed consent
- Ethics and informed-consent mentions were identified by searching full texts for participant terms combined with ethics- or consent-related terms, then comparing group proportions with chi-square tests.The search may miss nonstandard terminology or misidentify references to participants in other studies, so mention rates may be underestimated.Source: §3.3.6 · p. 8
3.3.7. Incorrectly reported p-values
- Statcheck recalculated APA-format p-values and flagged cases where misreporting changed the statistical-significance decision; the analysis counted articles with at least one such error among those reporting an APA test statistic.Non-APA formatting, incomplete reporting, and PDF-to-text conversion may prevent assessment, so error prevalence likely is underestimated.Source: §3.3.7 · pp. 8–9
4. Results
Across psychology journal groups, QJ articles were generally shorter and had fewer references and citations, while ethics and consent reporting also differed. Other differences were limited: readability and spelling-error rates were similar in practical terms, and statistical decision errors did not differ significantly.
4. Results
- QJ abstracts averaged 186 words and full texts 4,229, shorter than WoS (196; 5,434) and mid-tier (201; 5,143) articles.WoS full texts were also longer than mid-tier full texts.Source: §4 · p. 9, Figure 2 · p. 10
- Full-text readability differed statistically, but median FRE scores across groups ranged only from 42.7 to 43.9; QJ abstracts had more spelling errors, though all groups were at 2.2% or less.Title spelling-error proportions were comparable across groups.Source: §4 · p. 9, Figure 2 · p. 10
- QJ articles averaged 34.6 references, versus 46.3 in mid-tier and 50.3 in WoS articles; they also received fewer citations at three years (0.8) and overall (3.3) than mid-tier articles (2.4; 12.4).WoS articles received the most citations: 3.8 at three years and 16.7 overall.Source: §4 · p. 9, Figure 3 · p. 11
- WoS articles had the highest self-reference share (6.8%), followed by mid-tier (5.6%) and QJ articles (4.2%); self-citation shares were 17.6%, 16.2%, and 15.9%, respectively.Source: §4 · pp. 9–10, Figure 3 · p. 11
- QJ articles averaged 2.7 authors, fewer than mid-tier (3.3) and WoS (3.8); their mean country count was 1.1 versus 1.3 in both comparison groups, a limited practical difference.Source: §4 · p. 10, Figure 4 · p. 11
- Ethics was mentioned in 25.7% of QJ articles with participants, versus 34.5% of mid-tier and 46.9% of WoS articles; consent was reported in 44.9%, 48.4%, and 61.2%, respectively.The QJ–mid-tier consent difference was not significant; within each group, consent was mentioned more often than ethics.Source: §4 · p. 12, Figure 5 · p. 12
- Misreported p-values produced significance-decision errors in 12.7% of assessable QJ articles, 10.5% of WoS, and 7.9% of mid-tier articles, with no significant group difference.Fewer than one-third of articles in each group reported an assessable statistical test.Source: §4 · p. 12, Figure 5 · p. 12
5. Discussion
The discussion assesses whether quantitative indicators distinguish psychology articles in questionable journals (QJs) from those in mid-tier and Web of Science (WoS) journals, and what those differences imply about article quality. It concludes that QJ articles diverge on standards-related indicators, while the measures cannot provide a comprehensive account of quality.
5. Discussion
- Across each group, a similar percentage of articles reporting statistical tests made erroneous decisions due to misreported p-values; in WoS journals, self-citations comprised nearly 20% of all citations.Source: §5 · p. 13
5.1. Limitations
- PDF text extraction may underestimate ethics and consent mentions and p-value errors, and affect readability and full-text length estimates, but the same process leaves group comparisons suitable.The full-text corpus could not be cleaned for OCR and formatting errors.Source: §5.1 · p. 13
- Unaccounted-for journal policies may affect abstract and full-text lengths, statistical reporting formats, and ethics and consent statements, so their effects cannot be estimated.WoS abstracts showed a sharp cut-off at approximately 400 words.Source: §5.1 · p. 13
- Quantitative indicators cannot replace qualitative review of method suitability, conclusion validity, or originality, although automation enabled objective analysis of several indicators in a large sample.Source: §5.1 · p. 14
5. Discussion
- QJ articles diverged from psychology peer-reviewed standards: they were shorter, cited less literature, received fewer citations, involved fewer authors, and less often reported ethics and consent.The conclusion describes these indicators as suggesting lower plausibility for QJ articles, while noting lower visibility may also explain fewer citations.Source: §5 · p. 13, §5.2 · p. 14
5.2. Conclusions
- The study is an initial feasibility examination; further qualitative research and work on journal classification, geopolitical influences, and research-evaluation effects on publishing practices are needed.The authors call for a comprehensive understanding of article quality in QJs.Source: §5.2 · p. 14