SCOPE: Source Credibility and Outlet Profiling Engine

1 Introduction

• Readers, researchers, journalists, and policy analysts face a complex, fragmented information landscape when evaluating news sources at scale.The challenge includes assessing reliability, political leaning, and broader influence.Source: §1 · p. 1
• Real-world decisions may require understanding an outlet’s topical focus, ownership, citation practices, audience positioning, and past behavior.This outlet-level context complements work focused on individual claims, misleading articles, and manipulation techniques.Source: §1 · p. 1
• Outlet-level patterns provide useful context when claims are new, ambiguous, or difficult to verify directly.Relevant evidence is distributed across outlet articles, fact-checking platforms, social accounts, public records, and independent reports.Source: §1 · p. 1
• A single model may struggle to gather and assess distributed evidence, reconcile conflicting signals, and produce an inspectable profile.Media profiling involves retrieval, extraction, comparison, reasoning, and explanation, not classification alone.Source: §1 · p. 1
• SCOPE distributes media profiling among specialized agents for acquisition, content analysis, evidence search, and report generation.The coordinated agents combine complementary evidence streams into structured profiles.Source: §1 · p. 1
• The system aims to improve coverage, reduce the burden on any single model, and expose intermediate evidence and agent outputs.These design aims make the profiling process more transparent.Source: §1 · p. 1

2.1 Methodology

• MBFC reviews at least ten headlines and five full articles, considering language, framing, sourcing, balance, verified information, and available ownership and transparency details.SCOPE follows this methodology as a systematic basis for outlet assessments.Source: §2.1 · p. 2
• MBFC-style bias combines economic ideology, social values, straight-news balance, and editorial bias, while factual reliability combines failed checks, sourcing, transparency, and one-sidedness or propaganda.Failed fact checks receive the greatest factuality weight; lower factuality scores indicate higher reliability.Source: §2.1 · p. 2

2.2 System and Components

• SCOPE coordinates acquisition, nine content-analysis agents, evidence search, and report generation to approximate MBFC-style human review.The stages scrape articles, assess MBFC dimensions, retrieve outlet information and failed fact-check evidence, then synthesize an MBFC-style profile.Source: §2.2 · p. 2
• The fixed-order orchestrator checks evidence sufficiency and computes bias, factuality, and credibility scores from validated, schema-constrained agent outputs.Separating evidence collection, analysis, scoring, and reporting makes intermediate decisions inspectable and profiles auditable.Source: §2.2 · p. 2
• Economic and social ideology are scored separately from −10 to +10, with article relevance checked before assigning scores on either axis.The method also checks for loaded language and manipulation techniques; its two-axis framing treats ideology as more than a single left–right continuum.Source: §2.2 · p. 3
• SCOPE supplements scraped articles with web-search evidence about outlet ownership, funding, editorial history, and third-party fact-checks.It excludes media-rating aggregators to prevent gold-label leakage and uses external reach and press-freedom sources in credibility adjustment.Source: §2.2 · p. 3

2.3 Demonstration interface

• The web application’s Overview, Evidence, Analyzer Trace, and Compare tabs present verdicts, audit evidence, agent outputs, and side-by-side outlet profiles.The Evidence tab includes article quotes, analyzer findings, propaganda spans with confidence scores, and fact-check results.Source: §2.3 · p. 3

3 Experiments and Results

• The comparisons include LLM-only, 15-article, and web-search LLM baselines, plus MBC, adapted to use the experiments’ search tool and LLM answers.Source: §3 · p. 4

3.1 Data

• The 200 outlets were sampled round-robin across seven bias categories and five factuality levels; MBFC labels are gold annotations and its free-text fields are report references.Source: §3.1 · p. 4

3.2 Setup and metrics

• Evaluation uses ordinal MAE for bias and factuality, FACTScore precision, recall, and error rate, failed-check detection, METEOR, and ROUGE-L.Only failed checks of content published by the outlet count, not checks about the outlet itself.Source: §3.2 · p. 4
• Figure 3’s BBC profile displays an overview verdict alongside failed fact-check signals, analyzed-article phrases, and bias-component scores.Source: Figure 3 · p. 5

3.3 Main results

• 0.65 vs. 1.17 for LLM-only is SCOPE’s lowest bias MAE; it also records factuality MAE 0.67, perfect fact-check detection (1.00), and error rate 0.52.SCOPE also has the best ROUGE-L (0.16) and highest FACTScore precision (0.48).Source: §3.3 · p. 5
• Search raises fact recall (0.36) and METEOR (0.26) but has the highest error rate (0.62); offline articles improve bias MAE (0.89 vs. 1.17) while slightly hurting factuality.Source: §3.3 · p. 5
• On factuality, 42% of predictions are exact and 90% are within one ordinal class; on bias, 50% are exact and 90% are within one class.Figure 5 shows these ordinal-axis results.Source: §3.3 · p. 5
• $0.054 per outlet is the full pipeline cost, comparable to LLM-only ($0.050) and roughly half the cost of Search ($0.101).Source: §3.3 · p. 5

4 Conclusion

• SCOPE turns an outlet URL into an evidence-grounded profile of political bias, factual reliability, and credibility, with visible structured agent outputs tracing verdicts to evidence.Source: §4 · p. 5
• 90% of predictions fall within one ordinal class on both bias and factuality; SCOPE also has the lowest MAEs and perfect failed-fact-check detection among evaluated baselines.Source: §4 · p. 5
• Evidence remains useful even when the final label is wrong, especially for conspiracy and pseudoscience outlets that do not fit a left–right scale.Source: §4 · p. 5
• The full SCOPE system wins six of eight ablation metrics, including both ordinal MAEs, failed-fact-check detection, and error rate; Search leads on fact recall and METEOR.MBC has no Bias or Factuality evaluation because it provides descriptions rather than direct labels.Source: Figure 4 · p. 6
• Future work aims to extend beyond English-language and U.S.-centric outlets and apply the agent scaffolding to adjacent entities.Examples named are politicians, pollsters, and government bodies.Source: §4 · p. 6

Limitations

• The evaluation covers U.S.-centric, English-language outlets, so results may not fully generalize to non-Western, non-English, or low-visibility outlets.Labeled fact-checking and media-profiling data are scarce for other languages and regions; SCOPE itself is not limited to English, but its multilingual performance was not evaluated.Source: Limitations · p. 6
• Indirect leakage remains possible because other websites may reproduce or cite ratings from excluded media-rating platforms.The authors manually inspected retrieved search traces and found no evidence of leakage, but cannot rule it out completely.Source: Limitations · p. 6
• The scraper removes hyperlink targets, so the sourcing analyzer may miss citations hidden behind displayed anchor text.Manual testing sometimes found the system reporting missing evidence even when an article linked to a relevant source.Source: Limitations · p. 6
• Conspiracy and pseudoscience outlets can be hard to place on a left–right scale, and SCOPE may assign the wrong political direction to ideologically mixed or anti-establishment content.SCOPE usually captures these outlets’ low factuality and credibility.Source: Limitations · p. 6

Ethics and Broader Impact

• Outlet scores can be misused to dismiss or amplify sources.Source: Ethics and Broader Impact · p. 6
• Each score is backed by inspectable intermediate evidence, and withheld when the evidence-sufficiency gate is not met.Source: Ethics and Broader Impact · p. 6
• SCOPE is intended as an auditable assistive tool for researchers, journalists, and analysts—not as an arbiter of truth.Source: Ethics and Broader Impact · p. 6

A Model Selection

• Five candidate models were evaluated on the same approximately 50 outlets with a fixed token cap, so quality-per-dollar differences reflect model pricing rather than usage variation.Source: A Model Selection · p. 9
• Quality-per-dollar is the quality score divided by cost, with quality defined as an equal-weight, min–max-normalised composite of task metrics.The table caption specifies that the composite covers five task metrics.Source: A Model Selection · p. 9, Table 1 · p. 9
• 12.1 quality-per-dollar makes GPT-5 mini Pareto-optimal; its score is about 3× higher than premium models such as GPT-5 and GPT-5.2.This comparison is based on the LLM-only setting.Source: A Model Selection · p. 9, Table 1 · p. 9

B Ordinal Accuracy

• Figure 5 provides the per-class breakdown behind the aggregate within-one-class figures reported in Section 3.3.Source: B Ordinal Accuracy · p. 9, Figure 5 · p. 9
• Figure 5 shows the shares of outlets predicted exactly, off by one class, or at least two classes from the gold label, separately for bias and factuality.Bias is shown on the left and factuality on the right.Source: Figure 5 · p. 9
• Roughly 90% of SCOPE predictions fall within one class for both factuality and bias.Source: B Ordinal Accuracy · p. 9, Figure 5 · p. 9

C Failed Fact-Check Detection Example

• Figure 6 shows SCOPE’s “Signals at a Glance” panel for NaturalNews.com.Source: C Failed Fact-Check Detection Example · p. 9, Figure 6 · p. 9
• The NaturalNews.com panel surfaces four failed fact-checks published by LeadStories and Snopes.Source: Figure 6 · p. 9

• The bank spans 17 policy domains—7 economic and 10 social—with each domain offering four left/progressive, one centrist, and four right/conservative positions.Source: D The Ideology Item Bank · p. 9, Table 2 · p. 10
• Each item describes an observable editorial stance mapped to a calibrated point on the corresponding −10 to +10 axis and supported by a primary or scholarly source in the released bank.Source: D The Ideology Item Bank · p. 9, Table 2 · p. 10
• The scoring prompt embeds both axis scales and domain summaries as fixed reference points before assigning each outlet’s economic and social scores in one step.Source: D The Ideology Item Bank · p. 9
• The 17 domain summaries are distilled into one scoring prompt rather than executed individually for each outlet.Source: D The Ideology Item Bank · p. 9, Table 2 · p. 10