PaperlayerView exact revision on arXiv

Source-linked AI summary

Clustering scientific publications based on citation relations: A systematic comparison of different methods

Lovro Šubelj, Nees Jan van Eck, Ludo Waltman

arXiv:1512.09023v2cs.DLcs.SIphysics.data-an

TL;DR

The paper asks which graph-partitioning and community-detection methods perform best for clustering publications through citation relations. It systematically compares methods using statistical analyses and expert assessment, finding support for Infomap and related methods while noting costs and trade-offs.

  • Problem

    Which graph-partitioning and community-detection methods perform best is unclear, and the answer may depend on network type and the partition desired.

    Source: Introduction · p. 1

  • Method

    The study compares many clustering methods on publication citation networks using statistical properties and an expert assessment of scientometrics results.

    Source: Introduction · p. 2, Discussion · p. 18

  • Results

    Infomap and related Metimap receive the strongest support; Infomap performs best in the expert assessment and yields quite robust results.

    Source: Discussion · p. 19

  • Takeaways & Limitations

    Infomap is supported for citation-network clustering, while Metimap can address its relatively high computing time on larger networks.

    Source: Discussion · p. 19

    Methods were run with default parameters rather than optimized settings, so some methods may have performed better with tuning.

    Source: Discussion · p. 20

Abstract

from arXiv · show

Clustering methods are applied regularly in the bibliometric literature to identify research areas or scientific fields. These methods are for instance used to group publications into clusters based on their relations in a citation network. In the network science literature, many clustering methods, often referred to as graph partitioning or community detection techniques, have been developed. Focusing on the problem of clustering the publications in a citation network, we present a systematic comparison of the performance of a large number of these clustering methods. Using a number of different citation networks, some of them relatively small and others very large, we extensively study the statistical properties of the results provided by different methods. In addition, we also carry out an expert-based assessment of the results produced by different methods. The expert-based assessment focuses on publications in the field of scientometrics. Our findings seem to indicate that there is a trade-off between different properties that may be considered desirable for a good clustering of publications. Overall, map equation methods appear to perform best in our analysis, suggesting that these methods deserve more attention from the bibliometric community.

Introduction

The introduction asks which graph-partitioning and community-detection methods work best, noting that the answer may depend on network type and the desired partition. It motivates a systematic comparison for publication citation networks using both statistical properties and expert assessment.

Introduction

  • The best-performing graph-partitioning method is unclear, and performance may depend on network type and the partition sought.The paper addresses this question specifically for clustering publications by citation relations.Source: Introduction · p. 1
  • Direct citation relations enable efficient clustering of large publication sets, including millions of publications.Prior work cited in the passage clustered ten million publications using about one hundred million citation relations.Source: Introduction · p. 2
  • The study systematically compares spectral, modularity, map equation, matrix factorization, statistical, link-clustering, label-propagation, random-walk, clique-percolation, and expansion methods.The comparison covers methods proposed in graph partitioning and community detection.Source: Introduction · p. 2
  • The evaluation combines statistical analysis of clustering properties with expert assessment focused on scientometrics publications.The expert assessment concerns the scientometrics subfield within library and information science.Source: Introduction · p. 2

Methods

The study compares graph-clustering methods across citation networks using statistical, bibliometric, robustness, and expert-based criteria. It evaluates publication networks from several fields and the Web of Science, with network preprocessing and algorithm settings specified.

Comparison criteria

  • Clustering methods are assessed on cluster size, small-cluster prevalence, stability across runs and link perturbations, computing time, and expert judgments of topic interpretability.Experts assess whether clusters can be interpreted as meaningful research topics; this criterion is not evaluated quantitatively.Source: Methods · pp. 2–3

Networks

  • The four citation networks cover Scientometrics, Library & Information Science, Physics, and the entire Web of Science, using articles and reviews from three Web of Science indexes.Citation graphs are made undirected and unweighted by removing citation direction, duplicate links, self-citations, and isolated publications.Source: Methods · pp. 3–4, Table 1 · p. 3

Derived methods

  • Derived methods partition networks into subgraphs before applying Infomap, accepting each subgraph clustering only when it improves the overall log-likelihood.For subgraphs larger than 50 nodes, Infomap is applied; named variants use Graclus, METIS, Louvain, or label propagation as the first-stage method.Source: Methods · pp. 4–5

Clustering methods

  • The comparison includes 30 methods: 20 listed methods, five alternative cluster-count settings, and five derived methods spanning graph-partitioning and community-detection approaches.Algorithms generally use author-provided code and default parameters; overlapping-cluster nodes are assigned to the first returned cluster.Source: Methods · pp. 4–5, Table 2 · p. 4

Post-processing

  • A two-step post-processing procedure splits clusters larger than 10^4 nodes and merges clusters smaller than 15 with neighboring clusters based on log-likelihood.The split step reapplies the original method and accepts the result by log-likelihood; tiny clusters are assessed in random order.Source: Methods · p. 5

Graph and stability measures

  • Graph-structure measures include internal and external degree, the Flake function, modularity, likelihood, variation of information, and robustness under link rewiring.The robustness measure compares a clustering with one obtained after randomly rewiring network links.Source: Methods · pp. 5–6

Bibliometric criteria

  • Bibliometric criteria include average cluster size, covered-link fraction, effective orders of cluster sizes, and 90-percentile effective cluster diameter.The effective cluster diameter is estimated from 1,000 randomly selected seed nodes; reported values average varying numbers of realizations across the four networks.Source: Methods · p. 6

Results

The results compare citation-clustering methods across network characteristics and expert assessment, describing structural and bibliometric trade-offs, cluster-size and coverage patterns, and method-specific effects of post-processing. Expert reviewers found Metimap and Infomap most satisfactory for scientometrics, with only a weak preference for Infomap over Metimap.

Structural clustering analysis

  • Cluster-size distributions depended on method: spectral methods lacked heavy tails, while modularity, label-propagation, and link-clustering distributions followed power-law scaling over several orders.Map-equation and statistical methods deviated from power-law form at small and moderate cluster sizes; link clustering also produced a few very large clusters.Source: Figure 2 · p. 8, Results · pp. 8–9
  • Most methods returned just under 2,000 clusters, but Louvain and dynamical methods returned fewer, while link clustering and COPRA returned more.Most methods had K≫E; Flake F was low for power-law methods with very large clusters, high for homogeneous spectral methods, and intermediate for Infomap and OSLOM. Louvain had the highest modularity Q, whereas Infomap had the highest log-likelihood log L, which is the map-equation criterion.Source: Table 3 · p. 9, Results · p. 9

Robustness and complexity

  • Robustness varied: Graclus and OSLOM were fragile, Infomap and larger-network Louvain were stable, while Links barely changed after 30% rewiring because giant clusters dominated.Spectral and statistical methods had uncertainty U ≈0.4, link clustering, Walktrap, and SCP had U ≈0, and other methods had U ≈0.2. Infomap took about ten seconds on the Library & Information Science network but almost three hours on Physics.Source: Figure 3 · p. 10, Results · pp. 10–11, Table 4 · p. 11

Bibliometric clustering analysis

  • Power-law methods covered over 80% of links, spectral and statistical methods as little as 30%, and Infomap around 60%.For most methods O5≫1, whereas spectral methods were the exception; clique-based methods and link clustering had D90 > 10. Most methods had small D90.Source: Results · pp. 10–11, Table 4 · p. 11

Structural clustering analysis

  • Infomap and spectral/statistical methods lacked giant clusters, whereas Louvain and BPA produced very large clusters and link clustering and SCP could form one covering almost all nodes.These tendencies were assessed across the Library & Information Science and Physics networks.Source: Figure 4 · p. 12, Results · p. 12

Expert-based clustering assessment

  • The expert comparison favored OSLOM, Infomap, Metimap, and Louvain(10) for topic-level resolution, while spectral variants were too fine and BPA and Walktrap too coarse.BPA may suit identifying subfields, but not topics; Louvain(10) uses a resolution parameter set to 10.Source: Results · pp. 12–13
  • OSLOM produced only three clusters with more than 100 scientometric publications, splitting webometrics and patents across two approximately equal-sized clusters each; Louvain(10) included large, heterogeneous clusters lacking easily recognizable topics.Some Louvain(10) clusters included many publications relating to a specific country or geographical region, such as China or Africa.Source: Results · p. 13
  • Expert reviewers found the Metimap and Infomap clusterings most satisfactory for identifying topics within scientometrics.Table 5 lists their scientometric clusters with at least 50 publications.Source: Results · p. 13, Table 5 · p. 14

Expert-based assessment

  • In the scientometrics assessment, Infomap was only weakly preferred over Metimap: Metimap’s higher resolution split some topics across clusters, including bibliometric networks.Both methods produced reasonable views of the field, and their clusterings differed only slightly.Source: Results · p. 14
  • Clusters with fewer than 100 publications often included a significant share of publications weakly related to the cluster’s main topic, around 25% in some cases.This issue occurred even though Metimap and Infomap produced the best clusterings in the study.Source: Results · pp. 14–15
  • On the Library & Information Science network, post-processing reassigned very few publications from small to larger clusters and had no significant observed effect on cluster quality.The network’s relatively small size limited the effect on the main Metimap and Infomap clusters.Source: Results · p. 15

Large-scale clustering analysis

  • On Physics, Louvain and BPA covered around 80% of links, while Metimap stayed below 40%; method uncertainty U remained about the same as network size increased, while complexity T rose.Average cluster size S and effective orders O5 increased, while effective diameter D90 and coverage K/k remained comparable to the Library & Information Science results.Source: Results · pp. 15–16, Table 6 · p. 16
  • In Physics, post-processing left the largest Metimap, Metilus, and BPA clusters apparently unchanged but partitioned Louvain’s giant clusters, lowering S, O5, D90, and K/k while raising U.It merged tiny clusters, generally increasing S while leaving other statistics roughly unchanged for Metimap, Metilus, and BPA; degeneracy diagrams show it failed to partition BPA’s giant cluster.Source: Table 6 · p. 16, Results · p. 16, Figure 6 · p. 17
  • On All Fields, average cluster size S and effective orders O5 rose while coverage K/k stayed roughly the same; post-processing had the same effects as on Physics and added little runtime.The differences between methods were also visible in average internal degree K.Source: Results · p. 16, Table 7 · p. 17

Discussion

The discussion weighs citation-clustering methods against practical and bibliometric criteria, favoring Infomap overall while noting trade-offs. The assessment’s scope and setup limit how broadly its findings can be interpreted.

Discussion

  • Most methods produce clusters differing by several orders of magnitude; spectral methods yield more uniform sizes, while coverage ranges from less than 30% to more than 85% in Library & Information Science.Graclus and Infomap each assign more than 25% of publications to clusters with fewer than 15 publications.Source: Discussion · pp. 18–19
  • Clustering the Library & Information Science network takes more than 100 times longer with the slowest method than the fastest; Louvain, BPA, and Graclus are among the fastest.The authors note that slower methods are less suitable for large citation networks.Source: Discussion · p. 19
  • Infomap receives the strongest overall support: it performs best in the expert assessment and is relatively robust, though it takes more computing time and yields many small clusters.Metimap is a larger-network alternative; the authors say the effect of post-processing small clusters on clustering quality is unclear.Source: Discussion · p. 19
  • The findings align with earlier network-science results, while Infomap remains less widely used in bibliometrics than modularity-based methods.The authors suggest exploring methods beyond modularity, particularly Infomap, and say future studies should test whether it consistently performs well on citation networks.Source: Discussion · p. 19
  • The method comparison is incomplete, and default parameter settings were used without optimization, so some methods may have performed better with tuned parameters.The analysis omits overlapping and multilevel clustering methods and some recently developed statistical-inference methods.Source: Discussion · p. 20
  • The analysis covers undirected, unweighted direct-citation networks, while the expert assessment is subjective and limited to scientometrics, leaving its generalizability uncertain.The experts could not assess the full Library & Information Science field, and the authors caution that conclusions from a relatively small area may not generalize.Source: Discussion · p. 20

Related papers

Loading 1512.09023v2…