Author Identification using Multi-headed Recurrent Neural Networks

1.2 The PAN@CLEF 2015 author identification task

• PAN 2015 asks whether an unknown document matches the author of one or more known documents across four language corpora.The task includes 100 or more questions per corpus, with exactly half answerable yes and half no.Source: §1 · p. 1, §1.2 · p. 4

1.1 Recurrent neural networks for language modelling

• A language model predicts each symbol from its predecessors, while character-based models are robust to novel words and require less text and training time than word-based models.The paper notes that character-based models generally do not perform as well as word-based ones.Source: §1.1 · p. 2
• The multi-headed model shares a recurrent state across authors while independent softmax groups learn output tendencies that reflect each author's writing.The shared layer models a combination of authors' texts, addressing the overfitting risk of conventional language models on only a few thousand characters.Source: §1.1 · p. 3

1.2 The PAN@CLEF 2015 author identification task

• Relative cross-entropy scores can support a pre-existing authorship question, but are not sufficient by themselves to decide authorship.The paper cautions that its methods target the specific artificial PAN task and may not be useful in general.Source: §1.2 · p. 4

2.1 Text preprocessing

• Text is converted to a reduced, language-specific character set, with rare punctuation collapsed and rare characters discarded to limit complexity and excessive weighting.The preprocessing includes Unicode normalization, collapsing whitespace, mapping digits to 7, and language-specific handling of Latin characters in Greek text.Source: §2.1 · pp. 5–6
• Character mapping can expose distinctions such as a Greek tau used in place of a visually identical Latin t, while runs longer than five identical characters are truncated.The mappings were settled before training, and their efficacy was not tested.Source: §2.1 · p. 6

2.2 Multi-headed recurrent neural network language model

• One recurrent network shares a collective language model across authors while separate softmax groups model each author’s character probabilities; a control-corpus group can also be included.The PAN 2014 training set serves as the control corpus, and overlap with the 2015 sets means it is not fully independent; empirically its effect seems very small and positive.Source: §2.2 · p. 6

2.3 ReSQRT activation

• ReSQRT is the recurrent-layer activation, and its zero response for non-positive inputs and intermediate scale are reported to work well for character language models.Training uses an Adagrad form with a monotonically decreasing per-weight learning rate.Source: §2.3 · pp. 6–7

2.4 Training

• Training proceeds in sub-epochs, with author texts cycled or concatenated; balanced runs draw documents until authors have been trained for roughly equal amounts.When a single known text is much shorter than the unknown text, the texts are swapped and the network is trained on the unknown text.Source: §2.4 · p. 7
• Early training leakage back-propagates some character errors to other authors’ output groups, then decays so later training specializes each group on its author’s style.The initial leakage rate is about 1/N; training uses batches of 40 characters and truncated back-propagation through time to depth 70 or until the gradient falls below an adaptive threshold.Source: §2.4 · p. 7

2.5 Ensemble of variations

• The final scores average several runs with varied meta-parameter configurations; on training tests, ensembles slightly outperformed most constituent runs.Evaluation ensembles used random selections from configurations chosen per language by a haphazard search.Source: §2.5 · p. 8

2.6 Interpretation

• Each author’s output group receives a normalized relative cross-entropy score, whose rank is used as evidence for authorship and mapped to a probability around a median threshold of 0.5.For Spanish, Greek, and English, a small radius around the median also receives 0.5; on training data this slightly increased C@1 and had little effect on AUC.Source: §2.6 · pp. 8–9

3 Results

• Greek’s evaluation score was higher than its training score, Spanish’s was lower, and English and Dutch results were in the expected range.Table 2 presents selected 2015 PAN evaluation results sorted by macro-average; Table 3 details this system’s AUC, C@1, and margins against other teams.Source: §3 · p. 9, Table 2 · p. 9, Table 3 · p. 10

• Dutch results were consistently poor and seem to reflect a drastic genre difference; explanations for teams’ varying success are explicitly speculative.Source: §4 · p. 10
• English performance exceeded other teams by a significant margin, while results were very good on closer genre/topic matches without specialist NLP.Source: §4 · p. 10
• With recurrence accidentally removed, the model reduced to author-specific unigram character probabilities yet achieved training AUC 0.85 for English and 0.91 for Spanish, supporting the scoring methods.Source: §4 · p. 10
• The authors frame text-focused language modeling as the method’s strength, contrasting it with a more direct approach that may be more distracted.Source: §4 · p. 11
• Whether deeper LSTM or GRU architectures help remains open; long-range context may be of little use on 300-character texts, and their computational cost is problematic for PAN hardware.Source: §4 · p. 11
• The work demonstrates that even small RNNs can be useful in authorship identification while avoiding feature-selection and dimensionality difficulties.Source: §4 · p. 11