PaperlayerView exact revision on arXiv

Source-linked AI summary

LFTK: Handcrafted Features in Computational Linguistics

Bruce W. Lee, Jason Hyung-Jong Lee

arXiv:2305.15878v2cs.CLcs.LG

TL;DR

Handcrafted feature research faces inconsistent implementations, missing categorization standards, and limited multilingual open-source extraction tools. The paper categorizes features, releases an expandable toolkit, and analyzes feature correlations across task-specific datasets. Its baseline analyses report feature predictive abilities across four NLP tasks and preliminary correlations, including a negative association between verb count and text difficulty.

  • Problem

    Handcrafted feature implementations vary across research, categorization standards are absent, and no open-source extraction system works multilingual.

    Source: §1 · p. 2

  • Method

    The authors categorize more than 200 handcrafted features, build the expandable LFTK extraction toolkit, and conduct baseline correlation analyses on task-specific datasets.

    Source: §1 · p. 2

  • Results

    Baseline analyses report feature predictive abilities across four NLP tasks and preliminary correlations, including a negative association between n_verb and text difficulty.

    Source: §1 · p. 2, §4 · p. 7

  • Takeaways & Limitations

    LFTK is an expandable starting point for researchers to extract pre-implemented features and create research-specific versions, while large-scale extraction can help reveal novel correlations.

    Source: §4 · p. 7, §5 · p. 8

    The authors do not guarantee that a feature works similarly across languages, so its usability in a new language requires individual exploration.

    Source: §3.2.3 · p. 5

Abstract

from arXiv · show

Past research has identified a rich set of handcrafted linguistic features that can potentially assist various tasks. However, their extensive number makes it difficult to effectively select and utilize existing handcrafted features. Coupled with the problem of inconsistent implementation across research works, there has been no categorization scheme or generally-accepted feature names. This creates unwanted confusion. Also, most existing handcrafted feature extraction libraries are not open-source or not actively maintained. As a result, a researcher often has to build such an extraction system from the ground up. We collect and categorize more than 220 popular handcrafted features grounded on past literature. Then, we conduct a correlation analysis study on several task-specific datasets and report the potential use cases of each feature. Lastly, we devise a multilingual handcrafted linguistic feature extraction system in a systematically expandable manner. We open-source our system for public access to a rich set of pre-implemented handcrafted features. Our system is coined LFTK and is the largest of its kind. Find it at github.com/brucewlee/lftk.

1 Introduction

Handcrafted features remain in use alongside automatically generated representations, but inconsistent implementations, absent categorization standards, and limited multilingual open-source tooling complicate their use. The paper presents a categorized feature toolkit and baseline correlation analyses across task-specific datasets.

Motivation

  • Handcrafted features remain in current NLP research alongside automatically generated features such as Word2Vec and BERT embeddings.The introduction cites recent work using both kinds of features.Source: §1 · p. 1
  • Handcrafted features provide wider numerical representations spanning syntax and discourse, and are used with semantic embeddings in some applications.The introduction gives readability assessment and essay scoring as examples of hybrid use.Source: §1 · p. 1
  • The practices lack consistent implementations and categorization standards, while no open-source extraction system works multilingual.The introduction notes that the average-words-per-sentence feature has been implemented differently across studies.Source: §1 · p. 2

Contributions

  • The authors collect and categorize more than 200 handcrafted features, building LFTK as an expandable toolkit with language availability labels.Features are organized into four linguistic branches and 12 families; the system is built on spaCy.Source: §1 · p. 2
  • Baseline correlation analyses on task-specific datasets report feature predictive abilities across four NLP tasks and identify previously unreported correlations.The authors describe the analysis as preliminary and a basis for future in-depth studies.Source: §1 · p. 2

2 Related Work

The related work defines handcrafted linguistic features as scalar outputs of identifiable methods, describes their hybrid use with neural representations, and surveys feature-rich NLP studies, especially readability research.

2.1 What are Handcrafted Features?

  • A handcrafted linguistic feature is a single numerical value produced by a uniquely identifiable method on natural language.The discussion notes that features can combine textual properties with simple operations and return a float or integer.Source: §2.1 · pp. 2–3

2.2 Hybridization of Handcrafted Features

  • Handcrafted features can add syntax and discourse information to semantic embeddings or neural models in tasks requiring holistic text understanding.The related work gives readability assessment and essay scoring as examples of hybridization.Source: §2.2 · p. 3

2.3 Handcrafted Features in Recent Studies

  • Recent handcrafted-feature studies span several NLP tasks, with readability research especially prominent and using 80–255 features across semantics, discourse, and syntax.The cited examples include fake-news detection, bias detection, and learner-based reading-passage selection.Source: §2.3 · p. 3

3.1 Overview

The toolkit collects and categorizes more than 200 features from prior research, using a foundation-and-derivation design and a framework intended to support future expansion.

3.1 Overview

  • Table 1 presents the linguistic branches available in the software and gives abbreviated feature-name examples, using # to mean “number of”.The caption describes the table’s scope and example-column notation.Source: Table 1 · p. 4
  • Table 2 lists linguistic families and distinguishes foundation (F.) from derivation (D.) families.Its caption also identifies UPOS as Universal POS.Source: Table 2 · p. 4
  • The authors collected and categorized over 200 features, mostly from readability assessment, essay scoring, fake-news detection, and paraphrase detection.They note that the selected research areas reflect both their relationship to handcrafted features and the authors’ limited expertise scope.Source: §3.1 · p. 4

3.2 Categorization

LFTK organizes features as foundation building blocks and derivations, groups them by linguistic properties and families, and assesses language applicability through their components.

3.2.1 Formulation

  • Derivation features combine one or more foundation features; LFTK builds 155 derivation features from 65 foundation features.A foundation feature cannot be split into smaller building blocks, while a derivation can reuse foundations in multiple features.Source: §3.2.1 · p. 4

3.2.2 Linguistic Property

  • Features are grouped into lexico-semantics, syntax, discourse, and surface branches, with families providing smaller search-oriented categories.Each family contains features of one formulation type and also serves as a building block in the extraction system.Source: §3.2.2 · pp. 4–5

3.2.3 Applicable Language

  • A derivation feature is classified as generally applicable only when all its foundation components are generally applicable.For example, average nouns per sentence qualifies using UPOS tagging, whereas FKGL does not because its syllable counter is English-specific; feature behavior across languages still requires individual exploration.Source: §3.2.3 · p. 5

3.3 Feature Details by Linguistic Family

This section outlines feature counts by branch and family, then describes families spanning text counts, word difficulty, grammatical and discourse properties, lexical richness, and readability formulas.

3.3 Feature Details by Linguistic Family

  • Tables 4 and 5 report implemented feature counts by linguistic branch and family; the full feature list is in the appendices.The section explains motivations and implementations behind the feature families.Source: §3.3 · p. 5, Table 4 · p. 5, Table 5 · p. 5

3.3.1 WordSent & AvgWordSent

  • WordSent provides foundation counts for characters, syllables, words, and sentences, while AvgWordSent derives averages from those counts.WordSent relies heavily on spaCy tokenization except for syllable counts, which use a custom algorithm.Source: §3.3.1 · p. 5

3.3.2 WordDiff & AvgWordDiff

  • WordDiff covers word difficulty, including age-of-acquisition and corpus-based frequency statistics, while AvgWordDiff averages these features by word or sentence counts.The implemented Kuperman age-of-acquisition ratings cover over 30,000 words.Source: §3.3.2 · pp. 5–6

3.3.3 PartOfSpeech & AvgPartOfSpeech

  • PartOfSpeech counts token-level POS properties using spaCy dependency parsing and the multilingual UPOS scheme; AvgPartOfSpeech averages these counts by words or sentences.Average verbs per sentence is one example of an averaged feature.Source: §3.3.3 · p. 6

3.3.4 Entity & AvgEntity

  • Entity features count entities used to represent discourse characteristics, while AvgEntity expresses those counts per word or sentence.An example is the number of organization entities per sentence.Source: §3.3.4 · p. 6

3.3.5 LexicalVariation

  • LexicalVariation includes measures such as unique verbs divided by total verbs, with additional variants using squares or roots.The unique-verbs ratio is often called verb variation in other literature.Source: §3.3.5 · p. 6

3.3.6 TypeTokenRatio

  • TypeTokenRatio is treated as a separate family from LexicalVariation because of its distinctive prevalence, despite their similar intended characteristics.TTR features occur in second- and child-language acquisition research.Source: §3.3.6 · p. 6

3.3.7 ReadFormula

  • ReadFormula implements traditional readability formulas that recent work has also applied to tasks including fake-news classification and authorship attribution.Earlier readability assessment used linear formulas to quantify text readability.Source: §3.3.7 · p. 6

3.4 LFTK in Context

LFTK supports feature search through domain, family, and language attributes and uses a Python-based extraction architecture. Its reported extraction speed and reusable feature structure support large-scale corpus use.

3.4 LFTK in Context

  • LFTK tags features by domain, family, and language so researchers can sort and extract features by those attributes.The extraction system is implemented in Python, which the authors say makes it more expandable and customizable in the community.Source: §3.4 · p. 6
  • Excluding spaCy processing, LFTK extracts 220 handcrafted features from a 1,000-word dummy text in an average of 10 seconds, or about 0.01 seconds per word.The average was measured over 20 randomized 1,000-word texts on a single-core 2.3 GHz Intel Core i9.Source: §3.4 · pp. 6–7
  • The architecture links linguistic-family modules through foundation and derivation collectors before the main extractor inherits the derivation collector.Derivations built on foundations allow a calculated foundation feature to be saved and reused, reducing duplicate calculations.Source: Figure 4 · p. 7, §3.4 · p. 7

4 Which applies to which? Task-Feature Correlation Analysis

This section uses correlations to help researchers identify potentially useful handcrafted features for task setups. Its preliminary analysis covers four NLP tasks and reports some rarely noted associations.

4 Which applies to which? Task-Feature Correlation Analysis

  • Correlations use CLEAR readability (4,724 instances), ASAP essay scoring (1,783), LIAR fake-news detection (10,420), and SemEval hate-speech detection (9,000) datasets.Pearson correlations are calculated between features and dataset labels.Source: §4 · p. 7
  • n_verb negatively correlates with text difficulty, and the preliminary analysis reports rarely noted feature-label correlations.The authors describe the analysis as limited and say there is room for further exploration.Source: §4 · p. 7
  • Table 7 reports the top and bottom ten correlated features, while the full correlation analysis is in the appendices.The table caption specifies the positive-label interpretation for each task.Source: §4 · p. 7, Table 7 · p. 8

5 Conclusion

LFTK is an open-source, large-scale handcrafted-feature extraction system designed to be expandable. The authors position it as a general-purpose starting point across NLP tasks.

5 Conclusion

  • LFTK is open-source and expandable, with foundation features that support building modified, research-specific extraction programs.The authors give URL counts for Twitter bot detection and grammatical-error counts for automated essay scoring as newer task-specific features within their definition.Source: §5 · p. 8
  • The system is intended as a starting point for work incorporating linguistic features into neural or nonneural models, where breadth of coverage can matter as much as selection.The authors cite Lee et al. (2021), Yaneva et al. (2021), Klebanov and Madnani (2020), and Horbach et al. (2013).Source: §5 · p. 8
  • The authors describe LFTK as their first general-purpose handcrafted-feature extraction system, comparable or larger in size than Coh-Metrix and L2 Syntactic Complexity Analyzer.Expandability and multilingualism were considered in its architecture, grounded in systematic feature categorization.Source: §5 · p. 8

A All implemented features

The appendix records the software versions and the versioning and retrieval details for LFTK.

A All implemented features

  • The appendix reports spaCy en_core_web_sm 3.0.5 and Pandas 1.1.4 for the described implementation and correlation calculations.The passage says these versions reflect the most recent updates in the respective libraries.Source: A All implemented features · p. 12
  • LFTK version 1.0.9 is the version referenced in Tables 8–11.The appendix reports only linguistic families there due to space restrictions.Source: A All implemented features · p. 12
  • The published LFTK version is retrievable through PyPI or GitHub, and its feature description is updated with version changes.Source: A All implemented features · p. 12

B Feature correlations

The appendix provides full feature-correlation tables and tables listing implemented features by key, name, and branch.

B Feature correlations

Loading 2305.15878v2…