Source-linked AI summary
openXBOW - Introducing the Passau Open-Source Crossmodal Bag-of-Words Toolkit
Maximilian Schmitt, Björn W. Schuller
TL;DR
BoW tools traditionally focus on text, while crossmodal use requires representing numeric streams such as audio or video alongside text. openXBOW addresses this gap with an open-source toolkit that quantises and combines multimodal inputs, and its examples report strong results in speech emotion recognition and text classification.
Problem
Existing BoW practice does not capture word order, and publicly available tooling for generating crossmodal BoW representations was lacking.
Method
openXBOW generates BoW representations from text and numeric features by quantising low-level descriptors, processing modalities, and combining their subbags.
Results
The authors report outperforming the state of the art on a speech-based emotion-recognition task and achieving state-of-the-art results in text classification.
Takeaways & Limitations
The toolkit demonstrates potential for crossmodal BoW applications, although the reported examples use separate audio and text tasks rather than actual multimodal inputs.
Takeaways & Limitations
The introductory evaluation uses separate audio and text tasks because few suitable multimodal databases were available; actual crossmodal results were left to future work.
Abstract
from arXiv · showhide
We introduce openXBOW, an open-source toolkit for the generation of bag-of-words (BoW) representations from multimodal input. In the BoW principle, word histograms were first used as features in document classification, but the idea was and can easily be adapted to, e.g., acoustic or visual low-level descriptors, introducing a prior step of vector quantisation. The openXBOW toolkit supports arbitrary numeric input features and text input and concatenates computed subbags to a final bag. It provides a variety of extensions and options. To our knowledge, openXBOW is the first publicly available toolkit for the generation of crossmodal bags-of-words. The capabilities of the tool are exemplified in two sample scenarios: time-continuous speech-based emotion recognition and sentiment analysis in tweets where improved results over other feature representation forms were observed.
1. Introduction
openXBOW extends the bag-of-words principle from text to numeric and multimodal inputs by quantising low-level descriptors and combining modality-specific representations. It is presented as an open-source toolkit with configurable processing options and two application examples.
- Background: BoW represents documents as word-frequency histograms that can be supplied to classifiers handling sparse, potentially irrelevant features.The approach counts dictionary-word frequencies and produces a vector for methods such as naïve Bayes or SVM.
- Background: A major limitation of standard BoW is that it ignores word order, motivating sequence-based alternatives such as n-grams.N-grams count sequences of words or characters rather than isolated words.
- Crossmodal extension: BoW generalizes to visual and audio domains by quantising local image features or acoustic descriptors against a learned codebook before histogram construction.In audio, frame-level descriptors such as MFCCs are quantised using codebooks generated by methods including k-means, random sampling, or EM clustering.
- Contribution: openXBOW introduces an open-source toolkit for generating BoW representations across modalities and fusing information from different sources.The toolkit targets symbolic inputs such as text and numeric streams such as audio or video features.
- Contribution: The toolkit offers configurable vector quantisation, codebook generation, term-frequency weighting, and natural-language processing methods for textual features.The paper describes these options as addressing the generation of fused BoW-based representations.
- Evaluation: The paper evaluates the principle in time-continuous speech emotion recognition and tweet sentiment analysis, then outlines the toolkit structure and future developments.The introduction identifies these as two exemplary application scenarios.
2. Overview
openXBOW processes numeric and textual inputs through optional preprocessing, quantisation, weighting, normalisation, and fusion into BoW feature vectors. Its overview describes broad configuration support, multiple codebook strategies, and interoperable input and output formats.
- Implementation: openXBOW is implemented in Java, publicly distributed with source and a compiled JAR, and freely usable by the research community for non-commercial purposes.The implementation is intended to run on any platform supporting Java.
- Implementation: The toolkit supports ARFF, CSV, and LIBSVM formats for exchanging low-level descriptors, text, and BoW feature vectors.LIBSVM is supported for output only.
- Preprocessing: An optional preprocessing stage removes low-activity low-level descriptors before further processing, which can exclude non-voice background noise in speech.This stage is especially relevant when voice activity determines which acoustic frames are retained.
- Preprocessing: Normalisation or standardisation is essential when combined feature types have substantially different value ranges, and parameters are stored for consistent test-time application.The stored parameters support an online application of the same transformation to test files.
- Codebook generation: Codebooks can be generated by random sampling, random sampling++, k-means-style updates, or supervised class-specific construction.The supervised option concatenates separately learned class codebooks into a super-codebook.
- Quantisation: Split vector quantisation handles large descriptors by quantising subvectors, while manual splitting can create separate codebooks for different feature types.The resulting subvector indexes are then processed in the usual scheme.
- Quantisation: Soft assignment can weight term frequencies by codebook distance and assign each input to multiple nearby codewords.Gaussian encoding provides a non-EM route to soft quantisation, with N_a closest words considered under multiple assignment.
- Fusion and post-processing: Text processing includes stopping, n-grams, and n-character-grams, while fused domain representations can receive logarithmic TF or TFIDF weighting.The final histograms can also be normalised when input instances differ in size or activity duration.
3. Experiments
The experiments apply openXBOW to speech emotion recognition and Twitter sentiment analysis, using task-specific BoW/BoAW configurations. The toolkit outperforms the speech baseline on nearly all reported partitions and slightly exceeds the reported sentiment-analysis state of the art.
- Speech emotion recognition: openXBOW is evaluated on time-continuous speech emotion recognition using RECOLA audio, with codebook learning, validation, and test partitions.The setup uses MFCCs and log-energy with linear-kernel SVR; CCC is the evaluation metric.
- Speech emotion recognition: A 1,000-word codebook is generated after standardisation, with each LLD assigned to its 20 closest codebook words and logarithmic TF weighting applied.The learned standardisation and weighting parameters are retained for validation and test processing.
- Speech emotion recognition: The optimum prediction windows are 10.0 seconds for valence and 8.0 seconds for arousal, with a 4.0-second label shift.CCC accounts for scaling of the outputs and is also the AVEC 2016 competition measure.
- Speech emotion recognition: openXBOW significantly outperforms the AVEC 2016 baseline on validation and test sets, except for arousal validation, where performance is similar.The comparison uses official validation and test sets, with statistically significant improvements identified at significance level 0.01.
- Twitter sentiment analysis: The Twitter experiment uses 1,578,627 English tweets with binary sentiment labels and evaluates BoW representations against a reported 75% state-of-the-art accuracy.The corpus is split into 1,000,000 training and 578,627 test instances; linear-kernel SVM is used for the high-dimensional vectors.
- Twitter sentiment analysis: 77.28% weighted accuracy and 77.29% unweighted accuracy are achieved, slightly matching or outperforming the reported state of the art.The configuration uses a dictionary of 1,875 terms, with term-frequency stopping and log-TF-IDF weighting described for the sample call.
4. Conclusions and outlook
The paper introduces openXBOW as a first-of-its-kind toolkit for BoW representations from multimodal symbolic and numeric inputs. Its monomodal demonstrations show strong speech-emotion and text-classification performance, while broader crossmodal capability remains future work.
- Contribution: openXBOW is presented as a first-of-its-kind toolkit for generating BoW representations from multimodal symbolic and numeric information.The toolkit targets inputs including text, audio, and video feature streams.
- Results: The experiments demonstrate the toolkit’s potential by outperforming the state of the art on a speech-based emotion-recognition competition task.The conclusion describes the examples as monomodal demonstrations of the underlying BoXW principle.
- Outlook: The paper identifies actual crossmodal tasks as the setting in which openXBOW’s full potential is likely to be revealed.The authors note that the presented examples use separate audio and text tasks because suitable multimodal databases are scarce.
- Results: The toolkit already provides state-of-the-art results in text classification.This conclusion is stated separately from the speech-emotion result.
- Outlook: Future development includes additional soft vector-quantisation methods, order-sensitive representations for numeric features, and a graphical configuration interface.Proposed methods include EM clustering, non-negative matrix factorisation-based soft clustering, temporal augmentation, and numeric n-grams.