PaperlayerView exact revision on arXiv

Source-linked AI summary

A Large-Scale Dataset for Multi-Class Financial News Classification in Brazilian Portuguese

Leomax Bandeira, Gabriel Aguiar, Dalton Cézane Gomes Valadares, Luiz Pereira, Helder Fernando de Araújo Oliveira, Gustavo Xavier, Andson M. Balieiro

arXiv:2610.20630v1cs.CLcs.LG

TL;DR

Brazilian Portuguese Financial NLP lacks domain-specific datasets and research on dataset construction and weak supervision. This work builds a large news corpus and evaluates semantic-similarity classifiers using manual and LLM-generated reference labels. KNN performs best on manual labels, while the LLM-label setting combines roughly 50% accuracy with F1-scores near 20%.

  • Problem

    Research on dataset construction and weak supervision for languages beyond English remains limited, despite noise in large-scale web text collection.

    Source: §II · p. 2

  • Method

    The study evaluates cosine similarity, KNN, and FAISS on semantic embeddings, using manual labels or LLM-generated labels as reference data and a manually labeled test set as ground truth.

    Source: §V · pp. 4–5

  • Results

    Manual-label KNN achieved 47.3% accuracy and 31.9% F1-score, while LLM-label FAISS achieved 50.8% accuracy and 20.1% F1-score.

    Source: §V · p. 5

  • Takeaways & Limitations

    The results indicate that label quality and class distribution affect classification outcomes, and that F1-score is more informative than accuracy under imbalanced labels.

    Source: §V · pp. 5–6

    The manually annotated subset is small relative to the full corpus, and its curated training subset may not represent the corpus’s natural class distribution.

    Source: §V · p. 5

Abstract

from arXiv · show

The development of Financial Natural Language Processing (NLP) systems for Brazilian Portuguese is hindered by the scarcity of domain-specific datasets. To address this, we present a large-scale corpus of approximately 160,000 Brazilian Portuguese financial news articles collected from web sources. To establish initial benchmarks for a weakly supervised multi-class classification approach based on semantic similarity, we annotated subsets of this corpus both manually and using a Large Language Model (LLM). We evaluated three similarity-based methods -- cosine similarity, K-Nearest Neighbors (KNN), and FAISS-based nearest-neighbor search -- using multilingual semantic embeddings. Our experiments reveal that the task is highly sensitive to label quality and class distribution. On the manually labeled subset, KNN achieved the best performance (47.3% accuracy, 31.9% F1-score), closely followed by FAISS. Conversely, on the LLM-annotated subset, while accuracy reached approximately 50%, F1-scores dropped to near 20%, highlighting the negative impact of label imbalance and noisy weak annotations. These findings establish foundational baselines for a challenging task, underscoring the critical need for high-quality manual annotations. Ultimately, this dataset and our preliminary results provide a reference point for future research in Brazilian Portuguese Financial NLP.

I. INTRODUCTION

Timely classification of financial news can support financial decision-making, risk assessment, and market monitoring. This work addresses the scarcity of Brazilian Portuguese financial-news resources with a large multi-source corpus and similarity-based classification baselines.

I. INTRODUCTION

  • Accurate, timely categorization of financial news can benefit financial decision-making, risk assessment, and market monitoring.The introduction points to semantic representation models for Portuguese, including BERTimbau, as expanding automated text-understanding possibilities.Source: §I · p. 1
  • Portuguese financial NLP lacks resources comparable to English financial text datasets, and existing Portuguese resources can be difficult to access, narrow in scope, or time-limited.The authors identify a need for an open, reproducible, domain-appropriate benchmark for Brazilian financial-news classification.Source: §I · p. 1
  • The work introduces a corpus of approximately 160,000 Brazilian Portuguese news articles and evaluates cosine similarity, KNN, and FAISS as baseline classifiers.The manually labeled subset contains 2,300 instances across six domain categories; 1,800 serve as reference data and 500 as a held-out test set.Source: §I · p. 1

II. RELATED WORKS

Prior Financial NLP benchmarks largely address English sentiment tasks, while Brazilian Portuguese financial-news resources and weak-supervision strategies remain limited. Related work also highlights data noise and the computational trade-offs of similarity search.

II. RELATED WORKS

  • Web collection can yield irrelevant or weakly related documents, and noisy text can reduce classifier robustness, as discussed by Vinciarelli.The paper notes that much prior work assumes clean, structured text, unlike large-scale web collection.Source: §II · pp. 1–2
  • English resources such as Financial PhraseBank and SemEval-2017 Task 5 support sentiment-oriented evaluation, but not sector-level classification of Brazilian financial news.The related-work discussion identifies limited research on dataset construction and weak supervision for languages beyond English.Source: §II · p. 2
  • Sentence-BERT embeddings can be compared by cosine similarity, and a small labeled reference set can propagate labels through nearest-neighbor search.The discussion frames this approach as useful when manual annotation is expensive or annotated resources are scarce.Source: §II · p. 2
  • Exact nearest-neighbor search becomes costly for large embedding collections, while FAISS provides efficient indexing structures.K-Nearest Neighbors assigns labels from nearby labeled examples; SimHash offers a compact-signature alternative compared using Hamming distance.Source: §II · p. 2
  • Mason describes corpus-based extraction of metaphorical mappings, illustrating large-scale corpus analysis beyond finance.The related-work passage connects this approach to extracting structured knowledge from text.Source: §II · p. 2

III. DATASET DESCRIPTION

The dataset comprises Brazilian Portuguese news collected through broad, sector-related web searches. The collection approach accommodates partially relevant and unrelated results, which motivates an out-of-domain label.

A. Data Scraping

  • Approximately 160,000 articles were collected from January 1, 2020 to December 30, 2025, with each article’s original URL and publication date recorded.Search terms covered Steel Industry, Oil, Bank, Fintech, and General Market Impact.Source: A. Data Scraping · p. 2
  • Broad search-based retrieval can return partially related, weakly related, or unrelated articles, so the labeling scheme includes an Unrelated category.The resulting dataset is intended to support both filtering and classification tasks.Source: A. Data Scraping · p. 2

B. Dataset Structure

The dataset records article-level identifiers and labeling fields while limiting the released article text. Its manually labeled subset is split into reference and held-out test sets.

B. Dataset Structure

  • The URL is the unique identifier, while human-assigned and LLM-assigned labels are stored in separate columns.The full news text is not released; the dataset provides the first 100 characters and links to the complete articles to respect copyright restrictions.Source: B. Dataset Structure · p. 3
  • The 2,300 manually labeled instances are split into 1,800 reference examples and 500 held-out test examples.Cosine similarity, KNN, and FAISS predictions are experimental outputs, not primary dataset columns.Source: B. Dataset Structure · p. 3

C. Label Schema

The paper defines six mutually exclusive financial-news categories and uses human labels alongside a zero-shot LLM-labeled subset to examine similarity-based classification and labeling trade-offs. Its baseline pipeline evaluates classification against human annotations, while the collected corpus may contain partially related or unrelated articles.

C. Label Schema

  • The six mutually exclusive categories cover four financial sectors, cross-sector market-impact news, and articles unrelated to the investigated domain.General Market Impact includes potential cross-sector effects such as macroeconomic announcements, regulatory changes, and major political or geopolitical events.Source: C. Label Schema · p. 3
  • Manual annotation used keyword searches to find underrepresented-sector candidates, but annotators made the final category assignments.The searches addressed the corpus's overrepresentation of general-market and out-of-domain articles.Source: C. Label Schema · p. 3
  • LLM labels came from zero-shot Llama 3.22 prompts on arbitrarily selected articles, unlike the curated manual subset.The authors note that selection differences may have contributed to class-proportion discrepancies between the subsets.Source: C. Label Schema · p. 3
  • The two reference-labeling scenarios let the evaluation examine classifier performance and trade-offs between manual and automated labeling.Source: C. Label Schema · p. 3

IV. BASELINE EXPERIMENTS

  • Search-based collection yielded approximately 160,000 articles, but can include documents only partially related or unrelated to the search terms.The paper identifies this heterogeneity as noise in large-scale news-mining datasets.Source: §IV · p. 3
  • The baseline pipeline defines reference and test subsets, generates embeddings, classifies by similarity, and evaluates against human annotations.It assesses classification into predefined sector-level categories for Brazilian Portuguese financial news.Source: §IV · p. 3

A. Experimental Setup

The setup uses a manually labeled reference set and a held-out human-labeled test set, with a separate LLM-labeled reference scenario. Both scenarios are evaluated against the same test set using accuracy and F1-score.

A. Experimental Setup

  • 2,300 articles were manually annotated using random sampling and term-assisted candidate selection to address initial class imbalance.Searches targeted underrepresented sectors; final category assignments remained manual.Source: A. Experimental Setup · pp. 3–4
  • 1,800 manual labels formed the reference set, while 500 articles were reserved as a held-out test set not used for label propagation.Source: A. Experimental Setup · p. 4
  • The LLM-reference scenario used zero-shot labels, with both scenarios evaluated against the held-out manual test set using accuracy and F1-score.LLM-generated labels were not treated as ground truth, and test-set URLs were removed from both reference sets to avoid leakage.Source: A. Experimental Setup · p. 4

B. Embedding Generation

The study represents articles with multilingual SentenceTransformer embeddings, which serve as the shared text representation for all evaluated similarity-based classifiers.

B. Embedding Generation

  • The study used paraphrase-multilingual-MiniLM-L12-v2, producing 384-dimensional embeddings.The model was selected for a balance between computational efficiency and semantic representation quality for multilingual texts.Source: B. Embedding Generation · p. 4
  • All embeddings were L2-normalized so cosine similarity could be computed as the inner product of normalized vectors.Source: B. Embedding Generation · p. 4
  • The embeddings provided the common representation for every similarity-based classification method evaluated.Source: B. Embedding Generation · p. 4

C. Similarity-Based Classification

The study compares cosine-similarity nearest-neighbor classification, majority-vote KNN, and FAISS-based neighbor retrieval over the embedding space.

C. Similarity-Based Classification

D. Evaluation Protocol

Both labeling scenarios use the same 500-article manually labeled test set as ground truth. The protocol compares manual and LLM-generated reference labels using accuracy and F1-score while excluding test URLs from reference data.

D. Evaluation Protocol

  • All methods were evaluated against the same held-out set of 500 manually labeled articles, with manual annotations as ground truth.Held-out test URLs were removed from the reference sets to prevent data leakage.Source: D. Evaluation Protocol · p. 4
  • The evaluation compared reference sets built from manually labeled articles with sets using LLM-generated weak labels.Both scenarios were scored using accuracy and F1-score against the held-out test set.Source: D. Evaluation Protocol · p. 4
  • The resulting predictions and metrics provide baseline results for future studies using the dataset.The passage characterizes these outputs as reference baselines.Source: D. Evaluation Protocol · p. 4

V. RESULTS AND DISCUSSION

The similarity-based classifiers perform differently across manual and LLM-based reference-label settings, with accuracy and F1-score revealing distinct patterns. The results also highlight the dataset’s collection and annotation boundaries, the computational trade-off of FAISS, and the sensitivity of performance to label quality and class distribution.

V. RESULTS AND DISCUSSION

  • The corpus contains approximately 160,000 Brazilian Portuguese news articles, but search-based collection can include articles only partially related or unrelated to their query sector.The corpus was assembled using search queries related to financial-market sectors.Source: §V · p. 4
  • The experiments compare semantic-similarity classification using manual reference articles against an alternative scenario with LLM-generated weak labels.Cosine similarity uses the nearest reference article’s label, KNN uses majority voting among nearest examples, and FAISS uses indexed search over embeddings.Source: §V · pp. 4–5
  • Fig. 2 compares classifier results across annotation settings; manual-label KNN achieved 47.3% accuracy and 31.9% F1-score, while LLM-label FAISS reached 50.8% accuracy and 20.1% F1-score.Manual-label FAISS scored 46.6% accuracy and 31.2% F1-score; cosine similarity scored 43.7% accuracy and 29.2% F1-score. In the LLM-label setting, KNN scored 50.6% accuracy and 20.2% F1-score, and cosine similarity scored 39.4% accuracy and 19.9% F1-score.Source: Figure 2 · p. 5, §V · p. 5
  • LLM-generated labels concentrate 48.1% of instances in Unrelated, whereas the curated manual subset is more evenly distributed and does not represent the corpus’s natural distribution.Automated keyword searches retrieved candidate articles from underrepresented sectors because the original corpus overrepresented general-market and out-of-domain articles; final assignment remained manual.Source: Figure 1 · p. 4, Figure 2 · p. 5
  • The LLM-label setting’s higher accuracy does not indicate balanced performance: label imbalance affects accuracy, while F1-score better reflects multi-class performance across categories.The LLM-generated labels are concentrated in Unrelated; the manual-label setting has a more balanced class distribution.Source: §V · p. 5
  • The authors identify label noise and semantic ambiguity between categories as explanations for low LLM-setting F1, and propose expanding manual annotations and validating LLM labels.Future work also includes supervised transformer classifiers and integration of financial news with structured financial data.Source: §V · pp. 5–6
  • KNN and FAISS performed similarly and outperformed cosine similarity in both settings; exact KNN search is costly as reference sets grow, while FAISS indexing reduces search cost.FAISS is described as more suitable for large-scale applications despite performance close to KNN in this experiment.Source: §V · p. 5
  • Interpretation is limited by a relatively small manual subset, its intentional curation for class diversity, and dependence on embedding quality and reference-label reliability.Similarity-based methods also do not learn complex decision boundaries as supervised neural classifiers do.Source: §V · pp. 5–6

VI. CONCLUSIONS

The study introduces a large Brazilian Portuguese financial-news corpus and benchmarks semantic-similarity classification using manual and LLM labels. KNN performs best with manual labels, while LLM-label results underscore concerns about label quality and class distribution.

VI. CONCLUSIONS

  • 47.3% accuracy and 31.9% F1-score for KNN led the manual-label results, followed by FAISS at 46.6% accuracy and 31.2% F1-score, and cosine similarity at 43.7% accuracy and 29.2% F1-score.The authors also identify class ambiguity and the limited size of the manually labeled subset as challenges.Source: §VI · p. 6
  • LLM-labeled evaluation yielded around 50% accuracy but F1-scores near 20%, with labels concentrated in Unrelated rather than distributed across the six classes.The authors caution that accuracy alone can mislead under imbalanced label distributions.Source: §VI · p. 6
  • FAISS performed close to exact KNN while providing a more scalable retrieval strategy for large collections of dense vectors.The authors identify it as particularly suitable for large-scale applications.Source: §VI · p. 6
  • The work contributes an approximately 160,000-article corpus, a manually labeled six-category subset, and comparisons of manual and LLM labels with three similarity-based methods.Manual labels in a held-out test set served as ground truth for quantitative evaluation.Source: §VI · p. 6
  • Future work includes expanding manual annotations, validating LLM labels, testing supervised transformer classifiers, and integrating news with structured financial data.The proposed structured data include stock prices and economic indicators.Source: §VI · p. 6

Related papers

Loading 2610.20630v1…