PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Augmented Classification of Environmental Mitigations in Hydropower Licensing Documents

Hong-Jun Yoon, Tom Ruggles, Joanna Lee, Debjani Singh

arXiv:2608.23241v1cs.IRcs.AI

TL;DR

The paper addresses severe label scarcity in automated classification of environmental mitigation obligations across a 135-category taxonomy. It introduces retrieval-augmented classification and a hybrid BERT-detection/RAG-classification pipeline, which achieves a Micro F1 of 0.524 and outperforms BERT-only and RAG-only systems across all training-support buckets.

  • Problem

    Severe label scarcity limits supervised classification: 40 of 135 mitigation categories have no training examples, while 26 have fewer than five.

  • Method

    The paper retrieves category definitions for RAG classification and combines RAG classification with a high-recall BERT detector in a hybrid pipeline.

  • Results

    The hybrid achieves a Micro F1 of 0.524, outperforming the BERT-only pipeline (0.477) and the RAG-only pipeline (0.416) across every training-support bucket.

  • Takeaways & Limitations

    RAG provides zero-shot coverage for rare and unseen mitigation categories, while the hybrid combines this coverage with BERT detection performance on well-represented categories.

  • Takeaways & Limitations

    Hard-negative retraining improves detector precision by 0.064 but reduces recall by 0.135, increasing missed mitigation paragraphs from 60 to 161; the original detector is retained.

Abstract

from arXiv · show

Identifying and classifying environmental mitigation obligations in Federal Energy Regulatory Commission hydropower licensing documents is a labor-intensive task requiring deep domain expertise. We formulate this as a multi-label classification problem over a structured 135-category taxonomy and address the central challenge of severe label scarcity: 40 of 135 categories have no training examples, and 26 have fewer than five. A supervised Bidirectional Encoder Representations from Transformers (BERT)-based pipeline, while effective on well-represented categories, achieves F1 of zero on unseen classes regardless of augmentation strategy. We introduce a Retrieval-Augmented Generation (RAG) pipeline that conditions classification on retrieved category definitions, enabling zero-shot generalization across the full label space. We further propose a hybrid system that combines BERT detection with RAG classification, exploiting the high recall of fine-tuned detection and the zero-shot coverage of retrieval-augmented reasoning. Evaluated on the full set of 2017 license documents (5,860 paragraphs, 135 categories), the hybrid achieves a Micro F1 of 0.524, outperforming the BERT-only pipeline (0.477) and the RAG-only pipeline (0.416) across all training-support buckets.

I. INTRODUCTION

FERC hydropower licenses contain extensive, scattered environmental mitigation obligations that must be extracted across a 135-category taxonomy. Severe label scarcity motivates retrieval-augmented classification and a hybrid BERT–RAG pipeline, which achieves the strongest reported performance.

  • FERC oversees more than 2,000 non-federal hydropower facilities, whose licenses contain mitigation requirements scattered through documents often exceeding 15,000 words.
  • The task is fine-grained multi-label classification over 135 mitigation categories, and manual compilation takes trained annotators over three hours per document.
  • 40 of 135 categories have no training examples, while 26 have fewer than five, creating a severe long-tail label distribution.
  • RAG retrieves category definitions and supplies them to an LLM, enabling classification without category-specific examples across all 135 categories.
  • The hybrid pairs high-recall BERT detection with RAG classification to combine common-category performance with long-tail coverage.
  • 0.524 Micro F1 is achieved by the hybrid on 2017 license documents, exceeding BERT-only at 0.477 and RAG-only at 0.416 across every training-support bucket.

II. BACKGROUND

Prior work frames RAG as retrieval-conditioned language modeling and shows its relevance to imbalanced multi-label classification. These findings motivate combining retrieval-based long-tail coverage with encoder-based performance on frequent categories in FERC licensing documents.

  • RAG combines a retriever that selects passages from an external knowledge base with an LLM conditioned on the query and retrieved context.
  • Retrieval augmentation has improved performance on infrequent labels in legal and biomedical multi-label classification.
  • EURLEX57K established that label imbalance and zero-shot categories are endemic to legal document classification.
  • The impact of domain-specific pre-training remains an open question because FERC licensing is narrower and more specialized than broad legal corpora.
  • Patent-classification results report complementary strengths: retrieval-augmented LLMs on rare and zero-shot categories, and encoder models on frequent categories.
  • The present work extends an earlier BERT-based FERC extraction system by incorporating RAG for zero-shot and rare-category labels.

III. METHODS

The study evaluates BERT, RAG, and hybrid pipelines on expert-annotated FERC documents using a held-out 2017 benchmark. Its methods explicitly address severe training-support imbalance and metric-coverage constraints.

  • Three configurations—a BERT pipeline, a RAG pipeline, and a hybrid—are evaluated on a shared FERC hydropower licensing dataset.
  • The dataset contains paragraph-level annotations from a structured 135-class taxonomy spanning six resource groups.
  • Training and validation use 2014–2016 documents, while the held-out 2017 test set contains 5,860 paragraphs across 21 license documents.
  • 40 categories have no training examples and 26 have fewer than five, together accounting for nearly half of the taxonomy.
  • Table I summarizes training support across the 135 mitigation categories.
  • Only 53 of 135 categories appear in the 2017 test set, creating a coverage gap relevant to metric selection.

B. Models

All systems use detection followed by classification, separating whether a paragraph contains an obligation from which mitigation categories apply. The BERT pipeline implements both stages with fine-tuned BERT models.

  • Detection determines whether a paragraph contains a forward-looking environmental mitigation obligation, while classification determines its applicable categories.
  • The BERT detector is a binary sequence classifier based on bert-base-uncased, fine-tuned to identify paragraphs containing any mitigation obligation.
  • The BERT classifier uses 135 sigmoid outputs, allowing each paragraph to receive any combination of category labels.
  • A category is predicted when its sigmoid output exceeds a threshold of 0.5, with both models trained on 2014–2016 documents and evaluated on 2017.

2) RAG Pipeline:

The RAG pipeline retrieves category definitions relevant to each paragraph and uses an LLM to determine mitigation applicability and category assignments. The hybrid instead uses BERT for recall-oriented detection before applying RAG classification to candidate paragraphs.

  • RAG Pipeline:: RAG replaces both BERT stages with retrieval-augmented LLM inference over paragraph and category-definition embeddings.Retrieved definitions are used as context for determining whether a paragraph contains an obligation and which categories apply.
  • RAG Pipeline:: Five annotated examples provide task grounding, while definition-conditioned classification enables generalization to zero-shot and rare categories.The pipeline conditions classification on retrieved category definitions rather than learned label embeddings.
  • RAG Pipeline:: The hybrid combines BERT detection with RAG classification, passing BERT-positive paragraphs directly to RAG and bypassing LLM detection.BERT acts as a recall-oriented filter over the full document, while RAG handles the full label space.
  • RAG Pipeline:: BERT training uses bert-base-uncased models fine-tuned on 2014–2016 documents with a fixed multi-label threshold of 0.5.Both detector and classifier use the same pretrained initialization and are optimized with AdamW.
  • RAG Pipeline:: RAG retrieves the top k = 25 category definitions and discards paragraphs below a maximum embedding-similarity threshold of 0.46 before LLM inference.The embedding-based pre-filter serves as a fast screening step.

D. Performance Metrics

The evaluation prioritizes Micro F1 for end-to-end utility while reporting Macro F1 and support-bucket performance. Macro F1 is constrained by categories absent from the test set, and detector precision–recall trade-offs matter for downstream classification.

  • D. Performance Metrics: Micro F1 is the primary metric because it aggregates instance-level decisions across categories and reflects practical utility.Macro F1 gives equal weight to each category instead.
  • D. Performance Metrics: Macro F1 has a theoretical ceiling of 53/135 ≈0.393 because only 53 of 135 categories appear in the 2017 test set.Categories without test instances contribute an F1 of zero by definition.
  • D. Performance Metrics: The Macro F1 ceiling reflects the data split rather than a model limitation and must be interpreted accordingly.This qualification applies when comparing Macro F1 values.
  • D. Performance Metrics: Precision, recall, and F1 are reported separately for the BERT detector because their trade-off affects downstream classification performance.Detector operating characteristics influence which paragraphs reach classification.
  • D. Performance Metrics: Macro F1 is additionally stratified by zero-shot, rare, medium, and common training-support buckets over test-present categories.This isolates each system’s long-tail coverage.

IV. RESULTS

The study evaluates all three pipelines on the 2017 test set using Micro F1, Macro F1, and detector metrics. The hybrid obtains the strongest reported end-to-end Micro F1, while detector recall is high but produces false positives for downstream classification.

  • IV. RESULTS: All three pipelines are evaluated on the 2017 test set using Micro F1, Macro F1, and per-bucket Macro F1 over test-present categories.Micro F1 is the primary end-to-end metric.
  • IV. RESULTS: The hybrid achieves the highest Micro F1 of 0.524, outperforming BERT-only at 0.477 and both RAG configurations.Among RAG variants, five-shot prompting reaches 0.416 versus 0.350 for zero-shot prompting.
  • IV. RESULTS: All Macro F1 scores remain below the theoretical ceiling of 0.396, reflecting difficulty under sparse label coverage.The ceiling is imposed by the test-set category distribution.
  • IV. RESULTS: The BERT detector achieves recall of 0.920, missing 60 of 753 positive paragraphs while passing 406 false positives downstream.The reported asymmetry favors avoiding missed mitigation obligations over minimizing candidate over-generation.

C. Performance by Training Support

Performance differs sharply by training support: RAG and the hybrid extend coverage to zero-shot and rare categories, while BERT remains competitive on common categories. The hybrid outperforms both components in every reported bucket.

  • C. Performance by Training Support: The hybrid outperforms both BERT and RAG in every training-support bucket evaluated over test-present categories.Buckets are stratified by the amount of training support available for each category.
  • C. Performance by Training Support: BERT achieves F1 of zero on zero-shot and rare categories because it cannot predict labels absent from training.This limitation follows from discriminative supervised training.
  • C. Performance by Training Support: RAG covers zero-shot and rare categories through definition-driven reasoning, while the hybrid reduces the false-positive load passed to classification.The hybrid combines BERT’s detection with RAG’s long-tail coverage.

D. Augmentation Experiments

Augmentation experiments show that supervised data synthesis provides limited benefit under severe label scarcity, while RAG recovers unseen categories and complements BERT’s strengths across support levels.

  • Supervised Augmentation: BERT achieves F1 of zero on zero-shot and rare categories because it cannot predict labels absent from training.Category-description augmentation introduces noise without providing the reasoning capacity needed for unseen labels.
  • RAG Classification: 0.556 zero-shot bucket Macro F1 demonstrates that RAG can recover categories that supervised augmentation cannot.RAG conditions classification on retrieved category definitions at inference time.
  • Complementarity: 0.448 versus 0.400 common-category F1 shows BERT outperforming RAG when categories have more than 20 training examples.The hybrid combines BERT’s learned category-specific patterns with RAG’s full-label-space coverage.
  • Complementarity: The hybrid achieves the best performance in every training-support bucket by combining high-recall BERT detection with RAG classification.RAG handles zero-shot and rare categories, while BERT provides broad candidate coverage.

B. Limitations

Augmentation does not remove the pipeline’s structural constraints: classifier gains are small, detection errors limit end-to-end performance, and precision improvements can reduce coverage.

  • Classifier Augmentation: Description augmentation reduces oracle Micro F1 from 0.660 to 0.651, while co-occurrence synthesis improves oracle performance by only +0.010.In the full pipeline, the synthesis gain shrinks to +0.006 (0.483 versus 0.477) because detection errors limit downstream benefit.
  • Detection Bottleneck: 406 spurious candidates from the BERT detector dilute classifier-side gains in the full pipeline.The detection false-positive bottleneck prevents oracle improvements from transferring proportionally to end-to-end performance.
  • Structural Limits: 81 of 135 categories are absent from the 2017 test documents, and detection false positives set a ceiling on downstream classification gains.These constraints indicate that the main bottleneck is structural rather than solely data-driven.

VI. CONCLUSION

The paper concludes that a hybrid BERT–RAG pipeline enables automated mitigation-obligation classification across the label space, with strongest results on the full 2017-document benchmark. Future work targets broader annotation coverage and more principled detector improvement.

  • Conclusion: The proposed system pairs a BERT binary detector with a retrieval-augmented LLM classifier for environmental mitigation obligations.The design combines discriminative performance on represented categories with definition-driven zero-shot reasoning on the long tail.
  • Limitations: Supervised augmentation alone cannot resolve label scarcity, while hard-negative retraining improves precision at an unacceptable recall cost.The stated structural limits include 81 of 135 categories absent from the test set and irreducible detection false positives.
  • Future Work: Future work will expand annotations across more than 2,000 FERC-licensed facilities and use active learning to improve precision without sacrificing recall.Active learning would select informative paragraphs through model uncertainty for human annotation.
Loading 2608.23241v1…