Automated Detection and Structuring of Social Tipping Point Evidence in Climate related Documents: A Modular AI Framework

1 Introduction

• Climate-related publications grew roughly tenfold from 1991 to 2010 and continued increasing, while manual review could not keep pace with the literature.The paper identifies passage-level evidence as a particular challenge for existing document- and sentence-oriented tools.Source: §1 · p. 2
• The proposed system finds and structures social tipping point evidence passage by passage across research papers, policy reports, audiovisual content, and news.Its components can be replaced independently, and the workflow is designed for local, self-managed deployment and structured export.Source: §1 · p. 3
• The study asks whether passage-level segmentation improves STP evidence identification, iterative augmentation supports reliable classification with limited expert labels, and a modular framework supports downstream climate intelligence.These questions organize evaluation of the framework and its components.Source: §1 · p. 3
• The paper combines segmentation, classification, rewriting, and criterion-based rating in an open workflow, with targeted augmentation for low-resource classification.The system is wrapped in a configurable Streamlit interface backed by MinIO object storage.Source: §1 · pp. 3–4

2.1 Social tipping points and why they are hard to detect

• Social tipping work describes rapid social change and criteria that distinguish it from ordinary fast change, but reviews note that measured evidence is costly and text evidence remains little used as an early signal.Machine-learning work on tipping has focused on time-series early-warning signs rather than written STP evidence.Source: §2.1 · pp. 4–5

2.2 Machine learning for climate evidence synthesis

• Climate evidence systems map large document collections or classify sentence-level claims, while recent surveys and benchmarks identify detailed passage-level concept detection as an open need.Nearby work supports open, modular infrastructure because closed pipelines limit reuse.Source: §2.2 · pp. 5–6

2.3 Text splitting, domain models, and data augmentation

• Climate-domain pretraining often helps broad climate tasks, but gains can shrink or reverse on fine-grained semantic tasks; targeted augmentation instead uses error analysis to address model weak spots.Document splitting also matters for retrieval, while long-sequence modelling remains challenging.Source: §2.3 · pp. 6–7

2.4 Research gap

• Existing work supplies STP criteria and climate evidence systems, but none combines passage-level segmentation, concept classification, and criterion-based read-out in one deployable evidence-synthesis system.The cited gap spans document-level mapping, sentence-level claim detection, and the absence of passage-level concept tasks.Source: §2.4 · p. 7

3.1 System overview

• The workflow cleans text, segments it into passages, classifies each passage, then rewrites and rates detected passages against criteria.The architecture connects document ingestion to downstream retrieval systems.Source: §3.1 · p. 10, Figure 2 · p. 10
• Raw PDFs enter through a Streamlit interface and MinIO storage; DistilBERT segments passages, RoBERTa classifies them, and Mistral 7B and LLaMA 3.2 3B perform rewriting and criterion-based rating.Figure 3 describes the internal pipeline stages.Source: Figure 3 · p. 10
• MiniLM-L6-v2 embeddings and criterion ratings are stored in Milvus, where downstream systems can query them; the Streamlit application exposes configurable panels for each stage.The classifier is deployed as an ONNX export for inference.Source: §3.1 · pp. 10–11

3.2 Building the dataset

• The positive training set contains 90 hand-labelled passages from co-design activities and covers four environmental topics, with each passage marked against five qualifying factors.The factors are felt social consequences, shared awareness, shared understanding of causes and effects, demand for habit change, and demand for political action.Source: §3.2 · p. 11
• Ninety negative examples from the ClimateBERT climate-specificity database make the classifier distinguish STP passages from ordinary climate writing.The study targets few-shot, low-resource conditions because expert annotation requires substantial domain expertise and is costly.Source: §3.2 · p. 11
• The held-out benchmark began with 163 GPT-4.1-labelled passages reviewed by researchers; after excluding two LOW-confidence passages, Set A contains 72 HIGH-confidence cases and Set B contains 161 HIGH- and MEDIUM-confidence cases.Set A evaluates clear cases; Set B includes borderline cases.Source: §3.2 · pp. 11–12, Figure 4 · p. 12

3.3 Choosing the models

• RoBERTa is the primary classifier, ClimateBERT the climate-trained baseline, and the tuned RoBERTa is exported to ONNX for fast inference.The selection also considered performance with minimal tuning data and reduced bias on a small training signal.Source: §3.3 · p. 12
• Mistral 7B handles rewriting, LLaMA 3.2 3B rates criteria, and MiniLM-L6-v2 produces 384-dimensional embeddings.The smaller tuned rating model gave more consistent structured output than larger untuned models within the deployment compute budget.Source: §3.3 · pp. 12–13

3.4 Pre-processing: text extraction and cleaning

• PDF extraction keeps body text, titles, and list items, drops tables, figures, repeated page elements, and fragments, then applies cleaning and removes detected reference sections.Cleaning fixes character and spacing errors, protects named terms, and splits run-together words.Source: §3.4 · p. 13
• A transition-word rule rejoins sentences broken across PDF lines or pages, producing whole sentences for segmentation.The resulting text is described as a clean stream of whole sentences.Source: §3.4 · p. 13

3.5 Segmentation: splitting documents into passages

• Tipping-point evidence is often only one or two paragraphs within longer sections, where token-based and fixed-window methods can break it up.Such passages sit among related but ordinary climate content.Source: §3.5 · p. 13
• A DistilBERT classifier decides for each neighbouring sentence pair whether both sentences belong to the same passage, rather than relying on fixed windows or topic clusters.The resulting passages are intended to keep topics together across document layouts.Source: §3.5 · p. 13
• DistilBERT boundary finding respects topic edges and assigns all 13 parts, while BERTopic leaves parts out and fixed windows cut across topics.The figure uses fill colour to show each part’s true topic.Source: Figure 5 · p. 14

3.6.1 Training setup

• Classifier versions were trained for three epochs, and the final model was selected using tipping-point F1, Cohen’s kappa, and steadiness across confidence levels, valuing class balance over majority-class accuracy.Cross-validation was used to check overfitting on the small labelled set; the selected classifier was exported to ONNX.Source: §3.6.1 · p. 14

3.6.2 Targeted augmentation

• Across four augmentation rounds, synthetic examples targeted known classification errors while the training set retained a 70:30 real-to-synthetic mix.Each synthetic stream received a stratified hand check of at least 10% before inclusion.Source: §3.6.2 · p. 14, Figure 6 · p. 15
• Round 1 added 90 synthetic examples—45 short positives and 45 long negatives—to counter length bias; Round 2 added 30 synthetic examples to distinguish fast tipping-point onset from slow long-run climate processes.Source: §3.6.2 · p. 14
• Round 3’s 65 examples made the model over-cautious and were excluded, while Round 4 added ten negative reference-list examples that early tests had misclassified.Round 3 targeted confusion between theoretical discussions and documented tipping points, which use similar vocabulary.Source: §3.6.2 · p. 15

3.7.1 Rewriting passages for clarity

• Rewriting addresses fragmented, dense passages by making each a stand-alone note while retaining its facts and meaning.The Mistral 7B prompt permits sentence reordering for clarity but instructs the model to add no source-absent information.Source: §3.7.1 · p. 15, §3.7 · p. 15

3.7.2 Rating the five criteria

• LLaMA 3.2 3B rates each rewritten passage against five criteria: social consequences, shared awareness, causal understanding, habit change, and political action.The model names only criteria supported by the text and returns a confidence level and short evidence note.Source: §3.7.2 · pp. 15–16

3.7 Post-processing: rewriting and rating

• Each criterion receives a rating of Strong, Moderate, Weak, or Not evident, and the rewriting and rating steps together form a structured tipping-point summary.The stated intended uses are expert review and policy use.Source: §3.7 · p. 15, §3.7.2 · p. 16

3.8 User interface and deployment

• Each pipeline stage has its own configuration panel, covering storage, chunking, boundary threshold, classification, rephrasing, criterion rating, vector storage, and processing parameters.The splitter threshold controls whether neighbouring sentences remain in the same passage; model panels expose selectable models or endpoints.Source: §3.8 · pp. 16–17, Figure 7 · p. 16
• The interface connects to MinIO, segments and classifies passages, rates flagged passages, previews results, and exports selected fields as JSON, CSV, or Excel.Detected passages, rewrites, and criterion ratings are also embedded in Milvus for retrieval-augmented systems.Source: §3.8 · p. 17
• After one run, 89 of 233 segmented passages were detected as social tipping points (38.2%), and all 89 were rephrased and rated.Of detected passages, 68.5% had no quality flag, 18% were flagged for poor qualifying factors, and 13.5% for overlong rephrasing.Source: Figure 8 · p. 17
• The four model components can be replaced independently, while shared object and vector stores let downstream consumers use stored outputs without rerunning upstream stages.The stated downstream examples include retrieval-augmented question answering and policy-facing dashboards.Source: §3.8 · p. 18

3.9 Performance measurement

• Nine normalized measures are summed into one segmentation score, spanning clustering quality, within-passage coherence, coverage, semantic density, size balance, and page continuity.The measures include silhouette, Calinski–Harabasz, inverted Davies–Bouldin, within-passage cosine similarity, topic coherence, coverage, semantic density, inverted Gini, and page continuity.Source: §3.9 · pp. 18–19
• Segmentation criteria receive equal weight because prior work does not establish their relative importance, and a method should perform consistently across multiple quality aspects.All component scores are normalized to [0,1], except coverage, which can slightly exceed 1.0 under assignment-reward scaling.Source: §3.9 · p. 18
• Classification is measured with accuracy, precision, recall, per-class F1, macro and weighted F1, and Cohen’s kappa.Macro F1 averages class F1 scores equally, weighted F1 weights them by class support, and kappa adjusts observed agreement for chance agreement.Source: §3.9 · p. 19
• Kappa is the main measure for comparing augmentation rounds because the test sets contain more examples of one class than the other.It measures model-label agreement after removing agreement expected by chance.Source: §3.9 · p. 19
• Rewriting is evaluated by source-to-rewrite embedding cosine similarity, word-overlap Jaccard similarity, and lexical richness as type-token ratio.High cosine similarity together with low Jaccard similarity marks a real rewrite rather than a near-copy.Source: §3.9 · pp. 19–20

4.2 Classification performance

• Tuned RoBERTa reached 71.4% accuracy and Cohen’s kappa 0.337 on Set B, and 87.5% accuracy and kappa 0.742 on Set A, leading balanced detection.On Set B, kappa rose overall from 0.284 to 0.337 but dipped at V0.3; the later rounds traded some clear-case performance for hard-case gains.Source: §4.2 · pp. 21, 23, Table 3 · p. 22, Table 4 · p. 22

4.1 Document chunking performance

• 6.137 was the highest combined nine-metric score for DistilBERT boundary splitting, ahead of BERTopic (5.609), semantic similarity (4.457), and fixed windows (2.367).DistilBERT was selected on the joint coherence-and-coverage comparison; its 27 passages averaged 2.9 document parts, and all 149 parts were assigned.Source: §4.1 · p. 20, Table 2 · p. 20, Figure 9

4.3 Checking against human experts

• Against human agreement, GPT-4 reached 90.0% accuracy and tipping-point F1 0.84, versus RoBERTa’s 80.0% accuracy and tipping-point F1 0.69.On five passages where RoBERTa was unsure, human raters agreed with RoBERTa on three, while GPT-4 was sure on all five.Source: Table 5 · p. 24, §4.3 · p. 24

4.4 Rewriting and rating results

• Rewrites averaged a length ratio of 1.15; 6 of 57 (10.5%) crossed the 1.5-times-length flag, while cosine similarity averaged 0.84 and Jaccard similarity 0.28.Lexical richness remained at a mean of 0.66 for both source and rewritten passages.Source: §4.4 · pp. 24, 26, 27, Table 6 · p. 26, Figure 14 · p. 26
• Shared awareness (94.7%) and understanding (93.0%) were rated Strong or Moderate most often, while habit-change (49.1%) and political-action demands (45.6%) appeared in fewer than half.The authors relate this pattern to documents more often describing problems and causes than advocating particular responses.Source: Figure 12 · p. 25, §4.4 · p. 25
• 94.7% of passages cleared the two-criterion rating bar; combined length and low-rating flags placed 15.8% of detected passages in the expert-review queue.The 5.3% below the bar were flagged as likely false positives from the sorter.Source: Figure 13 · p. 26, §4.4 · p. 26

5.2 Why the splitting method matters

• DistilBERT splitting balanced coherence and coverage, whereas tight similarity omitted 39% of documents and BERTopic mixed tipping-point with ordinary content.The fixed-window method also performed worst; DistilBERT scored 6.137 against the best rival’s 5.609.Source: §5.2 · pp. 27–28, §5.1 · p. 27

5.3 Why a general model beats a climate-specific one

• ClimateBERT’s climate training did not help this narrow task: it predicted positives with recall 1.00 and precision 0.31, while general RoBERTa achieved better balance.GPT-4 scored 90% against expert agreement, compared with 80% for improved RoBERTa; the discussion frames the tuned open model as workable at collection scale and GPT-4 as a second-pass check.Source: §5.3 · p. 28

5.4 Uneven Gain Progression Across Augmentation Rounds

• Four targeted augmentation rounds raised Set B kappa from 0.284 to 0.337, but the third round dipped to 0.270 as a temporal-confusion fix affected class balance.On Set A, V0.2 peaked before later hard-case rounds gave back ground on clear cases; tracking kappa can reveal shifts accuracy alone hides.Source: §5.4 · pp. 28–29

5.5 Implications for applied pipeline design

• The framework’s modular components can be replaced independently, while its rating step routes 15.8% of detected passages to human review and can be updated as definitions evolve.For large corpora, the discussion favors an open tuned encoder for first-pass detection and GPT-4-scale review for harder cases; outputs should be read against the fact that documents may understate demand for action.Source: §5.5 · pp. 29–30
• The evidence record reflects what processed documents state: shared awareness and understanding appear in over 90% of detected passages, while demand for action appears in fewer than half.The authors caution against interpreting absent stated demand as proof that no demand exists.Source: §5.5 · p. 30

5.6 Limitations and future research

• The positive training set contains only 90 examples from a few domains, and the expanded data relies substantially on synthetic examples; transfer to unseen domains and real-world narrative diversity may be limited.Test labels came from GPT-4.1, with only a 51-passage human check; rewriting and rating were not validated against expert-written summaries, and readability and end-user usefulness remain unevaluated.Source: §5.6 · pp. 30–31
• The discussion proposes expert labeling and GPT-4.1–human agreement measurement to strengthen the benchmark and reduce reliance on synthetic data.The conclusion also describes the framework as a deployed workflow combining segmentation, classification, rewriting, criterion rating, and retrieval.Source: §5.6 · p. 31, §5.7 · p. 31

5.7 Conclusion

• 6.137 was the boundary classifier’s highest composite score among four segmentation methods; tuned RoBERTa reached 71.4 percent accuracy and kappa 0.337 on the full benchmark, and 87.5 percent accuracy and kappa 0.742 on clear cases.The classifier outperformed ClimateBERT and untuned LLaMA 3.2 3B; the full benchmark contains 163 passages.Source: §5.7 · p. 31

• Funding details are on a separate title page, while consent for publication and materials availability are listed as not applicable.Source: Statements and Declarations · pp. 31–32
• The authors report no relevant competing interests; ethics approval and consent to participate are not applicable because the study analyzes published text and expert annotations without human participants or identifiable information.Source: Statements and Declarations · p. 32
• The STP dataset is awaiting formal publication and will be made available after that process is complete.Source: Statements and Declarations · p. 32
• The study code is not publicly available.Source: Statements and Declarations · p. 32
• GPT-4.1 generated benchmark labels, GPT-4 and Claude Sonnet produced augmentation batches, and Mistral 7B and LLaMA 3.2 3B handled rewriting and criterion rating.Source: Statements and Declarations · p. 32