Source-linked AI summary
Quantifying Organizational Environmental Action from Web Data and Large Language Models
Quinn Reynolds, Daniel Shore, Vianey Leos Barajas, Tanhum Yoreh, Meredith Franklin
TL;DR
Environmental actions are often recorded in unstructured web content, and filtering text before classification may remove relevant evidence. This study builds a framework to measure organizational environmental action from websites and compares retrieval-based and direct LLM classification. Direct classification provides the broadest congregation-level coverage, while retrieval approaches reduce computational demands but risk information loss.
Problem
Environmental actions are dispersed across unstructured web content, and varied terminology and website architectures complicate systematic extraction and measurement.
Source: §1 · pp. 1–2Method
Results
Direct LLM classification provided the broadest congregation-level coverage, while keyword retrieval had the lowest expert agreement and semantic retrieval improved agreement but could exclude relevant content before classification.
Source: §4 · pp. 15–16Takeaways & Limitations
The framework offers a reproducible approach for converting organizational web content into structured environmental information, while highlighting that preliminary retrieval trades computational efficiency against possible information loss.
Source: §4 · pp. 15–16Website-based measurement captures documented rather than all undertaken activity, and website content does not independently verify whether activities occurred or their environmental impact.
Source: §4.5 · p. 18
Abstract
from arXiv · showhide
Quantifying organizational environmental action from publicly available web content remains a challenging environmental data science problem because relevant information can be dispersed across multiple webpages and is primarily communicated through unstructured text. We present a scalable computational framework for transforming organizational web content into structured measures of environmental action and demonstrate the approach using Jewish congregations in the United States. We constructed a national database of 4,964 congregations by integrating multiple geospatial, knowledge-base, directory, and manually reviewed sources. Of these, 2,657 had active websites that were successfully crawled, producing a corpus of 154,454 webpages. We compared three approaches for detecting environmental actions: keyword retrieval followed by large language model (LLM) classification, semantic vector retrieval followed by LLM classification, and direct LLM classification classification without preliminary retrieval. Agreement with an expert human reviewer was lowest for keyword retrieval ($\kappa$ = 0.26), higher for semantic vector retrieval ($\kappa$ = 0.42), and similar for direct LLM classification ($\kappa$ = 0.40). Although semantic retrieval achieved the highest agreement, its retrieval recall was 0.87, indicating loss of relevant content before classification. Applied to the complete corpus, direct LLM classification identified at least one environmental action at 1,398 congregations (53%), providing greater coverage than either retrieval-based approach. These results demonstrate that preliminary retrieval can reduce computational cost but may exclude relevant information before it reaches the classifier. The framework provides a reproducible approach for extracting organization-level environmental information from unstructured web content that can be adapted to other institutions.
1 Introduction
Environmental actions are often documented in unstructured organizational web content, where terminology and site structures vary and preliminary filtering can exclude relevant information. This study develops a scalable framework and compares three LLM-based approaches using U.S. Jewish congregations.
1 Introduction
- Environmental actions are difficult to measure systematically because organizations use varied terminology, writing styles, and website architectures, while keyword searches can miss implicit or context-dependent language.Filtering before LLM classification may reduce computational demands but exclude relevant content before model evaluation.Source: §1 · p. 2
- Organizational websites offer organization-specific evidence of environmental initiatives, but systematic national-scale measurement directly from these sites has received less attention.Congregation-level evidence is dispersed across formats and rarely collected in centralized databases.Source: §1 · p. 2
- Keyword retrieval, semantic vector retrieval, and direct LLM classification are compared against expert human classification, with tradeoffs assessed across agreement, recall, cost, and congregation-level coverage.The selected approach is also used to characterize action types and framing and examine geographic and organizational variation.Source: §1 · p. 2
- The framework combines web-scale acquisition, LLM-based extraction, human validation, and structured classification to convert organizational web content into analysis-ready environmental information.It is demonstrated using Jewish congregations in the United States and is described as adaptable to other institutions.Source: §1 · pp. 2–3
2 Methodology
The methodology operationalizes organizational environmentalism as actions undertaken or encouraged by an organization and applies the framework to U.S. Jewish places of worship. It combines multi-source database construction with website crawling and archiving, subject to access and coverage constraints.
2 Methodology
- Environmentalism is operationalized through organizational actions, defined broadly as practices, programs, or activities with actual or intended environmental relevance.The unit of analysis is an individual place of worship; the empirical application focuses on U.S. Jewish congregations.Source: §2 · p. 3
2.1 Database Construction
- Five complementary sources, including geospatial data, knowledge bases, synagogue lists, directories, and Google Places, were merged and deduplicated to construct the congregation database.Adaptive quadtree subdivision of Google Places searches was used to improve coverage in densely populated areas.Source: §2.1 · p. 3
2.2 Web Crawling and Archiving
- Active congregation websites were crawled through in-domain links, with a limit of 150 pages per site, and their content was archived as a consistent snapshot for classification.The corpus retained full webpage content for subsequent alternative retrieval and classification analyses.Source: §2.2 · pp. 3–4
- Automated-access restrictions prevented successful crawling of some websites, and only 68% of congregations with an identified website were successfully reached.A slower recovery crawl recovered additional sites, but the crawler did not circumvent access controls.Source: §2.2 · p. 4
2.3 Environmental Action Classification
Environmental actions are identified and categorized by an LLM using a 76-type taxonomy, with three text-selection approaches designed to contrast reduced processing with the risk of missing actions. Expert review supplies the reference classifications for validation.
2.3 Environmental Action Classification
- Three approaches select text differently for LLM classification: keyword retrieval, vector similarity retrieval, and direct classification without a retrieval filter.The first two are intended to reduce processing cost, while direct classification makes all retrieved website content eligible for classification.Source: §2.3 · p. 4, §2.3.4 · p. 6
2.3.1 Environmental Action Taxonomy
- The action taxonomy contains 76 types organized into nine categories, including Community, Spirituality and Worship, Energy, and Environmental and Climate Justice.Identified actions are assigned to an action type and aggregated into congregation-level counts by type and category.Source: §2.3.1 · pp. 5–6
2.3.2 Keyword Retrieval and LLM Classification
- Keyword retrieval uses a dictionary of 104 words or phrases and 39 stems to locate candidate passages, which the LLM then classifies against the action taxonomy.Each candidate chunk extends up to 500 characters before and after a keyword match; keyword occurrences themselves are not treated as evidence of action.Source: §2.3.2 · p. 6
2.3.3 Vector Similarity Retrieval and LLM Classification
- Vector retrieval embeds webpages and all 76 action types, retaining pages whose maximum cosine similarity exceeds 0.25 for LLM classification.The threshold was selected during method development to balance retrieval precision and recall.Source: §2.3.3 · p. 6
2.3.4 Direct LLM Classification
- Direct classification divides extracted webpage text into consecutive chunks of approximately 500 characters and submits each chunk to the LLM without candidate retrieval.This avoids requiring predefined keywords or a similarity threshold but submits substantially more text to the model.Source: §2.3.4 · p. 6
2.3.5 Human Validation and Method Selection
- An expert reviewer’s classifications serve as the reference standard; agreement is assessed with raw percentage agreement and Cohen’s κ, while retrieval recall is assessed for filtered approaches.A random sample of 1,200 chunks was selected after power analysis; retrieval recall measures human-positive observations retained for LLM classification.Source: §2.3.5 · p. 7
2.4 Secondary Classification and Contextual Variables
The study adds denomination, environmental framing, and congressional-district voting as contextual variables for analyzing congregation-level actions. LLM classification is used when denomination cannot be determined directly and to assign framing categories.
2.4 Secondary Classification and Contextual Variables
- Congregations are grouped into nine denominational categories, with unresolved affiliations excluded from denomination-specific analyses.When an affiliation is not explicit in names, websites, or metadata, a zero-shot LLM classification infers it from available website content and is evaluated against expert classifications.Source: §2.4 · p. 7
- Identified actions are classified as explicit, implicit, or embedded, distinguishing stated environmental motivation from environmentally relevant activity without stated motivation or activity integrated into religious life.The framing classifier uses one labeled example per category and is validated against expert human classifications.Source: §2.4 · p. 8
- Congregations are linked geographically to congressional districts classified by the winner of the 2024 federal election, and framing-category action counts are compared across districts won by each party.The district measure describes local political context, not the political affiliation of individual congregations or members.Source: §2.4 · p. 8
2.5 Statistical Analysis
The analysis reduced the influence of extreme action counts and used nonparametric tests for group comparisons, with adjustments for multiple post-hoc comparisons.
2.5 Statistical Analysis
- |z| > 3 on log(1 + x) action counts identified observations for removal to reduce the influence of unusually extreme values on group comparisons.Counts were discrete and right-skewed; x is the observed action count.Source: §2.5 · p. 8
- Mann–Whitney U, Kruskal–Wallis, and Friedman tests compared independent or related groups; Dunn’s and Nemenyi post-hoc tests followed significant omnibus results.Benjamini–Hochberg adjustment controlled the false discovery rate for post-hoc p-values; tests were two-sided with significance at p < 0.05.Source: §2.5 · p. 8
- Analyses were conducted using Python 3.11.5.Source: §2.5 · p. 8
3 Results
The analysis covered 2,657 congregations with active websites and compared three detection approaches before characterizing environmental actions across categories, denominations, and framing types.
3.2 Environmental Action Classification Method Comparison and Validation
- Direct LLM classification cost $153.57 and took an estimated 4.6 hours, versus $85.80 and 2.7 hours for vector retrieval and $28.77 and 1.1 hours for keyword retrieval.Retrieval reduced computational demands by limiting content submitted to the LLM, but had lower retrieval recall.Source: §3.2 · p. 10, Table 2 · p. 10
- Direct LLM classification identified actions at 1,398 congregations versus 1,137 for vector retrieval and 1,241 for keyword retrieval; agreement was κ = 0.40, 0.42, and 0.26, respectively.Vector retrieval recall was 0.87, so some human-identified content was lost before classification.Source: §3.2 · pp. 9–10, Table 2 · p. 10
3.3 Secondary Classification Validation
- Denominational classification achieved 100% raw agreement and κ = 1 across 20 congregations; framing classification achieved 91.5% agreement and κ = 0.70 across 106 actions.Framing categories were explicit, implicit, or embedded.Source: §3.3 · p. 10, Table 3 · p. 11
3.4 Distribution of Environmental Actions
- Across 2,657 congregations, the mean was 5.1 actions per congregation, while the median was one (IQR 0–4); among active congregations, the mean was 10.0.Action counts varied geographically; among the ten largest clusters, Seattle averaged 13.7 actions per congregation and Daytona Beach 1.6.Source: §3.4 · p. 11, Figure 2 · p. 11
- Spirituality and Worship and Community were the most prevalent categories, present in 77% and 62% of environmentally active congregations, and together accounted for 80% of actions.Counts were strongly right-skewed across categories, with especially pronounced long tails for these two categories.Source: §3.4 · pp. 12–13, Figure 3 · p. 12
3.5 Environmental Action by Denominational Affiliation
- Action counts differed significantly across denominations (H(8) = 413.8, p < 0.001); Orthodox congregations had significantly lower counts than all other denominations.Chabad counts were also lower than those of several named denominations, while Jewish Renewal and Reconstructionist counts exceeded those of several other groups.Source: §3.5 · p. 13, Figure 4 · p. 13
3.6 Environmental Framing
- Mean counts were similar across explicit, implicit, and embedded framing, but category composition differed; embedded actions were dominated by Spirituality and Worship, while explicit actions had more Community actions.Explicit and embedded distributions differed significantly (p = 0.045); implicit differed from neither explicit nor embedded.Source: §3.6 · p. 14, Figure 5 · p. 14
- Explicit action means were 1.78 in Democratic-won districts and 1.05 in Republican-won districts, while implicit means were nearly identical at 1.72 and 1.71.Action count distributions differed significantly by district context for explicit (p = 0.01) and implicit actions (p < 0.001).Source: §3.6 · pp. 14–15, Figure 6 · p. 15
4 Discussion
The discussion finds substantial but heterogeneous documented environmental activity among U.S. Jewish congregations and weighs the coverage, cost, and measurement limits of extracting it from websites. It argues that web-based extraction can complement structured environmental data while requiring careful validation and scope-aware interpretation.
4 Discussion
- Direct LLM classification used more computational resources but offered the broadest congregation-level coverage; retrieval reduced costs while risking relevant-content loss.Keyword retrieval had the lowest expert agreement, while semantic retrieval improved agreement but still excluded relevant content before classification.Source: §4 · p. 15
4.1 Measuring Environmental Action from Web Data
- Environmental actions were documented across U.S. Jewish congregations, with counts strongly right-skewed and activity varying in type, geography, denomination, and framing.Most congregations had relatively few identifiable actions, while a smaller number showed substantially greater documented activity.Source: §4 · p. 15, §4.1 · p. 16
- Spirituality and Worship and Community dominated identified actions, but routine religious practices may be underrepresented when websites omit them.The observed prevalence of embedded activity is evidence of documented practice, not a complete census of religious environmental practice.Source: §4.1 · p. 16
4.2 Denominational Differences
- Environmental action counts differed across denominational groups, but these associations do not establish whether theology, resources, demographics, or other correlated factors explain the differences.Orthodox congregations generally showed fewer identified actions than most other groups; Chabad congregations had fewer than Reconstructionist, Reform, and Conservative congregations.Source: §4.2 · p. 17
4.3 Environmental Framing
- Embedded actions centered on Spirituality and Worship, explicit actions included more Community activity, and implicit actions spanned operational categories despite similar overall mean frequencies.Environmental framing appeared more related to activity type than overall quantity.Source: §4.3 · p. 17
4.4 Political Context and Environmental Action
- Explicit environmental actions were less frequent in Republican-won districts than Democratic-won districts, but district voting cannot establish congregation or member political affiliation.Differences in implicit actions were substantially smaller; congregation-level political and demographic data are needed to distinguish context from composition.Source: §4.4 · pp. 17–18
4.5 Limitations
- Website-based measures capture reported activity rather than verified occurrence, duration, or environmental impact; action counts may also reflect website architecture and communication practices.The corpus excludes congregations without accessible websites, and automated crawling reached only 68% of congregations with identified websites.Source: §4.5 · p. 18
- Classification and processing have additional limits: validation is not error-free, API constraints shaped processing time, and transfer beyond U.S. Jewish congregations requires adaptation and renewed validation.The study also proposes extensions including longitudinal collection, contextual data linkage, alternative models, and validation against surveys, interviews, or direct observation.Source: §4.5 · p. 19, §4.6 · pp. 19–20
A.1 Environmental Action Classification
The classifier assigns environmental actions to labels from a supplied list, returning N/A when a page contains no actual environmental action. The prompt illustrates a broad range of congregational practices.
A.1 Environmental Action Classification
- The classifier must return exactly one label from the supplied list, or N/A when the page contains no actual environmental action.Source: §A.1 · p. 23
- The listed practices include worship activities, community engagement, and environmental projects such as gardens and climate-focused education.Source: §A.1 · p. 23
- Examples also cover operational actions, including bike racks, energy-efficient lighting, renewable energy, and recycling.Source: §A.1 · pp. 23–25
A.2 Denomination Classification
The prompt classifies congregations by denomination, requiring one label from a list of denomination categories or Unknown.
A.2 Denomination Classification
- The task requires exactly one denomination label for each classification.Source: §A.2 · p. 26
- Unknown is included as a classification option when a denomination does not fit a listed category.Source: §A.2 · p. 26
- The listed categories include Reconstructionist, Reform, Jewish Renewal, Conservative, and Orthodox denominations.Source: §A.2 · p. 26
A.3 Framing Classification
The framing classification distinguishes actions by whether they state an environmental purpose, are embedded in religious practice, or are described operationally. The prompt establishes a priority order for assigning these labels.
A.3 Framing Classification
- A stated environmental, sustainability, climate, or nature-related purpose makes an action Explicit, including when it occurs in worship.Source: §A.3 · p. 27
- Embedded applies when an action is part of religious practice or tradition without a stated environmental reason; Implicit applies to practical descriptions lacking religious or environmental framing.Source: §A.3 · p. 27
- The decision order prioritizes Explicit first, then Embedded, and otherwise assigns Implicit.Source: §A.3 · p. 27
B.1 Environmental Action Taxonomy
The environmental action taxonomy groups congregational practices across environmental and climate justice, operations, energy, water, waste, and food-related activities.
B.1 Environmental Action Taxonomy
- The taxonomy includes environmental and climate justice activities, youth opportunities, and support for local food initiatives.Source: Table 4 · p. 28, §B.1 · pp. 29, 31
- Operations and maintenance examples include paper-reduction policies, sustainability teams, and green cleaning or maintenance practices.Source: §B.1 · p. 29
- Energy-related examples range from tracking utility use and installing efficient lighting to renewable energy systems.Source: §B.1 · p. 29
- Water-related examples include conservation measures, efficient fixtures, rainwater capture, and water-conscious landscaping.Source: §B.1 · pp. 29–30
- Waste practices include recycling, reuse, composting, and waste-reduction planning.Source: §B.1 · p. 30
- Food-related practices include local sourcing, plant-based or seasonal options, reusable dishes, and community gardens.Source: §B.1 · p. 30