Source-linked AI summary
AI-Ready Research Workflows in Computational Social Science: Lessons on Building a Shared Language for Interdisciplinary Collaboration
Joan Giner-Miguelez, Alexandra Málaga, Felipe Gómez-Cortés, Adrian Carrascosa, Mariona Coll-Ardanuy, Andrés F. Castro-Torres, Raül Sirvent, Rosa M. Badia, Clara Guasch, Mercè Crosas
TL;DR
SSH lacks practical, scalable workflows for deploying AI on large scholarly datasets. This paper reports a two-year co-design effort that found shared vocabulary, adapted collaboration practices, and model-driven engineering were central to bridging domain research and HPC engineering.
Problem
SSH lacks practical frameworks for deploying large-scale AI, leaving teams reliant on ad hoc pipelines amid computational, querying, and validation bottlenecks.
Method
The authors designed a shared OpenAlex workflow through adapted Agile collaboration, co-design, and model-driven engineering for STS researchers.
Results
The main challenge was translating domain questions into engineering requirements; co-designed models created shared vocabulary while abstracting HPC complexity.
Takeaways & Limitations
Co-designed, model-driven vocabularies can support interdisciplinary AI workflow building beyond the motivating STS use case.
Takeaways & Limitations
Trustworthy, reproducible, and reusable results were only partially achieved because validation was under-resourced, reproducibility partly automated, and metadata largely manual.
Abstract
from arXiv · showhide
Artificial intelligence (AI) is gaining traction in the social sciences and humanities (SSH). However, adoption remains limited by technical barriers to high-performance computing (HPC), validation processes that lag behind AI's rapid progress, and reproducibility standards that most SSH teams cannot meet. Research workflows--common in the life sciences--address these problems via encoding and abstracting technical complexity into repeatable routines; yet, accounts of how to build them in SSH remain scarce. We report on a two-year effort to build a workflow that enables a Science and Technology Studies unit to query, analyze, and enrich OpenAlex--a database of some 460 million scholarly records--on the MareNostrum supercomputer, using methods ranging from large-scale bibliometrics to LLM-based classification. We found the main challenge was translating domain-specific research questions into engineering requirements -- bridging two distinct methodological languages, with implications that were both organizational and technical. Organizationally, it meant adopting and adapting Agile to the research rhythm and pace, and reframing collaboration from a service arrangement to a co-design process. Technically, model-driven engineering was as valuable for collaboration as it was for automation; co-building the model facilitated both the creation of a shared vocabulary and the abstraction of HPC complexity. Finally, we highlight limitations we found in validation, reproducibility, and FAIR metadata -- beyond what any single project can sustain -- calling for coordinated, cross-institutional investment in the tooling and standards needed for AI-ready SSH workflows sustainable at scale.
1. Introduction
AI and expanding data access are creating new possibilities for SSH, but adoption remains constrained by HPC barriers, methodological risks, and the lack of scalable, shared workflows. This paper reports a roughly two-year effort to build an organizationally and technically integrated workflow for querying, analyzing, and enriching OpenAlex at scale.
- Introduction: AI and increased data availability are opening new research possibilities across the social sciences and humanities.Examples include large-scale analysis of social-network behavior, long-term cultural and semantic shifts, and archaeological-site detection from imagery and maps.
- Introduction: Integrating modern AI into SSH remains challenging because state-of-the-art models require intensive computation and many researchers face substantial technical barriers.The introduction identifies HPC tooling and technical culture as poorly aligned with social scientists and humanists.
- Introduction: Methodological failures, including flawed train-test splits and overinterpreted results, have become more common as AI reaches researchers unfamiliar with machine learning and data science principles.The passage presents these risks alongside the broader engineering bottlenecks surrounding AI adoption.
- Introduction: SSH lacks community-wide frameworks for deploying large-scale AI, leaving even reproducible tools dependent on ad hoc, localized pipelines for HPC-scale LLM classification.The passage contrasts SSH with workflow initiatives in the life sciences and highlights the difficulty of processing hundreds of millions of records.
- Introduction: The paper reports a roughly two-year effort to build a shared workflow that combines organizational practices and technical automation for scalable OpenAlex research.The workflow shields researchers from low-level supercomputing configurations while enabling them to query, analyze, and enrich the corpus at scale.
2. Context and running case
The paper’s running case is a two-year collaboration between STS and Methods and Engineering to build reusable OpenAlex infrastructure, culminating in large-scale classification of scholarly works. The lab’s cross-group organization and dual mandate shaped both its collaborative practices and its automation approach.
- Lab organization: The lab comprises Computational Social Science, Computational Humanities, and Methods and Engineering for SSH, with STS located in Computational Social Science.Methods and Engineering connects domain-focused groups to HPC infrastructure.
- Lab organization: The collaboration aimed both to deliver studies and leave reusable infrastructure for subsequent research, shaping team organization and work automation.This dual mandate is reflected in the paper’s discussion of collaboration and automation.
- Shared infrastructure: Over roughly two years, STS and Methods and Engineering built and iterated shared infrastructure over OpenAlex, an open catalog of 460 million scholarly records, including 60 million with full text.The infrastructure supported multiple STS studies.
- Running case: The running study classified around 220 million works using LLM-based classification and dictionary tagging, totaling some 0.9 billion model inferences.The study examined the prevalence of AI methods across scientific fields and world regions.
- Running case: At this scale, engineering choices became research choices, including validated model-size reduction and parallelized inference before full execution.The model-size reduction was validated against a stratified, human-evaluated sample.
3. Organising the collaboration
Organizing interdisciplinary AI-ready research required building a shared contact language and discussion spaces across the domain–engineering spectrum. The team adapted Agile, translated research needs into technical requirements, and treated uncertain methodological work as co-design rather than service delivery.
- Shared language: The central organizational challenge was building a shared “contact language” and discussion spaces that could unite domain and engineering team members.This framing followed earlier interdisciplinary work and guided the section’s organizational takeaways.
- Adopting and adapting Agile to research: Agile provided sprints, a shared JIRA backlog, and sprint reviews that imposed a rhythm and vocabulary for surfacing misunderstandings early.Its value lay less in delivery machinery than in creating recurring occasions to identify problems before they compounded.
- Adopting and adapting Agile to research: Agile could not be imported wholesale: domain scientists had to reshape messy, non-linear inquiry into committed deliverables and user stories, creating significant extra workload.The passage also cautions that Agile cannot be applied at scale without adaptation.
- From service to co-design approach: The collaboration shifted between co-design and service, with the appropriate balance depending on task uncertainty.Routine plumbing could be delegated, but when the method was an open question, engineering decisions became research decisions and required co-design.
4. Automating the workflow: a co-designed, model-driven vocabulary
The workflow uses model-driven engineering to co-design a shared vocabulary that lets domain scientists run corpus-scale analyses while encapsulating HPC and AI machinery. Built incrementally with the STS team, it organizes reusable data artifacts, domain actions, configuration axes, and execution concerns, with future work toward a domain-specific language.
- Co-designed vocabulary: The pipeline packages parallel execution, large-scale data management, and AI model deployment so domain scientists can perform corpus-scale analyses without handling the underlying machinery.Its shared conceptual space captures recurring elements of a use case while reusable automation handles implementation concerns.
- Co-designed vocabulary: Model-driven engineering made metamodel construction a co-design process that turned negotiations over workflow concepts and relations into a shared technical vocabulary.The approach served communication and organizational coordination as well as technical system design.
- Workflow vocabulary: Over two years, the team distilled recurring operations and requests into four reusable concepts: incremental data artifacts, domain actions, configuration axes, and execution concerns.These concepts emerged through pipeline development alongside the STS team and retrospective meetings, rather than being designed up front.
- Workflow vocabulary: Layered data artifacts separate raw corpora, enhanced feature outputs, and research-specific insights, with independent versioning that enables reuse and traceability to exact producing versions.Expensive lower layers can be computed once and reused across analyses.
- Workflow vocabulary: Domain actions expose operations such as feature building, country aggregation, corpus compaction, and dictionary tagging as single commands, while reusable launchers hide environments, resource requests, parallelization, and model deployment.Launchers submit jobs, distribute work across nodes with COMPSs, and consolidate per-task logs into reproducibility records.
- Future development: The proposed next step is a domain-specific language with code generation, enabling project models to evolve faster and more safely while supporting low-code and visual interfaces.This direction is intended to make research workflows more accessible.
5. Beyond results: trustworthy and reusable outputs
The project achieved only partial success in producing trustworthy, reproducible, and reusable results. Validation, reproducibility, and metadata each exposed limitations that exceeded what a single project could address.
- Beyond results: trustworthy and reusable outputs: Validation, reproducibility, and metadata limitations were largely beyond what a single project could close, requiring institutional, cross-institutional, and community-wide efforts.The section presents each limitation as a takeaway for future projects and broader investments in shared capabilities.
- Institutional infrastructure to power human validations: Validation was under-resourced despite being central to scientific work, especially when LLMs are applied to SSH tasks without suitable benchmarks.The project used LLMs to classify scientific abstracts, illustrating the challenge of evaluating models on tasks they were not previously assessed for.
- Composable reproducibility artifacts to match research processes: Reproducibility was only partially automated because domain tasks span several actions whose records must be composed into hierarchical RO-Crates.COMPSs can emit a reproducibility capsule for a single action, but representing a whole domain task requires composing action-level records.
- Shared tooling for findable, reusable metadata: Metadata progress was thinnest because Dataverse deployment was incomplete, DDI-CDI integration required non-generalizable custom work, and Croissant support was limited.The intended metadata stack combined Croissant for ML discovery and reuse with DDI-CDI 1 for social-science infrastructure.
6. Conclusions
The two-year effort to build an AI-ready STS research workflow revealed organizational and technical nuances in creating a shared vocabulary and collaborative spaces for interdisciplinary AI research. Bridging the gap required adapting Agile to research rhythms and moving collaboration toward co-design.
- Conclusions: The workflow effort exposed organizational and technical challenges in building a shared vocabulary for interdisciplinary AI research.These lessons emerged from two years of work with an STS unit.
- Conclusions: Adapting Agile to the rhythm of research helped bridge interdisciplinary collaboration gaps.The conclusion identifies this as one of three organizational measures that supported the effort.
- Conclusions: The collaboration shifted between service and co-design, with the team pushing it back toward co-design.Co-design was treated as a necessary direction for interdisciplinary workflow development.