Source-linked AI summary
Software Carpentry: Lessons Learned
Greg Wilson
TL;DR
Software Carpentry responds to scientists’ limited preparation for software development by teaching computational competence through concrete problems and tools. The program reports substantial post-workshop learning gains and satisfaction, while facing persistent difficulties with learner diversity, online teaching, and educational contribution. Its broader ambition is to improve scientists’ productivity and participation in changing scientific practices.
Problem
Most scientists are not taught how to write, test, debug, install, and maintain software, even as these tasks increasingly dominate computational work.
Method
The program teaches computational competence through specific scientific problems and tools, connecting scripts, pipelines, automated tests, version control, and related practices.
Results
130% improvement in test scores followed the workshop, while 85% learned what they hoped to learn and 95% would recommend it to others.
Takeaways & Limitations
The program aims not only to make scientists more productive but also to help them participate in changes transforming science.
Takeaways & Limitations
The approach remains constrained by limited educational contribution, difficulty applying lessons to researchers’ work, and low online-course completion.
Abstract
from arXiv · showhide
Over the last 15 years, Software Carpentry has evolved from a week-long training course at the US national laboratories into a worldwide volunteer effort to raise standards in scientific computing. This article explains what we have learned along the way the challenges we now face, and our plans for the future.
Introduction
Software Carpentry addresses a growing problem: scientists’ computational work is increasingly constrained by software development tasks, yet most are not taught how to perform them. The program reports substantial learning gains and satisfaction, while acknowledging that applying the lessons remains difficult for many researchers.
- Introduction: Scientists’ time to produce computational results is increasingly dominated by writing, testing, debugging, installing, and maintaining software.These software tasks increasingly matter for the “desktop majority” of scientists outside high-performance computing.
- Introduction: Most scientists are never taught how to develop software effectively, despite possible exposure to generic programming, statistics, or numerical methods.
- Introduction: 130% improvement in test scores after the workshop indicates more than a two-fold increase in measured computational understanding.
- Introduction: Participants also adopted tools and techniques considered standard practice in the software industry.
- Introduction: Many researchers still find it difficult to apply the material to their own work, and attempts to teach online have failed.
From Red to Green
Software Carpentry moved from unsuccessful long courses and fragmented institutional support toward shorter, practical workshops designed for graduate students and scalable delivery. The revised model focused on computational competence, recruited instructors from participants, and shifted costs toward host sites.
- Red Light: Week-long intensive courses became less effective as attendees’ capacity filled and learning declined during the final two days.The authors also found textbook software engineering poorly suited to scientists’ exploratory, often solo work.
- Red Light: Term-long interdisciplinary courses struggled because departments could not agree where computing belonged in already full curricula.The course consequently lacked an institutional mechanism for offering interdepartmental credit.
- Orange Light: The MOOC reached many starters but only 5-10% finished, and most completers already knew most of the material.These outcomes matched other MOOCs but still motivated a search for a scalable, sustainable model.
- Green Light: Two-day intensive workshops improved attendance and retention while narrowing content to essential tools and concepts for scientists.The shorter format also helped identify which skills learners really needed.
- Green Light: The program scaled by recruiting instructors from workshop participants and adopting a host-site-covers-costs model.This reduced dependence on one instructor and recurring funding assembled from many sources.
- What We Do: Workshops teach Unix, programming, version control, and databases as concrete routes to computational competence rather than ends in themselves.The lessons connect tools through pipelines, automated tests, and reusable functions so learners can apply the ideas afterward.
Small Things Add Up
Software Carpentry attributes its effectiveness to many small teaching and organizational practices, while experimenting with partnerships, new channels, and scalable delivery. These practices emphasize openness, adaptability, hands-on learning, and support for local needs.
- Small Things Add Up: Live coding makes instruction more convincing, responsive to “what if?” questions, and useful for lateral knowledge transfer.It also increases the burden on instructors, although most find it more fun.
- Small Things Add Up: Open lessons let users modify materials and return changes through GitHub pull requests, enabling faster and cheaper scaling and adaptation.This workflow remains unfamiliar to most educators.
- Small Things Add Up: The program supports learners through collaborative notes, minute cards, cross-platform instruction, pair programming, and local helpers.These practices provide feedback, assistance, peer clarification, and working environments tailored to learners’ needs.
- Small Things Add Up: Software Carpentry experiments with partnerships and locally adapted activities, including collaboration with the Software Sustainability Institute and a “driver’s license” for DiRAC users.The DiRAC assessment covers Unix shell skills, testing, Makefiles, and related competencies.
Instructor Training
Instructor training and workshop design aim to build computational competence, but Software Carpentry still faces learner diversity, sustainability, assessment, delivery, and teaching-practice challenges. The program uses pre-assessment and experimentation while acknowledging that several problems remain unresolved.
- Instructor Training: At least 20% of learners may be lost and a different 20% bored when instruction is paced uniformly, motivating a short pre-assessment questionnaire.The questionnaire asks about specific tasks because learners often misjudge their own knowledge.
- Instructor Training: Financial sustainability is a major problem because host-site payments support workshops but do not cover two full-time-equivalent coordinating positions.The organization is still seeking a long-term solution.
- Instructor Training: Long-term impact remains unproven because follow-up and immediate surveys have not been conducted systematically, and scientific productivity is difficult to measure.The authors are taking steps to address the assessment problem.
- Instructor Training: Online teaching attempts have had mediocre initial turnout that declined rapidly, including MOOC-style offerings, webcasts, and online office hours.The authors report the same pattern for high-profile MOOCs.
- Instructor Training: Workshop customization is valuable for local needs, but the program needs to vary content more systematically and provide more frequent, diverse, and appropriately leveled exercises.Exercises must balance realistic scientific workflows against small tasks whose failures have fewer knock-on effects.
Conclusions
Software Carpentry seeks to close unevenly distributed computing-skill gaps so scientists can participate in changing scientific practices. Its conclusion frames improved scientific productivity and broader participation as connected ambitions, while the effort continues to rely on external support.
- Conclusions: Computing skills are unevenly distributed: some scientists build data-intensive applications, while others still struggle with basic data and file-management tasks.The authors link basic computing skills to scientists’ ability to pursue possibilities their colleagues are exploring.
- Conclusions: Software Carpentry aims not only to make scientists more productive today but also to help them participate in changes transforming science.The article invites others to support this ambition.
- Conclusions: Software Carpentry is supported by a Sloan Foundation grant, and the author reports employment by the Mozilla Foundation.The article also lists several organizations that supported the project over the years.
A Pre-Assessment Questionnaire
The pre-assessment questionnaire measures learners’ backgrounds, computing experience, operating systems, disciplines, and ability to perform representative tasks. Its questions cover programming, version control, testing, SQL, command-line work, and data analysis.
- A Pre-Assessment Questionnaire: The questionnaire asks about career stage, discipline, operating system, programming frequency, programming complexity, and programming languages.It includes options spanning novice through professional and multiple scientific fields.
- A Pre-Assessment Questionnaire: Representative tasks test data analysis, GitHub changes, unit-test coverage, SQL aggregation, and command-line file searches.Responses distinguish inability from completion with documentation or search help and completion with little or no help.
- A Pre-Assessment Questionnaire: Learners report familiarity with Git, unit testing and code coverage, SQL, and the command line using graduated experience categories.The categories distinguish unfamiliarity, name recognition, prior use, and current use.