Source-linked AI summary
Twelve Quick Tips for Managing IT Disasters in Small Research Software Teams
Greg Wilson
TL;DR
Small research software teams face disruptions involving hosted services, software platforms, data storage, and the people who control credentials or operational knowledge. The paper’s guidance emphasizes planning, backups, cost assessment, institutional support, and attention to team members during crises. It also identifies unique individual knowledge and obligations around sensitive data as important boundaries on preparedness.
Problem
Research software teams depend on services, platforms, data stores, and individuals whose failure or unavailability can disrupt their work.
Method
The paper offers practical disaster-planning guidance centered on backups, downtime-cost assessment, institutional consultation, and support for affected team members.
Results
The paper identifies technical recovery and human support as parallel parts of responding to research-computing disasters.
Takeaways & Limitations
Teams should treat disaster preparedness as an operational responsibility that includes both system recovery and the people who depend on those systems.
Takeaways & Limitations
Preparedness is constrained by single-person knowledge or access and by possible obligations for sensitive data under institutional or funder requirements.
Abstract
from arXiv · showhide
In 2025, the US government launched an unprecedented series of attacks on its own scientific research groups. A year later GitHub dropped below 90% availability for the first time, while wildfires in Canada, France, Spain, and elsewhere forced researchers from the homes and labs. These events and others have reminded us just how fragile research computing systems can be, and that planning for disasters is one of the most effective ways to prevent them. This paper is a short guide to disaster planning and recovery for a small research software team. The tips assume you are doing everything yourself on top of your regular job, and that you aren't an experienced system administrator. Some of the tips do require that kind of expertise, but most research institutions have research computing groups, data librarians, and environmental health-and-safety offices whose entire job is to help with exactly these problems. This paper tells you what "done" looks like; they can often provide it.
Who You Are
Small research groups may depend on software services, published packages, and datasets whose access, hosting, credentials, or storage can fail. These dependencies include both technical systems and the people who maintain them.
- Online services: Research groups may run online dashboards or services that become unavailable when providers suspend accounts, maintainers leave, or dependencies shut down.
- Software packages: Published software can become inaccessible when registry credentials expire or a hosting platform suspends the group’s account.
- Data: Research data can be lost or become inaccessible after hardware failure, theft without backups, or repository closure.
- Overlapping dependencies: These categories overlap within one group, which may simultaneously maintain software, services, and long-term datasets.
Tip 1: Know your risks.
Risk assessment starts by inventorying every service and physical asset the team depends on, then assigning recovery targets and identifying dependencies that could fail.
- Inventory: Inventory every service and physical asset the team depends on, recording where each item is hosted or stored.A first inventory takes about an hour, with quarterly reviews taking 15–30 minutes.
- Recovery targets: Recovery Time Objective (RTO) measures how long the team can be without an asset before work stops.
- Recovery targets: Recovery Point Objective (RPO) measures how much recent data the team can afford to lose.
- Single points of failure: Identify controllable single points of failure, including unique operational knowledge, credentials, payment methods, and dependent systems.Research support teams can help estimate recovery targets and assess failure cascades.
Tip 2: Make a plan.
A disaster plan should be easy to find, stored redundantly, written in plain language, and organized into scenario-specific recovery checklists. Backups must cover code, configuration, data, and other critical assets, then be tested through restoration.
- Plan structure: Store the shared disaster plan in at least two locations that cannot fail simultaneously, and ensure every team member knows where to find it.State declaration criteria plainly and specify each person’s first action.
- Plan structure: Write each scenario as a numbered checklist of specific actions, including communication, restoration, and verification steps.An online-service example sends an incident message, restores a database snapshot, and checks that the dashboard returns data correctly.
- Team resilience: A lottery-factor problem occurs when only one person has the knowledge or access required for a recovery step.The paper identifies this as the hardest part of planning because the person holding unique knowledge may not recognize all of it.
- Backups: Follow the 3-2-1 rule for critical assets, combining a working copy, automated backup, and off-site deposit across different storage locations or media.The guidance applies to data, software releases, electronic lab notebooks, and configuration information.
- Backups: Back up source code and configuration separately, isolate backups from production credentials when possible, and test full restores at least annually.Restore procedures should be idempotent because real disasters may require repeated restoration.
Tip 4: Communicate clearly.
Teams should prepare an out-of-band communication channel and contact information before an incident. One designated communicator should provide regular updates while others focus on recovery.
- Fallback communication: Choose a fallback communication channel that does not depend on normal infrastructure, and record it in the disaster plan.Signal or a phone tree can substitute when Slack is unavailable.
- Contact coordination: Maintain printed or phone-stored contact information so the team can coordinate even when shared drives are unavailable.A contact tree should specify who calls whom during an outage.
- Incident updates: Designate one calm, reliable communicator to send regular stakeholder updates, even when there is no new resolution.Prepared messages can address the team, service users, collaborators, and the post-incident status.
Tip 5: Test the plan.
Test the recovery plan in a safe environment and use the exercise to find documentation, automation, and timing gaps. Compare restoration time with the team’s Recovery Time Objective (RTO) and adjust one when they conflict.
- Give the recovery checklist to the newest team member and have them execute it on a harmless test copy without help.Every point where they get stuck identifies a documentation or automation gap.
- Walk through the checklist using a realistic worst-case scenario, fixing anything missing, wrong, or stale immediately.If no one has recently joined, the whole team can conduct an hour-long walkthrough.
- Compare test duration with the RTO: if database restoration takes four hours but the RTO is two hours, fix the restoration process or adjust the objective.
Tip 6: Watch for trouble.
Monitor the failures and warning signs most likely to disrupt the team, including service availability, software distribution, and cloud spending. Use inexpensive monitoring and alerts where possible, and involve multiple people in budget notifications.
- Monitor the services that would wake the team at 3:00 a.m.; no-code free tiers such as Uptime Robot and Healthchecks.io are better than nothing.
- Check that dataset DOIs resolve and packages remain installable, treating a sudden drop to zero monthly downloads as a possible registry failure.
- Check the cloud bill monthly because a misconfigured resource can create a catastrophic bill even for a small team.
- Set budget alerts at 50% and 90% of normal monthly spending and send them to at least two people.Most cloud providers offer free-tier budget alerting adequate for a small team.
Tip 7: Lock down your accounts.
Reduce account and infrastructure risks with stronger authentication, shared credential management, redundant administrative access, concise data policies, timely patching, and basic physical protections. Prepare explicit procedures for offboarding and ransomware response, while consulting institutional specialists about data obligations.
- Enable multi-factor authentication (MFA) on every digital account; authenticator apps or passkeys are preferable to SMS, though SMS is better than none.
- Store every shared credential in a team password manager rather than chat or email, and keep its recovery code physically safe.
- Maintain domain ownership records, enabled auto-renewal, quarterly verification, and an offboarding checklist that revokes digital access and collects physical keys.
- Ensure at least two people can access critical administrative accounts, including cloud, domain, payment, and password-manager credentials.Do not tie these credentials to a single phone number or email address.
- Keep policies short and explicit, consult data-protection specialists about HIPAA, GDPR, or funder obligations, and enable automatic patching.For on-premises equipment, a UPS configured for clean shutdown and battery replacement every three years can reduce power-related risk.
- For ransomware, disconnect the affected machine from the network, contact the incident lead, and consult legal counsel before negotiating or paying.
Tip 9: Stabilize, then investigate.
During an incident, stabilize operations before investigating, assign clear response and communication roles, and maintain a timestamped log. Escalate suspected provider-side problems early, then review the incident promptly to identify system causes and assign corrective action without blame.
- Stabilize first and investigate later, with one person leading the technical response, one handling communication, and others following the leads.
- Create a shared incident log and timestamp every significant action to synchronize the team and support the post-incident review.A shared Google Doc or an out-of-band Signal thread is sufficient.
- Escalate suspected cloud-provider problems immediately by opening a support ticket, since time spent on external issues cannot be recovered.
- Conduct a short review within 48 hours to identify what in the system allowed the incident, rather than who made a mistake.
- Record one concrete corrective action, assign it to one person, and set a deadline; use the least-involved team member as review lead if necessary.
Tip 10: Count the cost.
Disaster planning has both direct digital expenses and less visible personnel costs. Estimating the cost of downtime helps teams judge whether preventive services are worth paying for.
- Disaster costs include backup storage, password management, renewals, managed databases, repository deposits, planning, drills, and incident-response time.Personnel costs also include quarterly reviews, annual restore tests, and cross-training.
- A day of downtime should be valued using lost experiment time, missed deadlines, and damaged collaborator trust.
- That downtime estimate informs whether to spend on services such as a $50/month managed database or a $200/year backup service.
- Teams should ask what backup, database, and monitoring services cost and which expenses the institution will cover.
Tip 11: Support your team during the crisis.
During a disaster, support the team as well as the technology by prioritizing safety, calm, concrete tasks, connection, and protection from blame and exhaustion. Teams should also identify available wellbeing support, leave provisions, and performance-review policies.
- Disasters shock teams and the people who depend on them, so handling human damage is as important as handling technical problems.
- During the acute phase, apply crisis-response principles: promote safety, calm the environment, build self-efficacy, and foster connectedness.
- Give each person a concrete checklist task and use fallback communication to keep people connected and useful rather than helpless.
- Protect people who may have caused the problem by assigning recovery work, postponing blame, and checking their wellbeing privately.
- Rotate responders after the acute phase because two people working four hours each make fewer mistakes than one person working eight hours straight.
- Teams should clarify access to crisis support, post-incident leave, and how crisis preparation and response enter annual performance reviews.
Tip 12: Help your team recover after the disaster.
Recovery continues after technical restoration: teams should make space for emotional processing, recognize different recovery speeds, support affected colleagues, and acknowledge the recovery itself. Follow-up conversations and institutional support help sustain that process.
- A review within 48 hours gathers facts, while a second conversation a week or two later focuses on how people are doing emotionally.
- If a team member caused the disaster, address the matter privately and prevent repeated discussion of the mistake in team meetings.
- Because people recover at different speeds, check in individually one week after the incident and again a month later.
- Teams can respond to survivors’ guilt by discussing it and, when possible, offering affected colleagues practical help such as backup space.
- A short thank-you message celebrating the recovery can support the team when future problems occur.
- Recovery planning should include longer-term counselling, institutional contacts, and capturing lessons learned for other teams.