PaperlayerView exact revision on arXiv

Source-linked AI summary

Twelve Quick Tips for Managing IT Disasters in Small Research Software Teams

Greg Wilson

arXiv:2608.27196v1cs.SE

TL;DR

Small research software teams face disruptions involving hosted services, software platforms, data storage, and the people who control credentials or operational knowledge. The paper’s guidance emphasizes planning, backups, cost assessment, institutional support, and attention to team members during crises. It also identifies unique individual knowledge and obligations around sensitive data as important boundaries on preparedness.

Abstract

from arXiv · show

In 2025, the US government launched an unprecedented series of attacks on its own scientific research groups. A year later GitHub dropped below 90% availability for the first time, while wildfires in Canada, France, Spain, and elsewhere forced researchers from the homes and labs. These events and others have reminded us just how fragile research computing systems can be, and that planning for disasters is one of the most effective ways to prevent them. This paper is a short guide to disaster planning and recovery for a small research software team. The tips assume you are doing everything yourself on top of your regular job, and that you aren't an experienced system administrator. Some of the tips do require that kind of expertise, but most research institutions have research computing groups, data librarians, and environmental health-and-safety offices whose entire job is to help with exactly these problems. This paper tells you what "done" looks like; they can often provide it.

Who You Are

Small research groups may depend on software services, published packages, and datasets whose access, hosting, credentials, or storage can fail. These dependencies include both technical systems and the people who maintain them.

Online services

  • Research groups may run online dashboards or services that become unavailable when providers suspend accounts, maintainers leave, or dependencies shut down.Source: Who You Are · p. 1

Software packages

  • Published software can become inaccessible when registry credentials expire or a hosting platform suspends the group’s account.Source: Who You Are · p. 1

Data

  • Research data can be lost or become inaccessible after hardware failure, theft without backups, or repository closure.Source: Who You Are · p. 2

Overlapping dependencies

  • These categories overlap within one group, which may simultaneously maintain software, services, and long-term datasets.Source: Who You Are · p. 2

Tip 1: Know your risks.

Risk assessment starts by inventorying every service and physical asset the team depends on, then assigning recovery targets and identifying dependencies that could fail.

Inventory

  • Inventory every service and physical asset the team depends on, recording where each item is hosted or stored.A first inventory takes about an hour, with quarterly reviews taking 15–30 minutes.Source: Tip 1: Know your risks · p. 2

Recovery targets

Single points of failure

  • Identify controllable single points of failure, including unique operational knowledge, credentials, payment methods, and dependent systems.Research support teams can help estimate recovery targets and assess failure cascades.Source: Tip 1: Know your risks · p. 2

Tip 2: Make a plan.

A disaster plan should be easy to find, stored redundantly, written in plain language, and organized into scenario-specific recovery checklists. Backups must cover code, configuration, data, and other critical assets, then be tested through restoration.

Plan structure

  • Store the shared disaster plan in at least two locations that cannot fail simultaneously, and ensure every team member knows where to find it.State declaration criteria plainly and specify each person’s first action.Source: Tip 2: Make a plan · p. 2
  • Write each scenario as a numbered checklist of specific actions, including communication, restoration, and verification steps.An online-service example sends an incident message, restores a database snapshot, and checks that the dashboard returns data correctly.Source: Tip 2: Make a plan · pp. 2–3

Team resilience

  • A lottery-factor problem occurs when only one person has the knowledge or access required for a recovery step.The paper identifies this as the hardest part of planning because the person holding unique knowledge may not recognize all of it.Source: Tip 2: Make a plan · p. 3

Backups

  • Follow the 3-2-1 rule for critical assets, combining a working copy, automated backup, and off-site deposit across different storage locations or media.The guidance applies to data, software releases, electronic lab notebooks, and configuration information.Source: Tip 3: Back up everything · p. 3
  • Back up source code and configuration separately, isolate backups from production credentials when possible, and test full restores at least annually.Restore procedures should be idempotent because real disasters may require repeated restoration.Source: Tip 3: Back up everything · p. 4

Tip 4: Communicate clearly.

Teams should prepare an out-of-band communication channel and contact information before an incident. One designated communicator should provide regular updates while others focus on recovery.

Fallback communication

  • Choose a fallback communication channel that does not depend on normal infrastructure, and record it in the disaster plan.Signal or a phone tree can substitute when Slack is unavailable.Source: Tip 4: Communicate clearly · pp. 4–5

Contact coordination

  • Maintain printed or phone-stored contact information so the team can coordinate even when shared drives are unavailable.A contact tree should specify who calls whom during an outage.Source: Tip 4: Communicate clearly · pp. 4–5

Incident updates

  • Designate one calm, reliable communicator to send regular stakeholder updates, even when there is no new resolution.Prepared messages can address the team, service users, collaborators, and the post-incident status.Source: Tip 4: Communicate clearly · pp. 4–5

Tip 5: Test the plan.

Test the recovery plan in a safe environment and use the exercise to find documentation, automation, and timing gaps. Compare restoration time with the team’s Recovery Time Objective (RTO) and adjust one when they conflict.

  • Give the recovery checklist to the newest team member and have them execute it on a harmless test copy without help.Every point where they get stuck identifies a documentation or automation gap.Source: Tip 5: Test the plan · p. 5
  • Walk through the checklist using a realistic worst-case scenario, fixing anything missing, wrong, or stale immediately.If no one has recently joined, the whole team can conduct an hour-long walkthrough.Source: Tip 5: Test the plan · p. 5
  • Compare test duration with the RTO: if database restoration takes four hours but the RTO is two hours, fix the restoration process or adjust the objective.Source: Tip 5: Test the plan · p. 5

Tip 6: Watch for trouble.

Monitor the failures and warning signs most likely to disrupt the team, including service availability, software distribution, and cloud spending. Use inexpensive monitoring and alerts where possible, and involve multiple people in budget notifications.

  • Monitor the services that would wake the team at 3:00 a.m.; no-code free tiers such as Uptime Robot and Healthchecks.io are better than nothing.Source: Tip 6: Watch for trouble · p. 5
  • Check that dataset DOIs resolve and packages remain installable, treating a sudden drop to zero monthly downloads as a possible registry failure.Source: Tip 6: Watch for trouble · p. 5
  • Check the cloud bill monthly because a misconfigured resource can create a catastrophic bill even for a small team.Source: Tip 6: Watch for trouble · p. 5
  • Set budget alerts at 50% and 90% of normal monthly spending and send them to at least two people.Most cloud providers offer free-tier budget alerting adequate for a small team.Source: Tip 6: Watch for trouble · p. 6

Tip 7: Lock down your accounts.

Reduce account and infrastructure risks with stronger authentication, shared credential management, redundant administrative access, concise data policies, timely patching, and basic physical protections. Prepare explicit procedures for offboarding and ransomware response, while consulting institutional specialists about data obligations.

  • Enable multi-factor authentication (MFA) on every digital account; authenticator apps or passkeys are preferable to SMS, though SMS is better than none.Source: Tip 7: Lock down your accounts · p. 6
  • Store every shared credential in a team password manager rather than chat or email, and keep its recovery code physically safe.Source: Tip 7: Lock down your accounts · p. 6
  • Maintain domain ownership records, enabled auto-renewal, quarterly verification, and an offboarding checklist that revokes digital access and collects physical keys.Source: Tip 7: Lock down your accounts · p. 6
  • Ensure at least two people can access critical administrative accounts, including cloud, domain, payment, and password-manager credentials.Do not tie these credentials to a single phone number or email address.Source: Tip 7: Lock down your accounts · p. 6
  • Keep policies short and explicit, consult data-protection specialists about HIPAA, GDPR, or funder obligations, and enable automatic patching.For on-premises equipment, a UPS configured for clean shutdown and battery replacement every three years can reduce power-related risk.Source: Tip 8: Protect your assets · pp. 6–7
  • For ransomware, disconnect the affected machine from the network, contact the incident lead, and consult legal counsel before negotiating or paying.Source: Tip 8: Protect your assets · p. 7

Tip 9: Stabilize, then investigate.

During an incident, stabilize operations before investigating, assign clear response and communication roles, and maintain a timestamped log. Escalate suspected provider-side problems early, then review the incident promptly to identify system causes and assign corrective action without blame.

Tip 10: Count the cost.

Disaster planning has both direct digital expenses and less visible personnel costs. Estimating the cost of downtime helps teams judge whether preventive services are worth paying for.

  • Disaster costs include backup storage, password management, renewals, managed databases, repository deposits, planning, drills, and incident-response time.Personnel costs also include quarterly reviews, annual restore tests, and cross-training.Source: Tip 10: Count the cost · p. 8
  • A day of downtime should be valued using lost experiment time, missed deadlines, and damaged collaborator trust.Source: Tip 10: Count the cost · p. 8
  • That downtime estimate informs whether to spend on services such as a $50/month managed database or a $200/year backup service.Source: Tip 10: Count the cost · p. 8
  • Teams should ask what backup, database, and monitoring services cost and which expenses the institution will cover.Source: Tip 10: Count the cost · p. 8

Tip 11: Support your team during the crisis.

During a disaster, support the team as well as the technology by prioritizing safety, calm, concrete tasks, connection, and protection from blame and exhaustion. Teams should also identify available wellbeing support, leave provisions, and performance-review policies.

Tip 12: Help your team recover after the disaster.

Recovery continues after technical restoration: teams should make space for emotional processing, recognize different recovery speeds, support affected colleagues, and acknowledge the recovery itself. Follow-up conversations and institutional support help sustain that process.

Loading 2608.27196v1…