Source-linked AI summary
Why Your IT Project Might Be Riskier Than You Think
Bent Flyvbjerg, Alexander Budzier
TL;DR
The article examines IT project budget overruns and the risk that averages obscure extreme outcomes. It finds that one in six studied projects was a black swan, with average cost overruns of 200% and schedule overruns of almost 70%, highlighting outliers as the central IT risk.
Problem
Managers and consultants have focused on averages instead of more damaging outliers, missing the real problem in IT projects.
Method
The article analyzes IT project budget overruns to reveal whether the distribution contains a large number of extreme overages.
Results
Fully one in six of the studied projects was a black swan, with a 200% average cost overrun and an almost 70% schedule overrun.
Takeaways & Limitations
The true pitfall of IT change initiatives is a disproportionate number of black swans rather than especially high average cost overruns.
Takeaways & Limitations
Readiness planning considers whether a company can absorb a 400% or larger budget overrun while realizing only 25% to 50% of projected benefits.
Abstract
from arXiv · showhide
Out-of-control information technology (IT) projects have ended the careers of top managers, such as EADS CEO Noel Forgeard and Levi Strauss' CIO David Bergen. Moreover, IT projects have brought down whole companies, like Kmart in the US and Auto Windscreen in the UK. Software and other IT is now such an integral part of most business processes and products that CEOs must know their IT risks, which are typically substantial and overlooked. The analysis of a sample of 1,471 IT projects showed that the average cost overrun was 27% - but that figure masks a far more alarming 'fat tail' risk. Fully one in six of the projects in the sample was a Black Swan, with a cost overrun of 200%, on average, and a schedule overrun of almost 70%. This highlights the true pitfall of IT change initiatives: It's not that they're particularly prone to high cost overruns on average - it is that there are a disproportionate number of Black Swans. By focusing on averages instead of the more damaging outliers, most managers and consultants have been missing the real risk in doing IT. In conclusion, the article outlines ideas as to what can be done to avoid Black Swans.
Why Your IT Project May Be Riskier Than You Think
Large IT projects can expose organizations to severe operational, financial, and leadership consequences, even when their initial risks appear small. A global study of 1,471 projects compared planned budgets and benefits with actual costs and results.
- Levi Strauss: $5 million project led to an almost $200 million loss at Levi Strauss, forcing CIO David Bergen to resign.The project also disrupted order fulfillment and required the company to close its three U.S. distribution centers for a week.
- Why the risk matters: IT projects pose substantial organizational risk because they have become large and embedded across many aspects of operations.The paper argues that CEOs overseeing significant IT projects should be acutely aware of these risks.
- Organizational consequences: IT projects can cost top managers their jobs, sink corporations, and create economic losses extending beyond the sponsoring organization.The paper cites EADS CEO Noël Forgeard, Kmart, Auto Windscreens, and Hong Kong’s airport problems as examples.
- Study scope: 1,471 projects spanning enterprise resource planning, management information, and customer relationship management systems formed the study sample.Researchers compared budgets and estimated performance benefits with actual costs and results; 92% of projects were from public agencies and 83% were U.S.-based.
The True IT Pitfall
The study’s central finding is that IT projects have a dangerous fat tail: a disproportionate share produces massive overruns. Average overruns therefore understate the risk posed by Black Swans.
- The True IT Pitfall: 27% was the average cost overrun, but the projects’ budget-overrun distribution contained a large number of gigantic overages.The authors describe this pattern as a “fat tail.”
- The True IT Pitfall: One in six projects was a Black Swan, with a 200% average cost overrun and an almost 70% schedule overrun.These figures identify the frequency and magnitude of the extreme outcomes in the sample.
- The True IT Pitfall: IT change initiatives are not unusually prone to high average cost overruns; their distinctive danger is the disproportionate number of massive overages.The authors say managers and consultants have missed this risk by focusing on averages instead of damaging outliers.
- Examples: $100 million worth of candy could not be shipped by Hershey in time for Halloween, contributing to an 18.6% drop in quarterly earnings.The example illustrates how a flawed technology project can produce operational and financial consequences.
- Examples: $1.4 billion and $600 million IT projects contributed to Kmart’s bankruptcy after the modernization effort and supply-chain software update went off the rails.Kmart later merged with Sears Holdings, shedding more than 600 stores and 67,000 employees.
- Examples: $10 billion in lost revenue was attributed in one estimate to Germany’s Toll Collect technology project.Developers struggled to combine different software systems while implementing the toll-collection technology.
Avoiding Black Swans
The paper recommends assessing whether a company can absorb extreme IT-project failures before proceeding, then reducing exposure through smaller projects, contingency planning, and reference-class forecasting.
- Avoiding Black Swans: 15% of medium-sized technology projects exceeding cost estimates by 200% is a second stress-test scenario.The paper emphasizes that these projects may be secondary initiatives that receive less executive attention.
- Avoiding Black Swans: The stress-test scenarios apply with uncomfortable frequency according to the research.The paper presents them as risks leaders should consider rather than as comfortably improbable events.
- Avoiding Black Swans: Breaking large projects into initiatives with limited size, complexity, and duration is recommended for reducing Black Swan exposure.The authors also recommend recognizing unavoidable risks and making contingency plans.
- Avoiding Black Swans: Reference-class forecasting uses outcomes from similar projects in other organizations to improve forecasting.The method is associated with the work of Daniel Kahneman and Amos Tversky and is mandatory for big public projects in the UK and Denmark.
<INSET> Success Story: How One Company Nailed a Tricky IT Project
Emirates Bank expanded a complex core-banking project after a merger but contained its risks through disciplined planning, scope control, modularization, and focused execution. The project finished with limited overruns despite doubling in size.
- Success Story: Managers framed the initiative as a business endeavor and measured activities against a single target: readiness to go live.The objectives were to avoid mission creep and go live as soon as possible.
- Success Story: 18 months was the deadline for a system serving both banks after Emirates Bank’s merger with Bank of Dubai.The project also required a “big bang” rollout across branches, ATMs, online banking, and call centers.
- Success Story: 7% schedule slippage and 18% cost growth were recorded when the project was completed in November 2009.The merger had doubled the project’s size.
- Success Story: Project leaders resisted scope changes, divided the work into discrete modules, assembled the right team, and prevented team-member turnover.These were among the key steps identified in the case.