Why Big IT Systems Fail

Small IT systems usually deliver successfully. They are delivered on time and on budget. When they are delivered, they usually meet the needs of the business, they are secure, they are reliable, and they are easy to change.

Large IT systems usually do not deliver successfully. They are delivered late and over budget, if they deliver at all. If delivered, they usually fail to meet the needs of the business, they are rife with security problems, they fail frequently, and they are hard to change.

Why are we so good at delivering small IT systems and so bad at delivering large ones? The obvious answer is that large systems are more complex than small systems. But that is not the problem. 

The problem is not the fact that IT systems get more complex as they get larger. The problem is how they get more complex. Or more specifically, the rate at which they get more complex.

The complexity of an IT system increases at a rate we describe as exponential. For most modern IT systems, such as service-oriented architectures, the exponential increase is driven by the number of dependencies in the system. As the system gets bigger, the number of dependencies increase. As the number of dependencies increase, the complexity increases. But the increase in complexity is not linear, it is exponential. 

The difference between a linear increase and an exponential increase is critical. 

An example of a problem that increases linearly is a leaky faucet. Say a faucet leaks at a rate of one ounce per hour and the water is going into a clogged sink that can hold 20 ounces. After three hours, a three ounce container will empty the sink. If you don't get to the sink for ten hours, you know you need to bring a ten ounce container to empty the sink. The water leaks out at a steady rate. It doesn't start leaking faster just because more water has leaked.

But think of a forest fire. Forest fires increase at an exponential rate. Say in the first minute of the fire it has engulfed one square foot. You cannot assume that in twenty minute the fire will have engulfed twenty square feet. That is because forest fires spread exponentially; the bigger they get, the faster they spread.

The mathematics of IT complexity follow the mathematics of forest fires. Say we are building an IT system at the rate of one function per week. It will take almost one year to reach 100,000 standard complexity units (SCUs). But it only takes 10 more weeks to reach the next 100,000 SCUs. And then only 7 more weeks to reach the next 100,000 SCUs. By the end of the second year we are adding more than 30,000 SCUs per week!

Except that we won't, because this rate of complexity increase is unsustainable. Just like a forest fire will eventually burn itself out once it has consumed all possible fuel, so will an IT system. It will grow until the resources are no longer available to support the massive complexity increase. At that point, it will do what all complex systems do when they reach a level that is no longer sustainable. They collapse.

Does this mean that it is impossible to build large IT systems? No, it doesn't. It does mean that we need to figure out how to attack the complexity growth. We can't prevent the IT system from getting more complex (that is impossible), but we do need to figure out how to make the complexity increase linearly rather exponentially. 

In other words, we need to figure out how to make IT systems behave more like leaky faucets and less like forest fires.

We will email you alerts when new IT complexity related blogs or white papers are available. Subscribe <here>.

You can learn more about our IT Simplification Initiative (ITSI) <here>.

Photo of the forest fire is by the U.S. Forest Service, Region 5, made available through Creative Commons and Flickr. The photo of the faucet is by John X, also made available through Creative Commons and Flickr.

Article Alert: The CIO Perspective on IT Complexity

A new article alert by Roger Sessions

Article Name: The IT Complexity Challenge: A Trustmarque study into the state of IT complexity from the CIOs’ perspective.

Authors: None given.

Publisher: Trustmarque

Date of Release: October 2015

Available at: http://www.trustmarque.com/it-complexity-report/

Registration requirements:  Nominal registration information required.

Main Points of Article

IT Complexity is a huge challenge for most CIOs. 93% of CIO’s believe that IT complexity has increased and 66% believe that the cloud has increased complexity, more than for any other cited factor. This is especially interesting given that the cloud was touted as way of simplifying IT.

Companies are now recognizing the problems complexity causes. (“Organizations are craving simplicity.”) CIOs don’t know how to deal with IT complexity (“For CIOs, the end result... is confusion over which technologies and services will actually help them simplify their IT landscape.”) But despite the confusion about how to simplify IT, 79% of CIOs think that simplifying IT is a priority.

IT complexity is causing many problems that directly impact the business. IT complexity is a major contributor to security problems with almost all CIOs (87%) agreeing that IT security is a challenge. IT complexity also makes it difficult to respond to the business needs with almost all CIOs (89%) agreeing that simplifying IT is at odds with driving innovation

IT simplification is not a luxury, it is a necessity. But it is a necessity few CIOs are equipped to deliver. 80% believe their organizations lack the in-house skills needed to deliver projects at the speed required. The article concludes “What the modern CIO needs is to simplify the IT at their disposal, yet this is a huge challenge for IT departments to do it all on their own.”

My complements

All in all, this is a well written article that makes clear the dual points that while CIOs understand they have a problem with IT complexity, they have little idea what to do about it.

I also agree that IT complexity causes security problems and makes responding to business needs difficult. I would add to these two problems a host of others, including reliability and cost of delivery. I also agree that few CIOs are equipped to respond to the challenges of IT complexity.

Most CIOs will benefit from reading this article, if only to understand that they are not the only ones struggling with the problem of IT complexity.

My criticisms

The article gives no guidance as to how to respond to the problem of IT complexity other than to engage Trustmarque and there is no information as to how Trustmarque will help organizations achieve IT simplification.

Disclaimer

Roger Sessions has no interest, financial or otherwise, in the article discussed above. The article was chosen based solely on his judgement as to the value of the article to his readers.

Do you know about a recently released article you think Roger should cover? Or do you have information about a recent highly complex project you think he should write about? Anonymity guaranteed if desired. Drop him a note at roger@rogersessions.com.

Subscribe

Subscribe to Roger Sessions’s alerts and articles at http://rogersessions.com/library/subscribe.

This article alert is by Roger Sessions, probably the leading expert on IT complexity. Our approach to IT simplification is our IT Simplification Initiative (ITSI™). You can read about ITSI at http://rogersessions.com/itsi.

The road to IT Simplification begins with a single ITSI step.