Tuesday, April 6, 2010

The Failure of Success

I have written quite a bit about the cost of IT failure, most recently in a white paper. A number of people have criticized my analysis saying that just because a large project "fails" in the sense of being late, over budget, and/or not delivering expected functionality, doesn't mean that the project doesn't deliver value.

This may be true. The real question is this: had the business known how the project would have gone, would they still have agreed to do the project? If the answer to this is yes, then the project was a success. If the answer is no, then the project is a failure.

However even when the project comes in on-time, on-budget, and delivering expected functionality, the project may still not be a  success. These three metrics (budget, time, functionality) tells us little about the project itself and much more about our ability to make predictions about the project.

Take a simple example. The business tells IT they need a system that delivers 100 functions. IT calculates the cost at $100K/function. They tell the business the project will cost $10M and three years to deliver. Business approves the project. IT delivers the project one month early and $1M below budget. The business is happy. IT looks good. The project is a success!

Or is it? All this tells us is that IT did what they said they would do. It doesn't tell us whether what IT said they would do was really reasonable.

In most projects that size, complexity is a major cost factor. If IT and the business take appropriate steps to manage that complexity, the cost/function can be greatly reduced. If a project can be delivered for $9M without complexity management, then it is highly likely that it could have been delivered for $5M with complexity management.

So is this project a success or a failure? According to most pundits (such as Standish) this project is an unqualified success. In my book, it is a $4M failure. This is one more reason I say that looking at the percentage of IT successes is a meaningless statistic. What we need to look at is the percentage of IT budgets that are wasted.


Waylon said...

I've had quite a bit of debate with my fellow Systems Analysts and Project Managers about the definitions of success. If we define success as time, money, and functionality, two of those success measures are against forecasts, or predictions of success. So at some point in time, we draw a line in the sand and say "we're going to meet these requirements in this time, and for this cost". And if we don't meet these, then the project is deemed a failure. But perhaps this is really a failure in our forecasting models? I knew a project manager who came under budget by 90%, only because the original budget estimate was wildly inaccurate. I guess forecasting is a continuing area of concern for myself.

Rolf said...

your line of reasoning seems odd to me. Assume we not only apply complexity management but also magic potion X to the 10M project. Say we would use only 4M then. Is a project with complexity management a 2M (or 6M) failure?
This is has no apparent end.
IMHO, succes is a human's invention. Your definition seems as right or wrong as Standish's.

I'm puzzled.


Roger Sessions said...

I agree with Waylon. Looking blindly at budget and time line is easily manipulated.

I see Rolf's point. But if you can show a convincing argument that by using X you would reduce the cost of the project by Y, then it is true that you are wasting Y by not using X. Look at this from the other perspective. Suppose a project said they were going to code a large system in machine language. Would you not argue that doing so would be a huge waste of IT budget?

When we spend money unnecessarily (as in unmanaged complexity or poor tool choice)we are contributing to IT failure. The CEO doesn't care WHY money was wasted. She only wants to know how MUCH money was wasted. From the CEO perspective, money spent with no return is failure.

Rolf said...

>code a large system in machine language. Would you not argue that doing so would be a huge waste of IT budget?

If you ask me TODAY, I would probably say it IS a huge waste. If you had asked me 30 years ago I would have said GOOD IDEA, go for it!

This points to the notion of state-of-the-art, which is something that carries a time stamp. Therefore, success or failure also carry a time stamp.

(Does that mean that the old Standish reports are more right today ... let's not dive into that :-D)

However, there's also a hidden promise in your reasoning: if we could just wait long enough, and give the state-of-the-art enough time to develop, we would solve the now 10M problem for zero cost. And there seems to be an actual trend in that direction, while "zero" is an extreme obviously.

Roger Sessions said...

I don't think we need to make it that complex! The question is, given when the project needs to be delivered, what is the most cost effective approach to delivering it. If it a large project, then definitely complexity management is an important (even critical) strategy. But there are other useful tools as well. I'm for them all.

Rolf said...


I like the discussion. You apply complexity management to it, I apply engineering think :-)

Let's put it this way: If a project is a failure in the CEO's eyes, the CEO would ask how much money was wasted. If it was a succes in her eyes, she would not think about cost.

And, I agree, finding the most cost effective approach to a project is critical to most projects. That is, the most cost effective approach that is available right now, to the people who do the project.



Roger Sessions said...

But cost is part of the equation. Suppose your wife sends you out to buy a new Toyota Camry. You come home with one. You ask her if she is happy with your purchase. She will probably ask you how much the car costs. If you say $15,000, she will probably say you got a good deal. If you say $50,000, she will probably look at you like you are crazy. Same car. Different results.

Anonymous said...

Hi Roger,

I like your book.

But I think you conflate "project failure" with "architecture failure". If a project delivers a solution on-time, in-budget and with the required functionality it is a success - whatever the quality of the solution. If it delivers the 'wrong', overly complex, solution - that is not a 'project failure' - that is an architecture failure. And architecture failures can be very, very expensive as you say - especially when repeated over time. Enterprise Architecture is about constraining the range of possible solution sets and creating organisational processes for picking the right solution from the solution set for the project.

Roger Sessions said...

Thanks for your comments!

I see Enterprise Architecture as the discipline of minimizing complexity. Once complexity has been minimized (by appropriate partitioning), then the stage is set for the best possible solution architecture and the best possible project management. Complexity management can only occur at the EA level, because it requires understandings of both business and IT.