Thursday, October 1, 2009

Why I Focus On Complexity

When it comes to IT failure, there is no lack of "the usual suspects". We can look to failures in project definition, project management, and needs assessment. We can point to flawed architectures, implementations, and testing procedures. We can focus on communications failures between the business and IT, between IT and the user community, and between different business units.

Yet given this extensive collection of failure factors any of which can doom an IT project, why do I focus almost exclusively on the issue of complexity?

I see all of the failure factors as falling into one or more of three categories:

1. The factor is caused by complexity.
2. The factor is greatly exacerbated by complexity.
3. The factor is solved as a side effect of solving the complexity problem.

Examples of failure factors that are directly caused by complexity are the various technical failure factors, such as poor security or scalability. It is very difficult to make a complex system secure or scalable. Solve the problem of complexity, and these problems become much easier to solve.

Examples of failure factors that are greatly exacerbated by complexity include those related to communications. As a project increases in complexity, people tend to fall more and more into specialized jargon which makes communications more difficult and adds yet more complexity to the already complex project. Different groups tend to see each other as the enemy. As a side effect of learning to solve complexity, the groups learn to see not each other as the enemy, but complexity as their common enemy. Their common language becomes the language of simplification.

Examples of factors that are solved as a side effect of solving the complexity problem include those related to organization. For example, a well known failure factor is the lack of executive sponsorship. But it is difficult to find sponsors for large complex expensive projects. Once a project is broken down into small, simpler, less expensive projects, finding executive sponsors for those projects is much easier.

The other reason I focus so much on complexity is that of all these failure factors, complexity is the only one that is universally present in all failed projects. The fact is that we are quite good at doing simple projects. Our skills just don't scale up to complex projects. So we can either tackle the failure factors piecemeal and try to figure out how to scale each up to higher levels of complexity, or we can try to figure out how to scale down the complex projects into simple projects that we already know how to solve.

So while I pay close attention to all of these failure factors, I continue to believe that the one that deserves our undivided attention is the problem of complexity.


Jon H Ayre said...

Good blog and it is true that once complexity creeps into a project (IT or otherwise) it reaches a critical mass where failure is inevitable. However, for me the solution is not to endeavour to avoid complexity. Complexity is the symptom not the cause. The primary cause of all project failure is the wrong people in the wrong jobs (bums on seats). This is especially rife in IT (as IT skills are hard to find/assess) and in IT lack of ability results in the development of complex solutions. (It takes intelligence and experience to find simple solutions to apprantly complex problems).

The real solution: Hire the right people, and allow no weak elements. Complexity will not arise in the first place and there will be no need to "focus" on it and remove it.

The Enterprising Architect

Unknown said...

Complexity is inherent in every problem. If that was not the case, then it would be trivial to solve, and it wouldn't really be called a problem.

I agree that people are part of the solution; but they are not the root cause. The root cause is a lack of knowledge on how to deal with complexity. Complexity is always going to be there, but it must be reduced as much as possible, and the complexity that just can't be reduced needs to managed.

People need to learn to manage complexity, and live with complexity.

People with poor problem solving techniques, always generate more complexity. Multiply by a number of weak links and you have got yourself a BIG problem.

Alex Espinoza

Jurgen Appelo said...

I agree with Alex.

Complexity is a given. You cannot remove it. It will always be there. Put two people together, give them one small problem to solve, and the system will already be a complex one.

I am using 'complexity' in the scientific sense here.

I think you actually mean complicatedness. I would agree that things should not be needlessly complicated. The simpler the better.

But no matter how simple the project, it will always be a complex system.

Ashley Moran said...

You could then argue that the problem is not the inherent complexity of the systems, but the inherent complexity of simplifying systems.

I don't believe that complexity should ever exist in systems. Humans are not good at dealing with complexity, it's why we invent abstractions. If we must live with complexity at some point, that will be the limit of our knowledge.

I've found BDD/TDD useful for identifying this, as while it's easy to write a complex system, it's very hard to describe one precisely. If a system can be described simply at the appropriate subsystem levels, there's a good chance it has been designed simply.

Jurgen Appelo said...

Again, Ashley is talking about things being complicated, not complex. It's a pity people always always mix up the terms.

See: Simple vs Complicated vs Complex vs Chaotic

Roger Sessions said...

Complexity will always be there, but our goal should be to minimize it. That is, built the least complex architecture possible that solves the business problem.

I agree with Jurgen in that complexity is different than complicated although the two concepts are not unrelated. But while we can measure complexity, I don't know how to measure complicated.

Anonymous said...

Software is complex, complicated and difficult - too many variables, limited ability to precisely articulate requirements, perceived ability of slipping in changes easily, etc. But I found the general principles (theorems) of Simon-Ando quite interesting to deal with complexity - hierarchical decomposition of systems, restricting most communication within subsystem, local equilibriums, and achieving global equilibrium (stability of the overall system) through local equilibrium.

Many concepts and approaches in EA and software engineering can be related to Simon-Ando's theorems.

I also found the seminal work of David Parnas very relevant to software architecture.

Roger, I am interested in your views of these works. SIP does not contradict either Simon-Ando or Parnas' work.

Roger Sessions said...

As far as I know, SIP is in full accord with Simon-Ando, although I admit to not being an expert in their work. SIP is trying to find a balance between complexity by virtue of functionality (how many functions are in a subsystems) and complexity by virtue of communications (how much communications occurs between subsystems. SIP uses the equivalence relationship "synergistic" to drive the optimal partitioning of functionality into subsets.

Please lme know how you find SIP compares with these other approaches.