Saturday, October 29, 2011

SIP Complexity Model

SIP, as you probably know, stands for Simple Iterative Partitions, a methodology for finding the least complex solution to a complex IT problem. I have written about SIP elsewhere, for example, in my White Paper The Mathematics of IT Simplification. My goal in this blog is to give a high level description of the model SIP has for IT complexity.

This blog is a work in progress. I will modify it based on suggestions and eventually turn this into a white paper. So please leave your suggestions as comments. If I use them, I will credit you in the eventual white paper. You might want to subscribe to this blog, so that you can follow the discussion.

Let's assume that we have a business problem, BP, that needs an IT solution. Let's say a solution, S001, is proposed. There are a number of attributes of S001 that we could think about. One obvious attribute is how much of the business problem, BP, does S001 solve? In other words, high closely aligned is the solution S001 to the problem BP?

Let's say we have a function A that measures the alignment of S001 to BP and returns a result as an integer between 0 and 100, where 0 is complete non-alignment (S001 doesn't solve any of BP) to 100 (S001 completely solves BP.)

We can write this as y = A(BP, S001)

Let's say we also have a function C that measures the complexity of S001 and returns as a result some number of complexity units. I won't try to describe these complexity units here, but I have described them in the above referenced Mathematics white paper.

We can write this as x = C(S001)

Note that A is a binary function (it takes two arguments) because it must compare the proposed solution to the original problem. C, on the other hand is a unary function (it takes one argument) because complexity is not dependent on the problem, only the proposed solution.

Since S001 now has both a y value, A(BP, S001), and an x value, C(S001), we can plot it on a graph in which y is alignment and x is complexity. This is shown in Figure 1.

Figure 1. Graph of S001, A Solution to BP

Now BP doesn't just have one possible solution, it has a whole bunch of solutions. Let's say we have identified 25 possible solutions to BP. We can name these solutions S001, S002, ... S025.

Just as we plotted S001 in Figure 1, we can plot each of the other solutions. This is shown in Figure 2.

Figure 2. Graph of S001-S025

In general, the values of Sx will be bounded by a tetragon, as shown in Figure 3.

Figure 3. Complexity Tetragon Bounding Sx

Why are the values bounded by this tetragon? At the upper edge, values are bounded by alignment. It is not possible to have more than 100% alignment, so that cuts off values at the top. The x axis is bounded at the low end by the lowest complexity observed in any of Sxs and at the high end by the highest complexity observed in any of the Sxs.

The upper left corner of the tetragon is bounded in the x direction by the lowest possible complexity that still gives the maximum alignment with BP. We can still find simpler Sx, but they can only get simpler by losing functionality that BP needs.

The upper right corner of the tetragon is bounded in the x direction by the highest possible complexity that still gives the maximum alignment with BP. We can still find more complex Sx, but their complexity will result in a loss of ability to meet the needs of BP.

We can further divide the Complexity Tetragon from Figure 3 into four quadrants, as shown in Figure 4.

Figure 4. Four Quadrants of Complexity Tetragon

The lines marking the quadrant boundaries are shown as fuzzy because the boundaries themselves are fuzzy. We can focus on these quadrants by removing the points and adding quadrant names, as shown in Figure 5.
Figure 5. The Four Quadrants With Names

I'll come back to the quadrant names in a moment, but let's just consider the characteristics of solutions that live in each quadrant. The characteristics are as follows:
• vital quadrant contains solutions that have low complexity and high alignment to the business problem.
• stagnant quadrant contains solutions that have high complexity and high alignment to the business problem.
• chaotic quadrant contains solutions that have high complexity and low alignment to the business problem.
• simplistic quadrant contains solutions that have low complexity and low alignment to the business problem.
Now you might wonder why I have focused on complexity in the X axis. Why not, say, agility? The reason is that a number of important characteristics are all directly related to complexity. Agility, for example, goes up as complexity goes down and goes down as complexity goes up. A partial list of these characteristics and their relationship to complexity are as follows:
• Agility, inversely related to complexity
• Security, inversely related to complexity
• Performance, inversely related to complexity
• Reliability, inversely related to complexity
• Cost, positively related to complexity
• Time to deliver, positively related to complexity
A quick examination of the above list will show you that desirable characteristics go down as complexity increases and undesirable characteristics go up as complexity increases. Therefore complexity is the most important characteristic to consider. Mathematically, we can say that complexity is the independent variable and agility, security, etc are the dependent variables. I will not try to prove this assertion here, but I have written about this elsewhere (for example, in my book, Simple Architectures for Complex Enterprises.)

Now I can describe why I have named the quadrants as I have.

Solutions in the vital quadrant have low complexity and good alignment to the problem. I call this quadrant vital because these solutions have the best agility of all the quadrants that are aligned with the business problem. Because solutions in this quadrant have high agility, they can change as needed.

Solutions in the stagnant quadrant also solve the problem, but at a high complexity cost. Among other problems, these solutions are hard to change as business needs change. I call this quadrant stagnant because solutions in this quadrant are stuck in time.

Solutions in the chaotic quadrant have no redeeming features. They do not solve the problem and they are highly complex. I call this quadrant chaotic because nothing useful comes out of it. Chaotic solutions get canceled someplace along the project life cycle before delivery, usually after considerable money is dumped into trying to make them work.

Solutions in the simplistic quadrant do not solve the business problem, but at least they are not burdened with overwhelming complexity. I call this quadrant simplistic because the solution is overly simplistic with respect to the need. A rock, for example is a simplistic car.

When considering a solution to a business problem, it is helpful to consider which quadrant that solution lives in.

If the solution lives in the vital quadrant, you are in good shape. You have a solution to the business problem and it has low complexity and is thus highly likely to have a number of important characteristics (agility, etc.)

If the solution lives in the stagnant quadrant, the solution has hope. At least it appears to solve the business problem. Now you just need to focus on removing the complexity that is going to rob you of the characteristics you want.

If the solution lives in the chaotic quadrant, the solution has no hope. The best thing you can do is to kill the solution as quickly as possible and start from scratch. Learn your lessons and be happy you didn't lose more money.

If the solution lives in the simplistic quadrant, the solution has hope. At least you are not mired in complexity. Now just try to meet more of the business needs without substantially adding to the complexity.

Clearly, our ideal solutions live in the vital quadrant. But not all solutions in the vital quadrant are equal. The ideal ideal solution is the one in the upper left corner, shown as a target in Figure 6. This solution is the one with the absolute lowest complexity that still meets all of the business needs. This is the solution we would like to find.
Figure 6. Complexity Tetragon With Ideal Solution

So far, I have just looked at the SIP Complexity Quadrants at a moment in time. We can also look at how things drift over time. There are two distinct undertows that are pulling at solutions. One undertow is pulling from greater alignment to lesser alignment. The second undertow is pulling from lesser complexity to greater complexity. These undertows are added to the tetragon in Figure 7.

Figure 7. Complexity Tetragon With Undertows

By understanding the undertows we can predict that a solution that is vital on the day it is delivered is going to drift over time to being less aligned and more complex. This tells us that we need to put ongoing energy into fighting these undertows. We want to not only start out with a solution in the vital quadrant, we want to stay in that quadrant for the life of the solution. In fact, the lifetime of the solution will be highly influenced by which quadrant the solutions starts in and how effectively we fight the undertows after the solution is delivered.

Figure 7 now gives us a complete picture of the complexity landscape of a proposed solution. It gives us considerable information about how good a solution is, what steps need to be taken to improve it, and what forces are going to be pulling at the implemented solution over the long term.

What do you think? Is this a good start to helping you understand the SIP Complexity Model? Keep in mind this is a draft and I will be adding to it as I respond to comments. Feel free to leave comments here, tweet me at @RSessions, or email me (userID: roger; domain: objectwatch.com).

Anonymous said...

Thanks for the article, very good model for making sense out of multiple solutions complexity-wise.

Interesting enough, similar model is used by investment managers to come up with optimal asset allocation within a portfolio, so called Capital asset pricing model (CAPM). In short, all possible combinations are plotted using axes of 'risk' (std.dev) and 'return', and then it is said that the ideal portfolio is the most top-left point in the plotted area. It doesn't take a lot of imagination to see similarity between CAPM and your model, as more complex solution means more risk and more alignment means better result.

Chris Chedgey said...

It might help to indicate the source of the undertow. If I have understood correctly, the downward undertow is due to the BP/requirements evolving, and therefor the alignment dropping unless the solution evolves. The rightward undertow is due to the engineering activity required to modify the solution to keep up with the BP changes. (I love the use of "undertow"!)

Mark said...

Hello,
Regardless of the academic excellence of this model, I can't really see any Architect using this in earnest in their daily work as it's quite complicated.

Sorry, but this is all far too academic for me.

Uli Wehr said...

I suggest to also consider:
- the economical impact of a solution (implementation and maintenance),
- the flexibility and adaptability of a solution,
- the impact on the portfolio of existing and planned business problems and solutions.

The least complex solution for a given business problem may not be the most economical choice.

Flexibility and adaptability may increase complexity initially, but it could turn out to be the less complex solution over time. It also would allow for faster responses to a changing business environment.

There is actually a whole portfolio of business problems and solutions. Depending on what level we choose to manage this portfolio, e.g. by BP owner, department, or the whole organization, the least complex solution for one business problem may not be the least complex solution for the portfolio.

Jan Lindstrom said...

The undertow notion addresses the question of "why are we adrift?" (intent usually is not to be not business aligned).

Should we also be concerned about dependencies? Ie solutions tend to drift since they're linkages to other solutions drift. If there is an undertow to chaos and non-alignment, is that force like entropy? or does entropy decrease elsewhere in solution space? example could be a legacy system being bent-out-of-shape and left to die, since more business-critical systems need be in the vital quadrant?

Roger Sessions said...

Ilobanov: Interesting parallel. And I think you are right, investment risk most likely correlates with investment complexity.

Chris Chedgey: Yes, the downward undertow is a result of the business evolution and the rightward undertow is due to engineering changes, which typically are not done with an awareness of complexity.

And thanks re "undertow." I was trying to give the idea that the pressure is constant, unrelenting, and, if not addressed, will pull you into a place you do not want to be.

Mark, I don't know what to say to you. Any architect who is not willing to seriously consider the impact of complexity in a methodical, rational way is not a competent architect.

What would you think of a bridge architect who was unwilling to consider the mathematics of gravity and material strength before implementing a bridge design? Or a rocket-ship designer who was unwilling to study the mathematics of planetary motion?

Can you imagine a rocket designer saying, "I don't want to study Newtonian physics, it's too academic."

I have no doubt there are many architects who consider complexity "too academic." I also have no doubt they are one of the causes of so many IT failures.

Uli Wehr: I believe that the economical impact of a solution in terms of implementation and maintenance is a function of the complexity of the solution. In mathematical terms, the economical impact is a dependent variable and the complexity is the independent variable. The same goes for flexibility and adaptability. Complex solutions are difficult to adapt.

One advantage of focusing on complexity rather than the other "ilities" is that complexity gives us a primary focal point. If my claim is right, that the other "ilities" are dependent variables, then we don't need to worry about them (at least, we don't need to worry about them very much.)

To give an analogy, if we get the flu we will get aching muscles, sneezing, and fever. But we don't try to take precautions against aching muscles, sneezing, and fever. We try to take precautions against the flu. Why? Because the aching muscles, sneezing, and fever are symptoms of the problem (or, mathematically, they are the dependent variables) not the problem itself. The flu is the problem (or, mathematically, the independent variable.)

In another blog, I suggested we adopt the term "simplility" to describe the intentional architecting of simplicity into an IT solution. My goal there is to get people thinking about complexity as a primary problem (an "ility") rather than an afterthought.

Jan Lindstrom: I agree, dependencies is a major factor in IT complexity. In my mathematics paper (pointer given in this blog), I discuss the mathematical components of complexity, which I consider to be the amount of functionality in a system and the number of dependencies on that system to other systems.

I think entropy is a good analogy. Left to their own devices, entropy in a system always increases unless positive energy is put into maintaining order. In this model, this would be represented by the ongoing energy (time, money, thought) that needs to be put into fighting the undertow.

I think large, legacy systems eventually end up in the chaotic region (too complex to change, too out of alignment with the business needs to be useful.)

A big problem with legacy systems in the chaotic region is that CIOs tend to look for a magic pixy dust (e.g. virtualization) that will solve their problem. But there is no magic pixy dust. The systems have been allowed to drift too far for too long. The only remaining question is comparative cost. Is it cheaper to restore the legacy systems to vitality or cheaper to redo it from scratch. Of course, if we choose to redo it from scratch, we should learn from our lessons and focus on designing to the vital quadrant from the beginning.

Thank you all!

Anonymous said...

After reading this article, it comes to applying "common sense" (which is not very common) while choosing solution. It will help to add a case study though which can support using the approach, else there is a tendency to pick a solution which seems right!

Nacsák Tamás said...

Simple wording issue uner quadrants drawing: "We can focus on the these quadrants"

Roger Sessions said...

Thanks Tamas, I'll correct that.

Anonymous said...

Interesting way to look at Enterprize Architecting. I guess the key question now is how do you quantify complexity. Number of functions, menus, lines of code, etc.?

It would be very useful to hear more on that.

Aleksei said...

1. As I get it right, at the end of the third paragraph you need in a phrase " In other words, high closely aligned is the solution S001 to the problem BP?" change "high" to "how".
2. According to the content of the article. I can agree that unnecessary complexity, if we talk about engineering in general, is what anyone dislikes nowadays ... That's right.

Where I see a system or maybe logical incompleteness of the article is that, as I read it's text, it is almost disconnected from Engineering as a general field of knowledge and practice. If we start talking about engineering some solution (car, mobile device, flat or house, or software system) to suit business needs (in general, maybe better to say 'customer needs,), we will normally talk about whether the solution is optimal or not. Whether it is optimal or not is, as I understand, the final criteria for all parties involved. Whether the solution that suits maximum needs is complex or less complex is a subset of a more general criterial. In my opinion the article should somehow encompass this idea and make connection to Engineering discipline in general.

Everything else OK. Thank's. Nice work.

Aleksei said...

1. As I get it right, at the end of the third paragraph you need in a phrase " In other words, high closely aligned is the solution S001 to the problem BP?" change "high" to "how".
2. According to the content of the article. I can agree that unnecessary complexity, if we talk about engineering in general, is what anyone dislikes nowadays ... That's right.

Where I see a system or maybe logical incompleteness of the article is that, as I read it's text, it is almost disconnected from Engineering as a general field of knowledge and practice. If we start talking about engineering some solution (car, mobile device, flat or house, or software system) to suit business needs (in general, maybe better to say 'customer needs,), we will normally talk about whether the solution is optimal or not. Whether it is optimal or not is, as I understand, the final criteria for all parties involved. Whether the solution that suits maximum needs is complex or less complex is a subset of a more general criterial. In my opinion the article should somehow encompass this idea and make connection to Engineering discipline in general.

Everything else OK. Thank's. Nice work.