Simple Architectures for Complex Enterprises

Web Short: The Relationship Between IT Project Size and Failure

2012-01-06T14:50:00.000-08:00

Large IT projects have bigger budgets than small projects. They also have more formal approaches to tracking milestones and budgets. Does this make them more likely to succeed? It turns out that there is a relationship between the cost of a project and the chances that that project will succeed, but it isn't what you might think.

Watch the presentation and then leave a comment.

You can see a full list of Roger's Web Shorts here.
Would you like to be notified of future Web Shorts and White Papers by Roger Sessions? Sign up here.

And thanks to AuthorStream for hosting this presentation.

This presentation includes narration, so be sure to have your speakers on.

The Relationship Between IT P...

More PowerPoint presentations from Roger Sessions

Web Short: The Mathematics of Cloud Optimization

2012-01-06T14:44:00.000-08:00

How do you minimize the cost of running your large mission critical application on a public cloud? Do you focus on finding a low cost cloud provider? It turns out that the answer to cost reduction may be closer than you think. And it starts by understanding the mathematics around Cloud optimization.

Watch the presentation and then leave a comment.

You can see a full list of Roger's Web Shorts here.
Would you like to be notified of future Web Shorts and White Papers by Roger Sessions? Sign up here.
And thanks to AuthorStream for hosting this presentation.

The video includes narration, so be sure you speakers are on!

The Mathematics of Cloud Opti...

More PowerPoint presentations from Roger Sessions

Web Short: SIP Methodology for Project Optimization

2012-01-06T13:01:00.000-08:00

IT Methodologies such as TOGAF, Zachman, FEAF, RUP, and Agile are all important tools for the enterprise and software architect. But it turns out that all of these methodologies share a common limitation: they don't scale. Each of these works well for projects less than about $1 million, but try to use any of them for projects in the $10M range, and you will find yourself in a murky land with dangers around every curve.

If you are building or maintaining a large IT system, you must start by understanding the principles of partitioning. This Web Short gives an overview of the SIP methodology, the only methodology focused exclusively on the issue of partitioning. SIP doesn't compete with these existing methodologies, it completes them. SIP is the missing ingredient in scalability.

You can see a full list of Roger's Web Shorts here.

Would you like to be notified of future Web Shorts and White Papers by Roger Sessions? Sign up here.
And thanks to AuthorStream for hosting this presentation.

SIP Methodology for Project Optimization

More PowerPoint presentations from Roger Sessions

The CRASH Quiz

2011-12-10T05:15:00.001-08:00

CAST software just published their 2011 CRASH (CAST Report on Application Software Health.) I know the CAST folks quite well. They are leaders in the field of software implementation complexity. Implementation complexity is complementary to my interests, organizational complexity. Organizational complexity comes from poor project partitioning. Implementation complexity comes from poor project coding. Both of these types of complexity cause severe problems for large IT systems.

While the full CRASH report must be purchased, considerable information is available in the free summary available here. I will highlight some of the most surprising, and in some cases, controversial findings. To make this more interesting, I will deliver my discussion as a quiz. So get ready!

CAST did this analysis using their software analysis tools. CAST produces tools that analyze software systems and rates them on various criteria of code quality. CAST, for example, can analyze the 50,000 lines of code that was just delivered from your outsourcing firm and rate it on maintainability, adaptability, security, and a number of other attributes that you probably care a great deal about.

The logic to doing this analysis is simple. Sooner or later, you are going to find out about how maintainable, adaptable, and secure this code is. Would you prefer to find out now, before you have accepted delivery, or later, after you have deployed this system to your trusting constituents?

To produce the CRASH report, CAST used their tools on a large collection of software systems ranging in size from 10K LOC (lines of code) to more than 5M LOC. They did this for a number of industries, programming systems, and development methodologies.

Okay, are you ready to take the CRASH quiz? Here goes! I will start with the questions and then give you the answers.

QUIZ

Q 1. Rank the following languages from most used to least used: ABAP, C, Java, .NET, Cobol.

Q 2. In the Government, which of the following languages is most popular: ABAP, C, Java, .NET, Oracle Forms?

Q3. Which of the following yield the worst security scores: Cobol, C++, or Java?

Q4. Which of the following have the highest complexity scores: Java, Oracle Forms, or Cobol?

Q5. Which industry has the highest complexity scores: Government, Financial Services, or Telecom?

Q6. Which code has a higher overall quality index, code produced in-house or code that is outsourced?

Q7. Which development approach produces code with a higher quality index, Agile/Iterative or Waterfall?

Q8. What is the "Technical Debt" in an average system, per line of code?
a. Less than $1.00 per line of code.
b. Between $1.01 and $2.00 per line of code.
c. Between $2.01 and $3.00 per line of code.
d. Between $3.01and $4.00 per line of code.

ANSWERS (No Peeking!)

A1. Overall, these languages rank (from most to least popular) Java, Cobol, ABAP, .NET, and C. Actually, C is rarely used. I include in the list just for nostalgia.

A2. In the Government, Oracle Forms is the most popular programming system, followed by Java. I note that Oracle Forms tends to be relatively small programs if one can even call them programs. So while Java is used for fewer "systems" I suspect (but can't tell from the data) that it is used for many more lines of code.

A3: From a security perspective, C++ and Java are in a virtual tie for worst security. Cobol code overall has a much better security score. This may reflect more on the industry than the language, since Cobol is used heavily in the Financial Services industry, where security is taken more seriously than, say Telecom where Java use predominates.

A4. The most complex code by far is found in the Cobol systems followed by Oracle Forms. Java wins for the least complex code of the three.

A5. The industry with the highest complexity scores is the Government. Financial Services is a distant second followed by Telecom. This result is surprising given than Java is popular in both Telecom and the Government. The implication seems to be that although the Government is using very good language tools, it is not maximizing their effectiveness.

A6. Code that is produced in-house has a better quality index than outsourced code, but the difference is marginal and probably not statistically significant.

A7. Overall, Waterfall development has a significantly higher quality index than does code produced using Agile/Iterative. Not only does Agile/Iterative score lower in overall quality, it also scores lower in transferability (the ability for other groups to understand the code) and changeability (the ability to modify the code.) I can hear the groans of protest already from the Agile community. Sorry, I'm just the messenger.

A8. The average "Technical Debt" in a system is $3.61 per line of code (answer d.) The technical debt looks at the number of problems, the severity of those problems, and the cost of fixing those problems.

Some of these answers are a bit surprising, aren't they? Feel free to read the summary report here. You will probably find another surprise or two.

CxO Executive Round Table on Business Risk and the Cloud in NYC

2011-11-22T12:03:00.001-08:00

CxO Round Table on Business Risk and the Cloud in NYC
With
Robert Youngjohns, President of Microsoft North America
Roger Sessions (Author and Thought Leader)

Note: In return for being the main speaker at this event, Microsoft has offered me ten seats. I am making them available on linked-in, my blog, and to my email list on a first come, first serve basis.

- Roger Sessions

Event Description
Robert Youngjohns will offer Microsoft’s Vision, commitment and investment in the Cloud. Then Roger Sessions will give a vendor neutral roundtable discussion on Business Risk and the Cloud. Sessions’s talk will be followed by an interactive roundtable discussion between you and your peers.

Sessions’s Abstract: The business is greatly impacted by the decisions IT makes on the Cloud. But in most organizations, business has little input in cloud discussions. How can the business be sure that its needs for regulatory compliance, data security, reliability, and ROI are being addressed? A new model must drive the IT/Business collaboration to ensure the Cloud delivers the highest possible value at the lowest possible cost with the least possible risk.

Location: New York City, Tuesday, December 6, 2011

Audience: This talk is directed at those on the business side (that is, CxOs and business leaders) of large (>$1B/year) organizations. In order to optimize participation, the audience is limited to 20.

Note: There is no cost for this event.

Agenda
8:30 - 9:15AM Registration and Breakfast
9:15 - 10:15AM Microsoft’s Cloud Strategy - Robert Youngjohns, President, Microsoft North America
10:15 - 11:00AM Business Risk and the Cloud - Roger Sessions, Author, Thought Leader
11:00 - 11:45AM Roundtable Discussion: Experiences and Strategies
11:45 - 12:00PM Next Steps and Closing

To register for this event, send an email to v-josyph; at microsoft.com.
As the subject, use:
Managing Risk in the Cloud Roundtable Dec 06-Registration
In the body of the email, include your name, title, company, and the note, “by invitation of Roger Sessions.”

Speaker Bios

Robert Youngjohns is President, Microsoft North America
Youngjohns oversees a sales force of over 8,500 employees across the continent. He brings more than 30 years of experience in sales, marketing and strategic business development. Prior to joining Microsoft, Youngjohns served as president and chief executive officer of Callidus Software, Inc., a publicly traded company and leading provider of sales management software based in San Jose, Calif. Before joining Callidus, Youngjohns spent 10 years at Sun Microsystems, where in his last role as executive vice president of Global Sales he was responsible for Sun's worldwide sales organization. He also spent 18 years in various roles at IBM.
Youngjohns has deep technical knowledge and sales experience across the breadth of products within the Microsoft portfolio. Current market trends of focus are cloud computing and the consumerization of IT. He is committed to helping customers and partners get the most value from their relationship with Microsoft.

Roger Sessions is the CTO of ObjectWatch
He has written seven books including Simple Architectures for Complex Enterprises and many articles. He is a past founding member of the Board of Directors of the International Association of Software Architects, Editor-in-Chief of Perspectives of the International Association of Software Architects, and a Microsoft(tm) recognized MVP in Enterprise Architecture. He has given talks in more than 30 countries, 70 cities and 100 conferences on the topic of Enterprise Architecture.
Sessions is a well respect thought leader in the topics of enterprise architecture, the Cloud, and Complexity/Risk management.

SIP Complexity Model

2011-10-29T13:45:00.000-07:00

SIP, as you probably know, stands for Simple Iterative Partitions, a methodology for finding the least complex solution to a complex IT problem. I have written about SIP elsewhere, for example, in my White Paper The Mathematics of IT Simplification. My goal in this blog is to give a high level description of the model SIP has for IT complexity.

This blog is a work in progress. I will modify it based on suggestions and eventually turn this into a white paper. So please leave your suggestions as comments. If I use them, I will credit you in the eventual white paper. You might want to subscribe to this blog, so that you can follow the discussion.

Let's assume that we have a business problem, BP, that needs an IT solution. Let's say a solution, S001, is proposed. There are a number of attributes of S001 that we could think about. One obvious attribute is how much of the business problem, BP, does S001 solve? In other words, high closely aligned is the solution S001 to the problem BP?

Let's say we have a function A that measures the alignment of S001 to BP and returns a result as an integer between 0 and 100, where 0 is complete non-alignment (S001 doesn't solve any of BP) to 100 (S001 completely solves BP.)

We can write this as y = A(BP, S001)

Let's say we also have a function C that measures the complexity of S001 and returns as a result some number of complexity units. I won't try to describe these complexity units here, but I have described them in the above referenced Mathematics white paper.

We can write this as x = C(S001)

Note that A is a binary function (it takes two arguments) because it must compare the proposed solution to the original problem. C, on the other hand is a unary function (it takes one argument) because complexity is not dependent on the problem, only the proposed solution.

Since S001 now has both a y value, A(BP, S001), and an x value, C(S001), we can plot it on a graph in which y is alignment and x is complexity. This is shown in Figure 1.

Figure 1. Graph of S001, A Solution to BP

Now BP doesn't just have one possible solution, it has a whole bunch of solutions. Let's say we have identified 25 possible solutions to BP. We can name these solutions S001, S002, ... S025.

Just as we plotted S001 in Figure 1, we can plot each of the other solutions. This is shown in Figure 2.

Figure 2. Graph of S001-S025

In general, the values of Sx will be bounded by a tetragon, as shown in Figure 3.

Figure 3. Complexity Tetragon Bounding Sx

Why are the values bounded by this tetragon? At the upper edge, values are bounded by alignment. It is not possible to have more than 100% alignment, so that cuts off values at the top. The x axis is bounded at the low end by the lowest complexity observed in any of Sxs and at the high end by the highest complexity observed in any of the Sxs.

The upper left corner of the tetragon is bounded in the x direction by the lowest possible complexity that still gives the maximum alignment with BP. We can still find simpler Sx, but they can only get simpler by losing functionality that BP needs.

The upper right corner of the tetragon is bounded in the x direction by the highest possible complexity that still gives the maximum alignment with BP. We can still find more complex Sx, but their complexity will result in a loss of ability to meet the needs of BP.

We can further divide the Complexity Tetragon from Figure 3 into four quadrants, as shown in Figure 4.

Figure 4. Four Quadrants of Complexity Tetragon

The lines marking the quadrant boundaries are shown as fuzzy because the boundaries themselves are fuzzy. We can focus on these quadrants by removing the points and adding quadrant names, as shown in Figure 5.

Figure 5. The Four Quadrants With Names

I'll come back to the quadrant names in a moment, but let's just consider the characteristics of solutions that live in each quadrant. The characteristics are as follows:

vital quadrant contains solutions that have low complexity and high alignment to the business problem.
stagnant quadrant contains solutions that have high complexity and high alignment to the business problem.
chaotic quadrant contains solutions that have high complexity and low alignment to the business problem.
simplistic quadrant contains solutions that have low complexity and low alignment to the business problem.

Now you might wonder why I have focused on complexity in the X axis. Why not, say, agility? The reason is that a number of important characteristics are all directly related to complexity. Agility, for example, goes up as complexity goes down and goes down as complexity goes up. A partial list of these characteristics and their relationship to complexity are as follows:

Agility, inversely related to complexity
Security, inversely related to complexity
Performance, inversely related to complexity
Reliability, inversely related to complexity
Cost, positively related to complexity
Time to deliver, positively related to complexity

A quick examination of the above list will show you that desirable characteristics go down as complexity increases and undesirable characteristics go up as complexity increases. Therefore complexity is the most important characteristic to consider. Mathematically, we can say that complexity is the independent variable and agility, security, etc are the dependent variables. I will not try to prove this assertion here, but I have written about this elsewhere (for example, in my book, Simple Architectures for Complex Enterprises.)

Now I can describe why I have named the quadrants as I have.

Solutions in the vital quadrant have low complexity and good alignment to the problem. I call this quadrant vital because these solutions have the best agility of all the quadrants that are aligned with the business problem. Because solutions in this quadrant have high agility, they can change as needed.

Solutions in the stagnant quadrant also solve the problem, but at a high complexity cost. Among other problems, these solutions are hard to change as business needs change. I call this quadrant stagnant because solutions in this quadrant are stuck in time.

Solutions in the chaotic quadrant have no redeeming features. They do not solve the problem and they are highly complex. I call this quadrant chaotic because nothing useful comes out of it. Chaotic solutions get canceled someplace along the project life cycle before delivery, usually after considerable money is dumped into trying to make them work.

Solutions in the simplistic quadrant do not solve the business problem, but at least they are not burdened with overwhelming complexity. I call this quadrant simplistic because the solution is overly simplistic with respect to the need. A rock, for example is a simplistic car.

When considering a solution to a business problem, it is helpful to consider which quadrant that solution lives in.

If the solution lives in the vital quadrant, you are in good shape. You have a solution to the business problem and it has low complexity and is thus highly likely to have a number of important characteristics (agility, etc.)

If the solution lives in the stagnant quadrant, the solution has hope. At least it appears to solve the business problem. Now you just need to focus on removing the complexity that is going to rob you of the characteristics you want.

If the solution lives in the chaotic quadrant, the solution has no hope. The best thing you can do is to kill the solution as quickly as possible and start from scratch. Learn your lessons and be happy you didn't lose more money.

If the solution lives in the simplistic quadrant, the solution has hope. At least you are not mired in complexity. Now just try to meet more of the business needs without substantially adding to the complexity.

Clearly, our ideal solutions live in the vital quadrant. But not all solutions in the vital quadrant are equal. The ideal ideal solution is the one in the upper left corner, shown as a target in Figure 6. This solution is the one with the absolute lowest complexity that still meets all of the business needs. This is the solution we would like to find.

Figure 6. Complexity Tetragon With Ideal Solution

So far, I have just looked at the SIP Complexity Quadrants at a moment in time. We can also look at how things drift over time. There are two distinct undertows that are pulling at solutions. One undertow is pulling from greater alignment to lesser alignment. The second undertow is pulling from lesser complexity to greater complexity. These undertows are added to the tetragon in Figure 7.

Figure 7. Complexity Tetragon With Undertows

By understanding the undertows we can predict that a solution that is vital on the day it is delivered is going to drift over time to being less aligned and more complex. This tells us that we need to put ongoing energy into fighting these undertows. We want to not only start out with a solution in the vital quadrant, we want to stay in that quadrant for the life of the solution. In fact, the lifetime of the solution will be highly influenced by which quadrant the solutions starts in and how effectively we fight the undertows after the solution is delivered.

Figure 7 now gives us a complete picture of the complexity landscape of a proposed solution. It gives us considerable information about how good a solution is, what steps need to be taken to improve it, and what forces are going to be pulling at the implemented solution over the long term.

What do you think? Is this a good start to helping you understand the SIP Complexity Model? Keep in mind this is a draft and I will be adding to it as I respond to comments. Feel free to leave comments here, tweet me at @RSessions, or email me (userID: roger; domain: objectwatch.com).

New Word: Simplility

2011-10-28T08:14:00.000-07:00

I am introducing a new word into the IT lexicon: simplility. Simplility is defined as the intentional architectural design of simplicity into a software application

Simplility follows a long tradition of "ilities" such as portability, reliability, and scalability. These all imply that somebody has made an intentional decision to include specific attributes in an application. Nobody expects an application to be portable, reliable, or secure by accident. We understand that it take skill and effort to give an application these attributes. And we understand that there are important reasons for doing so.

Why, you may ask, do we need a new word? Why not just use the word simplicity? The problem with simplicity is that we take it for granted. To say that an application is simple simply does not have the same cachet as saying the application is scalable. We understand that scalable has business value.

So we need a word that elevates the attribute of simplicity to the same level of the other ilities that we understand provide business value. We need a word that announces that simplicity is a business asset and that it takes skill, effort, and commitment to incorporate this attribute into applications.

There is one problem with the world simplility. It implies that simplility is equal in importance to portability, reliability, or security. In fact, simplility is more important. It is the primary architectural attribute, the one from which all other attributes flow.

Take security, for example. A system that has simplility baked in is one that will be much easier to make secure than one that lacks simplility. Complex systems are inherently insecure.

The same with reliability. The greater the simplility, the greater the reliability. Complex systems are inherently unreliable.

So if we are serious about IT architecture, we need to get serious about simplility. It is the most important ility of all. And starting today, it has its own word.

Public or Private Cloud? Wrong Question!

2011-10-25T05:04:00.000-07:00

Many people are asking if it is best to use a public or private cloud. They are asking the wrong question.

The bigger issue for most organizations is not whether to use a private or public cloud, but whether to use any cloud at all. The issue here is not whether the cloud is a good idea or a bad idea, but whether the organization has sufficient maturity to make effective use of any cloud. Most don't.

Most organizations use a pre-cloud architecture. The cloud places specific demands on a app with respect to efficiency, performance, reliability, and security. Very few organizations understand these demands and thus very few apps of any significant size are architected to meet those demands.

A good historical comparison is the switch from client/server programming to three-tier programming. A new generation of technologies enabled three-tier programming, namely transaction processing monitors (TPMs.) TPMs promised a radical improvement in the efficiency of computer resources, very similar to what the cloud promises today. Many organizations took their existing client/server applications and "ported" them to a TPM environment. They then discovered the hard way that a client/server architecture is totally unsuited for a TPM environment.

Organizations that take their existing three-tier, SOA, or even client/server apps and "port" them to the cloud are in for an equally rude awakening. They will find that these apps are expensive to run, highly fragile, insecure, and will have poor performance.

Asking if a public or private cloud is best for most apps is like asking if a public or private road is best for most boats.

Next Web Short: Size and IT Risk

2011-09-17T14:04:00.000-07:00

Announcing a new series of short focused web meetings with Roger Sessions. You are invited.

NEXT WEB SHORT
Topic: Size and IT Risk; The Relationship Between Project Size and Project Failure
Date: Wednesday, Sept 21
Time: 11:00 AM USA Central Time (16:00 GMT)
Recommended Audience: CIO, CIO Reports, Enterprise Architects
Format: 15 minute presentation followed by Q&A

To Register: send an email with subject: REGISTER to roger (domain: objectwatch.com). Be sure to include your name and the email to which you would like the log in information set.

REGISTRATION LIMITED TO 20 PARTICIPANTS.

There is no charge for registration. We invite you to forward this information to those in your organization that you think might be interested in attending.

Upcoming Web Short Topics

Complexity and the Cloud

The Challenge of Business/IT Alignment

Achieving Agility

The Problem of Procurement

The Fallacy of Reusability

SIP Methodology

If you would like to attend these meetings but the time (GMT 1600) is inconvenient because of local time zones, let us know and we will add other time slots.

For priority notification on future Web Shorts and White Papers by Roger Sessions, subscribe to the ObjectWatch News List here.

The Mathematics of IT Simplification

2011-04-05T11:47:00.000-07:00

01 April 2011 - I have written a 32 page white paper on The Mathematics of IT Simplification. This paper suggests a mathematically grounded approach to simplifying large complex IT systems. This approach is called synergistic partitioning. Synergistic partitioning is based on the mathematics of sets, equivalence relations, and partitions. This approach has several advantages over traditional decompositional design:

It produces the simplest possible partition for a given system.
It is directed, or deterministic, meaning that it always comes out with the same solution.
It can determine the optimal partitioning of a system with very little information about that system.

You can pick up a copy of the paper here (it's easy, no registration or cost) and discuss it by leaving comments right here.

Discussing Complexity

2010-10-15T07:40:00.000-07:00

This blog is for discussing the paper, Urban and Enterprise Architectures: A Cross-Disciplinary Look at Complexity by Roger Sessions and Nikos A. Salingaros

Complexity is a major problem in enterprise architecture. Complexity obstructs business/IT alignment. Complexity obscures project vision. Complexity delays schedules, drives cost overruns, and hinders delivery of value. For enterprise architecture, complexity is the enemy.

Complexity is also a problem for urban architecture. In urban architecture complexity causes habitats that are destructive to the human psyche. Complexity can cause a variety of stress responses such as increased heart rate, sweating, and pupil dilation. For urban architecture, complexity is not just a matter of aesthetics; it is a matter of social wellbeing.

Although the symptoms of unmanaged complexity differ in these two fields, the laws governing architectural complexity are universal. Successful architectures follow the same basic principles whether those architectures describe biological, urban, or enterprise systems. And in all cases, the cost of ignoring these principles is disorder, dissolution, and ultimately chaos.

This cross-disciplinary paper explores the universal principles of controlling complexity and draws on lessons from urban architecture to better understand how to design successful enterprise architecture.
This paper was originally a lecture by Roger Sessions and Nikos Salingaros and is presented here as lecture notes.

Nikos A. Salingaros is the author of Anti-Architecture and Deconstruction (2004), Principles of Urban Structure (2005), and A Theory of Architecture (2006), as well as numerous scientific papers. Dr. Salingaros collaborated with Christopher Alexander, helping to edit the four-volume The Nature of Order during its twenty-five-year gestation, a work that had a huge impact on the Patterns Movement in Software Architecture.

You can download the paper here. Look towards the end of the section.

SAPA Paradigm for IT Architecture

2010-09-01T09:56:00.000-07:00

This blog is intended to give a basic introduction to the SAPA paradigm. SAPA is a new approach to designing, building, and delivering IT systems.

SAPA stands for Simple As Possible Architecture. The driving idea behind the SAPA paradigm is that simplicity reigns supreme. Complexity is the enemy. It drives systems cost, increases system failures, and greatly adds to the lifetime cost of an IT system.

There are a number of stages to building a SAPA. I think of the stages as follows:

Stage 1: Capability Mapping. Here we are trying to understand how the high level business processes that will be supported by the IT system are broken down into simplest possible set of capabilities (groups of business functionality.)

Stage 2: Business Architecture. Here we are taking one of the capabilities identified in Stage 1 and specifying the the simplest possible implementation of that capability in terms of business processes and dependencies.

Stage 3: IT Organization. Here we are identifying the simplest possible collection of IT systems necessary to support the business architecture identified in Stage 2 and documenting the dependencies between those systems.

Stage 4: IT Architecture. Here we are designing the simplest possible architecture for each of the IT systems identified in Stage 3.

Stage 5: IT Implementation. Here we are implementing the IT system as simply as possible following the architecture created in Stage 4.

Each of these stages follows the basic SAPA pattern:

Understand the immediate problem you are tying to solve
Design the simplest possible solution to that problem.
Validate that there is no simpler solution.

This common SAPA pattern has a number of implications that cut across stages. First, there must be some way to compare two solutions to see which is the simplest. Second, there must be a methodology you can use to test a solution to see if it can be simplified. And both of these imply that there must be some rational way of measuring complexity.

My recent work in the SAPA paradigm has focused at stages 1-3. For a good example of my work in these three stages, see my "anti-complexity patent." You can read about it here.

SAPA builds on the paradigms that come before it, especially Service-Oriented Architecture (SOA). But while SOAs are a natural way to realize a SAPA (stages 4-5) SOAs are in no way required for SAPA.

SAPA is a key enabler for many desirable features of IT systems. SAPA systems are easier to test, implement, and make secure. They perform better and are more reliable. They are cheaper to build and easier to maintain.

SAPA has something for everybody. For those interested in business/IT alignment, SAPA systems are highly aligned with the business processes they support. For those interested in cloud deployment, SAPA systems are well designed for life in the cloud. For those interested in Agile development, SAPA systems are a necessary prerequisite to effectively scaling up Agile approaches.

If you care about improving the organization's bottom line through more effective IT investments, you should care about SAPA. The reason is simple. As Simple As Possible!

................................................
Would you like to keep up to date with Roger's blogs, white papers, newsletters, books, conference appearances, and webinars on SAPA and Complexity? Subscribe to the ObjectWatch alerts. Instructions here.

Defining IT Complexity Terminology

2010-05-03T10:39:00.000-07:00

My specialty is understanding how to measure, locate, and eliminate unnecessary IT complexity. One of the difficulties of this field is terminology. You would think that a word like simple would be simple to define. But there is no widespread agreement on what the words related to complexity mean.

So in this blog, I am going to define some of the terminology that is important in IT complexity reduction. I won't claim that these meanings are relevant to all fields, but when I use these words, this is what they will mean. This is my initial pass. I will give examples from my own work, since that is the domain with which I am most familiar.

I invite readers to comment. Soon I will consolidate the discussion into a terminology proposal for www.cuec.info (The Consortium for Untangling Enterprise Complexity.) I assume that better definitions will come out of this discussion.

Simplicity Framework: A simplicity framework is an approach, model, or methodology that seeks to find, measure, and remove complexity from some domain. Any simplicity framework should specify the domain for which it is intended. Example: SIP (Simple Iterative Partitions) is a simplicity framework for IT architecture.

Complexity: Complexity is a measure of an attribute of a system that makes that system difficult to understand, manage, and/or implement. Complexity is measured in some units appropriate for the domain and defined by a simplicity framework. Example: In SIP, complexity is measured in Standard Complexity Units (SCUs).

Solution Set: For some problem P in a given domain, the solution set is the set of all solutions that solve P. Note that a solution set only includes solutions to P. If a proposed solution would not solve P, then it is by definition not a member of the solution set. Example: When considering architectures for an inter-library loan system, we examined dozens of potential candidates from the solution set.

Simple Solution: A simple solution is the element of a solution set that has the least complexity of all elements in the solution set. Note that it is theoretically possible that more than one solution from the solution set share the lowest complexity. In this case, there are multiple simple solutions. Example: People were surprised that the simple solution to the inter-library loan system was not the one initially proposed.

Complex Solution: A complex solution is any of the elements of the solution set that are not the simple solution and whose complexity can be reduced through the simplicity framework. Example: In the inter-library loan system, all of the originally proposed SOA architectures were complex solutions.

Simplistic Solution: A simplistic solution is any proposed solution to a problem that lacks the minimum complexity necessary to solve that problem. By definition a simplistic solution is not a member of the solution set. Example: The catalog system we looked at turned out to be a simplistic solution to the inter-library loan system.

Chaotic Solution: A chaotic solution is a solution that is presumed to solve a problem, but whose complexity is so high that using (or continuing to use it) is not feasible and reducing it is not practical given the simplicity framework. Note that it is not always possible to determine if a chaotic solution is even a member of the solution set. Example: The present inter-library loan system is a chaotic solution.

Do you have any ideas for more terms that should be defined? Do you have issues with these definitions? Leave comments here or discuss with me on Twitter (@RSessions). And keep an eye on www.cuec.info for efforts to standardize some of these terms.

Planned Version 2 Changes:
@johanlindberg suggested using candidate to describe Simplistic and Chaotic. His argument: Simplistic and Chaotic are by definition not solutions. Good point! He also suggested giving comparison terms to show similar terms in complexity theory and SW development.

@HotFusionMan (Al Chou) suggested adding note about layers of complexity. I think this is a detail of the simplification framework, but I need to make clear the generic character of these definitions.

Add definitions for simplicity, simplification. Add description of which problem spaces these definitions are appropriate for (those that are striving for simplicity.) Make less specific to IT and general to any situation in which a move from complex to simple is desirable.

Incorporate Nikos's discussion on understandability.

And thanks to @JohnDCook for Twitter contributions!

The Failure of Success

2010-04-06T07:50:00.000-07:00

I have written quite a bit about the cost of IT failure, most recently in a white paper. A number of people have criticized my analysis saying that just because a large project "fails" in the sense of being late, over budget, and/or not delivering expected functionality, doesn't mean that the project doesn't deliver value.

This may be true. The real question is this: had the business known how the project would have gone, would they still have agreed to do the project? If the answer to this is yes, then the project was a success. If the answer is no, then the project is a failure.

However even when the project comes in on-time, on-budget, and delivering expected functionality, the project may still not be a success. These three metrics (budget, time, functionality) tells us little about the project itself and much more about our ability to make predictions about the project.

Take a simple example. The business tells IT they need a system that delivers 100 functions. IT calculates the cost at $100K/function. They tell the business the project will cost $10M and three years to deliver. Business approves the project. IT delivers the project one month early and $1M below budget. The business is happy. IT looks good. The project is a success!

Or is it? All this tells us is that IT did what they said they would do. It doesn't tell us whether what IT said they would do was really reasonable.

In most projects that size, complexity is a major cost factor. If IT and the business take appropriate steps to manage that complexity, the cost/function can be greatly reduced. If a project can be delivered for $9M without complexity management, then it is highly likely that it could have been delivered for $5M with complexity management.

So is this project a success or a failure? According to most pundits (such as Standish) this project is an unqualified success. In my book, it is a $4M failure. This is one more reason I say that looking at the percentage of IT successes is a meaningless statistic. What we need to look at is the percentage of IT budgets that are wasted.

The Fundamental Flaw of Complexity Management

2009-12-06T15:37:00.000-08:00

As systems add more functionality, they become more complex. As systems become more complex, the traditional processes we use to manage those systems become more strained. The typical response on the part of those building these more complex systems is to try to understand how to scale up the processes from those that can handle simple systems to those that can handle complex systems.

Consider a "simple" system A with only 100 functions. Say process X has been used to successfully manage A. Process X could be Agile Development, RUP, Earned Value Management, or any other of your favorite management processes.

Now we need to build a more complex system B with 1000 functions. Since B has 10X the functionality of A and we know X works for A, most assume that we can use X to manage B as well, although we assume that it will take 10X the effort.

The flaw in this reasoning is that the difficulty of applying X to B (regardless of what X is) is proportional to the complexity of B, not to the functionality of B. And when the functionality increases by 10X, the complexity, because of the exponential relationship between functionality and complexity, actually increases thousands of times. The exact number is highly dependent on the nature of the functions of B and how they are organized, but the number will typically be very large.

As long as we focus on how to better use X to manage B, we are doomed to failure. The complexity of B will quickly outpace our ability to apply X.

Instead, we need to focus on the real problem, the complexity of B. We need to understand how to architect B not as a single system with 1000 functions, but as a cooperating group of autonomous systems, each with some subset of the total functionality of B. So instead of B, we now have B1, B2, B3, ... etc. Our ability to use X on each of Bi where i = 1, 2, ... will be dependent on how closely the complexity of the most complex Bi is to the complexity of A (which is the most complex system on which X is known to be a viable process.).

The bottom line: if we want to know how to use process X on increasingly complex systems, we must focus not on scaling up the functionality of X, but on scaling down the complexity of the systems.

For more information on scaling down complexity in IT systems, see my white paper, "The IT Complexity Crisis" available at http://bit.ly/3O3GMp.

The IT Complexity Crisis: Danger and Opportunity

2009-11-08T19:44:00.000-08:00

Roger's new white paper, The IT Complexity Crisis: Danger and Opportunity is now available.

Overview
The world economy is losing over six trillion USD per year to IT failures and the problem is getting worse. This 22 page white paper analyzes the scope of the problem, diagnoses the cause of the problem, and describes a cure to the problem. And while the cost of ignoring this problem is frighteningly high, the opportunities that can be realized by addressing this problem are extremely compelling.

The benefits to understanding the causes and cures for out-of-control complexity can have a transformative impact on every sector of our society, from government to private to not-for-profit.

Downloading the White Paper
You can download the white paper, download an accompanying spreadsheet for analyzing architectural complexity, and view various blogs that have discussed this white paper here.

Would you like to discuss the white paper? Add a comment to this blog!

The Problem With Standish

2009-10-29T16:52:00.000-07:00

In my recent white paper, The IT Complexity Crisis, I discussed how much IT failures are costing the world economy. I calculated the worldwide cost to be over $6 trillion per year. You can read the white paper here.

In this white paper I discuss the Standish Chaos numbers, but many readers have continued to question whether my conclusions are in agreement with Standish. I think my conclusions are in agreement, but I also think the Standish numbers are flawed. So I have mixed feeling about them. Let me explain.

The Standish Group has been publishing their annual study of IT failure, their "CHAOS Report" since 1994 and it is widely cited throughout the industry. According to the 2009 report, 24% of all IT projects failed outright, 44% were "challenged", and only 32% were delivered on time, on budget, and with required features and functions.

To be honest, I have never read the Standish Report. Given the $1000 price tag, not many people have. So, like most people, I am basing my analysis of it on the limited information that Standish has made public.

The problem with the Standish Report is not that it is analyzing the numbers wrong. The problem is that Standish is looking at the wrong numbers. It analyzes the percentage of IT projects that are successful, challenged (late, overbudget, etc.), or outright failures. This sounds like useful information. It isn't.

The information we really need is not what percentage of projects are successful, but what percentage of IT budgets are successful.

What is the difference between percentage of projects and percentage of budget? A lot. Let me give you an example.

Suppose you are an IT department with a $1M budget. Say you have six IT projects completed this year, four that cost $50K, one that cost $100K, and one that cost $700K.

Which of these projects is most likely to fail? All other things equal, the $700K project is most likely to fail. It is the largest and most complex. The less the project costs, the simpler the project is. The simpler the project is, the more likely it is to succeed.

So let's assume that three of the four $50K projects succeed, the $100K project succeeds, and the $700K project fails.

Standish would report this as 4/6 success rate, or a 67% success, 23% failure rate. I look at these same numbers and see something quite different.

I look at the percentage of IT budget that was successfully invested. I see $250 K of $1M budget invested in successful projects and $750 K in failed projects. I report this as a 25% success rate, a 75% failure rate.

So both Standish and I are looking at the same numbers, yet we have almost exactly opposite conclusions. Whose interpretation is better?

I argue that, from the organizational perspective, my interpretation is much more reasonable. The CEO wants to know how much money is being spent and what return that money is delivering. The CEO doesn't care how well the IT department does one-off minor projects, which are the projects that dominate the Standish numbers.

So the bottom line is that I have major issues with the Standish Report. It isn't that the Standish analysis is wrong. It is just that it is irrelevant.

Notes from ITARC NYC Open Meeting on IT Failures

2009-10-15T07:15:00.000-07:00

At the recent ITARC conference in NYC, I facilitated a open meeting on IT Failures. We only had one hour, but some interesting ideas were discussed. Thanks to Eric Weinstein for taking these notes.

Reasons people gave for IT failures they experienced:

-Lack of change management

- Scope of the requirements too high level/incomplete or fleshed out leading to bad outcomes

- Analysis of cost estimation was wrong b/c the requirements were not fleshed out

- Cost estimation is an art, man power, resource time, are hard to estimate.

- Lack of accurate communication and feedback AND whether the project is understood

- Final delivery had no bearing on value for the customer - all communication from the developers back to the business stakeholders that came back was totally ignored

- Functional requirements get a lot of attention BUT the non-functional requirements is invisible/doesn't get credit, how to quantify the cost avoidance, non-functional requirements

-Trade off of quick vs. correctly - executive irresponsibility

-Business has unrealistic expectations of delivery dates OR tech people in general estimating time - may skimp out on upfront analysis or testing...

-Implementation side - developers failing - tools to control SDLC process - source control system (full integration of code
check in - what requirements that code is fulfilling - must be reviewed sign/off)

- Main causes of failure is managing complexity of large systems - failure vs complexity has high relationship - more complex a system, the harder it is to scope...must learn how to take big monolithic systems and break down to smaller systems

Solutions

- "The Wrench in the System" Book recomendation

- Ask the business to delineate the success criteria, prioritize in numbers

- Understand timeframe, scope - rescope

- White paper - US Gov't - 66% of IT budget is high risk projects and half of those will fail

Attacking Architectural Complexity

2009-10-04T15:13:00.000-07:00

When I advocate for reducing the complexity in a large IT system, I am recommending partitioning the system into subsystems such that the overall complexity of the union of sub-systems is as low as possible while still solving the business problem.

To give an example, say we want to build a system with 10 business functions, F1, F2, F3, ... F10. Before we start building the system we want to subdivide the system into subsystems. And we want to do it in the least complex collection of subsystems.

There are a number of ways we could partition F1, F2, ... F10. We could, for example, put F1, F2, F3, F4, and F5 in S1 (for subsystem 1) and F6, F7, F8, F9, and F10 in S2 (for subsystem 2). Let's call this A1, for Architecture 1. So A1 has two subsystems, S1 with F1-F5 and S2 with F6-10.

Or we could have five subsystems with F1, F2 in S1, F3, F4 in S2, etc.. Let's call this A2, for Architecture 2. So A2 has five subsystems, each with two business functions.

Which is simpler, A1 or A2? Or, to be more accurate, which is less complex, A1 or A2. Or, to be as accurate as possible, which has the least complexity, A1 or A2?

We can't answer this question without measuring the complexity of both A1 and A2. But once we have done so we know which of the two architectures has less complexity. Let's say, for example, that A1 weighs in at 1000 SCUs (Standard Complexity Units, a measure that I use for complexity) and A2 weighs in at 500 SCUs. Now we know which is least complex and by how much. We know that A2 has half the complexity of A1. All other things being equal, we can predict that A2 will cost half as much to build as A1, give twice the agility, and cost half as much to maintain.

But is A2 the best possible architecture? Perhaps there is another architecture, say A3, that is even less complex, say, 250 SCUs. Then A3 is better than either A1 or A2.

One way we can attack this problem is to generate a set of all possible architectures that solve the business problem. Let's call this set AR. Then AR = {A1, A2, A3, ... An}. Then measure the complexity of each element of AR. Now we can choose the element with the least complexity. This method is guaranteed to yield the least complex architectural solution.

But there is a problem with this. The number of possible architectures for a non-trivial problem is very large. Exactly how large is given by Bell's Number. I won't go through the equation for Bell's number, but I will give you the bottom line. For an architecture of 10 business functions, there are 21,147 possible solution architectures. By the time we increase the system to 20 business functions, the number of architectures in the set AR is more than 5 trillion.

So it isn't practical to exhaustively look at each possible architecture.

Another possibility is to hire the best possible architects we can find on the assumption that their experience will guide them to the least complex architecture. But this is largely wishful thinking. Given a 20 business function system, the chances that even experienced architects will just happen to stumble on the least complex architecture our of more than 5 trillion possibilities is slim at best. You have a much better chance of winning the Texas lottery.

So how can we find the simplest possible architecture? We need to follow a process that leads us to the architecture of least complexity. This process is called SIP, for Simple Iterative Partitions. SIP promises to lead us directly to the least complex architecture that still solves the business problem. SIP is not a process for architecting a solution. It is a process for partitioning a system into smaller subsystems that collectively represent the least complex collection of subsystems that solve the business problem.

In a nutshell, SIP focuses exclusively on the problem of architectural complexity. More on SIP later. Stay tuned.

Why I Focus On Complexity

2009-10-01T05:04:00.000-07:00

When it comes to IT failure, there is no lack of "the usual suspects". We can look to failures in project definition, project management, and needs assessment. We can point to flawed architectures, implementations, and testing procedures. We can focus on communications failures between the business and IT, between IT and the user community, and between different business units.

Yet given this extensive collection of failure factors any of which can doom an IT project, why do I focus almost exclusively on the issue of complexity?

I see all of the failure factors as falling into one or more of three categories:

1. The factor is caused by complexity.
2. The factor is greatly exacerbated by complexity.
3. The factor is solved as a side effect of solving the complexity problem.

Examples of failure factors that are directly caused by complexity are the various technical failure factors, such as poor security or scalability. It is very difficult to make a complex system secure or scalable. Solve the problem of complexity, and these problems become much easier to solve.

Examples of failure factors that are greatly exacerbated by complexity include those related to communications. As a project increases in complexity, people tend to fall more and more into specialized jargon which makes communications more difficult and adds yet more complexity to the already complex project. Different groups tend to see each other as the enemy. As a side effect of learning to solve complexity, the groups learn to see not each other as the enemy, but complexity as their common enemy. Their common language becomes the language of simplification.

Examples of factors that are solved as a side effect of solving the complexity problem include those related to organization. For example, a well known failure factor is the lack of executive sponsorship. But it is difficult to find sponsors for large complex expensive projects. Once a project is broken down into small, simpler, less expensive projects, finding executive sponsors for those projects is much easier.

The other reason I focus so much on complexity is that of all these failure factors, complexity is the only one that is universally present in all failed projects. The fact is that we are quite good at doing simple projects. Our skills just don't scale up to complex projects. So we can either tackle the failure factors piecemeal and try to figure out how to scale each up to higher levels of complexity, or we can try to figure out how to scale down the complex projects into simple projects that we already know how to solve.

So while I pay close attention to all of these failure factors, I continue to believe that the one that deserves our undivided attention is the problem of complexity.

Cost of IT Failure

2009-09-28T13:01:00.000-07:00

What does IT failure cost us annually? A lot.

According to the World Technology and Services Alliance, countries spend, on average, 6.4% of the Gross Domestic Product (GDP) on Information Communications Technology, with 43% of this spent on hardware, software, and services. This means that, on average, 6.4 X .43 = 2.75 % of GDP is spent on hardware, software, and services. I will lump hardware, software, and services together under the banner of IT.

According to the 2009 U.S. Budget, 66% of all Federal IT dollars are invested in projects that are “at risk”. I assume this number is representative of the rest of the world.

A large number of these will eventually fail. I assume the failure rate of an “at risk” project is between 50% and 80%. For this analysis, I’ll take the average: 65%.

Every project failure incurs both direct costs (the cost of the IT investment itself) and indirect costs (the lost “opportunity” costs). I assume that the ratio of indirect to direct costs is between 5:1 and 10:1. For this analysis, I’ll take the average: 7.5:1.

To find the predicted cost of annual IT failure, we then multiply these numbers together: .0275 (fraction of GDP on IT) X .66 (fraction of IT at risk) X .65 (failure rate of at risk) X 7.5 (indirect costs) = .089. To predict the cost of IT failure on any country, multiply its GDP by .089.

Based on this, the following gives the annual cost of IT failure on various regions of the world in billions of USD:


REGION        GDP (B USD)  Cost of IT Failure (B USD)
World         69,800       6,180
USA           13,840       1,225
New Zealand   44           3.90
UK            2,260        200
Texas         1,250        110

Sessions's Complexity Aphorisms

2009-09-20T03:52:00.000-07:00

- Complexity is like heat. We need just enough to solve the problem. Any more kills us.

- The best IT solution is the simplest IT solution that solves the business problem.

- The simplest IT solution is the NULL solution, but that fails the effectiveness test.

- Complexity is the enemy.

- The cloud is a platform. Not a complexity solution.

- 60% of all IT budgets are invested in projects that are at risk for unnecessary complexity.

- Occam's Razor Applied to IT: When you have two competing architectures that solve exactly the same business problem, the simpler one is the better.

- Consulting organizations perpetuate complexity through fear: This project is too risky. You will fail and be blamed. Let us fail and be blamed.

- IT Complexity is a tax paid by everybody that benefits nobody.

Three Rules for IT Simplification: Small, Separate, and Synergistic.

2009-07-06T05:44:00.000-07:00

The most important factor in building a successful IT project is simplification. Simplification means that the overall system should be as simple as it can possible be while still delivering the goals of the project.

In general, there are three rules you must follow to build a simple IT system.
- Keep it small.
- Keep it separate.
- Keep it synergistic.

Let's take a look at each of these.

Rule 1. Keep it small.
Keeping it small means that a system should have no more functionality than it needs to deliver its defined goals. Every ounce of IT functionality must be clearly traceable to the business goals of the project. If a function is not traceable, it should be removed.

This, of course, can only be done if the business goals are clearly understood. Usually they are not. So make sure that at the beginning of the project, the business folks and the IT folks have clearly documented each business goal of the system. And that every bit of planned functionality can be mapped back to one of those goals.

This goal is often at odds with people's desire to make systems "reusable". In order to make systems more "reusable", IT likes to make systems more configurable and more functional. IT often convinces the business that reusability is an important goal, thereby justifying their efforts. Efforts to achieve reusability almost always add unnecessary complexity, and, in the end, rarely succeed. Think long and hard before you sacrifice simplicity at the alter of reusability.

Rule 2. Keep it separate.
An IT organization will inevitably need many systems and these systems will need to interoperate. Interoperability is at odds with simplicity. From a simplicity perspective, the less interoperability between systems, the better. From a corporate perspective, the more interoperability between systems, the better.

You can see the conflict. It is important to strike a balance between these two objectives. Design as much interoperability as necessary, but no more than necessary and design it with as much respect as possible for the boundaries between systems.

Rules 3. Keep it synergistic.
One of the most critical issues, from a simplicity perspective, is how functionality is placed. Many systems suffer huge complexity problems because functionality that should be co-located is not, or, just as serious, functionality that should not be co-located, is. It is critical that functions be placed based on natural (that is, business-defined) synergies.

So if simplicity is your game, remember the three S's that govern simplicity: small, separate, and synergistic. You won't be sorry!

How Many Enterprise Architects Does It Take To Measure A Donkey?

2009-04-30T11:40:00.001-07:00

I don't know how this got started, but we were tweeting a discussion about enterprise architecture, and somehow the question came up...

How Many Enterprise Architects Does It Take To Measure A Donkey?

A: depends on where the datum is and which part of the donkey they measure!

A: and should not speed of donkey be considered too? For relativity effects I mean.

A: ahh yes, but it is velocity and in the direction of measurement! A jumping donkey!

A: jumping donkeys - can I have some of what you two are on?

A: EAs measuring the donkey? One measures required height of TO-BE donkey, one argues length and height are basically the same, ...

A: .. one says the problem is that the front half of the donkey and the rear half are not in alignment.

A: ... one says that we can't measure the donkey until we have completed the business case.

A: one say that we need an industry consortium to define the best practices for donkey measuring.

A: ... one says we don't need to measure the donkey. It's not strategic. We need to outsource it.

A: ... one says that we can't measure the donkey until we first partition it into little pieces so that it isn't so complex.

With contributions from @RSessions, @richardveryard,@seabird20, @taotwit, and @j4ngis. (Did I miss anybody?)

Factors Driving System Complexity

2009-04-15T12:10:00.000-07:00

IT Systems are failing at an alarming rate and the rate of failure is increasing. Approaches that we have used in the past to successfully deliver IT systems are no longer working.

Something has changed: the complexity of the systems. It should be no surprise to anybody in IT that the complexity of the systems we are being asked to build has dramatically increased. The only surprise is that we have not adapted the processes that we use to build IT systems to take into account this increase.

Processes that can drive simple IT systems development do not scale up as the complexity of the systems increases beyond a certain point, a point that we have long passed.

I do not advocate throwing out what we know about building simple systems. Approaches such as Agile Development are great, as long as the complexity of what we are building is manageable. The approach that I advocate is learning how to break large, complex systems that we do not know how to build into smaller, simpler systems that we do know how to build.

I call this process partitioning. It is important to remember that the goal is not just to create smaller systems, but smaller systems that are as simple as possible. The specific process that I advocate for doing this is called SIP, for Simple Iterative Partitions. SIP is designed from the start to generate systems that are both small and simple.

While the goal of partitioning is to create smaller simple systems, the process itself is neither small nor simple. Planning for simplicity paradoxically adds complexity to the planning process but pays huge dividends further on. The extra work involved in delivering complex systems is far greater than the extra work involved in planning for simplicity from the beginning.

Partitioning is a science, like medicine. It must be led by people who understand the nature of the disease and have been trained in how to manage it. The training is important because there are many things that can go wrong. When things do go wrong, complexity is inadequately removed, or, in some cases, made worse.

IT failures due to poor partitioning are epidemic in our industry. For example, there are many failed service-oriented architectures (SOAs). Behind almost all of these SOA failures is a partitioning failure.

I have been studying IT failures for a number of years now. I have concluded that partitioning failures are the primary cause of most IT failures. The bigger the failure, the more likely it is that incorrect partitioning is the root cause. The symptom of partitioning failure is always the same: the project seems swamped by complexity. If you have worked on a project whose complexity killed the project, then you have experienced firsthand the effects of partitioning failure.

I have seen so many partitioning failures that I have started to recognize underlying patterns or categories of partitioning failures. At this point, I have identified eleven categories, and the list is growing as I analyze more failures. The eleven categories are as follows:

Delayed partitioning: The partitioning did not begin early enough in the project life cycle. It should be mostly complete before the business architecture is begun.
Incomplete decomposition: The decompositional analysis of functions did not go far enough, resulting in too coarse a granularity of the functions being partitioned.
Excessive decomposition: The decompositional analysis of functions went too far, resulting in too fine a granularity of the functions being partitioned.
Bloated subsystems: Too many functions have been assigned to one or more subsystems.
Scant subsystems. Too few functions have been assigned to one or more subsystems.
Incorrect assignment: Multiple functions have been assigned to the same subsystem that do not belong together or have been assigned to different subsystems that do belong together.
Duplicated capabilities: Functions have been duplicated across different subsystems.
Unnecessary capabilities: Functions have been partitioned that should not have been included in the first place.
Technical partition mismatch: The technical partitioning does not match the business partitioning, a common problem in purchased systems and service-oriented architectures.
Inadequate depth: The technical partitioning does not extend far enough. In SOAs, this problem often manifests itself as multiple services sharing common data.
Boundary decay: The boundaries between subsystems in the partition are not strong enough, and functionality slips back and forth between subsystems.

As you can see, there are many opportunities for errors in complexity management. And complexity (and therefore IT failure) is highly sensitive to partitioning errors, much more so than to implementation errors. It is critical, therefore, to have a well defined process to guide you and check your results along the way.

(Thanks to Chris Bird for suggesting this topic with his innocent tweeted question: “What are contributors to complexity?”)

Adaptability of Large Systems

2009-04-02T15:25:00.000-07:00

As I was tweeting the other day, I came upon a tweet from Noel Dickover about large systems being less adaptable than small systems. This began a tweet exchange that I thought brought up some important issues about adaptability.

Many people believe that large systems are harder to adapt than smaller systems. And, in genera, this is true. But system adaptability is not a function of size, it is a function of architecture. When a large system is partitioned correctly so that it is composed of a number of smaller, autonomous systems, then it MAY be more adaptable than a single smaller system.

I say "may", because its adaptability depends on how well it has been partitioned. The key is not whether the partitioning is good technically (say, mapping well to an SOA), but how well the technical partitions overlay on top of the business partitions of the organizations.

In other words, when a large system is built of autonomous smaller systems AND those smaller systems map well to the autonomous processes that occur naturally within the business, then, and only then you have a large system which is highly adaptable.

The reason so many systems fail to achieve this (even when they do manage a reasonable technical partitioning) is that the technical partitioning MUST BE driven by the business partitioning. This requires a partitioning analysis of the business that is completed before the technical architecture of the system is even begun.

This business partitioning analysis is best done by representatives of both the business and the IT organization. The business group has the best understanding of how functions relate to each other and the technical group has the best understanding of how this business partitioning analysis will eventually drive the technical partitioning architecture.

Since both business and technical experts are involved in this exercise, I place this work in the common ground between business and technology, the watering hole that we call enterprise architecture. But it is enterprise architecture with a very specific focus: driving technical partitioning from business partitioning analysis with the eventual goal of highly flexible systems that are pegged closely to the business need and mirror closely the business organization.

The Cancer of Complexity

2009-03-12T18:51:00.000-07:00

After many years in IT, I am convinced that code complexity is a relatively unimportant issue. This may sound strange, coming from someone who is always reminding people that Complexity is the Enemy. How can I not care about code complexity?

What is much more important than code complexity is how that code is architecturally organized. We can deal with complex code if that code is well sequestered. It is the effectiveness of the sequestering rather than any pockets of code complexity that will determine the complexity of the larger system.

Complex code that is well sequestered has what I describe as "benign complexity". An example of a system with benign complexity is a web service that is poorly written (complex) but well encapsulated (sequestered). IT systems that have benign complexity may have localized problems, but these problems are rarely lethal to the system as a whole, and, if they can't be solved, can at least be surgically excised.

Complex code that is poorly sequestered has what I describe as “malignant complexity”. An example of a system with malignant complexity is collection of services that all share common persistent data. These systems are also complex, but now the complexity is not localized and is almost impossible to address. These systems are usually headed for serious problems.

There are many similarities between malignant cancer (that is, cancer that is progressive and uncontrolled) and malignant complexity. Here are some that I have identified in recent twitter conversations:

Both malignancies grow exponentially over time.
It is difficult or impossible to control the rate of growth. (Thanks to A. Jangbrand for this one.)
Both malignancies can be prevented much more easily than cured.
Left to their own devices, both malignancies will destroy their host.
When removal is attempted, it is easy to splinter the malignancy and form new malignancies that can themselves grow. (Thanks to Richard Veryard for this one.)
By the time symptoms are noticed, the malignancy has often reached an advanced and sometimes incurable state.
Both malignancies can spread to other locations that are only remotely connected with the original location.

So how complexity is organized, benign or malignant, is more important than the degree of complexity itself. Standard code complexity analysis approaches, such as cyclomatic analysis, fail to take into account how code is organized. This is why these metrics are rarely useful in any kind of large scale complexity analysis. They may tell us that a function, perhaps even an application, is complex, but they never tell us whether that complexity is limited to the function or application, or is about to spread to the larger system.

I do believe that complexity is the enemy. Until we better understand complexity, our chances of building better IT systems is limited. The first thing we must understand about complexity is that not all complexity is equal. And the complexity on which most people focus is probably the least complex complexity of all.

(Join the twitter conversation - @RSessions.)

Editorial: Obama's Information Technology Priority

2009-01-27T08:51:00.000-08:00

Federal IT projects fail at an alarming rate. The total cost to the U.S. economy? According to my calculations, at least $200 billion per year. This editorial that I wrote is reprinted from the Perspectives of the International Association of Software Architects (January 2009). It is an in-depth analysis of why so many Federal IT systems are in trouble and what steps the Obama administration needs to take to control this problem.

You can pick up a PDF version of the editorial here. And if that doesn't work, go to www.objectwatch.com and follow the whitepapers tab.

What do you think? Leave a comment and let me know.

ObjectWatch Newsletter: Enterprise Architecture in Hard Times

2008-12-03T07:30:00.000-08:00

We are facing some very difficult financial times. Can enterprise architecture survive? From the article: EA can survive hard times, even thrive. But first we must make some major changes in how we view EA. No more poorly defined payoffs that are years in the future. The payoffs must be immediate. They must be real. And they must be compelling. What are these necessary changes? Read the article here and find out.

Senate Bill S.3384

2008-11-25T08:03:00.000-08:00

In reading Michael Krigsman's blog, I was introduced to Senate bill S.3384, "Information Technology Investment Oversight Enhancement and Waste Prevention Act of 2008." The bill was placed on the Senate Legislative Calendar on 10/1/08.

This bill was introduced by Senator Thomas Carper (D-DE) and cosponsored by Sen. Norm Coleman (R-MN), Sen Joseph Lieberman (ID-CT), Sen. Susan Collins (R-ME), Sen Claire McCaskill (D-MO), and Sen. George Voinovich (R-OH). The bipartisan support (two democrats, two republicans, and an independent) in itself makes the bill noteworthy.

The bill requires that any government IT investment project that has "significantly deviated" from its projections must be reported up the command chain, ultimately ending with the responsible CIO. Any project that is off its projections by more than 20% is considered "significantly deviated." Projects that are in "gross deviation" are further reported to the appropriate congressional committee. "Gross deviation" is considered more than a 40% deviation from its projections.

Deviations are measured relative to the Earned Value Management (EVM) benchmark. This benchmark includes timely measures for project expenditure, project completion, and project deliverables.

S.3884 requires that this reporting be done on a quarterly basis. This means that deviations must be measured (and reported) long before the project is actually completed.

State governments often follow the lead of the federal government, especially in "watch-dog" type legislation. Therefore it seems likely that S.3884 may have an impact beyond just the federal government and into all areas of the public sector.

EVM benchmarks are highly dependent on the ability of an organization to accurately forecast both deliverables and costs. These are both areas in which the government has historically been weak. I have frequently written about governmental IT projects that have gone under-promise, over-budget, past-due, or all of the above. If S.3884 is going to be successful, it will require a major new approach in how government does IT.

The biggest problem that the government has with IT is its chronic failure to manage complexity. The more complex a project, the harder it is to project cost and predict deliverables, two absolute requirements for an EVM analysis. There is no possible way that S.3384 can be implemented successfully unless the government first takes steps to understand and manage IT complexity.

I strongly support any requirement that the government do a better job with IT. But I think the Senate needs to be realistic. Passing S.3384 without giving governmental IT bodies the tools they need to address IT complexity is setting them up for failure.

Let's phase in S.3384. First, introduce SIP (or some other equivalent methodology) to the government to show how highly complex systems can be organized as much smaller, simpler, autonomous systems. Then hold the government accountable for how it delivers those simpler systems.

If the government takes the two steps of IT complexity management and IT accountability. we can transform how the government does IT. We will have a government that delivers IT projects on-time, on-budget, and meeting the needs of the citizens that pay for those projects. That is you and me.

Carving up an Enterprise Architecture

2008-11-12T11:15:00.000-08:00

Melissa A. Cook recently published an article in ComputerWorld titled Enterprise Architecture: Lessons from the cutting board. In this article, Melissa likens developing an enterprise architecture to cutting up a chicken for dinner. She gives three lessons for creating an enterprise architecture.

Lesson 1: Cut through the joints, in other words, look for natural boundaries around business processes. As she points out, “if you get the natural boundaries right, it simplifies the number and complexity of interfaces. How do we do this? Melissa says “we need to take a gander at the enterprise goose, the whole thing.” A good check point is looking at data: “a ton of data elements or columns of data on an interface, you probably have the boundaries wrong.”

Lesson 2: Natural breakpoints or boundaries are similar for similar organizations. In other words, see what others that are in similar businesses have done and use this for a model.

Lesson 3: Boundaries don’t change based on size of the organization. Big turkeys carve much like little chickens.

I agree with Melissa strongly on two points. First, if you get your enterprise architecture right, you will make a great leap forward in managing the complexity of your systems. Second, your success in getting the architecture right is reflected in the data structures moving around your system.

Having said that, I feel several points need more comments. Let’s start with her thoughts on data. While I agree that poor enterprise architectures result in complex data patterns, I disagree that this is a good test to use to evaluate your enterprise architecture. The reason is simple. By the time you can notice the problems with data, it is far too late to do anything about the problems with the architecture. You have already implemented your systems and any changes in the enterprise architecture at this point are going to be very expensive.

So what is the solution? Any enterprise architecture should be tested against theoretical models for enterprise architectures based on the mathematical laws of complexity. These tests can be conducted BEFORE the architecture is implemented and can pinpoint problem areas before money is spent on their implementation at a time when they are still relatively inexpensive to fix.

Another area that we need to look at more closely is her lesson 2. While it is true that similar organizations have similar boundaries, this is not particularly helpful information. Why? Because “similar” is not good enough. Even small errors in “cutting up the chicken” can lead to huge problems with complexity. I personally have seen millions of dollars lost due to a single poor function placement. Again, the answer is to methodically check each function for correct placement, not based on a particular chef’s preferences, but based on the strict mathematics of synergies.

Don’t get me wrong. Theoretical analysis can only take you so far. But it can take you a lot further than we can go by guess-work, checking data patterns, or even looking at industry patterns.

The Job of the Enterprise Architect

2008-11-10T09:07:00.000-08:00

ZDNet recently published an article on My Awesome IT Job: Enterprise Architect, IBM. The article is an interview with Martine Combes, an Enterprise Architect working with the CIO Organization of IBM in France. In the article, Martine says, "It is very easy to become overwhelmed by complexity and it is very difficult to maintain simplicity, especially when the scope is large, the number of team members is high. So better to have a simple architecture from the start, clearly showing the benefits for the Business."

On one hand, I applaud Martine's awareness of the importance of simplicity when it comes to Enterprise Architecture. On the other hand, I feel there is some degree of naivety in this understanding of the nature of complexity.

Marine appears to be saying that if you build your architectures simple from the beginning, you will have addressed the problem of complexity. There are several problems with this understanding of complexity. First of all, making an architecture "simple" depends on the eyes of the viewer. What is simple to me may be complex to you. A better goal than making architectures "simple" is to make architectures "as simple as possible." It is not possible to prove that an architecture is "simple", since "simple" is subjective. But it is possible to show that an architecture is "as simple as possible." The test for "as simple as possible" involves showing that the functionality follows the rules of synergistic partitioning. If you can find a subset of functionality that violates synergistic partitioning, then you know the architecture can be made simpler. If there is no such subset, the architecture is as simple as it can possibly get, and any further efforts to simplify the architecture will actually make it more complex.

My second issue with Martine's understanding of complexity involves the implicit assumption that just because an architecture starts simple that it will stay simple. In fact, architectures tend towards increasing complexity unless ongoing effort is put into keeping them simple. It is harder to keep an architecture simple than it is to make it simple in the first place. It requires an continuing analysis of new functionality and rigorous placement of all new functionality according to the rules of synergistic partitioning.

Whose job is it to keep make sure than architectures start out "as simple as possible" and stay that way for the system's lifespan? In my view, this is the primary responsibility of the enterprise architect. Only this person has the business perspective needed to identify synergies and the technical perspective needed to understand partitioning.

An organization that lacks a viable program in enterprise architecture will pay a severe cost in IT complexity. Complexity is the enemy. Enterprise Architecture is the solution. The only solution.

- Roger Sessions

The Future of Enterprise Architecture

2008-10-30T08:15:00.000-07:00

Kristian Hjort-Madsen recently posted a blog entry about the future of Enterprise Architecture in the government. Having done some work on architectural simplification for several public sector clients, this got me thinking about not only the future of EA in the government, but the future of EA in general.

It is my belief that we need to rethink Enterprise Architecture from being something that is focused on understanding the Enterprise to something that is focused on delivering value to specific projects. To some extent, this calls into question the name, "Enterprise Architecture", but I still think the name has value because it implies that we are considering project level issues that can only be addressed at the enterprise level.

To go even further, the main value that "Enterprise Architecture" can deliver to projects is in reducing their overall complexity. There are a number of secondary issues on which we can also focus, such as improving business/IT communications, but I believe that most of these will greatly benefit just from reduced project complexity.

Why do we need to focus on complexity at the Enterprise Level? Why not the IT level or the business level? The answer to this requires an overview of the Science of Simplicity. This is a bit much to cover here, but is the topic of my last book, Simple Architectures for Complex Enterprises and numerous white papers at my web site. In brief, simplification requires a good understanding of both synergistic relationships (which are best understood by the business side of the enterprise) and strong partition boundaries (which are best understood by IT side of the enterprise). We can thus only address these issues at the juncture point between business and IT, and that juncture point is what we usually call "Enterprise Architecture".

I think that government is perhaps the area that can particularly benefit from a focus on simplification. Government projects tend to be highly complex and often involve multiple vendors. When complexity is not managed methodically and early on, failure is too often the result. An excellent example of the problems inherent in such projects is the UK NHS system known as NPfIT (National Programme for Information Technology). This system will likely cost 50-100 billion dollars, and most likely will end up in failure. Why? Catastrophic complexity.

Why is complexity such a problem? Several studies (and numerous personal observations) have shown that increasing the functionality of a system by 25% increases the complexity of the system by 100%, unless steps are taken to manage the complexity. This means that taking a system from 10 functions to 100 functions increases the complexity by over 1000 times! Clearly this is not acceptable. The only way we can dampen this increase is by focusing on the problem of complexity. And Enterprise Architecture is where we have the opportunity to do so.

So let me summarize my thoughts on where Enterprise Architecture needs to go in the near term, especially in the public sector. First, it needs to move from an enterprise focus to a project focus. Second, it needs to narrow its focus to those problems that can only be solved from an enterprise (that is, a broad business/IT) perspective. Third, it needs to hone in on the one major problem that can only be solved from an enterprise perspective: complexity.

If we, as Enterprise Architects, make these adaptations, we will start making important, measurable project contributions and delivering highly visible short term value. And, in doing so, ensure our own long term survival.

Can an EA be "correct"?

2008-10-21T07:53:00.000-07:00

In a recent blog, Nick Malik asserted that EA is not about building apps right, but is about building the right apps. And that the correctness of an EA was a point of view.

This was my response:

In my view, EA is not about either building apps right OR building the right apps. It is about creating a structured framework within which meaningful and collaborative dialogues can occur between the business and IT groups. The term that I use to describe this "structured framework" is Enterprise Architecture.

In most organizations, "meaningful and collaborative dialogues" between business and IT do not occur. You must ask why this is so. Is it because the business folks are overly demanding and unreasonable? Is it because the IT folks are unable to understand what business is saying?
In my view, the problem is neither the business nor IT. The problem is that EA has either not been done at all or, if it has been done, has been done incorrectly. Which brings us back to Nick's original point. Is there a "correct" way to do EA?

The answer is absolutely, yes. But the starting point needs to be an understanding of what we mean by "correct". I define "correct" as "as simple as possible". So of two EAs that both accurately define the enterprise, the more correct one is the simpler of the two. And the EA that is "absolutely" correct is the EA that is the simplest possible EA.

Why do I believe that simplicity is the raison d'etre for EA? The answer goes back to my original question. What is it that gets in the way of meaningful dialogues between business and IT? In my opinion, what gets in the way is complexity.

When IT blames the business for fuzzy requirements, IT is wrong. When the business blames IT for inability to deliver, the business is wrong. The problem is neither IT nor the business. The problem is complexity. Until complexity is solved, dialogue is impossible. Complexity is the enemy, and it is the common enemy of both IT and the business.

So can you determine if an EA is as simple as possible? Again, the answer is yes. But to do so requires both an understanding of the mathematics of complexity and a rational process for testing an EA for being "as simple as possible".

These are both topics I have covered extensively in my book, "Simple Architectures for Complex Enterprises", so if you are interested in these ideas, that is where to go. I also have a number of papers on this topic at my website, http://www.objectwatch.com/.

Best Wishes,
Roger Sessions

Complexity is not a technology problem

2008-08-21T10:03:00.000-07:00

In a recent copy of CIO Magazine, John Parkinson wrote an article titled Simplifying IT Management is Anything But. In this article, John discussed the difficulty in seeing through the vendor presentations to see which technologies would really make a difference to an organization.

In my view, if you are trying to understand what technologies to use to address complexity, you are probably too late. The solution to complexity is never found in technology. It is found in architecture. And architecture is mostly technology neutral. My perspective on complexity is that it can only be addressed at the enterprise architectural level, and then only if an organization specifically focuses on that as the overriding issue. In my book, Simple Architectures for Complex Enterprises, I address the issue of architectural complexity in depth. In summary, we must first understand complexity, then model it, then build processes that can be proven to address it. But the single most important factor is attitude. We need to view complexity as an enemy and simplicity as a core business asset. Every business analyst and IT architect needs to unite in the battle against complexity. Most IT systems fail, and the bigger they are, the harder they fall. When an IT system fails, the reason is almost always unmanaged complexity.

Simple Architectures - The Book (Discussion)

2008-08-15T11:47:00.000-07:00

Simple Architectures For Complex Enterprises is now available. You can order it at Amazon. This book introduces a new way of thinking about Enterprise Architecture. The focus of Enterprise Architecture should not be on efficient use of IT, improving alignment between IT and the business, or any of the other traditional concerns of Enterprise Architecture. Why? Because none of these are the real problem, they are merely symptoms of the problem. The real problem is enterprise complexity. This book discusses the nature of enterprise complexity, why complexity causes so many problems for IT, how complexity can be understood, and, most important, how complexity can be attacked.

Would you like to discuss some of the issues raised in Simple Architectures? This is as good as place as any!

The Five Causes of IT Complexity

2008-05-29T05:43:00.000-07:00

Complexity is a big problem for IT. Complex systems cost too much, fail too often, and usually do not meet basic business and architectural requirements.

I believe that all IT complexity can be traced to five causes. Eliminate the five root causes of complexity, and you can eliminate unnecessary complexity.

These root causes are as follows:

Partitioning Failures – that is, systems in which either data and/or functionality has been divided into subsets that do not represent true partitions.
Decompositional Failures – that is, systems that have not been decomposed into small enough subsets.
Recompositional Failures – that is, systems that have been decomposed into subsets that are too small and have not been recomposed appropriately.
Boundary Failures – that is, systems in which the boundaries between subsets are weak.
Synergistic Failures – that is, systems in which functionality and/or data placement does not pass the test for synergy.

I’m planning on exploring these five causes in my next ObjectWatch Newsletter, so if you are interested, stay tuned.

Can anybody think of a cause of IT complexity that is not covered above?

How Not To Make IRS Systems Secure

2008-01-22T09:23:00.000-08:00

The GAO recently came out with a study (GAO-08-211) that highlights a problem that is frequently associated with IT complexity: poor security. The GAO, for those of you who are not familiar with it, is the United States Government Accounting Office. This is the organization charged with making sure that the U.S. Government is spending our tax dollars wisely.

In this report, the GAO severely chastised the Internal Revenue Service for “pervasive weaknesses” in the security of the IRS IT systems. According to this report, these weaknesses “continue to threaten the confidentiality and availability of IRS’s financial processing systems and information, and limit assurance of the integrity and reliability of its financial and taxpayer information.”

Now you might wonder why the IRS would do such a poor job with IT security. Surely the IRS is aware of the need for IT security!

The reason for the IRS problems is simple. The IRS systems are highly complex. And highly complex systems are notoriously difficult to make secure. Among the problems noted by the GAO:

The IRS does not limit user rights to only what is needed to perform specific job functions
The IRS does not encrypt sensitive data
The IRS does not effectively monitor changes on its mainframe.

This is ironic, given how important the IRS considers security. But the GAO finding is an excellent illustration of a point I make many times: controlling complexity is more important than controlling security. A system whose complexity has been controlled can be made secure relatively easily. A system whose complexity has not been controlled cannot be made secure, regardless of how much effort is expended.

Those readers familiar with my approach to controlling complexity (SIP) know that I advocate a form of mathematical partitioning to greatly reduce an IT system’s complexity. This process results in a number of sets of synergistic business functionality. I call these sets ABCs for autonomous business capabilities.

These sets represent a mathematical partition and that partition extends through the data ownership. Because of this, it is relatively easy to fix all three of the problems noted above with the IRS system.

For example, it is relatively easy to ensure that a given user need be given no more access rights than they need to complete a specific job, since they are given rights to business functionality, not to data.

It is relatively easy to encrypt data, since data moves between business functions only though well-defined messages and these messages are easily encrypted.

It is relatively easy to effectively monitor all changes made to the mainframe and associate those changes with specific business events and specific users, since any data is owned by specific ABCs and is never visible outside the ABC (except by messaging contracts).

The good news is that the IRS has responded to the GAO report by stating that it is addressing all of the issues raised. The bad news is that it is going about this in exactly the wrong way.

According to Linda E. Stiff, Acting Commissioner of the IRS, “…the IRS has obtained additional expert-level technical support to assist in the development of a comprehensive security analysis of the architecture, processes, and operations of the mainframe computing center complex in order to develop a roadmap and strategy to address several of the issues noted by the GAO in the report.”

In other words, the IRS is going to continue making the same mistakes that led to its current problems: worrying about security and ignoring the real problem, complexity.

This is unfortunate. It means that for the near term, we can expect the IRS systems to continue to be “unprotected from individuals and groups with malicious intent who can intrude and use their access to obtain sensitive information, commit fraud, disrupt operations, or launch attacks against other computer systems and networks,” as the GAO describes the IRS systems today.

Feedback on The Top Seven Mistakes CIOs Will Make in 2008

2008-01-04T16:31:00.000-08:00

My article on The Top Seven Mistakes CIOs Will Make in 2008 drew many comments with excellent observations. Let me respond to what I have seen so far.

Mark Blomsma questions whether we can partition an enterprise architecture into subsets that do not influence each other. He points out that a small failure in one subset can lead to a massive failure in another subset.

Mark is very correct in that it is difficult to create subsets that do not influence each other. I call the interactions between subsets thin spots. Our job in enterprise architecture is to minimize these thin spots. Partitioning is based on the theoretical ideal that there is no interaction between subsets of enterprise functionality (no thin spots). Of course, we know that this ideal is not attainable. So we need to compromise, allowing interoperability between subsets while minimizing the thin spot degradations.

In general, I believe our best hope of minimizing thin spot degradation of an enterprise architecture comes from having interactions between subsets occur through software systems rather than through business processes. There are two reasons for this. First, the more we can automate workflow, the better off we are in general. Second, we have a better understanding of managing software thin spots than we do for managing business process thin spots. As one such example, see the post I did on data partitioning.

One of the primary tasks of the Enterprise Architect is boundary management, that is, on architecting the boundaries between subsets of the enterprise. The stronger the boundaries, the better the partition and the better we can manage overall complexity. In my upcoming book, Simple Architectures for Complex Enterprises, I have a whole chapter dedicated to managing software boundaries.

The problem the enterprise architect faces is that the boundaries between subsets are hard to define early in the architecture and under a never ending attack by good intentioned developers for the entire life cycle of the architecture. One of the best approaches I have found to boundary management is getting the enterprise as a whole to recognize the critical importance of complexity control. Once complexity control has been embraced, it is easier to get people to understand the role of strong boundaries in controlling complexity.

KCassio Macedo notes the importance of tying EA to business value.

This is a great point. An enterprise architecture that does not deliver business value is not worth doing. One of the advantages of partitioning an enterprise architecture is that we end up with autonomous subsets, that is, subsets of functionality that can be iteratively delivered. I call these subsets ABCs, for autonomous business capabilities. These ABCs are very compatible with Agile implementation approaches.

Partitioning is our best approach to managing complexity and mapping the enterprise into autonomous bite-size pieces that can be delivered in an agile and iterative manner.

I believe business value need to be measured not only in ROI (return on investment) but on TTV (time to value). TTV is a measure of how quickly the enterprise sees value in having undertaken the enterprise architecture exercise. Most organizations are initially skeptical of enterprise architecture, and the shorter you can keep the TTV, the sooner you can transform skepticism into support. This builds momentum and improves even more the chances of future success.

Mrinal Mitra notes the difficulty in partitioning the business process and points out that some partitioning might even require a business reorganization. This is an excellent point, partitioning does sometimes require business reorganization.

However, why are we doing an enterprise architecture in the first place? As KCassio Macedo points out in the last comment, enterprise architecture must be tied to business value.

There is rarely any business value in merely documenting what we have today. We are more often trying to find a better way of doing things. This is where the business value of an enterprise architecture comes in. A partitioned enterprise is less complex. A less complex enterprise is easier to understand. The easier we can understand our enterprise, the more effectively we can use IT to meet real business needs.

When I am analyzing an existing enterprise for problems, I usually don’t start by partitioning the enterprise, but instead, analyzing how well the enterprise is already partitioned. If it is already well partitioned, then there is little value in undertaking the partitioning exercise. If it isn’t, then we can usually find areas for improvement before we actually invest in the partitioning exercise.

Anonymous gives an analogy of barnacles as an accumulation of patches, reworks, and processes that gradually accumulate and, over time, reduce a system that may have been initially well designed into a morass of complexity.

This is an excellent point. It is not enough to create a simple architecture and then forget about it. Controlling complexity is a never ending job. It is a state of mind rather than a state of completion.

In physics, we have the Law of Entropy. The Law of Entropy states that all systems tend toward a maximal state of disorder unless energy is continually put into the system to control the disorder. This is just as true for enterprise and IT architectures as for other systems.

We have all seen IT systems that were initially well designed, and then a few years later, were a mess of what Anonymous calls barnacles. Enterprise architecture is not a goal that is attained and then forgotten. It is a continuing process. This is why governance is such an important topic in enterprise architecture. In my view, the most important job of the enterprise architecture group is understanding the complexity of the enterprise and advocating for its continual control.

Anonymous also points out that complex IT solutions often reflect poorly conceived business processes. Quite true. This dovetails nicely on Mrinal Mitra observation in the previous comment. And this is also why we need to see enterprise architecture as encompassing both business processes and IT systems.

Alracnirgen points out that the many people involved in projects, each with their own perspective and interpretation, is a major source of complexity.

This is another good point. There are enterprise architectural methodologies that focus specifically on addressing the issues of perspective, interpretation, and communications. Perspective, for example, is the main issue addressed by Zachman’s framework. Communications is the main issue addressed by VPEC-T.

My own belief is that we must first partition the enterprise, focusing exclusively on the goal of reducing complexity. Interpretation will still be an issue, but we can limit the disagreements about interpretation to only those that directly impact partitioning. Once we have a reasonable partition of subsets, then we can, within those autonomous regions, focus on perspective, interpretation, and communications in the broader picture.

Terry Young asks if the relationship between complexity and cost is linear. Terry’s experience is that “doubling the complexity more than doubles the cost to build and maintain the system.” This relationship between complexity and cost that Terry brings up is very important.

I agree with Terry that the relationship between complexity and cost often appears to be non-linear, however I believe that this is somewhat of an illusion. Let me explain.

I believe that the relationship between complexity and cost is, in fact, linear. However the relationship between functionality and complexity is non-linear. Adding a small amount of functionality to a system greatly increases the complexity of that system. And since complexity and cost are linear, adding a small amount of functionality to a system therefore greatly increases its cost. It is this exponential relationship between functionality and cost that I believe Terry is observing.

Just to give you one example, Cynthia Rettig wrote a paper in the MIT Sloan Management Review (The Trouble with Enterprise Software by Cynthia Rettig) in which she quoted studies showing that every 25% increase in the functionality of a software systems increase the complexity of the software system by 100%. In other words, adding 25% more functionality doubles the complexity of the software system.

Let’s assume that we start out with a system that has 100 different functions. Adding 25% more functionality to this would be equivalent to adding another 25 functions to the original 100. Rettig tells us that this 25% increase in functionality doubles the complexity, so the software with 125 functions is now twice as complex as the one with 100 functions. Another 25% increase in functionality gives us a total of 156 functions, with four times the original complexity. Another 25% increase in functionality gives us a total of 195 functions with eight times the original complexity.

This analysis shows us that by doubling the functionality in the system, we increase its complexity (and cost) by 800 per cent. If we continue this analysis, we discover that tripling the functionality of the original system increases its complexity by 126 times. In other words, system B with 3 times the functionality of system A is 126 times more complex than system A. And, since complexity and cost are linear, System B is also 126 time more expensive than system A.

But this is the pessimistic way of looking at things. The optimistic way is to note that by taking System B and partitioning into three sub-systems, all about equal in functionality to System A we reduce its complexity (and cost) to less than 3% of where it started.

If you are trying to explain the value of partitioning to your CEO, this argument is guaranteed to get his or her attention!

The Top Seven Mistakes CIOs Will Make in 2008

2008-01-02T19:48:00.000-08:00

CIO Magazine recently interviewed 250 CIOs from a variety of organizations and asked what their top ten goals are for 2008. As I read this article, I realized that of the top ten goals, seven have virtually no hope of being attained. Why is this?

In my January ObjectWatch Newsletter, I discussed why so many CIOs are heading down a path of failure in the coming year. The article is available without cost or registration at the ObjectWatch Web Site. Feel free to comment on the article here.

Data Partitioning

2007-12-30T09:26:00.001-08:00

Those who have been reading my newsletters and white papers know I believe that complexity in IT systems and business processes is best thought of as the amount of disorder in the system in question.

The best way to control this disorder is to partition the system into a small number of subsets. Mathematically, partitioning can be shown to have a huge impact on the amount of disorder in a system, and therefore, its complexity. For those interested in more information on this topic, see my briefing papers on SIP (Simple Iterative Partitions) at the ObjectWatch web site.

Recently, one of my readers sent me an email asking about my philosophy about data sharing. He asked

I understand the general philosophy [of partitioning] but am not sure how that would work at the implementation level for data(say database). Does your solution mean that HR database tables can never participate in queries with say planning module database tables?

In general, data sharing can be represented by the following diagram:

It should be pretty obvious from this diagram that data sharing severly breaks down the boundaries that logically separate applications. If application A decides to change how it stores data, application B will be broken.

Anytime we destabilize the boundaries between two applications, we add tremendously to the overall disorder of the system. Just a little bit of destabilization adds a huge amount to the disorder.

Some organizations attempt to deal with this by creating a common data access layer, but this doesn't significantly change the problem. If the implementation of A changes so that it no longer uses the common data access layer, application B is broken.

In order to ensure strong application boundaries (and therefore minimal system complexity), we take all possible steps to minimize dependencies between our applications. The best way to accomplish this is to insist that all interactions between applications go through intermediaries.

If application A wants to ask application B for some information, it makes the request through an intermediary that wraps A. That intermediary makes the request to an intermediary that wraps B. The intermediary on the requesting side is called an "envoy" and the intermediary on the receiving side is called a "guard".

How the envoy and guard send and receive requests is not the problem of either application, but the most common way of doing so is through the use of messages that are defined by the general family of standards known as web services.

Notice how this protects the two applications from implementation changes on the other side. If the implementation of A changes, it is likely that A's envoy might also need to change. But nothing on B's side needs to change.

In this architecture, we have strong boundaries between the two applications, and these boundaries serve to control the over all disorder of the system .

But still, sometimes application A needs access to application B's data. What do we do then?

There are two possibilities. The first is that application A can ask application B for the data it needs (using, of course, intermediaries). The second is that a third application can be created that manages the shared knowledge of the organization. Let's call this application the KnowledgeManager. When any application makes changes to data that is considered part of this shared knowledge, that application is responsible for letting the KnowlegeManager know about those changes (using intermediaries, of course).

Some may argue that this architecture is inefficient. Why not just go directly to the database for what you want? The answer has to do with our ultimate goals. I believe that our most important goal is controlling IT complexity. Controlling IT complexity is far more important that a few milliseconds of performance.

Complexity is expensive. Disk access is cheap.

Cures for Complexity Lacking

2007-12-28T09:50:00.000-08:00

A recent IT Business Canada discussed four approaches used by CIOs to reduce architectural complexity. According to the article, these approaches are:

Business Process-Driven Architecture
Good Governance
Defaulting to Simplicity
Continuous Improvement

These ideas all sound good and they might be. But we can't tell what, if anything, they have to do with reducing complexity because the article never defines complexity. There is no model, either explicit or implicit for what complexity looks like in an enterprise architecture, how we might measure it, or how we would know we had eliminated it. How can we eliminate something unless we know what that something is?

Imagine this dialogue:

Sam: My house has way too much wazine.

Dan: What's wazine?

Sam: Wazine is wazine. Everybody knows what wazine is.

Dan: How much wazine is in your house?

Sam: I have no idea.

Dan: How much is a good amount?

Sam: Beats me.

Dan: How do you measure wazine?

Sam: Don't know.

Dan: If you don't know what wazine is and you don't know how much of it you want and you don't know how to measure it, how are you going to get rid of it?

Sam: Simple. I'm going to exercise more, eat a better diet, call my mother twice a week, and balance my checkbook every month.

Dan: That all sounds great. What does that have to do with wazine?

Sam: What's wazine?

Silly, isn't it? And using a word that sounds impressive, like "complexity", doesn't make the conversation any less silly.

Now don't get me wrong. I am all for controlling complexity in enterprise architectures and IT systems. I just believe that you can't eliminate something until you understand what that something is.