Why Big IT Systems Fail

Small IT systems usually deliver successfully. They are delivered on time and on budget. When they are delivered, they usually meet the needs of the business, they are secure, they are reliable, and they are easy to change.

Large IT systems usually do not deliver successfully. They are delivered late and over budget, if they deliver at all. If delivered, they usually fail to meet the needs of the business, they are rife with security problems, they fail frequently, and they are hard to change.

Why are we so good at delivering small IT systems and so bad at delivering large ones? The obvious answer is that large systems are more complex than small systems. But that is not the problem. 

The problem is not the fact that IT systems get more complex as they get larger. The problem is how they get more complex. Or more specifically, the rate at which they get more complex.

The complexity of an IT system increases at a rate we describe as exponential. For most modern IT systems, such as service-oriented architectures, the exponential increase is driven by the number of dependencies in the system. As the system gets bigger, the number of dependencies increase. As the number of dependencies increase, the complexity increases. But the increase in complexity is not linear, it is exponential. 

The difference between a linear increase and an exponential increase is critical. 

An example of a problem that increases linearly is a leaky faucet. Say a faucet leaks at a rate of one ounce per hour and the water is going into a clogged sink that can hold 20 ounces. After three hours, a three ounce container will empty the sink. If you don't get to the sink for ten hours, you know you need to bring a ten ounce container to empty the sink. The water leaks out at a steady rate. It doesn't start leaking faster just because more water has leaked.

But think of a forest fire. Forest fires increase at an exponential rate. Say in the first minute of the fire it has engulfed one square foot. You cannot assume that in twenty minute the fire will have engulfed twenty square feet. That is because forest fires spread exponentially; the bigger they get, the faster they spread.

The mathematics of IT complexity follow the mathematics of forest fires. Say we are building an IT system at the rate of one function per week. It will take almost one year to reach 100,000 standard complexity units (SCUs). But it only takes 10 more weeks to reach the next 100,000 SCUs. And then only 7 more weeks to reach the next 100,000 SCUs. By the end of the second year we are adding more than 30,000 SCUs per week!

Except that we won't, because this rate of complexity increase is unsustainable. Just like a forest fire will eventually burn itself out once it has consumed all possible fuel, so will an IT system. It will grow until the resources are no longer available to support the massive complexity increase. At that point, it will do what all complex systems do when they reach a level that is no longer sustainable. They collapse.

Does this mean that it is impossible to build large IT systems? No, it doesn't. It does mean that we need to figure out how to attack the complexity growth. We can't prevent the IT system from getting more complex (that is impossible), but we do need to figure out how to make the complexity increase linearly rather exponentially. 

In other words, we need to figure out how to make IT systems behave more like leaky faucets and less like forest fires.

We will email you alerts when new IT complexity related blogs or white papers are available. Subscribe <here>.

You can learn more about our IT Simplification Initiative (ITSI) <here>.

Photo of the forest fire is by the U.S. Forest Service, Region 5, made available through Creative Commons and Flickr. The photo of the faucet is by John X, also made available through Creative Commons and Flickr.

Article Alert: The CIO Perspective on IT Complexity

A new article alert by Roger Sessions

Article Name: The IT Complexity Challenge: A Trustmarque study into the state of IT complexity from the CIOs’ perspective.

Authors: None given.

Publisher: Trustmarque

Date of Release: October 2015

Available at: http://www.trustmarque.com/it-complexity-report/

Registration requirements:  Nominal registration information required.

Main Points of Article

IT Complexity is a huge challenge for most CIOs. 93% of CIO’s believe that IT complexity has increased and 66% believe that the cloud has increased complexity, more than for any other cited factor. This is especially interesting given that the cloud was touted as way of simplifying IT.

Companies are now recognizing the problems complexity causes. (“Organizations are craving simplicity.”) CIOs don’t know how to deal with IT complexity (“For CIOs, the end result... is confusion over which technologies and services will actually help them simplify their IT landscape.”) But despite the confusion about how to simplify IT, 79% of CIOs think that simplifying IT is a priority.

IT complexity is causing many problems that directly impact the business. IT complexity is a major contributor to security problems with almost all CIOs (87%) agreeing that IT security is a challenge. IT complexity also makes it difficult to respond to the business needs with almost all CIOs (89%) agreeing that simplifying IT is at odds with driving innovation

IT simplification is not a luxury, it is a necessity. But it is a necessity few CIOs are equipped to deliver. 80% believe their organizations lack the in-house skills needed to deliver projects at the speed required. The article concludes “What the modern CIO needs is to simplify the IT at their disposal, yet this is a huge challenge for IT departments to do it all on their own.”

My complements

All in all, this is a well written article that makes clear the dual points that while CIOs understand they have a problem with IT complexity, they have little idea what to do about it.

I also agree that IT complexity causes security problems and makes responding to business needs difficult. I would add to these two problems a host of others, including reliability and cost of delivery. I also agree that few CIOs are equipped to respond to the challenges of IT complexity.

Most CIOs will benefit from reading this article, if only to understand that they are not the only ones struggling with the problem of IT complexity.

My criticisms

The article gives no guidance as to how to respond to the problem of IT complexity other than to engage Trustmarque and there is no information as to how Trustmarque will help organizations achieve IT simplification.

Disclaimer

Roger Sessions has no interest, financial or otherwise, in the article discussed above. The article was chosen based solely on his judgement as to the value of the article to his readers.

Do you know about a recently released article you think Roger should cover? Or do you have information about a recent highly complex project you think he should write about? Anonymity guaranteed if desired. Drop him a note at roger@rogersessions.com.

Subscribe

Subscribe to Roger Sessions’s alerts and articles at http://rogersessions.com/library/subscribe.

This article alert is by Roger Sessions, probably the leading expert on IT complexity. Our approach to IT simplification is our IT Simplification Initiative (ITSI™). You can read about ITSI at http://rogersessions.com/itsi.

The road to IT Simplification begins with a single ITSI step.

The Three Headed Dog

Hercules and Cerberus

Vulnerable, Unreliable, and Inflexible Systems: Three Symptoms, One Disease

As IT systems increase in size, three things happen. Systems get more vulnerable to security breaches. Systems suffer more from reliability issues. And it becomes more expensive and time consuming to try to make modifications to those systems. This should not be a surprise. If you are like most IT professionals, you have seen this many times.

What you have probably not noticed is an underlying pattern. These three undesirable features invariably come in threes. Insecure systems are invariably unreliable and difficult to modify. Secure systems, on the other hand, are also reliable and easy to modify.

This tells us something important. Vulnerability, unreliability, and inflexibility are not independent issues; they are all symptoms of one common disease. It is the disease that is the problem, not the symptoms.

The disease is complexity. System complexity not only results in vulnerability, unreliability, and inflexibility, but a host of other problems such as budget overruns, deadline slippages, and poor business alignment. If you have ever had to deliver large IT systems, you are also painfully familiar with these issues.

System complexity spreads like most diseases: exponentially. As system size increases, system complexity increases faster. The rule of thumb is that when a system grows by twenty-five percent in functionality, it doubles in complexity. Once complexity doubles, security breaches double, system outages double, and the cost of making future changes double.

We are fortunate. In the last decade, we have made tremendous strides in understanding IT complexity. We now have mathematically grounded models for understanding complexity. We have metrics for measuring complexity. And we have directed methodologies for maximizing desirable functional growth while minimizing undesirable complexity growth. With these models, metrics, and methodologies, we are finally in a position to make complexity related IT problems a distant memory.

As we better manage the disease of complexity, we also better manage the symptoms of complexity: vulnerability, unreliability, inflexibility, and an assortment of others.

The irony is that if you want to make systems more secure, flexible, and reliable, you won’t do it by making systems more secure, flexible, and reliable. At least, you won’t get far doing that. Sooner or later, you need to attack the disease that is the underlying problem. That disease is complexity.

Complexity is the cancer of IT. That is the bad news. The good news is that we now no longer need be victims of complexity. We have models to understand complexity, methodologies to eliminate it, the tools to make sure it doesn’t return. And that means that we can now create large IT systems that are also secure, reliable, and flexible. Not to mention less complex.

- Roger Sessions, Austin Texas

Wake Up Call for the Banking Industry

In a discussion thread on the LinkedIn group Simpler IT, Marc Lankhorst mentioned that the Dutch Bank (The Dutch banking regulatory board) recently came out with a new report that discussed the stability of Dutch banks. The report is titled Naar een Dienstbaar en Stabiel Bankwezen or To a Serviceable and Stable Banking System (as translated by Bing.)

Appendix 7 of the report discussed the critical relationship between banking IT systems, Enterprise Architecture, and complexity management. The report is in Dutch. I ran it through Google translate. The translation was very rough, but even in the Google translation, it was obvious that the report was nothing short of a wake up call to the banking industry.

One of the members of the Simpler IT group arranged for a group of humans to translate Appendix 7. This group did a great job and kindly gave me permission to reprint their translation here. The primary translator was Sasja McCann with help by Andi McCann and Peter Mcelwaine-Johnn.

The original (in Dutch) is by the Dutch Ministry of Finance and is here.

If you would like to discuss this report, I recommend discussing this on the appropriate thread at Simpler IT.

Appendix 7: Resolution and Recovery of IT systems in Banking

Information Technology (IT) is critical for many businesses and non-profit organisations. Business processes are so dependent on automated information systems that, in many cases, the performance of those systems are for a large part responsible for the success of an organisation. Many business procedures cannot operate anymore without the need for information systems. 

This is no different in the financial sector, and particularly the banking sector. Banks are sometimes referred to as glorified IT companies. IT plays a critical role in nearly all banking processes and has done so for a long time. Already in the 1960s, bulk processes were automated on a large scale, such as the automated payment system for instance. Over the years, most banks have created a variety of information systems, most of them geared towards automating administrative processes, but in the meantime, many other forms of information processing have been created.

Information systems in the Dutch banking sector, and the IT function that is responsible for those systems, have the following characteristics:

Information systems are an integral part of business, more so than in other sectors;

• The budgets for their development and maintenance, and budgets for the IT function in general, are therefore correspondingly higher;
• The proportion of older systems (legacy) is still relatively high, with relatively high additional maintenance costs;
• The complexity of systems is relatively high, partly due to their relative old age and the dependency of banking processes on those systems. “Everything is dependent on everything else.”
• The diversity of systems is relatively high, and also the number of systems is high;
• IT functions are generally quite mature. A lot of investments have been made in processes and personnel;
• Personnel working in the IT function are well-educated and have a high experience level;
• The management and maintenance of information systems is often outsourced to specialized IT companies (e.g. IBM, Accenture, Cognizant) and is often operated from India or Eastern Europe;
• Information systems extend to the customers, both business and retail customers. Much use is made of “electronic banking”
• The customer is an extension of the information systems of the bank, and partly for this reason IT in the banking sector needs to comply with strict security regulations / conditions. This aspect is an important component of the trust customers have in a bank

Information technology is, in general, characterised by big changes and fast dynamics. For the purpose of this report it would be too much to discuss social media, cloud computing and big data in detail, but obviously banks will continue to invest in these areas, not only to be able to offer an attractive proposition to their customers and shareholders, but also to continue to comply with laws and regulations. At the same time all these IT developments offer the chance to reduce the complexity of information systems and to enhance their effectiveness. However, these developments do all need to comply with the relevant IT Governance.

In the remainder of this text we discuss the resolution and recovery of IT systems in the context of M&A activity. We describe some principles (the preconditions which we discussed in the previous paragraph) to which information systems must comply in order to be capable of resolution and recovery (i.e. splitting IT systems). We start with a brief introduction to the discipline to which these principles belong, called Enterprise Architecture. Enterprise Architecture is a tool for the governance, including IT Governance, of an organisation.

Enterprise Architecture

An Enterprise Architecture (EA) is a coherent set of principles, models and patterns focused on the design, development and management of an organisation. An Enterprise Architecture is like a blueprint (or map) of an organisation and its information systems. Strictly speaking EA is not a specific IT tool – in practice, however, it is a key tool to assure IT Governance. It describes the business functions and processes, their relationships and their information needs, and it outlines the information systems that meet that need.

An Enterprise Architecture structures the IT landscape, makes it possible to describe the current and necessary information systems in an orderly and consistent manner, and to take decisions based on these descriptions. These decisions are aimed generally at the new development, modification or replacement of information systems.

The discipline that deals with EA has developed in recent years in response to the increasing complexity of existing information systems, and the associated problems of large, unmanageable IT projects and dilemmas that many organisations face as a consequence of the fast dynamics and speed of information technology.

Due to the structuring and complexity-reducing character of Enterprise Architecture, this instrument is the means to achieve resolution and recovery of information systems. 

Enterprise Architecture and Dutch banks

Because of the aforementioned characteristics of information systems in the banking sector, Enterprise Architecture is highly relevant to banks. This is the reason why most Dutch banks have invested in the development of Enterprise Architecture functions and departments.

In theory, the banks already have a tool that enables high quality of information systems. One of the aspects of high quality is that resolution and recovery of information can take place in a controlled manner. However, given the quality problems that banking systems face at the moment, the reality is often different: insufficient availability, high security risks and lack of maintainability. This contributes to high cost of maintenance and adjustment costs of banking information systems and also means that successful resolution and recovery is very difficult to achieve.

Why then, given all the promises, is Enterprise Architecture still underutilised? Reasons for this are:

1. Opportunism of the ‘business’: often driven by circumstances there are often “quick and dirty” information systems developed that do not conform to the EA. These systems usually live a longer life than originally envisaged. It is often these systems that cause the most problems. 
2. Backlog: we’ve already highlighted the legacy problems of banks. It takes a lot of time and effort to clean up and replace legacy systems. 
3. Unnecessary complexity: sometimes there is an atmosphere of mystique around Enterprise Architecture that makes it unnecessarily complicated, resulting in lack of understanding by the people that need to understand it. Furthermore, the programmes that are implemented through Enterprise Architecture are often large and complex, which increases the risk of failure. 
4. Insufficient overview: partly because of the complexity and scale of information systems there is no clear overview to actually develop a clear ‘map’. The result is often a very complex diagram that no one understands anymore. 
5. Mandate: The staff in the Enterprise Architecture discipline (“architects”) have insufficient mandate from the organisation to achieve effective “compliance” with the architecture. Sometimes architects are not sufficiently able to express (the importance of) EA. 
6. Contracts and Service Level Agreements: vendors are sometimes unable to comply with EA or do not want to comply, e.g. if cost justifications are introduced. Until recently, there were no standards for suppliers or banks to adhere to. 
7. Each bank has in the past tried to re-invent the wheel at least once, under the assumption that banking processes differ greatly. Obviously this is not the case. It has, however, led to costly programmes and projects that have resulted in a healthy apathy towards IT at senior levels within the banks. 

The last two reasons led to the realisation that there is a need for EA standards for banks. This standard has recently been developed by the Banking Industry Architecture Network (BIAN), established by a number of major banks, along with several established IT vendors[1]. In the Netherlands, ABN AMRO, ING and Rabobank are members of BIAN. Other members are several European, Asian and American banks and its membership is expanding rapidly. The standard, the so called BIAN model, describes all the services that a bank offers, including IT support required for this. The advantage of such a standard is that banks do not have to reinvent the wheel themselves. This not only reduces costs but also increases the quality of the IT landscape, and facilitates M&A activity amongst banks. Figure 1 shows the complete model (at the highest level)[2].

Figure 1 BIAN Service Landscape 2.5 

Resolution and Recovery Principles

Regarding unbundling, an Enterprise Architecture should give priority to the following three principles. This means that all the information systems of a bank are structured and arranged such that they conform to these principles. Note that the principles can be further “unravelled” - in order to avoid complexity as much as possible, we describe them at an aggregate level in this report.

We have sought to minimize the number of principles. This does not mean that we discourage additions or refinement of the above three principles – in practice, banks often use more principles. A minimal set enables clarity, and also allows for the acceptance and implementation of the EA principles.

Principle 1: Compartmentalisation of information systems.

The background of this principle is that business functions/departments must be able to operate as independently as possible from each other and that the information system of one function does not interfere with that of another function. The bank defines its business functions to be as detailed as possible, and also defines the relationships between business functions as clearly as possible. The information systems of a business function should not support other business functions, but communicate (via so-called “services”) and exchange data with information systems from other business functions – they are compartmentalised. Compartmentalisation is achieved in practice by, inter alia:

• Virtualisation of information systems, which means that users share hardware and software in a controlled way. Special software (virtualisation software) ensures the compartmentalisation; authorisation and authentication play an important role in this
• Develop and analyse information systems with a “service-oriented” view. “Service-orientation” refers to ensuring a system is developed with the end-user and the purpose of the service in mind.
• Developing information systems using components with well-defined functionality. Each component should have a clearly defined service. Components should be standardised and documented and may be reused.
• Multiple layers of information systems, in which, for example the presentation of data is separated from the processing of data. 

A beneficial side effect of compartmentalisation is the reduction of complexity, which in itself simplifies resolution and recovery. In addition, the number of links (interfaces) between systems is reduced, making maintenance easier. The success of compartmentalisation depends on carefully thought-through and well-documented Enterprise Architecture.

Principle 2: Data has one owner who is responsible for the storage, description, sharing and destruction of the data.

This principle should ensure the quality of the data of a bank; for example, it should prevent inconsistencies and data unreliability caused by copying data and then editing the copied data. The data owner is responsible for the quality of the data. Data quality is crucial in any potential split of activity. In the case of a resolution of an information system as a result of M&A activity, two cases can arise:

• The split entities are no longer part of the same holding. A predetermined copy of the data is made to be used by both entities. Each entity then applies principle 2 in their own entity. 
• The split entities are part of the same holding. In that case, they can use the same data, and principle 2 still applies, i.e. one data owner.

Principle 3: An information system has one owner, who is responsible for both the quality of the information system and its components, as well as the quality of the services provided by the information system.

The application of Principle 3 ensures clarity of ownership of an information system. In any split this clarity is crucial. Even if there is no split, it is important that an information system has an owner, with a budget to develop the information system to a required level and to keep it there, to ensure business processes are supported optimally and that resolution and recovery is possible. Incidentally, this is also one of the guiding principles of Sarbanes-Oxley (SOX).

Preconditions

Earlier in this text we have already stated that many banks already use Enterprise Architecture, including resolution and recovery principles, and that specific roles, disciplines and processes have been defined. We argue that the Enterprise Architecture discipline needs to take a stronger position within the bank. This means that:

• The staff in the discipline (architects) have excellent content and communication skills. They know the banking business, the information systems of the bank and the relevant information technology for the bank through and through, and can clearly convey that knowledge verbally and in writing. They are able to capture and define an Enterprise Architecture in understandable language and / or models, and can express the importance of Enterprise Architecture effectively.
• The discipline reports to top management in the bank. Enterprise Architecture comprises the entire bank and the information provision of the whole bank – it is therefore important that this broad scope is reflected in the weight of the discipline within the organisation. The discipline not only has a close relationship with IT in particular, but also with the operation of the bank in general, and with the risk management discipline. A close relationship with the COO and CRO, in addition to the relationship with the CIO, is therefore necessary.
• The Enterprise Architecture discipline has the mandate to test the current and future information systems against the Enterprise Architecture. The discipline also has the mandate to escalate to the highest level in the case of non-compliance, with the obligation to indicate what action should be taken to eliminate non-compliance. This mandate also extends to suppliers and vendors – it should be contractually specified that suppliers and vendors are to conform to the Enterprise Architecture.
• It is advisable to rest the accountability for the Enterprise Architecture discipline with one person: the Chief Enterprise Architect.

Measures

Following the above, we propose the following steps to ensure successful resolution and recovery of banking information systems:

1. An important means of ensuring resolution and recovery is to establish Enterprise Architecture disciplines. Establish a number of clear principles, wherein the three principles as described in this document are a minimum. Become a member of an industry body or adhere to a standard in this area – BIAN seems obvious.
2. Strengthen the Enterprise Architecture discipline in the bank by appointing a Chief Enterprise Architect with knowledge of the banking business and an overview of the IT landscape of the bank. 
3. Let the Chief Enterprise Architect report to top management. 
4. Make resolution and recovery the Chief Enterprise Architect’s responsibility, even if only with regard to the IT landscape
5. Give the Chief Enterprise Architect the mandate and the tools to assess changes and new developments in the IT landscape, to comment on them and, if necessary, to stop them.
6. Give the Chief Enterprise Architect the mandate and the tools, including a number of enterprise architects with excellent communication skills and experience in the banking industry, to initiate activities that enable successful resolution and recovery of the IT landscape.
7. Increase EA knowledge and skills of supervisors/senior management. This applies to risk management, the Supervisory Board and DNB (Dutch National Bank). It has been observed that the latter has little to no ability to test an Enterprise Architecture. In addition, currently there is no reference model to benchmark any testing. The aforementioned BIAN model can fulfill the role of this reference model.

Note that these measures are not only beneficial for the ability to successfully resolve and recover, but also increase the quality and maintainability of information systems in general.

[1] For more information, see www.bian.org

[2] Updated to show v2.5 of the BIAN model. Original report showed v2.0.

The Math of Agile Development and Snowmen

I was recently asked about the relationship between Agile Development and Snowmen. If you aren't familiar with The Snowman Architecture, see this earlier blog.

I'm not sure The Snowman Practice has much to offer Agile on small projects (say <$1M). These projects always seem to do reasonably on their own. However once a project goes much above $1M, the Agile approach can no longer keep up with the increasing project complexity.

This is where The Snowman Practice has much to offer Agile. The Snowman Practice offers a methodology to break a large project into smaller highly targeted autonomous projects that have minimal dependencies to each other and are highly aligned to the business architectural organization.

Smaller highly targeted autonomous projects 

There is actually a mathematical explanation as to why Agile over $1M needs The Snowman Practice. As projects increase in functionality, their complexity increases exponentially. Agile however is a linear solution. This means that the amount of work an Agile team can produce is at best linearly related to the size of the group.

At small sizes, a linear solution (Agile) can contain an exponential problem (complexity). But at some point the exponential problem crosses over the ability of the linear solution's ability to provide containment. For Agile projects, this seems to happen someplace around $1M.

At some point the exponential problem crosses over
the ability of the linear solution's ability to provide containment

The Snowman Practice solves this problem by keeping the size of each autonomous sub project under the magic $1M crossover point. And that is a project size that is well suited to the Agile development approach. Or any other linear constrained approach.

Acknowledgements

The Snowman photos are, in order of appearance, by Pat McGrath and smallape and made available via Flickr through licensing of Creative Commons.

The Thirteen Laws of Highly Complex IT Systems

The new White Paper, The Thirteen Laws of Highly Complex IT Systems is now available [here].

Two Roads Converge?

by Roger Sessions and Richard Hubert

Introduction

We (Richard Hubert and Roger Sessions) have a lot in common. We are both Fellows of the International Association of Software Architects (IASA). We have both written books and articles. And we are both well known proponents of a particular approach to Enterprise and IT Architectures. For Richard, this approach is called Convergence Architecture. For Roger, this approach is called The Snowman Practice. But it may turn out that we have more in common than we thought. Our approaches may complement each other in some interesting ways. But first, let’s take a look at what each of us has been doing.

Introduction to Richard’s work

Since the mid 1990’s I (Richard) have been developing and optimizing an architectural style that addresses the complexity of both the IT systems and the business processes. I call this holistic perspective Convergent Architecture (CA). I wrote about this in 2001 in my book Convergent Architecture (John Wiley N.Y. ISBN: 0471105600.) CA includes properties that  I consider to be inherent in any architectural style. The metamodel that I use includes the project design, the system design, and the business design. At a high level, this model is shown in the following diagram:

Figure 1. Coverage of a holistic architectural style

As you can see in the above diagram, the partitioning between Organization, Process, and Resource plays a significant role in the quality of the design. Experience and rules-of-thumb are adequate to handle many designs, but as systems get larger, a more formal approach is preferable, especially if it can be assisted by tools. This is where Roger’s work is a perfect fit.

Introduction  to Roger’s work

I (Roger) have been looking at how to validate an architecture. To do this, I have developed a mathematical model for what an ideal architecture looks like and a methodology for delivering an architecture that is as close to that ideal as possible. The starting point for this is to define what we mean by “ideal.” My definition of an ideal architecture is the least complex architecture that solves the business problem. This means that we also need a metric for measuring complexity, which, fortunately, comes out of the mathematical model. You can read about this mathematical model in this white paper.

It turns out that when you discover the ideal architecture for a given problem, it almost always has a characteristic shape: A collection of business functionality sitting on top of a collection of services sitting on top of a collection of data. In addition, these three tiers are separated from other collections by strong vertical partitions. There is a strong connection between business functions in the top tier, services in the middle tier, and data in the bottom tier. Where connections are required between tiers, these occur through asynchronous messages at the service level. This architecture is shown in the following diagram:

Figure 2. The Snowman Architecture Created By SIP

As you can see in the above diagram, the idealized architecture looks a lot like a snowman. The head, torso, and bottom of the snowman contain business functions, services, and data, respectively.

The methodology I (Roger) have developed to drive this architecture is called SIP, for Snowman Identification Process. Some of you may know it under its old name, Simple Iterative Partitions. You can get a good overview of The Snowman Practice from this video.

Synergy discovery

When we compared the architecture that is driven by the CA’s architectural metamodel (Figure 1)  to the architecture that is driven by the highly theoretical SIP (Figure 2) it was clear to us that significant commonalities are at hand. 

Both approaches are based on the fundamental Enterprise Architecture principle of IT-Business alignment. Both approaches define best practices concerning how this alignment can be effectively achieved and measured. Additionally, both approaches are based on rules and patterns that apply to the simplification of any system, whether large or small, organizational or purely technical. The Convergent Architecture, for instance, has been used to design IT-organizations which then use the same approach to design and simplify IT systems (this is conceptual isomorphism).

Lastly, and most important of all, we recognized a SIP approach can be applied to mathematically support and objectively drive both architectural styles. SIP thus enhances the design approach and process as both a tool and substantive mathematical proof needed to ascertain the simplest (least complex) of all possible system and organizational structures. . 

In essence, we now have CA showing that the SIP theory really does deliver a partition that stands up to the most demanding examination. And at the same time we have the SIP mathematics defining the vertical boundaries of a CA architecture that are mathematically not only sound, but as simple as possible.

The Future

Where will this take us? To be honest, we are still discussing this. But the possibilities are intriguing. Imagine, two mature methodologies that have such strong synergy where both the theoretical and the model-driven approaches seem to come up with such complementary solutions. Stay tuned for more information.

Addressing Data Center Complexity

If you have been following my work, you know how I feel about complexity. Complexity is the enemy. I have written a lot about how complexity causes software cost overruns, late deliveries, and poor business alignment.

In this blog, I decided to look at complexity from another perspective: the data center. This is the perspective of those that must support all of those complex systems the software group has managed to deliver.

The problem with complexity is that it magnifies as you move down the food chain. This is bad news for those at the bottom of the food chain, the data center.

Straightforward business processes become complex software systems. Complex software systems require very complex data stores. Very complex data stores run on extremely complex data centers. These extremely complex data centers are expensive to manage, run, and secure.

The numerous problems that complexity creates for data centers were highlighted in a recent survey by Symantec called State of the Data Center; Global Results [1]. The results of this survey should cause any CIO to break out in a cold sweat.

According to this survey, complexity is a huge problem for data centers. For example, the typical company surveyed had 16 data center outages per year at an average cost of $319K per outage or over $5M per year. And this does not include indirect costs, such as loss of customer confidence. The number and magnitude of these outages was directly attributed to data center complexity according to those surveyed. Complex data centers fail often and they fail hard.

But outages aren’t even the biggest complexity related headache for data centers. The most cited complexity related problem is the high cost of keeping the data center running on those increasingly rare days when there is no outage. Other problems attributed to data center complexity were security breaches, compliance incidents, missed service level agreements, lost data, and litigation exposure. Clearly complexity is a big problem for data centers.

How are data centers addressing this escalating complexity? According to this survey, the approach 90% of companies are taking is information governance. What is information governance? According to Debra Logan, a Gartner Research VP,

Information governance is the specification of decision rights and an accountability framework to encourage desirable behavior in the valuation, creation, storage, use, archival and deletion of information. It includes the processes, roles, standards and metrics that ensure the effective and efficient use of information in enabling an organization to achieve its goals [2]. 

Two points should be clear from the above definition. First, information governance is a vague concept.  Second, whatever information governance is, it has nothing to do with the problem that is vexing data centers, namely complexity. This is unfortunate, given that so many of the surveyed companies say they are pinning their hopes on information governance to solve their complexity related problems. These companies are headed for a major disappointment.

If information governance won’t solve complexity related data center problems, what will? The problem, as I stated earlier, is the magnification of complexity as it rolls down the food chain from business process to data center. This problem can only be solved with complexity governance. Complexity is the problem, not information.

How do I define complexity governance?

Complexity governance is a set of policies, guidelines, and procedures that ensure every business process is implemented with the simplest possible IT solution supported with the simplest possible data organization running on the simplest possible hardware configuration. 

This sounds good but what would it take to implement this?

Gartner’s Managing VP and Chief of Research, David Cappuccio, is on the right track when he says it is particularly important for more data center staff to understand the “cascade effect” of making changes in a complex environment [3]. Unfortunately, few IT staff are trained in IT complexity, a prerequisite to understanding the cascade effect to which Cappuccio eludes. And it stands to reason that if one does not understand how complexity cascades, one is woefully unprepared to do anything about it.

Here is my recommended plan for putting in place effective complexity governance.

1. Train everybody to understand the importance of controlling complexity. Every person on the IT food chain should be able to recite these words in their sleep: Complexity is the Enemy.
2. Train a select group that includes representatives from the business, IT, and data center in IT Complexity Analytics, the science of complexity as it relates to IT systems.
3. Give this group a mandate to put in place strong complexity governance.
4. Give this group the authority to enforce complexity governance.
5. Hold this group responsible for delivering simpler IT systems that run on simpler data centers.
6. Document the benefits complexity governance delivers.

I don’t claim that complexity governance is simple. The reality is that complexity governance requires a significant and sustained effort. But it is an effort that delivers substantial business value. If you don’t believe me, ask somebody who is in the middle of their tenth three hundred thousand dollar data center outage this year. They will tell you: Complexity is the enemy.

Sign Up

I'll be glad to let you know when I have new blogs, white papers, or videos available. Sign up here.

References

[1] https://hp.symantec.com/system/files/b-state-of-data-center-survey-global-results-09_2012.en-us.pdf

[2] http://blogs.gartner.com/debra_logan/2010/01/11/what-is-information-governance-and-why-is-it-so-hard/

[3] ] http://www.datacenterknowledge.com/archives/2012/12/04/gartner-it-complexity-staffing/

Acknowledgements

The photo is by Route79 on Flickr, licensed through Creative Commons. (http://www.flickr.com/photos/route79/)

Notices

This blog is copyright by Roger Sessions. This blog may be copied and reproduced as long as no changes are made and his authorship is acknowledged. All other rights are reserved.

Gartner: Complexity Reduction One of Top Ten Technical Trends

Gartner recently came out with their Top Ten Strategic Technical Trends Through 2015. Gartner states that one of the Top Ten Trends is IT Complexity. IT Complexity, Gartner says, is a major inhibitor “of an enterprise to get the most out of IT money spent.”

I have been writing for the last ten years about the need to reduce complexity and the bottom-line benefits of doing so. Gartner quotes one of my early papers, The Mathematics of IT Simplification. In this paper I introduced a modern, rational understanding of IT Complexity.

Until this white paper appeared, IT Complexity was largely ignored by Enterprise Architects. This is despite the fact that IT Complexity was responsible for numerous IT disasters and was costing the world economy trillions of dollars per year. IT Complexity was still treated in a random, ad hoc manner if it was treated at all.

This White Paper laid a new foundation for the science of IT Complexity. This paper showed how to model IT complexity, how to measure it, and how to manage it using verifiable, reproducible methodologies that are based on solid mathematics.

If we want to take IT Complexity as seriously as Gartner says we should, we need to start by understanding it. This paper is where it all began. Gartner read it. You should too.

You can find the White Paper, The Mathematics of IT Simplification [here].
You can view the Gartner presentation, The Top 10 Strategic Technology Trends for 2013 [here].
You can view a 20 minute overview of The Snowman Architecture, my answer to IT Complexity, [here]

Photo by Alan in Belfast, via Flickr and Creative Commons.

The 3000 Year Old IT Problem

It was first described 3000 years ago by Sun Tzu in his timeless book, The Art of War.

We can form a single united body, while the enemy must split up into fractions. Hence there will be a whole pitted against separate parts of a whole, which means that we shall be many to the enemy's few.¹

This is the first description of a problem solving technique that would be named 600 years later by Julius Caesar as divide et impera. Or, as we know it, divide and conquer.

The Art of War is frequently named as one of the top ten must reads in business². I have long been fascinated by The Art of War and especially the divide and conquer strategy. Back in the days when I was a researcher at the National Cancer Institute, divide and conquer was a common research strategy for testing drugs. Divide and conquer is used extensively in business, economics, legal, politics, and social sciences, to name a few.

Oddly enough, the only field I can think of that divide and conquer has not been used successfully is IT. Whereas virtually every other field has been able to solve large complex problems using a divide and conquer strategy, IT is alone in its failure. Try to divide and conquer a large IT system, and you get an interconnected web of small systems that have so many interdependencies that is practically impossible to coordinate their implementation.

This is the reason so many large IT systems are swamped by IT complexity³. The one problem solving strategy that universally works against complexity seems to be inexplicably inept when it comes to IT.

How can this be? What is so special about IT that makes it unable to apply this strategy that has found such widespread applicability?

There are only two possible answers. The first is that IT is completely different from every other field of human endeavor. The second is that IT doesn't understand how divide and conquer works. I think the second explanation is more likely.

To be fair, it is not all IT's fault. Some of the blame must be shared by Julius Caesar. He is the one, remember, who came up with the misleading name divide and conquer (or, as he said it, divide et impera.) Unfortunately, IT has taken this name all too literally.

What is the problem with the name divide and conquer? The name seems to imply that we conquer big problems by breaking them down into smaller problems. In fact, this is not really what we do. What we do do is learn to recognize the natural divisions that exist within the larger problem. So we aren't really dividing as much as we are observing.

Let's take a simple example of divide and conquer: delivering mail.

The U.S. Postal service delivers around 600 million letters each day. Any address can send a letter to any other address. The number of possible permutations of paths for any given letter are astronomical. So how has the Postal Service simplified this problem?

They have observed the natural boundaries and population densities in different areas of the country and divided the entire country up into about 43,000 geographic chunks of roughly equal postal load. Then they have assigned a unique zip code to each chunk.

In the following picture, you can see a zip code map for part of New York City.

Partial Zip Code Map for New York City

The areas that are assigned to zip codes are not equal sized. They vary depending on the population density. The area that includes Rockefeller Center (10020) is very dense, so the area is small. The area where I grew up (10011) has medium density and the area assigned to the zip code is consequently average sized.

If we blow up my home zip code we can see some other features of the system. Here is 10011 enlarged:

Zip Code 10011

You can see that the zip code boundary makes some interesting zigs and zags. For example, the left most boundary winds around the piers of the Hudson River. Towards the bottom, the boundary takes a sharp turn to the South about half way through the southern boundary. This is to follow Greenwich Avenue. On the right side, the boundary follows Fifth Avenue for quite a while until we hit the relatively chaotic Flatiron district at 20th street.

So zip codes are not randomly assigned. They take into account population densities, street layouts, and natural boundaries. The main point here is that a lot of observation takes place before the first zip code is assigned.

Suppose we created the zip code map by simply overlaying a regular grid on top of New York City. We would end up with zip codes in the middle of the Hudson River, zip codes darting back and forth across Fifth Avenue, and some zip codes with huge population densities while others would consist of only a few bored pigeons.

You can see why the so-called divide and conquer algorithm is probably better named observe and conquer.

The general rule of thumb with observe and conquer is that your ability to solve a large complex problem is highly dependent on your ability to observe the natural boundaries that define the sub problems.

Let's consider one other example, this time from warfare.

In about 50 BCE, Julius Caesar and 60,000 troops completed the defeat of Gaul, a region consisting of at least 300,000 troops. How did Caesar conquer 300,000 troops with one fifth that number? He did it through divide and conquer. Although the Gallic strength was 300,000 in total, this number was divided into a number of tribes that had a history of belligerence among themselves. Caesar was able to form alliances with some tribes, pit other tribes against each other, and pick off remaining tribes one by one. But he was only able to do this through carefully observing and understanding the "natural" political tribal boundaries. Had Caesar laid an arbitrary grid over Gaul and attempted to conquer one square at a time, he could never have been successful.

This brings us to the reason that divide and conquer has been so dismally unsuccessful in IT. Because IT hasn't understood that observation must occur before division.

If you don't believe me, try this simple experiment. The next time an IT person, say John,  suggests breaking up a large IT project into smaller pieces, ask John this question: what observations do we need to make to determine the natural boundaries of the smaller pieces? I can pretty much guarantee how John will respond: with a blank look. Not only will John not be able to answer the question, the chances are he will have no idea what you are talking about.

With such a result, is it any wonder that divide and conquer almost never works in IT? It is as if you assigned zip codes by throwing blobs of paint at a map of New York City and assigning all of the red blobs to one zip code and all of the green blobs to another.

For the first time in the history of IT, we now have a solution to this problem; a scientific, reproducible, and verifiable approach to gathering the observations necessary to make divide and conquer work. I call this approach synergistic partitioning. It is the basis for the IT architectural approach called The Snowman Practice. And if you are ever going to try to build a multi-million dollar IT system, Snowmen is where you need to start. I can assure you, it will work a lot better than throwing blobs of paint at the wall.

You can start reading about The Snowman Practice [here]. Or contact me [here]. What do Snowmen have to do with defining divide and conquer boundaries? Ask me. I'll be glad to tell you.

Subscription Information

Your one stop signup for information any new white papers, blogs, webcasts, or speaking engagements by Roger Sessions and The Snowman Methodology is [here].

References

(1) 6:14, translation from http://suntzusaid.com
(2) See, for example, Inc.'s list [here].
(3) I have written about the relationship between complexity and failure rates in a number of places, for example, my Web Short The Relationship Between IT Project Size and Failure available [here].

Legal Notices and Acknowledgements

Photograph from brainrotting on Flickr via Creative Commons.

This blog and all of these blogs are Copyright 2013 by Roger Sessions. This blog may be copied and reproduced as long as it is not altered in any way and that full attribution is given. 

SIP and TOGAF: A Natural Partnership

If you have ever heard me talk, you know the standard three-tier architecture is a bad approach for large complex IT projects. It has poor security. It is difficult to modify. It is highly likely to fail.

Instead, I recommend the Snowman Architecture. This architecture is great for large complex IT projects. It has excellent security. It is easy to modify. And it is highly likely to be delivered on time, on budget, and with all of the functionality the business needs.

I often show pictures contrasting these two different architectures, such as this:

The classic three tier architecture is generated by standard IT methodologies, such as TOGAF¹. The Snowman Architecture is generated by the methodology known as SIP (Snowman Identification Process.)

From this discussion you might assume that traditional methodologies such as TOGAF are in conflict with SIP. But this is not the case. The ideal architecture comes not from SIP instead of TOGAF, but from an intimate dance between SIP and TOGAF².

To see how this dance works, let me take you through the steps needed to define a Snowman Architecture and point out where SIP takes the lead and where TOGAF takes the lead.

The starting point for a Snowman Architecture is identifying the business functions. These are the smallest granular functions that are still recognizable to the business analysts. SIP drives the identification of these business functions.

The second step is the partitioning of these business functions into closely related groupings. We call these groupings synergy groups. SIP includes very precise mathematically based algorithms for identifying these synergy groups and it is critical to the project that this partitioning be done correctly.

The third step is the definition of the strong vertical boundaries that will separate the snowmen from each other. This again is SIP functionality. And now SIP takes a break and TOGAF takes over.

To review, these first three SIP lead steps are shown here:

Next we gather the requirements for the business architecture. The business architecture is the head of the snowman. From the SIP analysis, we know which functions are in each head, but we don't know much about their requirements. TOGAF has good processes in place for requirements gathering.

The fifth step is defining the technical architecture. TOGAF has been doing this for years, so SIP has nothing to add here.

The sixth step is defining the data architecture. Again, TOGAF does a great job. We don't need SIP for this.

These three steps, then, are the heart of the TOGAF contributions and are shown here:

The final two steps (seven and eight) involve tying the Snowmen together. First we define the business dependencies between synergy groups. While TOGAF has some information about how to define business dependencies, the SIP defined synergy groups are critical here because they greatly reduce the number of dependencies that will need to be implemented.

Once the business dependencies have been identified, then we are back to TOGAF to translate these business dependencies into a service oriented architecture. These last two steps are shown here:

As you can see, SIP and TOGAF complement each other. SIP depends on TOGAF to fill in the SIP defined Snowman structure. TOGAF depends on SIP to scale up to projects larger than one million dollars. Put the two together and we have a highly effective solution to building large, complex IT systems. The best of all worlds.

Footnotes

(1) TOGAF stands for The Open Group Architectural Framework. It is a trademark of the OMG.

(2) While SIP works well with TOGAF, SIP is  agnostic about which methodology is used to drive the non-SIP parts of the analysis. If your  organization is using some other architectural methodology rather than TOGAF, you can replace TOGAF in these diagrams and discussions by your favorite methodology.

The Snowman Architecture Part Three: The Technical Benefits

This is the third part of a four part blog.about the Snowman Architecture. The first part was The Snowman Architecture: An Overview. The second part was The Snowman Architecture: The Economic Benefits. In this part, I will be discussing the technical benefits of the Snowman Architecture. But don't read this until you have read the overview! And if you care about things like ROI, check out part two.

But whatever you do, don't miss this part! Over the next twelve pages or so (yes, I know it's a bit long) I'm going to take you through fourteen of the most compelling technical reasons why the Snowman Architecture is a huge improvement over today's approaches to large IT. You are now reading nothing less than my declaration of war on traditional IT methodologies.The first snowball has now been fired!

Review

I gave an overview of the Snowman Architecture in part one, but let's review briefly.

The Snowman Architecture breaks down a large IT system into small vertically partitioned subsystems called Snowmen. These snowmen interact with each other through asynchronous messages. Snowmen are designed to be as autonomous as possible using a design methodology known as Simple Iterative Partitions¹ (SIP).

Snowmen come in three layers. The head of the snowman consists of the business functions that make up a capability. The torso of the snowman consists of the technical systems that support those business functions. The bottom of the snowman consists of the data that is used by those technical systems. Each of these layers is strongly partitioned based on the business functions that make up the head.

Snowmen reach out to each other through their arms, the asynchronous messaging system. Often this is implemented as an SOA.

Snowmen reach out to each other through their arms.

Contrast to Traditional Architectures

A traditional IT architecture is also implemented in three layers. These layers are the same as those of the Snowman Architecture: business architecture, technical architecture, and data architecture. What differentiates the Snowman Architecture from a traditional IT architecture is the strong vertical partitioning, as shown in Figure 1.

Figure 1. Traditional IT Architecture vs. Snowman Architecture

It turns out that this strong vertical partitioning has a major impact on the effectiveness of the architecture. Let's take a look at fourteen key non-functional attributes of a large IT system. As you will see, every single one of them is improved by the strong vertical partitioning that characterizes the Snowman Architecture.

In this analysis, I assume that the system we are evaluating is a large (greater than ten million dollar) system. This is the point at which traditional IT architectural methodologies are no longer able to keep up with the exponential increases in system complexity². I also assume that SIP was used to assign business functions to the head of the snowman, an essential step to minimizing the overall complexity of the Snowman Architecture.

Okay, given these two assumptions, let's see why the Snowman Architecture outperforms all traditional approaches to large IT system design. I'll start by listing the fourteen attributes and then go through them one by one. The attributes I will look at are these:

• Business Alignment 
• Regulatory Compliance 
• Auditing 
• Security
• Agile Friendliness 
• Maintainability 
• Testability 
• Reliability 
• Recovery 
• Throughput 
• Scalability 
• Flexibility 
• Cloud Effectiveness 
• Vendor Lock in

You might as well do a quick check-point. Are any of these attributes important to your IT systems? If not, you might as well stop reading now. If one or more of these are of interest then keep reading.

Okay, now let's go through them one by one. Feel free to skip those you don't care about.

Business Alignment

A system is well aligned when it meets the needs of the business. You can think of business alignment as the Wow factor. When the system is delivered, does the business say, "Wow!" Or does it shake its collective head and reach for the nearest bottle of Tequila?

In any system design life cycle, there is a phase in which the business requirements are gathered. In the traditional approach (the left hand side of Figure 1) the requirements are gathered more or less immediately after the project has been approved and before the technical architecture is designed.

The size of the requirements document(s) is always proportional to the size of the project. Massive projects require lots of requirements documentation, often tens of thousands of pages. The larger the stack of requirements, the lower the chances are that those requirements accurately reflect what the business actually needs.

In SIP (the guiding design methodology for the Snowman Architecture) an additional project phase is introduced: the Partitioning Phase. This is when the basic shape of the Snowman is identified.  

This is when the basic shape of the Snowman is identified.

What is important from an alignment perspective is that this partitioning of the larger system into smaller autonomous snowmen takes place before the requirements have been gathered. Since a typical snowman rarely exceeds one million dollars in cost, it's requirements are modest. Since the requirements are relatively modest, it is more likely that those requirements accurately reflect the actual business need.

Since the Snowman Architecture has more accurate requirements than the traditional IT architecture, it is likely to actually meet the business need.

Regulatory Compliance 

An IT system is considered compliant when it can be shown to operate within the constraints of regulatory laws and regulations. Some enterprises such as the video gaming industry have few if any restraints. Others, such as financial organizations, have a complex web of laws and regulations. 

Our ability to show that a given IT system operates within its regulatory constrains is dependent on the complexity of the system. The more complex the system, the more difficult it is to prove compliance.

Large traditional IT systems (the left side of Figure 1.)  have a highly complex web of relationships between business functions, technical processes, and data. Thus it is very difficult to prove compliance.

The Snowman Architecture (the right side of Figure 1.) has a number of simple relationships between business functions, technical processes, and data. The architectural simplicity is guaranteed by the SIP directed partitioning of the business functions into snowmen heads and the strong vertical partitioning that occurs once the technical and data architectures are created.

Because each snowman is relatively simple, it is correspondingly easy to prove that it operates in compliance with any relevant regulatory constrains.

Auditing 

An IT system is considered auditable when we can accurately trace data changes back to technical processes, from there back to business functions, and from there back to human beings. The more paths there are to the data, the more difficult it is to trace these paths. 

The traditional IT architecture has a large number of complex paths to the data. It is nearly impossible for an examiner to determine which of these paths resulted in a particular item of data being updated .

The Snowman Architecture has a small number of simple paths to the data. An examiner can easily get the whole picture and then determine which of these few candidate paths resulted in a particular item of data being updated. 

An examiner can easily get the whole picture.

Security

We when talk about the security of a system, we are generally talking about our ability to protect data. Data, of course, resides in a database. So security comes down to our ability to configure the database so that unauthorized updates are not possible. In a traditional architecture, there are so many processes that need to update so many parts of the database under so many different circumstances that it is difficult to figure out a secure configuration. And even if one does manage a secure configuration, the next process that is added will change everything.

In the Snowman Architecture, database configuration is much easier. Because the partitioning of the head of the snowman (the business processes) dictates the partitioning of the technical layer and then the partitioning of the data layer, the only processes that will ever need to access the data in the snowman are the processes in the torso of the snowman. This makes configuration easy: allow the processes in the snowman torso to access the data in the snowman bottom and don't allow any process outside the snowman to access any data in the snowman. Viola. Done.

Agile Friendliness 

Many organizations are attracted to Agile Development Methodologies. I agree that Agile development has a lot of promise. However I also think it doesn't scale. 

A recent paper by Vikash Lalsing et al.³ indicates that Agile projects of 0.5 person years or less are excellent candidates for Agile development. They predict such projects will be less than 10% over budget. By the time the project  size reaches 3.6 person years, the budget overrun increases to 18%. And by the time the project size reaches 8.2 person years, the budget overrun increases to 66%.

The Snowman Architecture is ideally suited to projects of greater than $10M. This equates to an effort of close to 100 person years. This is more than ten times the project size that yielded the 66% budget overrun. 

In the Snowman Architecture, the larger project is broken down into relatively simple, autonomous chunks of project work. Each of these chunks becomes an individual snowman. The use of the SIP methodology ensures that not only is each snowman as simple as possible, but the relationships between snowmen are as simple as possible. 

The project size of any one snowmen is unlikely to exceed $1M and in many cases will be much less. A $1M project is around 7 person years. This is still large by Agile standards, but far closer to a workable agile number than a project that does not have the benefit of the Snowman Architecture.

Maintainability 

A system is maintainable when it is easy to locate the source of bugs. The more complex the system, the more difficult it is to find the source of bugs.

The complexity of the traditional architecture (the left side of Figure 1.) is much higher than the complexity of the Snowman Architecture. The maintainability of a traditional architecture is therefore much lower. The use of the SIP methodology guarantees that not only is the overall complexity of the Snowman Architecture low, it is as low as it can possibly be⁴. Simplicity is important when it comes to snowmen. A simple snowman is simple to maintain.

A simple snowman is simple to maintain.

Testability 

System bugs can manifest themselves at any point in the system life cycle. The later in the life cycle the bug is manifest, the more problems it causes. Our goal in system testing is to find bugs in the system as early as possible and definitely before the system is delivered to customers. The most common strategy for system testing is to write code or scripts that exercise the system and ensure it is working correctly. 

To be sure a system is working correctly you must write test code that exercises every possible logical path through the system. The more logical paths there are through the system, the more difficult it will be to create the test code and the more likely it will be that you will have missed an important path. This translates to a greater likelihood that you will ship buggy code.

There are two reasons the Snowman Architecture is more testable than the traditional architecture.  

The first reason the Snowman Architecture is more testable has to do with the number of paths. 

A traditional IT architecture has many possible paths. By the time the system reaches a few million dollars in size, it effectively has an infinite number of paths and there is no way they can all be tested.

In the Snowman Architecture, each snowman can be tested independently. Since each snowman is relatively small and simple, there are relatively few paths through the snowman. Once you have tested all of the snowman and the connections between them, you have effectively tested the system as a whole. Thus your chances of shipping buggy code are greatly reduced if you are using the Snowman Architecture.

The second reason the Snowman Architecture is more testable has to do with how pieces of the system are connected together. In a traditional IT architecture, segments of code are often connected by shared data in a database. In the Snowman Architecture, snowmen are almost always connected through asynchronous messages. 

These two approaches to connections are very different from a testability perspective. Shared data connections are almost impossible to test. There are just too many ways the data can be accessed. Asynchronous messages, in contrast, are very easy to test. One need only write a messaging harness, a common practice among service-oriented architectures, and the connection points become easily tested. 

So we see two reasons the Snowman Architecture is so easier to test than the traditional IT architecture. First, it has fewer code paths. Second, it uses asynchronous messages for its connection points. It is hard to test a traditional IT architecture. It is easy to test a Snowman Architecture.

It is easy to test a Snowman Architecture.

Reliability 

Reliability is a measure of the typical amount of time a system will remain running before it unexpectedly drops dead. Reliability is often described as mean time between failures. 

Reliability is related to testability, the last attribute I discussed. The more testable a system is, the less likely it is to have post-delivery bugs. It is these post-delivery bugs that cause systems to fail. The fewer bugs, the less likely the system is to fail. Since the Snowman Architecture is easier to test than the traditional IT architecture, is is likely to have fewer bugs and thus will be less likely to fail.

But there is another factor that favors reliability of the Snowman Architecture. This has to do with how easy it is to quarantine a bug. Consider Figure 2, which is a blowup of the left hand side of Figure 1 with some labels added for reference.

Figure 2. Blow-up of Traditional IT Architecture.

Assume that database D crashes. We have three processes dependent on D, namely, n, o, and p. So these three processes crash. Processes g and i are both dependent on n, so they both go down. Process i is also dependent on o, but since it has already crashed, we need not worry about it further. Processes i, d, and f are all dependent on p. Process i is already down, but now d and f join the fun. So now we have D, n, o, p, i, d, and f all down. This can corrupt any databases they are involved with which includes C and E. This brings down their dependent processes b and h. Which in turn... you get the picture. There is no quarantine, so when one part of a system catches a bug, that bug can rapidly propagate to the entire system.

Contrast this to Figure 3., which shows a closeup of the Snowman Architecture.

Figure 3. Closeup of Snowman Architecture

Assume in Figure 3. that database B crashes. It can bring down processes d, e, f, and g. But that's it. The boundaries of the snowman have effectively quarantined the bug from spreading further. The only connections between d, e, f, g and other processes are through asynchronous messages, and these channels can easily be protected. So while the bug in B may crash the entire snowman, there is no pathway for the bug to spread further.

The bottom line is that bugs occur less frequently with the Snowman Architecture (because it is easier to test) and when they do occur they tend to have only a local impact. In the traditional architecture, bugs occur more frequently (because it is harder to test) and when they do occur, they tend to have a global impact.

Recovery 

Recovery is related to reliability (the last section.) Whereas reliability measures how often the system fails, recovery measures how long the failure lasts. In an ideal system, we have high reliability and fast recovery, meaning that the system rarely crashes and when it does, the crash doesn't last long.

It is difficult to develop an effective recovery strategy for a traditional IT architecture. There are too many databases, too many processes, and too many ways everything can be related to each other. When this web of relationships goes down, what do you do? You try to protect the entire system but this is difficult because the system is a large, it is complex, and it is a moving target. 

In contrast, it is easy to develop an effective recovery strategy for a Snowman architecture. All you need to do is shadow any requests to the snowman to a backup snowman. Then if a failure occurs, reroute all new requests to the backup. This rerouting can occur as quickly as one can notice that the primary snowman has failed. This is shown if Figure 4.

Figure 4. Recovery Mechanism for Snowman

Taking this and the last two sections together, I can make the following claims about the Snowman Architecture relative to the traditional IT architecture:

• The Snowman Architecture will have fewer bugs.
• The bugs will have less impact.
• Recovery from that impact will be faster.

Throughput 

Throughput refers to the amount of work a system can process in a unit of time. Often we measure throughput in transactions per minute. Throughput should not be confused with response time which measures how long a single user waits for work to be completed.

Throughput is important because it directly influences cost. If a system has low throughput, then a lot of resources are needed to process a given workload. If a system has high throughput, the number of resources needed to process the same workload is much less. 

There are two architectural factors that strongly influence throughput: the number of synchronous connections and the amount of shared data. Synchronous connections slow down throughput by blocking processes until connected processes have completed their work. Shared data slows down throughput by blocking databases.

Both of these factors come together in the traditional IT architecture. These systems heavily favor synchronous connections and make extensive use of shared data. Between the two, throughput is substantially degraded.

In the Snowman Architecture, synchronous connections are only used within a snowman. All (or almost all) connections between snowman occur through asynchronous messaging. Which means no blocked processes.

In the Snowman Architecture, shared data is the equivalent of a multi-headed snowman. This is anathema to the Snowman Architecture. The only processes that are allowed to share data are those that live within a single snowman. Since only a few processes ever share data, database blocking is kept to a minimum.

Between the judicious use of asynchronous messaging and non-shared data, the Snowman Architecture performs at a much higher throughput that does the traditional IT architecture. This means lower costs per unit of work which means lower IT costs.

Shared data is the equivalent to a multi-headed snowman. This is anathema to the Snowman Architecture

Scalability 

Scalability refers to our ability to support larger and larger workloads. Say we have designed a system to support 100 concurrent customers and then our system become so popular that we must support 500 concurrent customers. Our ability to adapt to the higher customer load is dependent on our scalability.

In the past, scalability was seen as a hardware power problem. It was assumed that to allow a system to process larger and larger workloads, it had to run on more and more powerful hardware. When the current system could no longer support the workload, the hardware would be upgraded. This could involve faster processors, more memory, or larger disk drives. In the worst case, this involved replacing smaller cheaper machines by larger expensive machines.This is the model that served the power computer companies like IBM and Sun so well.

Today, scalability is seen as a hardware numbers problem rather than a hardware power problem. We now assume that to process a larger workload we don't replace cheap machines with expensive machines, instead we get more cheap machines. This is the model that powers the most scalable systems in the world today such as Google. Google runs its entire system on inexpensive throw-away hardware and has for more than a decade⁵.

Given this modern view of scalability, there are three factors that determine how scalable a system is.

The first scalability factor is the compactness of the system. Smaller, more compact systems are easier to scale. Larger, more disperse systems are harder to scale.

The second scalability factor is the usage of asynchronous messages. The judicious use of asynchronous messages goes a long way toward making a system scalable. Think of an asynchronous message system as like a mailbox. As mail comes in faster, one adds more receivers. As long as any of the receiver's can process the mail, scalability becomes limited only by the number of receivers you can support.

As mail comes in faster, one adds more receivers.

The third scalability factor is the size of the database on which the system depends. Because databases have such specialized hardware requirements, they are the most difficult part of a system to scale up.

To compare the scalability of the Snowman Architecture versus a traditional IT architecture, we must start by defining the unit of scalability. In the Snowman Architecture, the unit of scalability is an individual snowman. In a traditional IT architecture, it is the entire system.

In comparing the two architectures, we see the Snowman Architecture outperforming the traditional large IT architecture in all three scalability factors. First, it is much more compact. Second, it uses asynchronous messaging in all the right places, at the boundaries to the snowmen. Third, it minimizes the size of the data pool that must be scaled by enforcing the concept of strict vertical partitioning.

As a result, the Snowman Architecture is much more amenable to scaling using the modern efficient approach to scalability, scaling by numbers. The traditional IT architecture is largely consigned to the much more expensive and inefficient approach to scalability, scaling by power. Today, scaling by power seems as quaint as vinyl records.

Flexibility 

Flexibility refers to our ability to modify the system as our business needs evolve. Say we have build our payment system to take credit cards and we now want to take debit cards. How easy is it to update our system to take debit cards as well as credit cards?

Our ability to modify our system depends on how complex the system is. The more complex the system, the more difficult the modifications will be to implement  Traditional large IT systems are very complex. They are therefore very difficult to modify. Frequently changes in one part of the system causes unexpected problems in other parts of the system. 

The Snowman Architecture is composed of a series of autonomous, self-contained, relatively simple snowmen. Because of the synergy algorithms used by SIP to partition business functionality across snowmen, it is highly likely than any modifications necessary for a specific business change will all be located within a single snowman. Since any given snowman is simple (certainly relative to a traditional IT system) we can expect that the modifications will be much more straightforward than they would be with a traditional architecture.

Cloud Effectiveness

The cloud is an attractive platform because of its "pay for what you eat when you eat it" pricing model. But to leverage this platform, it is important to structure your systems so that you eat the least amount possible to accomplish your work.

Traditional large IT systems are poorly organized to leverage this model. Because of their sprawling nature, all or most of the system must be running on the cloud to accomplish even the most trivial of tasks. This means that you are paying for all or most of the system even when you are using only a small part of it. Even worse, when you need to add new instances to handle larger workload, you are adding sprawling new instances that quickly drive the cost out of sight.

The Snowman Architecture is a collection of smaller snowmen, each dedicated to a group of closely related ("synergistic") tasks. In most scenarios, a given workload will require only a single snowman. This means that you are paying only for the resources that that snowman requires. And when you add new instances, you add them in small, inexpensive, snowman sized amounts.

Figure 5 contrasts the traditional IT architecture and the Snowman Architecture running on the cloud.

Figure 5. The Cloud: Traditional IT Architecture versus
The Snowman Architecture.

Vendor Lock-in

A system exhibits vendor lock-in when it is dependent on a single vendor for some aspect of its life support. Usually this vendor is the one providing the software platform.

Vendor lock-in is either good or bad, depending on your perspective. If you are the client, vendor lock-in is bad. It puts you in a weak bargaining position with your vendor. If you are the software platform provider, vendor lock-in is good. It puts you in a strong bargaining position with your customer.

The standard customer approach to avoiding vendor lock-in is through the use of standards. If the customer builds a system on a standard API, then the customer can easily port the system to another software platform that supports that same API. Or at least, that is the logic.

How do vendors achieve lock-in in the face of a plethora of standards covering everything from data storage to virtual systems? Vendors achieve lock-in through the tried and true process called embrace and extend. Embrace and extend is a two part process. First, the vendors embrace a particular standard. Then the vendor extends the standard in vendor specific ways. These extensions are the bait that draws in the customer. The goal is to make the extensions so powerful that they are irresistible. Once the customer has taken the bait, they are trapped. Lock-in is complete.

I have seen many customers try to resist the bait with corporate edicts forbidding the use of any vendor extension. In the end, resistance is futile. You will be assimilated.

The larger and the more complex the system, the more difficult it is to locate and remove the vendor extensions. This mean it is more difficult to port the system to another vendor. If you can't take your code to another vendor, you are locked-in. And your future is now in the hands of a company whose main goal is wringing as much money as possible out of you in the next contract negotiation.

As I said, resisting vendor extensions is pointless. The best strategy for avoiding vendor lock-in is to make it as easy as possible to locate and rewrite those sections of a system that have used the vendor extensions. Your ability to locate and rewrite those sections is dependent on how small and simple the system is. We are dealing with the same issues I discussed in the section on Modifiability. Small, simple systems are easy to modify. Large, complex systems are not.

Thus small and simple is your best defense against vendor lock-in. And if you want small and simple, don't look to standards. Look to snowmen.

Summary

In part one of this blog, I introduced the Snowman Architecture. In part two, I discussed the non-technical advantages of this architecture. In this part, I have discussed the many technical advantages of this architectural approach.

If you are building a large IT system (say, over $10M) the Snowman Architecture offers a huge number of compelling advantages over traditional approaches. These advantages range from better security to improved reliability to lower cost to greater flexibility. In fact, there is not a single non-functional requirement that will not benefit from the Snowman Architecture.

If, at this point, you are preparing to build a large IT system and you aren't seriously considering the Snowman Architecture, then I don't know what else I can say. One of us is crazy.

One of us is crazy.

Stay tuned for part four of this blog, in which I will discuss the arguments against the Snowman Architecture and why they are all flawed.

- Roger Sessions
Houston, Texas

Did you find any errors (even spelling) in this blog? Let me know. I'd love to correct them.

Would you like to subscribe to notifications about my blogs, white papers, and webshorts? Sign up here.

References

(1) See, for example, the Web Short SIP Methodology for Project Optimization by Roger Sessions. Available here.

(2) See, for example, the Web Short The Relationship Between IT Project Size and Failure Rates by Roger Sessions. Available here.

(3) PEOPLE FACTORS IN AGILE SOFTWARE DEVELOPMENT AND PROJECT MANAGEMENT by Vikash Lalsing, Somveer Kishnah and Sameerchand Pudaruth in International Journal of Software Engineering & Applications (IJSEA), Vol.3, No.1, January 2012. Available here.

(4) The Mathematics of IT Optimization by Roger Sessions. (White Paper). Available here.

(5) WEB SEARCH FOR A PLANET: THE GOOGLE CLUSTER ARCHITECTURE by by Luiz André Barroso, Jeffrey Dean, and Urs Hölzle in IEEE Micro March/April 2003 Available here.

Acknowledgements

The snowman photos are all from Flickr under Creative Commons license. The photographers are, in order of appearance: 

• Mary Poppins Snowman: gertrudk
• Three Snowman Family: emueller
• Children and Snowman: stv
• Bird and Snowman: jb-london
• Snowman with Broken Eye: rubin110
• Fallen Snowman: maiac
• Two Headed Snowman: caroslines
• Snowman Army: smallape
• Crazy Snowman: jaybergesen

Legal Notices

This blog is copyright (c) 2012 by Roger Sessions. It may be copied, reposted, and printed as long as it is not modified in any way. Other than that, unauthorized usage prohibited. Ask, though. I'll probably agree.

SIP is a trademark (t) of ObjectWatch, Inc. ObjectWatch is a registered trademark of ObjectWatch, Inc. All other trademarks are owned by their respective companies.