01 April 2011 - I have written a 32 page white paper on The Mathematics of IT Simplification. This paper suggests a mathematically grounded approach to simplifying large complex IT systems. This approach is called synergistic partitioning. Synergistic partitioning is based on the mathematics of sets, equivalence relations, and partitions. This approach has several advantages over traditional decompositional design:
- It produces the simplest possible partition for a given system.
- It is directed, or deterministic, meaning that it always comes out with the same solution.
- It can determine the optimal partitioning of a system with very little information about that system.
You can pick up a copy of the paper here (it's easy, no registration or cost) and discuss it by leaving comments right here.
16 comments:
I think that there are a few things that need further clarification and explanation.
First this isn't about simple architectuures for complex ENTERPRISES, it is about partitioning of a fairly well defined and bounded space. It looks more like deployment/packaging than Enterprise.
Second, and I think more important is the question of granularity. That seems to me to be a bigger driver of complexity than anything else. The paper hints at it, but how do we decide what the "things in the sets" actually are. In other words at what level of abstraction do we do them? That I think is what trips up application designers as much as anything else. How do I know in your example that I might have to do something special with currency? better still when do I discover that? And then when do I modulkarize for it. Where should the module (?) reside.
Also unfortunately, pure reduction of over-complication doesn't deliver what we need always. So where I have a dependency between A and B, but according to partitioning work, they end up residing in different partitions I may end up with a performance overhead that I cannot tolerate in the system.
So while improving cohesion and reducing coupling (of various kinds) is desirable for some purposes, it isn't the only shearing force to consider.
Chris, I agree that this paper is not about simplifying enterprises, although I believe that the same mathematics (and probably even the same equivalence relation) could be used to do so. But this paper is about reducing the complexity of IT systems.
However I believe this methodology must be driven by the enterprise architecture group, because that is where the knowledge to do this properly resides.
I disagree that this is about partitioning a "fairly well defined and bounded space." Actually, this is a great way to help define an unbounded space. The reason is the property of conservation of structure, one of the properties of equivalence driven partitions. We are guaranteed that as we learn more and more about the IT system in question, we will only expand, not change or invalidate the so far discovered structure.
As far as your concern that packaging requirements might contradict the synergistic analysis, we need to look at some actual examples. But there are opportunities to optimize coupling at the design phase. For example, if A is performance dependent on B but B ends up in a different subset, then perhaps asynchronous messages are not the best coupling mechanism between A and B. Perhaps shared data in a database is a better way to couple. Of course, doing this results in much tighter coupling, but you gotta do what you gotta do.
You say this is more like deployment/packaging. I don't agree with this. One of the points of the methodology is that we do the partitioning very early in the life cycle, long before we know enough to be considering packaging and deployment issues. We don't even know more than a handful of the business functions, at this stage!
So packaging and deployment occur at the exact opposite end of the life cycle. And because packaging and deployment occur so late, they offer no benefits in terms of decreasing complexity at an early enough stage to help reduce costs or improve success rates.
As I suspect you would agree, large system failures occur long before we get to deployment or packaging.
Hi Roger,
As you suggested in our email exchange, I re-post my comments here. Looking forward to your reply.
* * *
I read your recent whitepaper. It generated quite a bit of interest in my company. There are a few crucial points I'd like to clarify with you.
1. When we develop a system, we start with one big block (the to-be-implemented system), work out how to decompose it into smaller routines and then group these routines into packages/modules. It appears your methodology starts with blocks with already well-defined boundaries (e.g. "Deposit", "Withdraw") and ends with packages grouping these blocks. It doesn't explain how we get these initial blocks in the first place. i.e. The methodology deals with composing but not decomposing. Is that the case?
2. I have the impression that the methodology works primarily at only one granularity. Is that the case? I think this ties back to the previous question. Without a mechanism to deal with decomposing, it seems it can't be applied recursively.
3. You says once we got an optimal partition, the solution will remain optimal even if you add new functions to the system. That's quite different from my own experience. When we accumulate enough new functionality, at some stage, we need to refactor the architecture in order to manage the newly added complexity. And this is the case at all different levels of granularity. What is your opinion on this?
4. Your methodology assumes synergy between two functions can be indicated by a boolean, with a yes/no answer. From my experience, there are actually degrees of synergy (or cohesion). Take a look at this business rule inside a UK masterfund system:
In an UK Personal Pension account in drawdown phase, if an investor is over age 75 or he has dependants under 25, any payment is tax free.
Where should this logic belong to? In the Payment module, the Pension module, the Client Management module or the Tax module? From my experience, we select one over the others not because the business rule has synergy with only one module but not the others. Rather, it's because the synergy is stronger with that module. Also, to me, there is no systematic way to work it out. Judging how strong the cohesion is is an intellectual exercise that is more an art than a science. (If you disagree, can you now tell me where to put this logic? ;-) What's your opinion on this?
5. This brings me to my last question. Look at the previous business rule again. I'm not sure business people can identify the synergy for the IT people. Business people will definitely tell you this rule has synergy with all the mentioned areas. Synergy, with your definition, encompasses technical criteria, not business criteria. Business people look at their operations in terms of workflows and business processes. How business operations are segregated along the workflow dimension is usually orthogonal to how the underlying system is partitioned.
Do you agree?
Hi John,
Thanks for your comment. Let me go through your questions.
1. You are right, I don't explain how to find the blocks. I assume that we are using functional decomposition to find what I call the Atomic Business Functions (ABF). An ABF is the lowest level business visible function possible. If we try to decompose an ABF, we get technical functions that are not recognizable to the business as useful. I didn't discuss this in this paper, but I have discussed this elsewehere.
2. I don't see this methodology as recursive, in that the methodology is never applied to previous results of running the methodology. But you will use recursive desomposition to identify the ABFs.
It could be confusing to hear me advocate decomposition, since I pointed out in the article that decomposition is a non-directed methodology. However I am just using decomposition to identify the ABFs. For that purpose, decomposition is acceptable. For determining modules, decomposition is not acceptable (because it is non-directed.)
3. I believe the reason that you need to refactor the architecture to manage the newly added functions is because you are not using the synergy algorithm I described. You are most likely using decompositional design. Decompositional design is not based on equivalence relations, and therefore does not have the property that I called "Conservation of Structure."
4. It looks to me like the logic you are describing is something like determine-tax-liability. Now I can't say where the synergy lies (since I am not a business expert in this field.) but I would suspect that it would end up in the Tax Module, since it doesn't make sense to have a tax without being able to determine if there is a tax liability. I do agree with you that synergy will not always be clear. When this happens, you escalate it through the business. It is important that technical folks do not try to make the call, because this is a business decision, not a technical decision (although it has technical implications.)
5. Part of the problem here is probably an unclear idea of synergy. When we are training in the technology, we train both the business and the technical people what synergy means, and do many training exercises around the concept. If A seems synergistic with B and C, then either A, B, and C should all be in the same subset (because of the property of transitivity), or A should be in a subset different from both A and B.
Now keep in mind that "same subset" does not necessarily mean that all will end up in the same service, or same library, or same executable. Lower level granularity packaging decisions may be made by the technical folks. But the important point is that whatever boundaries are defined at the business level must project onto the technical boundaries. What this means in practice is that the business defined boundaries (as defined by synergy analysis) must contain any smaller granularity packaging used by the technical folks.
Great questions! I hope this helps.
Hi Roger,
Thanks for the replies.
Your very last paragraph on business level boundaries vs technical level boundaries is an interest one.
Correct me if I'm wrong. I think what you are saying is: The SIP methodology works at the business level but not the lower technical level(s). If so, at what granularity do you see SIP is most applicable? Do you see it's most appropriate for partitioning the inter-enterprise SOA layer?
Hi John,
The SIP analysis works at the business level, but projects onto the technical level. It definitely projects onto the inter-enterprise SOA level, but it doesn't stop there. If it is to be effective it needs to continue the projection right through to the data layer. In fact, in my experience, more systems fail because of bad data design than for any other reason. For more on this, see my white paper with Salingaros at http://objectwatch.com/white_papers.htm#SessionsSalingaros
Hi Roger,
How does this business-level-onto-technical-level projection work? Lets do a thought experiment.
Say we need to expose a banker rounding routine as an SOA service so that all our trading partners are using the same rounding algorithm. The traditional way to look at this is, the rounding routine is recognized as a low-level utility function which can be shared at compiler time. So, the routine will be shared and compiled into the Deposit service, Withdrawal service, etc., and also in its own BankerRounding service.
How does SIP handle this? Exposing the rounding routine is a business requirement. So, do I go to talk to the business people and see what syngery they see between the services? What possible answers will they give me? And how do I incorporate their analysis into the SIP process and then project it onto the technical architecture of my system?
I hope I'm not going around in circle here. I'm trying to wrap my head around this.
Hi John,
This is a great question, and one that comes up frequently. So thanks for asking it!
When you are doing synergistic partitioning, there are two steps. First, you must identify the business functions. Second, you must partition them based on synergies.
Now what is a business function? It is a function that the business recognizes as providing business value.
I'm not an expert in the banking domain, but I suspect that the business folks in the bank would say that a rounding function provides them no business value. This is not to say that a rounding function provides no value, just that that value is not at the business level.
Since the rounding function fails the business value test, it is not subject to SIP analysis. Which, in this case, means it becomes an implementation detail at the technical level. Since SIP provides no guidance on how the technical level is implemented, the technical folks are free to implement rounding any way want, as long as that implementation doesn't violate the SIP architecture.
One implementation that WOULD violate the SIP architecture is to make the rounding function a service. This violates the rule that all connections at the technical level be mappable back to connections at the business level. It is highly unlikely there there is any connection at the business level that involves rounding.
One implementation that would NOT violate the SIP architecture is to make the rounding function a shared library that any partner that so desires can make use of (and any partner that does not so desire need not make use of). And this, as you point out, is the traditional way we would have handled this problem anyway.
This is a good example of where SIP protects you from bad implementations. Imagine what would have happened if some unlucky person had decided to implement rounding as a service. This would mean that every single operation involving rounding would require a roundtrip message invocation to a share service. This would be a performance nightmare and a major single point of failure.
Yet another example of where the simplest architecture (as defined by SIP) is the best architecture (as defined by run time metrics.)
Hi Roger,
I'm afraid you missed my point. I used rounding as an example to illustrate my point, a scenario that (1) there isn't a clear cut way to define "syngery" useful for partitioning, and (2) the distinction of the business layer and the technical layer blurs.
Besides, we (you and I) are not in a position to say whether a rounding routine provides direct business value or not. By the very definition of "business value", only business people can dictate that. Just imagine the CEO of our company tells us to expose the rounding routine as a service because our trading partners have sign the contract to pay a hefty subscription fee to use it (because after GFC the fed government only trusts our rounding logic :-). How can we argue it's not providing business value and tell the CEO it's architecturally incorrect?
(While this scenario may look unreasonable and artificial, it's not. Consider Amazon. Its database functions were elevated into the business layer when they started providing EC2 cloud-based services.)
So, my question stays. How to apply SIP to partition a system involving, say, a Rounding Service, a Deposit Service and a Withdrawal Service?
John,
If the business folks see business value in the rounding function, then it passes the first test I mentioned, it is a business function.
The next question is, where does synergy place the function in the partition. There are only two choices here. Either the function is synergistic with the functions in a particular subset or it isn't. If it isn't synergistic with any of the functions in any of the subsets, then it goes into a subset of its own. If synergy is applied correctly, it is not possible for a function to be synergistic with the functions in more than one subset.
In the example you give, where the business folks see business value in the rounding function, then it will likely end up in its own subset. Why? Because the test for synergy requires mutual need. So Synergy(A, B) is true if and only if A is not useful without B AND B is not useful without A. In other words, if you can identify typical business scenarios in which B could be useful even if A didn't exist, then B is not synergistic with A.
I hope that helps.
Hi Roger,
So, we will have Rounding in its own service. Am I correct?
The Deposit service etc requires the same rounding routine. So, here I have 2 choices: (1) let Deposit call the Rounding service. (2) Compile the the underlying rounding routine into the Deposit service. Am I right it is not within SIP's scope to provide me a methodology to decide between the two?
Hi John,
Actually, the concept of a "service" is at a lower level than SIP considers. I advocate that the synergistic collection of business functions be implemented within a single software fortresses, which will have the following characteristics:
1. If will have a asynchronous messaging interface (the "service" aspect of it.)
2. The messaging interface will be a projection of the business dependencies.
3. It will be a self-contained island of trust.
4. It will project onto the data architecture, meaning that no function outside of this fortress can access this fortress's data.
SIP does not consider a shared library a proper implementation of a synergistic collection of functionality.
But I do point out again that the decision as to whether Rounding ends up in its own software fortress is implied by the business decision that Rounding a) has business value and b)is not synergistic with any other business functionality. In a real world banking scenario, I find this hard to imagine.
Hi Roger,
Would you mind to explain why shared libraries aren't appropriate in SIP? How is sharing business logic different from sharing code in the low-level "plumbing" stack? (i.e. Each synergistic collection is working on the top of OS. The OS is shared.)
It's been a pretty long "conversation". Really appreciate that you've spent the time to explain to me the details.
Hi John,
SIP has no problem with shared libraries. From SIP's perspective, shared libraries is an implementation detail.
SIP focuses on packaging issues, not implementation issues. SIP says what lives in the package and how it communicates with other packages. How things inside the package are implemented is outside the scope of SIP (as long as the SIP rules aren't violated.)
Hi Roger,
Just read the MathOfITSimplificatin pdf, and it's definitely interesting to see how you attempt to make what I'd call "intuitions" fit into a more formal framework. I don't buy it 100% but at least I agree on your conclusion that a focus on synergies from a business perspective helps partitioning in a way that reduces complexity.
I'm fan of Domain-Driven Design (DDD), and some of what I've read in your paper resonates well with that approach. In case you don't know this book by Eric Evans I'd suggest you had a look, you may be interested in it, its subtitle is "Tackling complexity at the heart of software" after all...
First your focus on synergy *from a business perspective*, as opposed to IT-driven considerations, sounds like DDD. Focusing on words and concepts that make sense to the business is also at the core of DDD.
More importantly, DDD advocates splitting big models into separate, smaller, models, each for a particular purpose. My feeling in this case is that the overall complexity a several simple models is their sum, whereas the complexity of a big all-purpose model is more the product of each atomic complexity, or growing kind of exponentially. Again reminds me of your arithmetic of complexity.
The "snowman architecture" presented also reminds the typical "Composite Application" as promoted by Udi Dahan (in SOA) and also inspired by DDD. They looks really similar either because they are driven by similar considerations, or because they contrast in the same way with many established practices.
If I had to address this problem of deciding how to split projects, I'd use the the strategic design part of DDD, starting by recognizing the "Bounded Contexts", which are unit of consistency in the domain (business) language.
Knowing the "Context Map", a key tool in strategic design, is key to partitioning projects, understanding teams cooperation and even to decide whether to go bespoke or buy off the shelf. We learn to identify the Bounded Contexts by asking the right questions and listening carefully to the business (and IT people as well), as they give all the hints required. However it is totally not formal, and hard to teach... just like you suggest in your approach when you need to identify the synergies, the key to the problem is indeed in the conversations with the business. We have to learn listening and asking better questions. And in fact that's the hard part!
And of course I regret that you patented such "methodology", a trademark was not enough? Anyway such patent is close to inapplicable ;-)
Cheers,
Hello Roger,
I saw your online presentation on the SIP process and read the white paper on your approach to partitioning using synergy. Very nice!
I agree on the general approach; splitting complex systems in autonomous subsystems is exactly my view as well, including the separation of the sub-projects all the way down to the data level.
The synergy-criterium to create the partitions seems a very interesting mechanism.
I am now particularly interested in how you see some typical problems associated with such a system separation, because in the end, the solution usually needs to become "1 solution" again.
Below I describe 2 typical problems related to such an approach:
1) The business function of maintaining highly shared reference data; e.g. currencies, product catalog, policy parameters, ...
Most of the other sub systems have one-way dependencies on this facility, so no synergy; it will be a partition on its own.
Following the paradigm this would be a standalone system, with its own storage solution.
The problem that arises is how to couple:
- SOA messaging to retrieve a currency symbol is obviously slow
- A shared DB in fact breaks the paradigm of autonomous sub-projects and makes changes to modules a pain (cascade of changes to all other sub systems)
- Each system has its own copy of relevant reference data, getting update-messages from the central maintenance module. This maintains the paradigm and can make changes manageable. But as it creates right away more complexity (many independently versioned implementations of each set of reference data); that goes against what we want to achieve: reducing complexity.
Question: what is your view on this dilemma?
2) Your presentation in the SIP process takes the example of a SOA architecture. This is fine, but obviously most systems are not just integrating on the SOA level but also on user interface level in order to give users a coherent user interface to all capabilities.
This suggest to me that saying that each partition will be executed as a separate project and next simply integrated at the end seems a bit of an over-simplification. Creating a coherent user interface with a workflow and UI-logic supporting helping the user in coherently using the capabilities if each sub project is almost a project on its own.
Question: what is your view on how to deal with this in a SIP-like approach?
Thanks,
Joseph
Post a Comment