Integrated Infrastructure Blog

Rage against the term 'ESB' not the need for Integration

In my opinion the rage against ESBs is directed more at the term "Enterprise Service Bus" than at the need for integration within an enterprise SOA. Few bloggers would publicly discredit integration infrastructure in the same way that they rebuke the term ESB.  Corporate IT portfolios are ripe with IT systems needing to be integrated and line-of-business executives who remain disgruntled with the lack of agility shown by IT in response to their demand for business change.  So, if you replace the label "ESB" with the word "integration", I expect that the vitriol would turn to sweet loving respect if not desire.

Integration infrastructure buyers ("ESB" buyers) are learning that most ESBs that come packaged with a platform are really unadorned web service brokers – allowing simple transformations of SOAP messages but little more.  If they've done their homework, these same buyers have learned that many vendors re-label security appliances, EAI suites and even application servers as ESBs.  So there is little surprise that the term "ESB" itself has become the object of such animosity. 

Furthermore, most new SOA infrastructure applications don't need an ESB.  ESBs are designed to provide integration between applications and are often overkill when put between services within a single development team. And when real integration projects do arise the complex nature of legacy IT infrastructure makes exceeds the capabilities of these one-size-fits-all service brokers.

Many of the clients with whom I've spoken about integration recognize the vitality of this solution and are on record with their satisfaction:

• Andy Edwardson, Vice President of information technology at FAMI, commented: "In addition to tackling internal integration challenges, our goal is to create a highly available environment in which we can more easily interact with business partners over the Web. Our desire is to develop an enterprise architecture that is less 'hard-wired' and more service-oriented, allowing us to create reusable business objects resulting in a more agile business environment. The Sonic ESB will facilitate such an environment."
• Bruce Hogg, Enterprise Architect for Pacific Blue Cross, commented: "Sonic ESB allows us to leave existing processes in place while layering new business functionality on top, and it allows us to implement a messaging-based distributed infrastructure for migrating to an SOA, step-by-step."

These supporting quotes suggest that some customers have come to draw their own conclusions about what an "ESB" is and as such, support the term - provided that the solution meets their needs (and perhaps is implemented in the right way). It's frustrating that the market has put them in a position where they have to go to such lengths – it makes their job of rationalizing the options available to them that much more difficult. Perhaps in the end, we should admit that the battle of defining ESB properly is a lost cause; rather vendors like Progress focus on SOA integration in general and the value of a robust solution to this age old IT challenge.

Loose coupling versus decoupling ...

Last week Jeremy Suratt made an interesting point on the Infor Tech Talk blog.  He says that event-driven architecture (EDA) is different from traditional service-oriented architecture (SOA) in the manner in which applications are related - or not related.  In particular he says "key to event-driven SOA is the ability for decoupled, asynchronous operations."  I agree wholeheartedly with Jeremy and think this point is one of the least well understood aspects of modern enterprise architectures and the complexity inherent in large integration projects. 

There is a great book on this topic entitled "Software Fortresses" by Roger Sessions.  Though the book doesn't mention the term SOA (it largely predates SOA and the current hype), it does point to the fundamental need for decoupling of systems through asynchronous messaging.  In the book, one of the golden rules is "never let transactions cross software fortresses."  I may be misquoting this a bit ... but the essential point is that between application domains, which are referred to as fortresses, you should allow for no dependencies which might block a transaction's completion.  This rule flies in the face of what most SOA infrastructure advocates today suggest is the very purpose of this new architecture.  Aren't we to just make a Web Service call when we want to leverage components in other systems?  Roger says - just because you can doesn't mean you should.  Once you grok these simple architectural best practices and the power it brings through "isolation", you're fast on the road to understanding the difference between traditional SOA and event-driven SOA.  The latter abides by the golden rule by inverting the control of intra fortress communications - using an event publishing rather than a service invocation model.  The former decouples the applications and the latter couples them - albeit loosely through Web services. 

Perhaps this is a long winded "dah" moment for you, but for many organizations it's still something they wrestle with daily.   Progress chose this decoupled model for the design of our integration backbone, Sonic ESB.   As a result, applications integrated with Sonic ESB are naturally decoupled.  This means you can shut down any application without impact to the overall operation of the other applications.  Unlike systems that link through traditional SOA request/reply invocation... Sonic ensures that messages get through when the application comes back online.  Now this is very different from other ESBs on the market or available through open source today.  These ESBs employ a "traditional SOA" architecture - meaning that applications connected through them are coupled via service-proxy rather being decoupled through asynchronous message forwarding.  I don't want to belabor the point, but a loose coupling of Web services is an insufficient architecture for large integration projects.   If you're not familiar with this type of ESB, I encourage you to download our latest Sonic ESB 7.6 version and go through the tutorial on "batch to real-time" processing - which shows the power of event-driven integration in the form of a continuous pipeline processing example. 

Before you start writing you own ESB ...

Vincent Partington recently wrote on his blog "before you start writing your own services and your own ESB, have a look at what's available."  Vincent point out the challenges of migrating from a home-brew ESBs and recommends against a "not invented here" attidute.  Reminds me of a dinner conversation in Winnipeg, Manitoba several years ago following my presentation of a new SOA Maturity Model to a group of local IT directors and executives.  The gentleman next to me asked a simple question - of all the services in the world (SOA) which ones are the most important to do first?  I was a bit set back by the question since I'd never had someone ask such a straight forward and obvious question.  After chewing a bit on my dinner I admitted how profound it was and then suggested - file parsing, transformation, routing, and file writing were probably the services that everyone should invest in first.  These are precicely the same set of services now offered out of the box - in both commerical and open source ESB (enterprise service bus) implementations. 

At Progress we've gone a step further to document larger enterprise integration patterns such as "batch to real-time" and "remote data distribution" on our developers network, and we also offer public code share libraries for users of Sonic ESB.  So, long and short - consider downloading Sonic 7.6 and leveraging some of these prebuild integration services, code shares and higher level EAI patterns.  I think you'll see that Vincent was right on the money with his advice, "...have a look at what's available."