SOA Infrastructure Blog

The power of proactivity

Earlier this week (last Friday evening to be exact), I took off on a business trip to Australia.  I was scheduled to go from Boston to San Francisco, and from there to Sydney.  When I arrived at the airport, I checked in, cleared security, and meandered towards my gate.

When I got to the gate, the gate agent was calling out my name.  United's not my regular airline so I didn't have any special status that would get me upgraded automatically or anything like that.  So, when the gate agent calls your name in this situation, you usually think "uh oh". I went up to the gate to find out what was up.  Here's what they told me:  They said that the flight to San Francisco might leave late due to delays in San Francisco so my connection would be tight.  So, just to make sure, on the spot they re-booked me through Los Angeles on flights that left and arrived at around the same time as my original itinerary.

The last 15 years I've been Platinum or higher on multiple airlines - but this was the first time an airline had ever been this proactive.  As soon as I was re-booked, I checked and there was no delay listed for the San Francisco flight yet nor did the FAA show any general delays for San Francisco.  I was pretty impressed and am a very happy customer. I will also try to fly with them more in future (now if they'd only put electrical outlets in coach - but that's another story).

How does this translate to the IT world?  If you can anticipate or detect problems before your business users are even aware of them you, will become a hero.  Some people want to hide issues from their users, but don't be afraid to let your business users know there's an issue. If you are taking actions to address the issue, and the users see that, they will gain trust in you.

A question that often runs through people's minds when they think of this is, will your users think you're not on the ball if this happens too often (is it better to only react to the really bad issues proactively)? Would I have gotten mad at United if the original flight wasn't actually delayed?  Not at all.  The fact that they were thinking ahead was what mattered to me. 

Users don't expect perfection - they expect (and respect) honesty, empathy, responsiveness.  Give them that and they will be with you for the long haul.

How to Be Agile Without Falling on Your Face

Eric Knorr of InfoWorld recently wrote an article about how testing was the key to how to be agile without falling on your face.

While I agree that IT organizations are squeezed and that quality is often given short end of the stick, I don't think Eric's solution of "optimize the efficiency of your testing tools, environments, and methodologies" is necessarily the right way to think about the problem.

Most organizations think of quality as a role (the role of the tester) - and as a result most testing tools are just that: specialized tools designed with the needs of the professional tester in mind.  The problem is, with modern applications, quality doesn't start and the tester and doesn't end at the tester.  Quality is a responsibility not a role - a responsibility of everyone involved from development through production.    This is one of the reasons that developers really like our free Actional Diagnostics tool (used to be called Mindreef SOAPscope) - it's designed from the perspective of the needs of developers, but it focuses on improving quality. For example, we found in formal ROI analysis that deploying our products in the hands of developers reduced the number of production issues by 25%.

So, rather than focusing on how to make testers more efficient (which will only get you so far), it's better to focus on tools and SOA infrastructure to help developers and architects avoid defects before they are created.  But these aren't testing tools because developers don't use traditional testing tools.  If you have 10 defects, avoiding just one of these development (something which is eminently doable) is as beneficial as improving testing productivity by 10% (something which is probably a lot harder).

On the other end of the spectrum, with today's interconnected applications the number of defects which will show up in production is significantly higher than traditional applications, and you'd likely have to increase your testing by a factor of 10x to get to the same production-defect-rate as a traditional silo'd application.  Alternately, a 10% reduction in the time-and-cost of resolving issues in production easily pays off as much as a 10% improvement in testing efficiency.  And, achieving a 10% reduction in this is definitely achievable... Now imagine you could reduce the time-and-cost of resolving issues in product by 85%...

Oracle buys Sun: Let the revolution begin

First, Marie Antoinette said "let them eat cake."  Will Oracle buying Sun have a similar outcome?

If Oracle goes down one path, it could be a brilliant strategy.  Oracle was previously competing with two other enterprise software vendors: IBM and SAP.  Now, they only have one competitor: IBM.  Two vendors providing turnkey solutions: hardware, software, and services all rolled into one.  What's more, cloud computing is essentially a turnkey services business as well - so it positions Oracle to capture a huge slice of both traditional and cloud computing worlds.  Down this path, enterprises end up with the ultimate in flexibility: whether on-premise or cloud, powered by a common Java-based technology stack and hoards of consultants, the business needs rule the day.

On the other hand, if Oracle goes down the other path (taking Java, MySQL, and other Sun assets and clamping down on them), it risks alienating a large community of both vendors and customers.  Frankly, Oracle doesn't have a great reputation in this regard - so, even if they do everything right, people might still turn away because of (historically well justified) fear.  If this happens it could turn the continuous trickle of people moving from Java to other technologies (.NET, Ruby, PaaS, etc.) into a flood - a rebellion triggered by a ruthless but short sighted empire builder.  If this happens, the traditional enterprise software market as we know it could implode, to be replaced by fundamentally different technology base (almost certainly cloud native - can you say Google Apps?).

Which path do you think Oracle will take?

SOA saved by measurement?

Joe McKendrick over at eBizQ blogged about two reports that say SOA is as good as you measure it - in essence saying that the key to SOA success is measuring it.

I'm a big fan of measuring things.  If you can't measure the benefit of doing something, you need to think twice about whether you should be doing it at all.  But, these reports miss the mark in a number of ways.

The first is that measuring SOA doesn't matter if you can't get buy-in to do SOA.  You can build the best measurements in the world, but if your targets say that, after your large up-front investment, SOA will begin paying back in 1-2 years, you're probably not going to get the funding.  Time to value is a critical component that can't be underestimated (especially in this economy).  Risk / reward ratio is another critical component.  Small investments that yield big returns in short time frames are what you need to be looking at if you want to get SOA off the ground.

This brings us to the second issue:  Meaningful metrics.  If you're a service provider business, you can certainly measure things like revenue-per-service, "service vitality", and ROI for revenue bearing services.  But what the heck does this have to do with SOA?  People have been building and delivering  services that earn revenue long before SOA ever existed.  And, how do these metrics work for companies that don't sell their electronic services?  What's the directly measurable ROI for FedEx's free web service to do package tracking?

When you want to measure the ROI for SOA, you have to measure it versus alternative options.  Building a revenue bearing service with a SOA architecture doesn't mean that the entire benefit for the service is attributable to SOA infrastructure.  What you need to measure is the cost-to-build-with-SOA versus the cost-to-build-without-SOA (because, in either case, the revenue generated will likely be the same).  Whether FedEx uses SOA behind their package tracking service doesn't change the business impact of electronic package tracking.

This is where things get complicated: How do you separate the real benefit of SOA from alternative approaches?  This is where these reports further get it wrong.  Most organizations never do a complete accurate costing of how to build something in two ways (with SOA and without) - so you can't measure the SOA ROI for a service.  And, when people do generate these cost estimates, in reality they are often manipulated to ensure that the path they want is the one that looks good: so you can't trust these numbers.  Similarly, metrics such as service reuse rate (versus rate of developing new services) are easily unintentionally manipulated.  Build fewer services up front and you'll have better reuse rates (the metrics put pressure on not building services - exactly the opposite of what you want in a world where the culture is already to not build services).

There are certainly some metrics given in these reports which are meaningful.  For example "mean time to service development" and "mean time to service change".  Of course, these metrics aren't practical in the real world.  Every service has vastly different complexity, so to get a real mean, you need 100s of samples (100s of services to be built and changed) before you get any real numbers.  This will only happen years after you get started unfortunately.

So, all this talk about metrics is great, but don't be a measurbator.  Make sure you put on your realism hat:

• Don't focus on metrics that are "suspect" (e.g. ROI for SOA) - the business won't believe you.
• Don't focus on metrics which can't be measured until years later (e.g. mean-time-to...) - the business won't listen to you.
• The lower the costs, the less justification you'll need
• The lower the risk, the less justification you'll need
• The quicker you show perceived benefit, the less you'll need to worry about metrics

Metrics do have a place in SOA - to help refine it over the long run.  But, people won't believe the metrics until they see results, which is why you can't start out that way. The best way to get SOA infrastructure off the ground is the time-proven method: Get a bunch of smart do-ers (not talkers or thinkers) on an important project that, using SOA, you can deliver quickly - and use this to gain mind share of what SOA can do.

Progress Software Introduces Actional Diagnostics

Presented by Dan Foody, VP of Products for Actional at Progress Software

With Actional Diagnostics, you can accelerate Web service development, testing and validation with tools to improve service quality, performance, and compliance, and simplify time consuming XML-oriented tasks. Listen to this podcast and hear Dan talk talk about the features and benefits of introducing a service quality and validation tool into your SOA infrastructure initiative. Register to be alerted when the FREE download is available.

For more SOA podcasts, subscribe to our iTunes channel.

The problem with good governance

I saw an interesting story in the news two days ago.  It's about a company that has received multiple "good governance" awards.  Well, it turns out that their "good governance" has led to the chairman having to admit that their finances were falsified.  Know who I'm talking about?  Satyam.  Yes, that good governance award winner - the one that turned out to have over $1B of "falsified" cash on their balance sheet.

Sure, this is corporate governance, not IT governance - but it raises an interesting question for IT and SOA governance: How do you know your governance is not being bypassed? Because this is exactly what happened in the Satyam case.

The thing that likely got Satyam into the problem was likely one key thing: Their governance checks were manual, not automated.  This meant that they could be easily bypassed or avoided.  They were people processes, not automated processes.

SOA What? The moral of the story is that governance that's not automated (with the checks in the right, unavoidable, places) is governance that doesn't work.

Goodbye SOA, we hardly knew you.

Anne Thomas Manes of Burton Group posted how SOA is Dead; Long Live Services.  This one is too juicy not to comment on!

Firstly, I totally agree with Anne.  I think "SOA" is definitely dead in 2009.  And I also agree that the need for services will grow.  But, we still wrap too many things into the same terms.

At a simple level, I'll break it into four points, I'd like to talk about where I think things are going:

Service-oriented applications.  BPM, cloud computing, SaaS, PaaS, composites, mashups. All of these are doing well and will continue to do well even in the 2009 economy.  Why?  1) They can be leveraged as needed; 2) They can be done fast; and 3) Using them typically provides measurable benefit from day one (if it doesn't, then you should do something else shouldn't you!).  It's precisely the laser focus solution aspect of these that is their greatest strength.  Get in and get out while solving a problem that someone really cares about.

Service-oriented architecture (SOA).  This is where you look from a higher level above all your service-oriented applications to create a cohesive architecture that all of these follow.  Dead in 2009.  Why?  Because approaching this from an architectural perspective will always fail.  Is it a noble goal?  Sure.  But without other changes in your organization (see below), it just can't be successful.  Said another way, before you can start on a successful SOA infrastructure initiative you must finish the items below.

Service Delivery.  By service I don't mean what you're probably thinking (a thing you create).  I mean service in the most pure sense: as in "what value do you deliver to others" (with the follow up of "how do I continue to improve the value I deliver to others").  This requires a transformation for most IT organizations... and most haven't even started yet. To be fair, there are organizations that have figured this out (take S&P for example: their whole business is service delivery, so of course they understand it) - but the vast majority don't fit into this category - and if you do not pass go, you can't collect your $200: So this is the next step for most IT organizations that want to transform.  Think: "How do I create my first true service?" Here's a hint: you can't do it by creating a project, task force or cross-functional team (more on this in a subsequent post since this is a richer topic than I can cover here).

Service-oriented IT.  Once you've mastered service delivery on a small scale, then you can tackle it on a larger scale: turning IT from being project and application oriented to being service oriented.  Warning: this is a major transformation that can't be done without the CIO leading the charge.  Sorry, but no enterprise architect can make this happen.  And it can't happen until you've figured out service delivery.  This is not a 2009 project for most organizations.  It is a 2010+ transformation.

Most organizations that went down the SOA route tried to do SOA without first figuring out what it means to deliver a service, and without realizing they needed to get this right (and begin the service-oriented IT transformation) before they could never be successful at service-oriented architecture.

Service-oriented architecture is a technical and architectural transformation.  But it can't exist without first focusing on the organizational and cultural transformation of service delivery and service-oriented IT.

This is why SOA is dead in 2009 and why services will live on. But who knows... maybe once an organization figures out service-oriented IT, SOA will be resurrected - but don't wait up.

Goldilocks Governance

An article in the Wall Street Journal today on approaches to IT Outsourcing (The Goldilocks Strategy) has definite parallels to what I've seen with SOA Governance.

The article talks about an organization which first moves to outsourcing with a very loose contract, and ends up failing (poor service, mistrust, etc.).  Then, they move to an airtight contract that lays out every last detail.  But as their business changes, the rigid contract creates issues that result in... you guessed it: poor service and mistrust.  So, they finally switch their approach to one that's much more hand's on to build trust between their organization and the outsourcer, allowing them to more easily adapt to changing business climate... and they end up with a very successful IT outsourcing strategy.

The same parallel hold true for most SOA Governance initiatives.  Either there's none, or it's too overbearing (automated, rigid, and inflexible) - creating the exact problems that governance is intended to fix.  Moving from web services to REST?  "Sorry, our governance approach doesn't allow for that."

The right solution is often one that has little to do with tools, technologies, contracts, or mandates but instead is heavily based on the enterprise architecture team building a strong personal relationship with the rest of the organization.

Cloud Computing - The New SOA?

Presented by Dan Foody, VP of Products for Actional at Progress Software

Is SOA dead? It seems the technology buzz these days is all about cloud computing. What's the difference? Do they complement one another? Listen to this podcast and hear Dan talk candidly about how cloud computing and Web 2.0 are being integrated into today's enterprise architecture.

Subscribe to the SOA Infrastructure iTunes channel to hear more podcasts published by Progress Software.

Has availability outlived it's usefullness for SLAs?

Probably the most common metric used with Service Level Agreement (SLA) for enterprise applications (and SOA infrastructure) is availability.  While my primary interest is in figuring out how to define effective SLAs for distributed, transactional business applications (things like order management, reservations, etc.), I thought it would be interesting to explore the use of availability for more basic applications - for example email servers.

Most IT organizations will provide an SLA to their business for email.  A typical example would be "we guarantee at least 99.7% availability for our email systems."  That's basically less than 26 hours of downtime in a year.

So, this begs the question: Is this a measurement that business users care about?  Because, after all, the purpose of an SLA is to ensure you meet the expectations of your users.

Now, 26 hours a year could be 4 minutes a day at 4 a.m., or it could be one full day of downtime on the last day of a quarter.  Do you think a business user considers these equivalent?  Of course not.  Having email go down for a full day on the last day of a quarter could be extremely damaging to a business, whereas 4 minutes of downtime in the middle of the night each day would be irrelevant.

Let's look at a different variation: let's say we have 15 minutes of email downtime, right in the middle of the day, every 3rd day.  Would that be an issue?  Well, with Microsoft Exchange and Outlook, users wouldn't even notice this because Outlook caches data locally even when the server is online.  So, even if the server is down the user can be reading emails plus drafting and sending new ones.  In fact, with Outlook users have to look hard to even know if they are connected to the server or not - it's not obvious.  The end user perceives email as subjectively up or down based on whether they can interact with Outlook or not - not whether the server is actually up or down.  They also perceive that email is "slow" when Outlook is slow (not when the server is slow).

Now let's say the email server is up and running (so it's available), but the virus checker hooked into the email server is spinning in a loop, causing a 2 day delay for every sent or received email (something that happened to us at one point).  Even though email is available, clearly end users would be very unhappy.

So, it's pretty clear that % availability of an email server is a useless metric for email SLAs.  It doesn't correspond to what users care about.  Users care that the emails they send arrive in their recipients mail boxes in a reasonable amount of time (let's say less than 15 minutes).  Similarly, they care that email arrives in a timely manner as well.  So, the most meaningful email SLA would be something like "we guarantee that inbound or outbound email will arrive within 15 minutes."  They also care that they can constantly interact with the interface to the application (for email this is Outlook).

To be clear, I'm not saying that measuring availability is wrong.  Knowing when the email system is unavailable is a useful diagnostic.  It lets IT know that they might end up with a SLA problem.  But don't think that the business cares one bit about it as part of your SLA, it's just a diagnostic tool - an early warning sign of a subset of problems that might cause an SLA violation.

OK, now let's translate this to a few lessons about defining effective SLAs:

1. The best SLAs are ones that measure metrics (and guarantee results) that directly correspond to the expectations of users or to measurable business metrics (e.g. revenue).
2. The synchronous parts of an application (specifically, the part the user directly interacts with) should be measured with different metrics than the asynchronous/background parts of an application.
3. For the synchronous parts of an application outages and performance issues should be weighted by when they occur (mid-day or at night, beginning of quarter or end-of-quarter, etc.) and how long they take - a raw total or % is not that meaningful.
4. For the asynchronous parts of an application, SLAs are most effective when they relate to missing important deadlines (for example, not being able to recognize revenue by the end of the quarter, not being able to ship a next-day-delivery-guaranteed product by the end of the day, the expectation of an email arriving within 15 minutes, etc.)

So, yes, defining a good SLA is hard.  You need to understand your users and your business.  You need to measure things which aren't necessarily that easy to measure.  Many IT organizations take the easy road out and measure metrics like % availability and call it an SLA (no wonder business people view IT as out-of-alignment with them).  Providing SLAs which follow the lessons above will ensure that you meet the expectations of your users, customers, and partners.  It may be hard, but the result is priceless.

What does that web service mean to you?

Do you have the right SOA infrastructure tools in place that will help you achieve better business visibility?

SOA management tools that aid in runtime governance and service quality will help. Listen as Dan Foody describes how a popular industry automobile website had some troubles because users were learning how to "game" their system.

Interested in more? Visit and subscribe to our iTunes podcast channel.

When do you need SOA governance?

I was at Software AG's SOA Governance Summit last week and got a chance to see Gartner analyst Frank Kenney give a presentation - on SOA Governance of course.  As always, Frank did a great job - but there is one area that I feel he got wrong.

Frank said that you don't need governance products until you have more than 10-12 services (up to that point Excel will do fine to track things).  Now, Frank was partly doing this because he wanted to show he was independent of the vendor sponsoring the event - so whether he believes this or not, I don't know.

But, here's an example to show why this isn't true: Let's say you have just one service but that service has 1000 different applications consuming it.  Do you think can get by with Excel?  Do you think Excel will help ensure you meet the service levels of each of your 1000 consumers?  Of course not.

In the end, the number of services is irrelevant to whether you need SOA governance or not.  What really matters is the number of relationships.  Having 1 service consumed 1000 times is just as complex as having 1000 services each consumed once.  But, it's definitely a different kind of complexity and so how you govern these two scenarios is very different (with 1000 services you need design time governance first but with 1000 consumers you need runtime governance first).

If you're with me so far, let's bring this to the next level...

SOA Management. Do you really need it?

What the cost is for not having it.

Did you forget to think about SOA management when you developed your SOA infrastructure deployment plan? If you haven't already, listen to this podcast presented by Dan Foody, VP of Products for Actional at Progress Software, and hear an example of a company that did just that and regretted it. Make your SOA more agile by making sure that the right SOA management tools are in place before you deloy your SOA infrastructure.

The Problem with SOA is Architecture

As I read David Linthicum's post on SOA vendors focus too much on integration... and not enough on architecture, this really clarified one of the messes we've gotten into with SOA.  The problem is "architecture."  Dave's definition of architecture (from the post) is:

"the orderly creation, placement, configuration, and management of IT assets."

The problem is that it's only one definition of architecture.  Dave is referring to enterprise architecture:

Wikipedia: "a process for describing an enterprise and its information systems and planning changes to improve the integrity and flexibility of the enterprise"

The other type of common architecture in the software world is application architecture (or you might call it software architecture):

Wikipedia: "the structure or structures of the system, which comprise software components, the externally visible properties of those components, and the relationships between them."

I think that one of the things that gets people so confused with SOA is that the term never really clarifies what type of architecture it's referring to.

So, maybe we really need two terms:

• Service Oriented Enterprise Architecture (SOEA)
• Service Oriented Application Architecture (SOAA)

Once you recognize this, a number of things start to fall in place.  Both of these have value... but they are different.

Arguably, most enterprise architects think of SOEA and so the term SOA (at least among the purists) has drifted towards meaning SOEA.  Unfortunately, this has alienated a lot of people, because a lot of people experience the value of SOAA every day (web 2.0, SaaS, and cloud computing are great examples of SOAA but have little to do with SOEA) but don't think of it as SOA.

Now, I'm not recommending we create two new term (SOEA and SOAA).  The world is already confusing enough.  Maybe what we need to do, as an industry, is consolidate on using "SOA" to represent service oriented enterprise architecture, and just refer to everything else (SOAA) as "service oriented."

Actional's Strategic Partnership with Software AG

Today, Software AG announced the strategic OEM partnership between our two companies.  What makes this especially noteworthy is that, up until now, if someone wanted to buy full-lifecycle SOA governance, they only had two choices:

• Buy all the pieces, design and runtime, from one vendor (knowing that one or the other was weak - since no vendor had even a half decent offering on both sides of the fence).
• Buy market leading components - but have to buy them from different vendors

This partnership will allow customers to buy full-lifecycle SOA governance from one vendor, without having to compromise.

One other important note is that Actional's direct competitors have tried to create FUD in the market by claiming that Actional is tightly tied with Sonic.  Of course we do work well with Sonic, but a very large percentage of Actional customers use us with platforms such as BEA, TIBCO, IBM, Microsoft, JBoss, and others - no Sonic in sight. Our competitors certainly liked to paint a picture that Actional was "Sonic centric" even though this was far from the truth.

To have a successful SOA management and runtime governance offering:

• Your products need to be vendor neutral - working equally well with the SOA infrastructure of many vendors.
• As many more organizations are scaling up their SOA initiatives - building mission critical applications that are service oriented - SOA management can't cause your applications to slow down, or cause you to lose transactions.  It needs to be scalable and reliable.

These were two important factors in Software AG's selection, and this partnership clearly demonstrates Actional's leadership in these areas.

SOA is like a financial market part II

Glen Daniels added a good comment on one of my last posts.  Glen's point was that in a financial market everyone tries to achieve their own goals, but it's different in an enterprise - an enterprise is more like a sports team, with everyone working towards a common goal.

Sorry, Glen.  I can't agree. The majority of people care most about closest to them (i.e. their health and welfare, etc.).  And, sometimes this aligns with their organization (or team) - just ask any Red Sox fan whether Manny Ramirez was always acting in the best interest of the team and you'll see this is true.

Here's a business example that's more relevant: SOA goals are often long term goals.  And these can often conflict with short-term goals of getting a project delivered quickly.  It might be in the best interest of the organization to follow the "SOA route" and delay project delivery - but the IT team would risk being fired if they did that.  So the majority of IT teams focus on the short term goals.

The way I look at it, your assumption out of the door should be that no one is aligned with your SOA goals (just like in a financial market you're acting like the market regulator).  Yes, this is the paranoid approach: they are out to get you.  Even if only 20% of them are out to get you (and the other 80% are with you) the problem is you usually don't know which 20% to worry about!

So, what does this mean?  You should always focus on ensuring the goals of your SOA are never in conflict with other goals that people might have (e.g. short term project delivery deadlines).

WOA governance

Joe McKendrick wrote an interesting post on WOA winning the personality contest versus SOA.

That got me to thinking: If we need governance for SOA, do we need it for WOA also?

Some  would argue that it's the complexity of SOA itself (driven by the enterprise top-down focus) that creates the need for a formalized SOA governance initiative.  Without formal SOA governance you can't hope to succeed with SOA because it's too easy to get it wrong.

But WOA removes or bypasses many of the complexities of SOA (no need for complex tools, no need for WS-splat and all the arcane requirements that go along with that). So do you still need governance for it?

The answer is probably yes (if you're an enterprise architect, you can stop holding your breath now).  But, I think the approach to "WOA governance" is going to be fundamentally different than that of SOA governance (OK, time for the EA's to hold their breath again).

In traditional SOA governance, enterprise architects set rules for the providers and consumers of services to follow - they set the rules that govern both sides of the interaction.  This works fine in an enterprise where everyone ends up reporting to one common person when you look far enough up the chain.

But, let's look at a WOA example: Progress Software consumers services published by Salesforce.com.  If we were to apply a traditional SOA governance approach to this, we'd first appoint an "Enterprise Architect for the Internet" who would set all the policies that both Progress and Salesforce.com would have to follow.  Simple really.  Except the part about "appointing an EA for the Internet".  That might be a bit tricky. So, you can see, the top-down approach of SOA governance totally falls down when you look at WOA.

But, if a top-down approach doesn't work, what does?  Do we not have to solve some fundamental "governance" problems still?  Problems like:

• How can a provider make it easier to on-board customers and keep them happy (all while changing the service frequently)?
• How can a consumer establish and build trust in their service provider (that's trust as in "trust but verify")?

WOA absolutely has to address these challenges.  So, for WOA we need to achieve many of the same goals that SOA governance hopes to achieve - but we'll have to achieve them in a fundamentally different way.

Roles and Responsibilities in SOA

Stephan Lahanas posted on the top-10 reasons SOA projects fail.  I think many of his points are very valid (especially the one about how it's the business that matters, not the standards or the hype).

But, I think one of his points is a bit off the mark:

"Design drives development not the other way around, we wouldn't want the drywall guy designing a skyscraper, yet that happens in SOA implementations (figuratively speaking)...You can't really do 'Agile SOA'..."

Well, I certainly wouldn't want the architect of a skyscraper putting up my drywall.

But seriously, creating a SOA infrastructure isn't like creating a skyscraper. Creating a SOA is not an orderly progression of activities that lead to a defined end result.  It's more like a financial market: anyone can do whatever they want to make themselves wealthier, as long as they don't violate the rules (i.e. the law).  There is no defined end-result for a financial market: everyone is free to try and achieve their own optimal end result.  And many financial markets are extremely agile.

So, can you do "Agile SOA?" Absolutely.  As long as you've optimized the rules - your SOA governance - to allow for agility (which many people get horribly wrong, of course).  Provided the rules are not cumbersome, if someone else has built the services you can very quickly create consumers for those services without any muss-or-fuss.  No waterfall model of software development needed; no systems engineers needed.

If, on the other hand, you think of your organization's SOA as one big project, then almost certainly there is no way to be agile.  But, if you're thinking about it this way, you've already lost anyways. Central planning never has been (nor ever will be) agile.

Less Governance is Good Governance

Todd Biske wrote a good post the other day that focused on how governance doesn't mean command and control.

Let me add a few important things to the discussion though: no amount of "management by consensus" will make everyone happy.  There will always be someone that's unhappy with something.  The real issue is that different people will be unhappy about different things - so you'll end up with everyone unhappy in one way or another anyways.

What this means for SOA governance is actually pretty simple: less governance is good governance.  It sounds pretty simplistic but the fewer policies you put in place, the fewer things there are for people to be unhappy about - and therefore the easier it will be to come to consensus, and the more people will be happy being governed.

As much governance as is absolutely necessary, but no more.

When Design-time Becomes Redesign-time

I continue to be surprised by the number of enterprise architects I talk to that think that the way to implement design-time governance is to have a checkpoint, before a service goes into production, to validate the service meets the requirements and rules.

The key problem with this approach is that it's already too late.  By the time a service is ready to be deployed, it's well past "design-time" (technically, it's actually "deployment time").  So, one of two things happen: The service meets the requirements so it can go ahead, or it doesn't. If it doesn't, it's back to the drawing board for a redesign (or the project gets deployed anyways because it's more important than following the rules set by the enterprise architect).

While it's definitely valuable to have deployment-time validation "checkpoint," you shouldn't confuse this with design-time.  Real design-time SOA governance happens when the service is being designed (surprise!), so that you avoid problems before they are baked in and expensive to change.

If your design-time governance approach consists only of checkpoints at deployment-time, you should really call it what it is: "redesign-time governance".

What type of CIO do you need for SOA?

There's been a lot of talk over the last week around Burton group's report on SOA adoption.  Most recently I read Dave Linthicum's Should you Fire your CIO blog entry and I totally agree.  Not that I mean you should fire your CIO, but there are definitely different styles of CIO's - and if you have the wrong kind maybe you need a change.

What I find most valuable about the Burton report is that, unlike many of the other recent SOA surveys, it doesn't rely on self evaluation.  After all, if I'm asked whether I'm ugly and smell bad, my answer would be "no" (whether it's accurate or not) - this is what people smarter than me would call "cognitive dissonance". So self evaluation isn't a very reliable way to measure SOA success.

That said, back to the topic of CIOs.  There's a pattern I've seen: the "change and go" CIO.  Certain CIOs make their living coming into an organization to shake it up, change it, get it going... and then they leave onto the next company that needs their help.  In a number of cases, I've seen SOA infrastructure initiatives kick started by a "change and go" CIO.

Why does this work?  These CIOs only go to companies that recognize they need to change - so you've got a fundamental ingredient for success: a mandate for change, driven from the top - and someone to make that change happen, with no desire to carry sentimental baggage forward if it doesn't make sense.  Now, I don't have data to support this, but I'd hazard to guess that more than a few of the (small number) of success cases that Burton found were a result of one of these "change and go" CIOs.

If your organization is looking for a change, it may make sense to look for one of these unique breeds of CIOs.  The one thing you'll want to do, however, is interview the CIOs that came after the "change and go" CIO to make sure that they didn't achieve their short term results at the expense of long term pain (which only the next CIO will be able to tell you!).

The secret is out: We acquired Mindreef

Well, my intrepid investigative reporter friend, Jeff Schneider, broke the news.  We've acquired Mindreef.  You can never time these things perfectly, and we didn't want to have news of Mindreef get lost in all the noise around some of the other acquisition stories.  So, we had been planning to announce it a bit later.  Of course Jeff, never one to let go of a breaking story, couldn't hold off :-)

On the Actional side, we first started partnering with Mindreef back in 2003 or so (long before Actional was acquired by Progress).  One of the first things that attracted us to work with Mindreef was the way their products make working with XML and web services easy.  As Mindreef expanded their product line they added the ability for a team of SOA stakeholders to collaborate, working together to ensuring that services are delivered successfully - something that still separates Mindreef from any other SOA quality or validation product.

In the SOA space, there's often been a clear dividing line between products that focused on design time concerns and those that focused on runtime concerns. But, SOA isn't so clear cut.  When I've built a service that's in production, and you want to design a consumer for that service, is this design time or runtime?  What if I've built a consumer that's in production and you want to change your service, is this design time or runtime? Well, it's both really.  So why then do we have such a divide between design time and runtime SOA governance products?

By bringing together Mindreef and Actional we're doing what's only natural - removing the artificial dividing line between design time and runtime, addressing the complete SOA lifecycle with technologies that are leaders in both.

Are your services dominant or submissive?

A lot of business relationships exist where one side is much more powerful than the other.  Walmart's relationship with their suppliers is a text book example - Walmart is definitely the dominant one: "Here's how you're going to work with us and, if you don't like it, find someone else to sell to."  And, their suppliers definitely do what Walmart wants, no matter how painful: "Thank you sir, may I have another."

What's interesting is that this exact same pattern appears with services. People that build services seem to naturally either take a role of assuming dominance (assuming they are the gorilla) or one of submission (assuming they need to cater to the whims of each of the consumers of their services).  Of course, there's nothing holding back the exact same service provider from going down either one of these paths - in the end it's all a state of mind really.

The important question, though, is whether one is better than the other?

Welcome Iona!

As you've no doubt heard, Progress announced the acquisition of IONA yesterday.

The Actional team has been partnering with IONA since well before this acquisition business started.  It was actually a very good match because a large percentage of IONA's customers run mission critical applications - applications that typically not only have high volume of use, but which are composites of many different platforms and technologies... as most real world applications are.  In many cases, these are applications where revenue depends on the success of the business transactions.  This type of real-world application is something Actional excels at, so the partnership is a great fit and we've generated a significant amount of momentum together.

Of course what made the partnership really work, though, is that both of our teams worked really well together.  The team at IONA have been great to work with all along.  So, let me wish them all a warm welcome. I look forward to working with the IONA team when the acquisition is approved.  'Nuf said!

Implementing SOA during a Recession

By now we all know that service-oriented architecture (SOA) isn’t something you buy out of the box. It’s also a long term initiative that involves planning, implementation and fine-tuning. But when budgets are tight, many companies may slow down or even stop their SOA initiatives. Listen to the podcast, Implementing SOA during a Recession, presented by Dan Foody, VP of Actional products at Progress Software. During this podcast Dan talks about the importance of maintaining your initiatives during an economic downturn by looking for the quick wins and hits that will allow you continually enhance your SOA infrastructure. He talks about business-critical issues such as security and compliance, and SOA governance that can’t be ignored even during a recession.

Subscribe to our iTunes channel for more SOA Infrastructure podcasts from Progress Software.

CIO as an Architect of the Business

John Soat wrote a summary of one of the panel discussions at the MIT Sloan CIO Symposium.  One of the interesting questions raised was whether the CIO should move from being a architect of the company's technology to being an architect of its business processes.  The response was "I would not trust the CIO to re-architect the organization."  The highlight is mine.

If you have a CIO that is both an expert in running the business and an expert in the technology, I can't agree with this comment.  But the reality is that any CIO that has this level of skill is probably already on the CEO track.  Why stay in an "IT role" when you could be running a business (especially since there's almost a stigma against CIOs being viewed as business leaders - the CIO's glass ceiling)?  So, this is a very rare breed.

Assuming you actually want your business processes automated (a problem that spans IT and business), the question for me, then, is who would you trust?

You can't trust the CIO because they are not an expert in the business - they won't understand the subtle trade-offs in the ways the business processes need to work.  You can't trust the business leader because they are not an expert in IT - they won't understand the subtle trade-offs in the way the IT applications and SOA infrastructure need to work.

So who do you trust?  You trust the person that listens instead of talks.  The person that takes input instead of dictating.  The person that recognizes they don't have all the answers, and is comfortable with that.  The person who has built a team, top to bottom, that works this way as well.

A lot of the better CIOs exhibit many of these qualities - unfortunately the one thing they almost invariably lack is these same qualities embedded, top to bottom, in the DNA of their IT organization.  Until CIOs can address this, I have to agree with the panel: you can't trust the CIO.

On the other hand, can you trust the business owner instead?

WOA is SOA - Take 2

Wow, Ron Schmelzer reacted, how should I put it, somewhat unkindly to my last post.  Yes Ron, I did read the whole article (and Nick Gall's too!)

I guess my point was probably too subtle.  I wasn't reacting to the fact (yes fact) that WOA is a "more concrete" form of SOA (see I did read your article Ron :-).  I'm in total agreement with that.  My point was, from a naming perspective, that's irrelevant.

Let me use an analogy.  Let's say we discovered intelligent carbon-based life on mars.  Would it be better to talk about it as "intelligent life" or as "a more specific type of carbon-based organic"?  Both are absolutely true, but which one matters more?  Obviously "intelligent" is the most important attribute.  The fact that it's carbon based is interesting but, in comparison, is not nearly as important (except to a few dedicated biologists who could care less about the practical consequences of it being intelligent).

For me, the same holds true for WOA.  What's the most important attribute, that it's "web-oriented" or that it's a "more concrete form of SOA"?  The most important attribute is that it's web-oriented.  That's where the power of it comes from.  The fact that it's a more concrete form of SOA is interesting, but not nearly as important.

That's why I think we should refer to it as WOA, not "web-oriented SOA."  We need to put the emPHASis on the right syllABle... Unless you'd prefer that I call you "carbon-based organic Ron" from now on. :-)

WOA is SOA?

Ron Schmelzer of ZapThink recently wrote an article on WOA (web oriented architecture), saying that we should really think of WOA as a subset of SOA, and that maybe it should be called "Web oriented SOA".

In theory, I tend to agree with Ron.  But we all know that, in theory, there is no difference between theory and practice, but in practice they are different.

Ron is basically saying "because WOA is a specialization of SOA, it should be considered part of SOA".  That is, in theory the more general concept is more important than the specialization.

Here's my problem though: In practice, when dealing with many parties, architecture simplifications or specializations are always more powerful and useful than generalizations.

Don't believe me?  The reason this is true is the network effect. The simpler the requirements to become part of a network, the easier it is to grow the network, the more valuable the network becomes, and the more parties that want to join it.

In contrast, generalizations work against the network effect - the more variations that are allowed, the more variations will occur, the harder it becomes to get value from the network, and the more slowly an overall network will grow (if it grows at all).

The power of WOA comes directly from its web-oriented simplicity, it's minimalism - not from the fact that  it happens to follow a service oriented paradigm.  Let's put the emphasis where it belongs: practice is more important than theory if you're trying to get results.

SOA Ironic

If you were listening to the webinar I gave a few days ago, you probably remember the phone system going haywire with feedback (don't worry, if you listen to the recording we're posting, this will have been edited out).

One of the topics I was talking about right at the time the feedback occurred was the "deer in the headlights" problem that IT faces when a business user tells them something is wrong with the applications since they often have no idea that a) that there is a problem, b) where the problem is, and c) what the problem is.  If you remember my theme, it was that good visibility is the solution.

What you didn't get to hear was that, after the webinar was finished, we talked to the vendor that ran the webinar about the feedback problem.  What was their answer? I quote, "We don't know where the problem is."  They, of course, promised us they would have their quality assurance specialist look into it.  And their answer was... "We're not sure what happened."

SOA What? Well, the moral of the story is that business service failures don't only occur with SOA infrastructure, but the same approach can be used to address them with SOA and without: visibility.

Live Webinar with Dan Foody

The Risks, Strategies and Payoff: SOA Runtime Governance
Tuesday, April 22, 2008 at 11:00 am EST
Register to participate >

Cut through all of the confusion around SOA governance to learn the ins-and-outs of the two key aspects of SOA runtime governance: visibility and policy enforcement. Hear about the challenges and strategies for addressing them. By learning how SOA runtime governance can directly impact your business, you will have the key information needed to build a business case to justify the investment in SOA solutions.

Join Dan Foody tomorrow, April 22nd, for a live webinar entitled, The Risks, Strategies and Payoff: SOA Runtime Governance.  Listen to what Dan believes are SOA best practices for company's looking to gain complete visibility into all the services that their applications consume. He'll also talk about how system and business professionals can easily monitor and ensure quality of service (QoS) that their business-critical applications - and customers - demand.

Other upcoming SOA infrastructure webinars you may be interested in, include:

• 1+1=3: Marrying Best-of-Breed Tools for Business Process Governance
  Tuesday, April 29, 2008 at 11:00 am EST
  Presented by: Marc Smith, Lombardi Software, Director of Product Marketing and David Bressler, Actional Product Evangelist, Progress Software
• Visibility and Tracking for Critical Applications – Optimize Application Performance and Increase Revenue
  Tuesday, May 6, 2008 at 11:00 am EST
  Presented by: Julianna Cammarano, Sr. Product Marketing Manager, Progress Software

A Problem Common to SaaS and SOA

I read a blog posting from the other day by a frustrated software as a service (SaaS) vendor venting that:

• They need rapid release cycles... but,
• They can't find all issues in their pre-production environments

For example, the author said "There are multiple points of failure and I can’t seem to think of how we could avoid these types of issues going forward."

Welcome to the world of running a service.  Whether the service is SaaS, a service you provide inside your organization (e.g. a SOA service), or a traditional business service (e.g. like those provided by Reuters, Bloomberg, and Thomson), the issues are the same: You can never anticipate all the ways your consumers will use the services in advance.  This means the risk of consumers experiencing problems is much higher than in other situations.

It also means you can't operate in a "let's get it right, ship it, and go on to the next project" traditional development-shop mentality.  You need to think about solving the problem in a fundamentally different way.

The first step to addressing this problem is being able to insulate consumers from one another (even when they use the same instances of your service).  The most important thing is to ensure that the actions of one consumer can't cause problems for another consumer.  For example, if a consumer goes into an endless loop making requests of your service, you need to ensure this doesn't impact your other consumers (for example, using infrastructure to implement request throttling controls).  Basically you're controlling the ripple effect.

Once you've figured out how to insulate consumers from each other, you can now think about "onboarding" each consumer as a separate project (even though you are in production your consumers aren't).  What assistance do you need to provide to them so they can successfully complete the project of going live with your infrastructure?  If, for example, they use REST or SOAP services you provide, how do you help them to ensure that they are using them correctly and appropriately?

Thirdly (and really this is just an optimization of the past point): How do you empower your consumers to do this for you (so you don't have to do all the work of making them successful).  You'll never be able to fully avoid the responsibility of "owning" consumer satisfaction (and fixing it if it's bad) - but you can put measures in-place so that your consumers can be more successful with less effort.  This can be as simple as providing much better documentation, samples, and tutorials - all the way to providing an interactive environment for the consumer to explore how to best use your service.

SOA on ACID

Tony Baer reported a recent IBM event where Steve Mills talked about how the next stage for SOA was to add ACID reliability and fault tolerance to it.  Am I the only one that doesn't understand this?

If IBM really means ACID (as in formal database-type transactions), then maybe they need a bit of re-education.  It's commonly agreed that ACID is not appropriate for a loosely coupled environment.  Instead, guaranteed messaging and compensation are the appropriate means.

But, if Steve Mills didn't mean ACID - and just means "reliable", then I'm also confused because there are a lot of organizations doing this today.  We have dozens of customers running high volume mission critical applications - where revenue depends on the reliability of their transactions... and they are doing this on SOA.

Now, I suspect that IBM isn't really planning on trying to run every application inside DB2 to get ACID properties (I suspect only Larry Ellison would propose a vision like that).

So, maybe, and this is just a thought, maybe what they mean is that when you stitch together 60+ CDs worth of different products and call it a "unified solution" (after all, they all have the same brand, so it must be a unified solution), it might not be the most robust approach.  If this is the case, I applaud IBM's desire to reduce the complexity of their product lines... but let's not talk about this as a deficiency of SOA infrastructure.

A power shift in organizations

I read an interesting article in the Wall Street Journal today.  It was on a topic totally unrelated to SOA, but carried an important message hidden within it:  Power is shifting to the users and away from IT departments.

The article talked about IT organizations trying to impose governance restrictions on their end-users, and that this may be a losing battle (at least in the case of keeping the iPhone out of IT).  Why is this a losing battle? In the end, productivity of end users is viewed as a very high priority to organizations.  They typically view IT's governance role as one of "first do no harm" (to productivity that is).  IT needs to make sure technologies used by employees are safe, secure, risk-managed, etc. - but it's rarely within IT's mandate to say "you can't use it" (regardless of what many IT organizations might wish), except in heavily regulated industries.

This battle is repeating itself in the SOA world as well: Excel is the worlds most common consumer of web services.  SalesForce.com and Google are among the most commonly leveraged web services in organizations.  As IT organizations try and "lock down" their SOA with governance rules that prohibit ad-hoc use of Excel or prohibit the use of external services "because they don't meet the corporate policies" a backlash will happen.  It's not if, but when.  And, in the end, the users will win, and IT will lose.

What I learned about SOA from my Macbook Air

I figured that since David Bressler did a gratuitous blog posting about his Macbook Air that I needed to do the same about mine.

Now, I'm not a Mac fanboy.  This is the first Mac I've owned and I got it mostly for its size and weight since I travel a lot.  To be honest, it's probably less reliable than my old XP laptop.  The machine itself locks up or crashes about the same amount, and some of the apps (like MacMail) clearly have some annoying bugs in them compared to what I ran on Windows (Outlook).

But, the one thing it beats my old box on is responsiveness.  I used to be able to go out for coffee while XP was coming out of sleep or hibernation.  But, with the Mac, it takes me longer to type in my password than it takes for the machine to wake from sleep.  Starting applications follows the same pattern.  Slow on XP versus lightening fast on the Mac.

What does this have to do with SOA?  Very little (that's why I called it a gratuitous blog entry after all).  No, in reality, it teaches some important lessons: In you can focus on excelling in quality of service, it's probably more important than having the most bug-free services and applications.

Users will reward you for recognizing that their time is important.  They can usually figure out how to work around bugs.  And, after a while, the workaround becomes second nature to them.  But, when something is slow across the board, there are no workarounds:  Given the opportunity, users will switch or just not use the system altogether.

So, the next time you're building a service, think carefully about where you put your focus and resources.  You have the opportunity to make your service consumers a lot happier with you.

Is the "A" in SOA the Problem?

My discussion with Tony Baer at onStrategies the other day, along with Joe McKendrick's discussion with Ron Schmelzer, where Ron said (to paraphrase) "It's always a great time to invest in architecture," got me thinking; and that's always a dangerous thing.

There are a lot of people preaching the line "SOA is an architecture (not a solution or end state)."  And, yes, I agree SOA is not a solution or an end-state.

But, I've got to wonder whether the "SOA is an architecture" mantra is just a cop out?  Is it just a simple way to say "expect no measurable return in any time frame you'll care about so stop bugging me about it"?  The benefits of architecture are fundamentally unmeasurable unless you can do a direct A-B comparison... which you can't.  And, one thing I've learned is that if you can't figure out how to objectively measure something, you probably shouldn't be investing a lot in it.

The problem with focusing on "SOA is an architecture" is that architecture is perceived as needing to be complete before you begin building.  First, getting an architecture "right" takes a lot of time (probably time that an organization doesn't really have).  Second, things change - so what was "right" this year is probably not "right" then the next year.

There's clearly also a fallacy in saying "once we get the right architecture locked in, we'll be agile."  Why is this wrong?  Because if your architecture itself isn't agile (i.e. quickly changeable and adaptable to unforeseen circumstances), then fundamentally nothing you build on it can be agile in the long run either.  If this doesn't make sense, I've blogged on a closely related topic in the past that might shed some light on the subject.

If enterprise architecture is your thing, that's good.  I like architecture.  Some of my best friends are enterprise architects. But, instead of focusing on SOA infrastructure as key to your architectural success, maybe you should be thinking "How do I build an architecture that itself is agile?"  Maybe you should be thinking, "How do I build an architecture that will make it easy for me to adapt the next wave of architecture (whether it's EDA, RESTy, or unknown)?"

Now, on the other hand, if you're building applications to solve problems for your business, you should be thinking about how you harness the tools and techniques that are available to you to get your job done faster and better for the business stakeholders - regardless of what worked best on the last project.  "Horses for courses" as my Irish friends would say.

These two recommendations (one to enterprise architects and one to the project) may sound disjointed, but they come together: An architecture that itself is agile can easily fit with projects based on different architectural foundations.  Agility at both the project and enterprise level.  Nothing better!

Protecting SOA During the Recession

As of today, the general consensus of economists is that we're now in a recession.  So, it's probably a good time to reopen the discussion on ensuring that your SOA initiative survives the recession.

While we all know that SOA is about agility, it's probably a bit strange that I'm now telling you to never utter those words again inside your company - at least until we're out of the recession.  During a recession executives aren't thinking about agility, they are thinking about cost.

Now is the time to be positioning SOA infrastructure as instrumental to cost containment for the IT organization.  Consistency, avoiding duplication (notice I didn't say "reuse" even if that's what I was referring to), and consolidation are all instrumental to managing costs.  And SOA gives you these.

If you can, you should also position server and application consolidation initiatives, as well as storage and server virtualization as part of the umbrella SOA initiative, which you can talk about as focused on reducing waste at every layer of the infrastructure -- from storage, through servers, through applications, to business processes and services.  Elimination of duplication and sharing of common resources at every layer of the infrastructure is a message that is tailor made for a recession.

The parallels between concepts like application consolidation, server virtualization, storage virtualization and SOA are clear - in many ways they are the same thing, just at a different layers of IT.  Embrace this big picture as "SOA" and you can create a much more compelling picture of how SOA benefits the entire enterprise.

Is this a bastardization of what "SOA" means?  Perhaps, but the reality is that by aligning with these other initiatives, you can provide a more holistic picture to the CFO about the overall IT benefits that "SOA" is driving - ensuring that you protect your SOA initiative from getting put on the butcher block.

Of course, so far I've only talked about positioning SOA.  Next we have to talk about how the recession impacts your execution tactics.

Agile governance?

Lars Hansen asked a question about my last post, Are agility and chaos two sides of the same coin? He asked whether I really meant "chaos", or perhaps I meant "complexity".

I definitely meant chaos.  Complexity and agility are most definitely opposites.  However, the thesis of my last post was that perhaps you need chaos to get agility - that chaos and agility are closely related (and in fact inseparable).

To be a word-weenie for a moment, here's specifically the definition of chaos I used:

"The behavior of systems that follow deterministic laws but appear random and unpredictable. Chaotic systems are very sensitive to initial conditions; small changes in those conditions can lead to quite different outcomes. One example of chaotic behavior is the flow of air in conditions of turbulence."

So, what makes the chaotic nature of dynamically unstable aircraft desirable?  When the fly-by-wire system detects a random movement that's in the right direction, the plane takes advantage of it (and even lets it amplify itself).  But, when the system detects a random movement that's undesirable it quickly compensates.

So, the key factors that make dynamically unstable aircraft possible are:

• Complete visibility into what's happening at any moment;
• Knowledge of the desired outcome (what the pilot wants the plane to do);
• Extremely fast control that ties the visibility to achieving the desired outcome.

Notice that I didn't say "Control of all the possible outcomes."  The plane can't do this (because it's inherently unstable).  The control systems of dynamically unstable planes are "going with the flow" or performing "dive and catch" response to what they observe... all many thousands of times per second.  These planes are only possible because they can sense and respond extremely quickly, not because they can control what happens next.

SOA What?

Are agility and chaos two sides of the same coin?

It's conventional wisdom that business agility results from providing a significant amount of control and direction around a SOA initiative to ensure predictable results.  To achieve this, strong enterprise architecture and SOA governance are assumed to play a key role.

Of course, it's always important to ask whether conventional wisdom is really true...

It was long believed that, in aircraft design, the most agile aircraft was the most stable and predictable.  This conventional wisdom existed because engineers believed that certain things just weren't possible. For example, they believed a dynamically unstable aircraft couldn't fly.

With the advent of fly-by-wire though, the conventional wisdom was turned on its head.  Aeronautic engineers now know beyond a shadow of a doubt that the most agile aircraft are also the most unstable.  In aircraft, chaos and unpredictability are key to achieving the best agility.

So, if the most chaotic "architecture" for aircraft provides the most agility, could the most chaotic "architecture" for SOA provide the most business agility? And, if the answer is "yes", what's the SOA infrastructure equivalent to fly-by-wire that makes it possible?

The Ungoverned Service Consumer

In a previous post, I promised I'd explain how to avoid governing service consumers.  Now, let me say that not governing consumers is of critical importance because consumers just won't use a service that is complex to get started with (registration processes, design reviews on load requirements, etc.).  They will find all sorts of novel explanations for why they can't reuse the service (ones you won't be able to refute easily)... even if reusing it would be the right thing to do.

So, I'm saying you need to avoid governing consumers... without losing control.  Impossible you say?  Impossible until you take advantage of the fact that perception is reality.

From the perspective of the service provider, you obviously want consumption of your service to be well governed.  The key, however, is that you want the perception of the consumer to be that you're not governed... until they want to be.

Here's a great real-world example:  There are a number of sites where you can get free storage on the web.  It's quick, easy, and with many it's anonymous.  But with all of them, if you go through the formal sign-up process you can then get a higher quality of service (QoS) - whether it's more bandwidth or better uptime.  So, if it worked the exact same way for service consumers, you would want to allow easy "ungoverned" use and support a higher quality of service if they submit to being governed.

The key with the "ungoverned" consumers is that you never want them to impact the customers that have signed up for the higher quality of service.  Some examples of what you want to do without the consumers realizing it:

• Throttle their usage so they can't swamp the system.  Amazon S3 didn't do this, and guess what happened?
• Validate their messages so they can't cause the service to fail by passing in bad data.

Notice that both of these can be done without consumers ever knowing about it - a very good thing.  When the consumer decides they want to sign up for the "higher quality of service," what you are actually doing is removing some of these checks-and-balances (once you're satisfied they are already compliant).

Interestingly, signing up for "better governance", in reality, results in removing governance checks!  Consumers will love that you offer both "ungoverned" and "governed" use models.  But it's probably not important for them to realize that "ungoverned" is more heavily governed than "governed".

Perception is reality.

What does the weather have to do with SOA?

Have you ever noticed that weather reports used to say things like "20% chance of rain", but now almost all just say "chance of rain" or even "rain likely"?  (Where I am, today its "snow likely.") It can't be that weather forecasting has gotten less reliable can it?

Sure, you could explain it as part of the "dumbing dumming down of America" (or whatever country you live in).  But there's actually a different reason behind this... setting expectations properly.

No one minds a pleasant surprise and everyone hates a nasty one.  If you're going to surprise someone, you better make it a pleasant surprise.  The terrible snow storm you expected didn't happen?  "Boy, that was lucky."  Stuck in a snow storm you didn't expect?  "All those useless weather forecasters should be fired!"

That's what weather forecasters do now - they give a near-worst-case forecast so that any deviation is almost certainly a pleasant surprise... the weather was better than expected... and their consumers are happier with the job they did.

SOA What? In SOA, service consumers have expectations of the services they use, just like you have expectations of the weather service you use.  Should you ignore the expectations of your consumers?  No.

Managing the expectations of consumers is critical to the success of a service... which is why I'm amazed that all too often service providers set the expectations high and then under-perform (a nasty surprise for the consumer).

So, if you're a service provider, repeat after me, "under promise and over deliver."  If you follow this advice, you'll have loyal and happy service consumers... consumers love the occasional pleasant surprise.

Spring Cleaning for SOA Governance

Jon Williams, blogging over at InfoWorld in his New York CTO blog, posted an entry on worthless process.  In it he talked about, for every process/project you need to think "will this make a difference to the business."

I have to say, I wholeheartedly agree with what he's saying - especially when applied to SOA governance policies.  For every policy you add, you should be able to clearly articulate it's value and impact on the business.  If you can't, then maybe you should reconsider adding it.

He goes on to say you also need to look at existing policies/processes/projects and do a spring cleaning of the ones that are no longer relevant.  How often do people assume that policies and processes that are already in-place are still relevant today?  It's been my experience that policies and processes never die.  "policy cruft" is certainly one of the things that turns good governance into a barrier to agility.

Remember, less is the new more.

A Makeover for SOA Governance

Looks like my previous posting on this topic was pretty timely.  Joe McKendrick (over at ZDNet) just today talked about SOA governance stifling agility.

In Joe's article, he asked the question of whether or not we need a new name for SOA Governance (and that he's always thought that SOA Management fit the bill). If we wanted to give "SOA Governance" a marketing makeover, here's a few slogans SOA Governance advocates might use:

• With SOA Governance, make the A in SOA mean Agility.
• A lack of Service Oriented Behavior got you down?  Try SOA Governance.
• Tired of project teams ignoring you?Implement SOA Governance to show them who's in charge.
• Policies? We don't need no stinking policies, we've got SOA Governance.
• Can't stomach that last wafer thin policy?  Better get a bucket of SOA Governance.
• You want the truth?  You can't handle the truth... unless you've got SOA Governance to pretty it up.

OK, so maybe it's hard to come up with catchy slogans for SOA Governance.  None of this solves the real problem with SOA Governance:  SOA governance is often implemented as too much stick and not enough carrot.  Too much of a barrier and not enough of an enabler.  Too many policies with too little measurable value.  Unless these issues are addressed, no amount of marketing, magic quadrants, or vendor testimonials are going to make SOA Governance successful in most organizations.

Governance and the Downfall of Business Agility

It's pretty well accepted that business agility is the holy grail of SOA. And most people agree that strong (SOA) governance is a key requirement for an effective SOA infrastructure. So, why am I here telling you that governance is a barrier to business agility?

Historically, strong controls have always been a barrier to agility... and the business finds ways around those barriers. Is the mainframe too tightly controlled by IT to get your job done? Why not move to client/server. Are the client/server deployments are too tightly controlled by IT? Why not move to building situational business apps in Excel. You can, of course, see the exact same pattern emerging between SOA and the move to Web 2.0.

Governance and other forms of IT control almost, by definition, stifle innovation. Why? Because innovation is about thinking outside the box, whereas governance is about defining the box that you're not allowed to think outside. Since innovation goes hand-in-hand with agility, if you stifle innovation, you limit agility.

Of course, I'm definitely not arguing that you follow the "wild wild west" deployment model in your IT organization. What I am advocating is that you carefully think about how and where you want to allow innovation (and thus optimize agility) and ensure that your governance is extremely flexible in these areas. Where you don't need innovation, you can put stricter SOA governance controls in-place.

Let's try and put this into practice. In many cases, it's safe to assume that innovations will come via the creative use of exiting services, not from creating services themselves. If you believe this is true for your organization, you should put more governance controls around service creation, but severely limit the controls that are put around service use. How you do this, without losing control altogether, I'll leave to a follow-up post.

When Downtime Doesn't Matter

SOA, by its very nature, results in many more moving parts in any IT system.  As a result, many organizations obsess over ensuring that SOA infrastructure doesn't impact their business' availability - and rightly so.

So, it's interesting to look at Apple's website (the Apple Store section in particular) and see their approach to downtime.  When they introduce new products, they take the entire site down - often for more than a few hours. In fact, they took their site down this morning to launch the new 16GB iPhone and 32GB iPod Touch.

What I first found so surprising is that Apple Store downtime has a cult following - people look for it and, in fact, jump for joy when the store goes down.  There's always a celebration among Apple users when the Apple Store goes down.

Sound a little different than your executives' and customers' response to downtime with your applications?  Why is it that Apple gets lauded for downtime but, in your organization, people's jobs are at risk?

The answer: It's all about setting expectations.

Apple users have a clear expectation of what happens when the apple store goes down.  Not only that, they see a clear benefit to it - there's something in it for them.  In their case, having to wait the 1.5 heart wrenching hours is hard, but it means there will be new products for them to buy when it's over.

So, as you roll out more and more SOA based applications, you need to carefully consider how you properly set and meet expectations - and how to make those expectations meaningful and valuable to your end users.

Maybe Reuse Isn't Always Such a Good Thing

I was flying from Montreal to Boston yesterday and we had a flight delay.  Not that unusual, but I did what I always do when that happens - I went to the FAA's website to see what was happening.

I was expecting to see the FAA say it was a weather or volume problem.  But, what I saw was "EQ:Automation" as the cause.  I dug in, and it turned out that this meant the radar systems had gone down (the applications behind it had crashed).

What's even more interesting is the same problem happened at JFK, Laguardia, and Philadelphia at the same time! Now, since nothing popped up in the news, I'm presuming it wasn't a coordinated terrorist attack on the FAA (if it was, the terrorists did a poor job since the systems were only down for about 90 minutes).

But, assuming it was a software glitch, this is a great example of why reuse isn't always a good thing.  Even though they would have had separate instances because they were all running the same software, it behaved the exact same way - and some common condition caused all of them to crash.  Had they each been running different radar control systems, at worst, one of these airports would have been shut instead of 4.

Of course, the problem gets even worse when you reuse the same instances of an application as you do in SOA.  A condition caused by only one of the consumers can bring down the service for all of them.

The advantage of not reusing services would be that you can never run into these situations.  Now, I'm not advocating avoiding reuse altogether - just the contrary - reuse is generally a good thing. But it's also often good to allow for a bit of duplication (maybe 2 to 3 different variations of the same service).  Healthy competition generally results in better service for everyone - and you end up with some protection from systemic failures of one common implementation.

If only the FAA had been thinking this way, maybe I and tens of thousands of other passengers wouldn't have been delayed yesterday.

SOA Strategy and Tactics

As I was reading Dave Linthicum's flip-flopping denial, I kept asking myself how someone could both "resist the temptation to redirect resources toward tactical business needs" while at the same time "focus on short term objectives that are directly related to the generation of revenue"?

Dave clearly believes these statements are consistent with one another. So, to understand Dave's point of view I realized I had to delve a little bit deeper into the interplay between strategy and tactics.

First, it's important to recognize that tactics are the set of actions taken, and short term objectives met, to achieve (longer term) objectives set by strategy.  That is, "tactical" actions unrelated to the strategy aren't tactical at all - they are just "doing stuff".  I'll assume, though, that Dave really means tactical (not "doing stuff") when he says "tactical".

Here's the tricky part: your organization will have both a business strategy and a SOA infrastructure strategy.

If your business strategy and SOA strategy are out of alignment, then a tactic to achieve one of the strategies may negatively impact the other strategy.  That is a tactic to achieve your SOA strategy may negatively impact your business strategy (and vice versa).  If your SOA strategy is out of alignment with your business strategy, then you better fix one of them... it's probably going to be your SOA strategy.

On the other hand, if the two strategies are in complete alignment with one another, then any tactic taken to achieve either one should further the other strategy or, at a minimum, be neutral to it.  That is, any tactic used to achieve your SOA strategy will never negatively impact the business strategy - provided the SOA strategy aligns with the business strategy.

This still leaves open the question of how to prioritize the short term objectives within a given strategy. More specifically, what measure do you use to prioritize the tactics used to achieve your SOA strategy?

So Dave, correct me if I'm wrong, here's how I should interpret your two statements:

With "resist the temptation to redirect resources toward tactical business needs," you were really saying to prioritize short term SOA objectives based on how much they benefit the SOA strategy.

But, when you later said "focus on short term objectives that are directly related to the generation of revenue," you were saying that, in bad times, you should redirect your resources to assist in executing short term business objectives, but prioritize which ones you address based on their net impact to the SOA strategy.

Did I get that right?

If so, while maybe we can agree you're not a flip-flopper, my perspective on prioritization is very different:

• If your SOA and business strategies aren't aligned, fix your SOA strategy, then...
• Prioritize your short term business objectives based on their net impact to the business strategy
• Prioritize your short term SOA objectives based on their net impact to the business strategy

The business impact should always drive priority.  Period.  To me, any justification as to why you should ever prioritize based on the impact to the SOA strategy is backwards thinking. Why? If the business strategy doesn't meet its objectives, the success of your SOA strategy is irrelevant... unless your goal is to become a talking head, like me and Dave :-).  Just make sure you know where your bread is buttered first.

The sky is falling (for SOA)

There's been a lot of discussion recently on how the upcoming recession will impact SOA. My rule of thumb is that in good times businesses care about agility while in bad times they care about efficiency.  So, if you want your SOA initiative to survive the recession, you better deliver (not talk about) efficiency and cost savings.

It's only two months ago that Dave Linthicum gave me a lot of flack when I criticized his statement to (and I quote) "resist the temptation to redirect resources toward tactical business needs."

However, recently Dave changed his tune when he said, "the best approach is to focus on short term objectives that are directly related to the generation of revenue."  Dave, it's good to see you're listening... even though it took a while :-)

While commenting more on Dave's flip-flopping would be rubbing salt in the wound, I think the advice I gave in the series of responses (1, 2, 3, 4) to Dave's original post are still appropriate as we enter the upcoming recession.

The Legacy of BEA

A lot has been said about this acquisition over the last few months, though I don't know why anyone doubted Oracle would acquire BEA. What Larry wants, Larry gets.

But, now that it's official, overnight Oracle has created a new product category: Legacy SOA Infrastructure.

Like it or not, it would be bad business for Oracle to have two app servers, two ESBs, two process engines, etc. So, in each category, Oracle are going to have to pick one to go forward.  Because of this, customers that made a different choice will end up with legacy SOA infrastructure products that they will have to deal with.

Of course, the $19.375 question (for each product category) is "which one wins?"

There's more to SOA than services

Late last week I had a last minute flight change and switched to JetBlue for my round-trip to Washington D.C.  By last minute, I mean last minute.  Waiting for a flight, on an airline that will remain United nameless, our plane had multiple mechanical failures to fix, including a fuel leak.  We would have departed after my meeting was over.

So, I called my travel agent and while I was walking between terminals, I had them book me on the only flight that would get me there in time -- on JetBlue.  I was still on the mobile phone with the travel agent when I arrived in front of the JetBlue check-in kiosks 30 minutes before the flight was due to depart.  I waited to get my confirmation code, and immediately punched it into the kiosk to get my boarding pass... and it worked.

I was impressed, because their central reservation system was sync'd with the airport systems in real-time.

Because I was doing a day trip, I could even check-in for my return flight immediately.  At the kiosk I also changed my seat on the return to get a better one - it still amazes me that airlines automatically assign you last-row-middle when there are lots of other seats open.

And, it's on my return where the fun began...

The beginning of the end for UDDI

Posted by Dan Foody

I recently had a discussion with one of the key industry analysts covering the SOA governance space.  He said that the need for UDDI is going away – to be clear, registries and repositories are very important, but the UDDI technical standard is no longer a key driver.

UDDI had promised to be the universal standard for communicating with registries, and there were no real competing standards.  So, what’s led UDDI to be on its deathbed? A few things, I think:

• Repositories have become much more important than registries and UDDI wasn’t really great for repositories.
• UDDI tried to be all things to all people which made it way too complex.  What, you say you don’t want visions of tModels dancing in your head?
• At the same time, the UDDI mechanisms for searching and change notification were very "primitive" – this meant that federation across repositories, and end user tools for using registries were all proprietary approaches.
• UDDI never got a critical mass of tools using it for access to registries – this was probably due to all the issues above.

This is a long time coming but welcome news for anyone that’s ever had the “pleasure” of using UDDI. On the other hand, if you’re a UDDI vendor, take a big breath and remember the 5 stages you’ll go through:  Denial, anger, bargaining, depression, and acceptance.

No doubt this does leave a gap – we have no widely adopted standard for integrating with or federating registries and (more importantly) repositories.  Hopefully all the registry/repository vendors will get together and fill the gap soon – preferably with something that’s simple and useful.  The status quo (with or without UDDI) where every SOA infrastructure product must custom-integrate with every registry/repository product just can’t be sustained.

Rest in peace UDDI.  Just do it quickly, please.