SOA saved by measurement?

Joe McKendrick over at eBizQ blogged about two reports that say SOA is as good as you measure it - in essence saying that the key to SOA success is measuring it.

I'm a big fan of measuring things.  If you can't measure the benefit of doing something, you need to think twice about whether you should be doing it at all.  But, these reports miss the mark in a number of ways.

The first is that measuring SOA doesn't matter if you can't get buy-in to do SOA.  You can build the best measurements in the world, but if your targets say that, after your large up-front investment, SOA will begin paying back in 1-2 years, you're probably not going to get the funding.  Time to value is a critical component that can't be underestimated (especially in this economy).  Risk / reward ratio is another critical component.  Small investments that yield big returns in short time frames are what you need to be looking at if you want to get SOA off the ground.

This brings us to the second issue:  Meaningful metrics.  If you're a service provider business, you can certainly measure things like revenue-per-service, "service vitality", and ROI for revenue bearing services.  But what the heck does this have to do with SOA?  People have been building and delivering  services that earn revenue long before SOA ever existed.  And, how do these metrics work for companies that don't sell their electronic services?  What's the directly measurable ROI for FedEx's free web service to do package tracking?

When you want to measure the ROI for SOA, you have to measure it versus alternative options.  Building a revenue bearing service with a SOA architecture doesn't mean that the entire benefit for the service is attributable to SOA infrastructure.  What you need to measure is the cost-to-build-with-SOA versus the cost-to-build-without-SOA (because, in either case, the revenue generated will likely be the same).  Whether FedEx uses SOA behind their package tracking service doesn't change the business impact of electronic package tracking.

This is where things get complicated: How do you separate the real benefit of SOA from alternative approaches?  This is where these reports further get it wrong.  Most organizations never do a complete accurate costing of how to build something in two ways (with SOA and without) - so you can't measure the SOA ROI for a service.  And, when people do generate these cost estimates, in reality they are often manipulated to ensure that the path they want is the one that looks good: so you can't trust these numbers.  Similarly, metrics such as service reuse rate (versus rate of developing new services) are easily unintentionally manipulated.  Build fewer services up front and you'll have better reuse rates (the metrics put pressure on not building services - exactly the opposite of what you want in a world where the culture is already to not build services).

There are certainly some metrics given in these reports which are meaningful.  For example "mean time to service development" and "mean time to service change".  Of course, these metrics aren't practical in the real world.  Every service has vastly different complexity, so to get a real mean, you need 100s of samples (100s of services to be built and changed) before you get any real numbers.  This will only happen years after you get started unfortunately.

So, all this talk about metrics is great, but don't be a measurbator.  Make sure you put on your realism hat:

• Don't focus on metrics that are "suspect" (e.g. ROI for SOA) - the business won't believe you.
• Don't focus on metrics which can't be measured until years later (e.g. mean-time-to...) - the business won't listen to you.
• The lower the costs, the less justification you'll need
• The lower the risk, the less justification you'll need
• The quicker you show perceived benefit, the less you'll need to worry about metrics

Metrics do have a place in SOA - to help refine it over the long run.  But, people won't believe the metrics until they see results, which is why you can't start out that way. The best way to get SOA infrastructure off the ground is the time-proven method: Get a bunch of smart do-ers (not talkers or thinkers) on an important project that, using SOA, you can deliver quickly - and use this to gain mind share of what SOA can do.

Comments

[Ryan Burke]

Dan,
I have recently begun to check out blogs (especially what is being said about SOA today), and I love your work and writing. Could you please send me an email so we could discuss a few issues for the future?

Thanks,
Ryan