Integrated Infrastructure Blog

The great cloud crash of 2009

I've been following the recent story of how Microsoft's Danger division and the T-Mobile Sidekick.  If you don't know much about Danger, it's basically a cloud service tied to a hardware device (the Sidekick).  All the data is stored in the cloud (email, contacts, etc.) and cached on the device so that when the device resets, it starts up empty and reloads its data from the cloud.  Unfortunately, many customers found out recently that when they reset their device, all their data was gone.  Poof.  No more contacts, no more calendar entries, no more emails.  It turned out they didn't have any real backups and their data redundancy was foiled by human error.  Whoops.  While they have recovered most of people's data now, there are some important lessons to be learned from it.

One of the articles I read on this was titled Don't Blame Cloud Computing for the T-Mobile Mess.  While the author's heart is in the right place (he really likes cloud computing), I can't agree: Cloud computing is absolutely to blame for this fiasco.

Let me explain by starting with an analogy.  Recently we had a little event that some people call the "great market crash of 2008".  The root cause of this is pretty well known now:  A lack of transparency into different financial instruments (e.g. CDOs) made it impossible to accurately assess risk.  And, in the absence of an accurate risk assessment, people assume things will be ok and focus on their short term gain.

It seems to me that we have the exact same situation with cloud computing.  There's essentially no transparency into any cloud provider's integration infrastructure, processes, or planning.   As a result no user can accurately assess the risk of using one cloud provider over another.  Do you think that if sidekick users had know "Danger doesn't do backups" they would have trusted the service with their data?  Of course not, most users assumed everything was OK.  They assume their cloud provider is doing the right thing.  It was only a matter of time before a crash would happen (and this won't be the last one).  In the immortal words of Otter from Animal House, "You f*cked up - you trusted us".

Cloud providers, unfortunately, think that it's not in their best interest to be transparent because, frankly, customers are conditioned to just assume everything is OK so why rock the boat.  When was the last time you walked into a grocery store to buy apples and said, "Can you cut this one open so I can see whether it's OK on the inside?"  No, you probably look at the shiny skin of the apple and assume everything is OK with the inside.

Before cloud computing can become mainstream, users of cloud services must have the ability to accurately assess risk for themselves.  In order to do this, cloud providers must provide transparency. if not users must demand it by speaking with their wallets.  Don't let the cool, shiny UI fool you into assuming everything is OK under the skin.

Business Transaction Management with SAP

If you follow Actional, you may have seen that we announced Actional 8.1 today.  The most interesting part of the release is our new support for SAP.  Yes, we already supported SAP NetWeaver (like most of the other people in our space) so unless you're an SAP aficionado, you probably won't recognize the importance of natively supporting SAP ABAP - which is what we've announced.

SAP supports two main application server environments: one based on Java and one based on ABAP, SAP's own programming language (you might also hear the term "basis" which is another name for the ABAP application server stack).  OK, with me so far?  While SAP support both, almost all of the SAP packaged applications are written using the ABAP stack - Java is primarily used for infrastructure services (things like their portal).

With this new version, we've added the ability to trace business transactions into and through ABAP - so we can detect problems even within the ABAP portion of a business transaction.  This is critical for many SAP customers because - without the ability to do this - SAP packaged application logic is seen as a black box silo.  And, with 100's of millions of lines of code in the SAP packaged applications, that a pretty big area to have a blind spot.

While there are a lot of business transaction management vendors out there that support SAP, they are usually referring to the Java side of SAP. As a result, once a transaction hits ABAP (and almost all of them do) it enters the black box and can't be seen again until it leaves ABAP.

As you might have guessed, we're really excited to extend Actional's patented transaction tracing to SAP's core platform.  If you're an SAP user, and need to ensure the success of business transactions that span SAP and other applications, platforms, and middleware then hopefully you'll get a chance to see whether this unique Actional capability can help you.

Is the Travel Industry Out of Touch with Our Changing Business Climate?

Did you knw that the travel industry loses $11.5 million per year through failed transactions?

Last week Progress Software released the results of a survey commissioned with an independent specialist technology market research company, Vanson Bourne Research. The researchers surveyed 149 global travel businesses and found that 67% of respondents have noticed their transaction failures soar by almost a third – even though their transactions have only increased by 12%.

Dan Foody, VP of Products at Progress Software, sites: “This research clearly highlights the far-reaching effects of transaction failures within the travel industry. The trend of increasing IT complexity is further compounding revenue loss through increased lost transactions, customer churn and inefficient use of IT resources with 97% of respondents concluding that transaction failures were increasing operational costs."

 “Companies should consider introducing a more streamlined approach to monitoring transaction flows across their IT environments, delivering the ability to respond to changing conditions and customer interactions as they occur. This will enable business leaders to capitalize on opportunities, drive efficiencies and reduce the risk of impacting customer experience. The more responsive approach will provide visibility and increase understanding of the impact IT failures have on their own IT services and their customers,”

Get your copy of the report! The report IT Impact on Travel & Leisure Industry Reservation Management, examines the causes and consequences of this situation. Read this report and learn about some of the underlying causes for the increase in transaction failures, what the consequences are, and the limitations of most traditional monitoring and management systems.

Share your experiences and thoughts by posting a Comment to this entry.

The power of proactivity

Earlier this week (last Friday evening to be exact), I took off on a business trip to Australia.  I was scheduled to go from Boston to San Francisco, and from there to Sydney.  When I arrived at the airport, I checked in, cleared security, and meandered towards my gate.

When I got to the gate, the gate agent was calling out my name.  United's not my regular airline so I didn't have any special status that would get me upgraded automatically or anything like that.  So, when the gate agent calls your name in this situation, you usually think "uh oh". I went up to the gate to find out what was up.  Here's what they told me:  They said that the flight to San Francisco might leave late due to delays in San Francisco so my connection would be tight.  So, just to make sure, on the spot they re-booked me through Los Angeles on flights that left and arrived at around the same time as my original itinerary.

The last 15 years I've been Platinum or higher on multiple airlines - but this was the first time an airline had ever been this proactive.  As soon as I was re-booked, I checked and there was no delay listed for the San Francisco flight yet nor did the FAA show any general delays for San Francisco.  I was pretty impressed and am a very happy customer. I will also try to fly with them more in future (now if they'd only put electrical outlets in coach - but that's another story).

How does this translate to the IT world?  If you can anticipate or detect problems before your business users are even aware of them you, will become a hero.  Some people want to hide issues from their users, but don't be afraid to let your business users know there's an issue. If you are taking actions to address the issue, and the users see that, they will gain trust in you.

A question that often runs through people's minds when they think of this is, will your users think you're not on the ball if this happens too often (is it better to only react to the really bad issues proactively)? Would I have gotten mad at United if the original flight wasn't actually delayed?  Not at all.  The fact that they were thinking ahead was what mattered to me. 

Users don't expect perfection - they expect (and respect) honesty, empathy, responsiveness.  Give them that and they will be with you for the long haul.

How to Be Agile Without Falling on Your Face

Eric Knorr of InfoWorld recently wrote an article about how testing was the key to how to be agile without falling on your face.

While I agree that IT organizations are squeezed and that quality is often given short end of the stick, I don't think Eric's solution of "optimize the efficiency of your testing tools, environments, and methodologies" is necessarily the right way to think about the problem.

Most organizations think of quality as a role (the role of the tester) - and as a result most testing tools are just that: specialized tools designed with the needs of the professional tester in mind.  The problem is, with modern applications, quality doesn't start and the tester and doesn't end at the tester.  Quality is a responsibility not a role - a responsibility of everyone involved from development through production.    This is one of the reasons that developers really like our free Actional Diagnostics tool (used to be called Mindreef SOAPscope) - it's designed from the perspective of the needs of developers, but it focuses on improving quality. For example, we found in formal ROI analysis that deploying our products in the hands of developers reduced the number of production issues by 25%.

So, rather than focusing on how to make testers more efficient (which will only get you so far), it's better to focus on tools and SOA infrastructure to help developers and architects avoid defects before they are created.  But these aren't testing tools because developers don't use traditional testing tools.  If you have 10 defects, avoiding just one of these development (something which is eminently doable) is as beneficial as improving testing productivity by 10% (something which is probably a lot harder).

On the other end of the spectrum, with today's interconnected applications the number of defects which will show up in production is significantly higher than traditional applications, and you'd likely have to increase your testing by a factor of 10x to get to the same production-defect-rate as a traditional silo'd application.  Alternately, a 10% reduction in the time-and-cost of resolving issues in production easily pays off as much as a 10% improvement in testing efficiency.  And, achieving a 10% reduction in this is definitely achievable... Now imagine you could reduce the time-and-cost of resolving issues in product by 85%...

Embedded Not-so-lightweight ESBs

David Linthicum touched upon a very prevalent problem of technology not living up to its expectations set during the hype curve! He says, "With cloud computing, SOA, or whatever we come up with next, we still need to figure out core issues within our enterprises that no set of technologies or emerging trends can solve." In the service-oriented architecture (SOA) case, we all know that the widely professed use cases of systems "automatically" looking up service registries on the web/intranet, locating the required services, getting the interfaces, picking the right one, and actually making a service request is just a pipe dream! The reality is nowhere close! It is difficult to even find enterprise wide service buses, much less the dynamic on-the-fly service-look up-and-use!

I recently came across a scenario where even a large international bank has SOA infrastructure solving regional problems - and not necessarily as a corporate backbone! I was at this tech discussion with the regional technology head of a large bank, exploring how Apama (Progress' Complex Event Processing (CEP) engine) fits into a biz requirement they have. Specifically, he wanted to understand how Apama works and how it is a good fit for their biz problem. They were trying to build a business performance tracking solution (a la GRC) that involves tracking key biz activities, ensure due process is followed, and then proactively detect potential service-level agreement (SLA) failures so they can be averted. A classic use case for Apama!

Now, when we started to see where the business events information came from and how they could be "wired" to emit the required events into Apama, we suggested that they could just tie into their enterprise service bus (ESB)—we knew they had standardized on one. As we dwell further, it turns out they are using an ESB to solve departmental integration requirements, and there are multiple instances and no central ESB that can help integrate these "departmental integrations".

Now that was a revelation. (I am not complaining because suddenly our opportunity now included Sonic ESB, along with Apama). While we have been reading (and I myself have been blogging and talking) about how SOA failed to live up to its expectations, I still expected at least a large number of banks to have standardized backbones. But not to be! I guess, like in many large enterprises, even here SOA adoption has started purely to solve departmental integration problems—more as localized islands of "integration". And its nothing like the enterprise-wide integration platforms or service bus that we have so much noise about since 2004. This reality check at this bank was a clear validation of this failure! It seemed clear that the technology promise was something, and the actual on the ground realization was something totally different!

Come to think of it, it does seem like a good model for use cases such as BAM or GRC - to have a complete platform that in addition to the CEP engine and dashboard, it also includes a "local/embedded" ESB. Now, embedded is not about being lightweight or smallas it is about being available out of the box when you get the platform. In this case, the ESB is available along with CEP, well integrated - so no surprises as you start using it. And the purpose is to "wire" exchanges from existing ESBs or applications in the enterprise to the CEP environment.

Not a bad idea at all! Embedded ESBs!

Chapter 4: The Data Integration Dialogues (Part 1) is Now Available

Chapter 4 of The Semantic Dialogues, The Data Integration Dialogues (Part I), is available now. Here's an excerpt: "There's a new kid on the block, and Patrice is shaking things up. She wants to write off National Networks' investment in an enterprise data model she hasn't even seen, and replace it with an industry standard that most of the team is unfamiliar with. Cliff is openly skeptical about the upside to standardization. Naturally he's attached to his previous work, but he's also worried about adopting a complex model that looks a lot different from his. Will Jerry's vote for a do-over prevail? "

Click here to read Chapter 4, The Data Integration Dialogues (Part I), today. And if you’re enjoying The Semantic Dialogues, be sure to share it with a colleague.

The Semantic Dialogues tells the story of how National Networks, a fictional telecommunications service provider, pursued and achieved data interoperability within their SOA infrastructure. The story begins with an urgent request from Operations for a fix to a problem that's keeping them from billing for new services. Assured that extensions can be made to the common data model (the SID) for the COTS billing system that's causing the headache, the problem is resolved in short order. The story then goes back a year to a time when the lack of a data interoperability layer made similar problems seem insurmountable and a tiger team is formed to put in place an OSS/BSS architecture that will change the way the CSP does business.

Progress Software Introduces Actional Diagnostics

Presented by Dan Foody, VP of Products for Actional at Progress Software

With Actional Diagnostics, you can accelerate Web service development, testing and validation with tools to improve service quality, performance, and compliance, and simplify time consuming XML-oriented tasks. Listen to this podcast and hear Dan talk talk about the features and benefits of introducing a service quality and validation tool into your SOA infrastructure initiative. Register to be alerted when the FREE download is available.

For more SOA podcasts, subscribe to our iTunes channel.