Integrated Infrastructure Blog

Now That's a Real Forklift Upgrade

I have to admit... I don’t really know how our customers use OpenEdge. I do know there are a ton of customers - over 65,000. And, if that weren’t enough, there are over 1,500 partners too. What's more, many of them are in-production with SaaS offerings.

Damn, that’s a lot.

(If any analysts are reading... just think about the opportunity of selling Actional into that installed base even if we never got another “new logo” sale.)

This week’s press release follows on from several months of a beta period where about 20 or 30 OpenEdge customers tested the newly released Actional integration.

As TVH Forklift Parts realized, knowing what’s happening in their integrated infrastructure, and being able to assure a consistent level of service has tremendous value to a distributed and shared infrastructure.

Why is this important?

It’s about the business context. Without that context, solutions are just technology (we have good technology too… but that’s not enough).

That’s the difference between assurance and management. Assurance implies business-technology coordination to achieve a business result. Management implies your technical components are up and running. Big whoop. Just today I spent 2 hours on the phone with T-Mobile. All the technical components were up, but it still wasn’t working. I know you can relate.

Colleen points out that our partners are being viewed more and more as business partners, not just technology providers. Simply put, our partners need technology to understand the business impact of “events” within their infrastructure.

Understanding the business impact means that we (technology infrastructure providers) need to provide an awareness of the business context when problems occur. The only way to do that is to track business context all the time.

I’ve heard a few times recently of prospects who have a “competitive” solution in place to track business assurance… but when I probe, it seems they don’t run it all the time because (pick one):

1. It impacts performance of my applications. (it doesn’t scale)
2. It collects too much information. (it doesn’t scale)
3. It requires too much CPU on my app servers. (it doesn’t scale)

I don’t understand how people think a solution that doesn’t run all the time can do the job.

Let me rephrase.

If it’s not running all the time and collecting context of your business, how are you using the context of the business to make better run-time decisions?

Simply put, you’re not.

I’m glad to welcome TVH Forklift Parts to the Actional family. And, if you’re reading, thanks for sharing your story.

Is the Travel Industry Out of Touch with Our Changing Business Climate?

Did you knw that the travel industry loses $11.5 million per year through failed transactions?

Last week Progress Software released the results of a survey commissioned with an independent specialist technology market research company, Vanson Bourne Research. The researchers surveyed 149 global travel businesses and found that 67% of respondents have noticed their transaction failures soar by almost a third – even though their transactions have only increased by 12%.

Dan Foody, VP of Products at Progress Software, sites: “This research clearly highlights the far-reaching effects of transaction failures within the travel industry. The trend of increasing IT complexity is further compounding revenue loss through increased lost transactions, customer churn and inefficient use of IT resources with 97% of respondents concluding that transaction failures were increasing operational costs."

 “Companies should consider introducing a more streamlined approach to monitoring transaction flows across their IT environments, delivering the ability to respond to changing conditions and customer interactions as they occur. This will enable business leaders to capitalize on opportunities, drive efficiencies and reduce the risk of impacting customer experience. The more responsive approach will provide visibility and increase understanding of the impact IT failures have on their own IT services and their customers,”

Get your copy of the report! The report IT Impact on Travel & Leisure Industry Reservation Management, examines the causes and consequences of this situation. Read this report and learn about some of the underlying causes for the increase in transaction failures, what the consequences are, and the limitations of most traditional monitoring and management systems.

Share your experiences and thoughts by posting a Comment to this entry.

Astounding Statistics Regarding Order Fallout & Revenue Leakage

Sometimes it’s really nice working for a profitable and established public company. With the launch of Actional 8, we’ve put some discipline on our focus around Business Transaction Assurance (BTA). We’ve rolled out lots of internal materials to train our field to understand the “problem” from a customer perspective, so that they can engage in conversations around the problem, rather than “pitch products.”

In any case, I’m reading through some research around order fallout (revenue leakage) Progress just sponsored that was completed by Vanson Bourne. Vanson Bourne carried out over 200 company interviews (these, to my knowledge, were not all Progress customers). The companies are in the USA, Europe, and Asia/Pacific and all had a minimum revenue of $200M. The study was not limited to the Telecommunications industry, but does call out specifics about both the Telecommunications and Hospitality industries because of their heavy reliance on consumer transactions.

Even realizing that we sponsored the research and so it is perhaps somewhat suspect, I can’t help but be amazed at the raw statistics the report showed. Some examples:

• 86% of telecommunications companies believe that order fallout causes delayed or lost revenues; 64% believe it causes customer churn also
• 98% Say that order fallout “definitely” or “probably” causes increased operational costs
• 75% say that having to deal with increased order failures increases demand on resources (warning: circular reference here... increased demand on resources causes order fallout to worsen, and so on...)
• Transaction volumes increased in 2008 by 16% on average, yet transaction failures increased 35% on average over the same period of time
• 86% of companies reporting high levels of IT complexity admitted to an increase of transaction failures; those with the most complex environments had their transaction failures increase by almost half in 2008
• On average, these companies have 11 full-time employees manually finding/fixing transactions, and it takes approximately 2 hours on average to fix them; Telcos have teams of 18 people, on average.

There are phenomenal implications to these numbers. The inefficiency is outstanding. And, there is plenty more where that came from!

I’m a student of life, and in particular of software companies. I’m always amazed when companies spend huge efforts to win new business, only to discard the relationship once the deal is won. Sure, no one does this purposely, but... through compensation and other more subtle motivations, there aren’t too many people in companies whose job it is to deal with happy customers without problems to keep them happy. I find this curious, because “relationship revenue” has a much lower cost of sale than “RFP revenue” and puts much less burden on every aspect of the company.

I digress a little, only to make the point that the companies in this survey are LOSING MONEY THEY HAVE ALREADY EARNED. Therefore, these lost transactions represent the most profitable ones they process! It’s like there’s a queue for service, with someone at the head saying “I’ll take yours, I’ll take yours, I’ll take yours, Nope - you, drop that on the floor, but don’t worry, keep our service.”

The report goes on to talk about some of the interesting “management” facts that usually are part of press releases about “new software versions” but often don’t actually make it into the software release (hasn't anyone developed a compiler that can compile press releases yet... it's a product some companies really need). It seems like there are a lot of difficulties with the software meant to solve these problems. Such as:

• The performance of the apps being monitored is negatively impacted (>60% companies reported this to be the case)
• The management system requires additional servers (~48%)
• Management systems need additional people to run them  (~47%)
• Can’t run all the management/monitoring features in production due to their resource requirements (~42%)
• They use up too many CPUs (~38%)
• 44% of companies don’t manage the end-to-end order process because of the above reasons!

All that said, I still have a personal favorite...

When asked “How often do you lose orders even though these systems say everything is OK?”, fully 35% of companies admit this happens “often” or “frequently.” The number soars to 56% of the very large companies (revenue >$2B) and 69% in companies that have the highest IT complexity scores. (78% of companies self-classify themselves into the highest complexity ratings)

It goes without saying that I believe Actional solves the problems in the first list, without introducing the ones in the second.

Who cares what I think though when you have Forrester studying two of our IN PRODUCTION customers, and finding the same results:

• Staff required to fix root-cause messaging problems was reduced by 85%
• Service quality reports are available in near real time, and created automatically; in the past they were difficult to prepare and were not readily available
• Of the two companies studies (admittedly, not a large sample!), one had a payback of <12 months, the other around 20 months

I’ve blogged before about the importance of the scalability and performance of the management system, without which features don’t really matter. This report validates my earlier post, and calls out specifically that many management features can’t be used in production because of the impact on performance. I wonder if the vendors of these management systems didn’t oversell their products... I wish they had asked that question in the survey! (And, when looking for that link, I found another post that was also mentioned in the report... the importance of actually knowing you have a problem.)

Thank you for reading this far. Email me if you like a copy of the Vanson Bourne report and the Forrester case studies. (These case studies are anonymous, but all research was completed by and validated by Forrester Research).

SOA and Events - Two Sides of BAM?

I came across a prospect recently (new markets so engineering help is always welcomed :-) ) - a domain consulting company now building a Governance Risk and Compliance (GRC) solution. Now, this very much being in the area of BAM, one would expect that traditional Business Activity Monitoring (BAM) would be used. But as we were discussing it more, it turns out that more often than not, GRC is about triggering the risk assessment and compliance/governance checks as business events occur. Here again, the trite assumption (that even I made) is that the organization has SOA imbibed all over and so it can easily tie in BAM and use that BAM environment to 'detect' and trigger the necessary checks/rules. And here again, I was proven wrong because SOA is not as widely prevalent as we all would like to believe! The consulting company, a group with a strong domain expertise in Governance & Risk/Performance, who is building the solution said they wanted an "extremely loose" integration, but their prospects were unlikely to have a mature SOA infrastructure. Which means, they cannot really expect BAM to be practical.

After looking at the biz analysis they had already performed, and taking the time to breakdown the GRC system into a bunch of flow-charts (one for each scenario), they found a pattern:

• a business event occurs as part of ongoing business processing
• the event includes the relevant business data
• a series of GRC checks and rules are executed
• any violation is recorded and an alert raised

This consulting company has gone a step further and also modeled the whole biz domain (they were targeting a specific sector in the financial space) - in a very interesting manner. As a set of key business events, anyone in the domain can immediately understand what each of these events mean. Each business event includes a clear definition of the accompanying data. In effect, they flipped the complete domain business ops as a set of business events! Hmmm...

So far, we have seen models that show the enterprise as a bunch of business processes that invoke a set of services. With the above flipped view, the enterprise modeled as a bunch of biz events is very unique. The former is a view of defining the business flows (processes) that form the enterprise's biz processing. The latter view is more of a snapshot of the enterprise post the fact - as business events that "occur" in the enterprise. Both very clearly model the purpose and the data involved, but in the former method, it is the data that flows through. And with the latter approach, the data that gets "generated" post a business activity.

From this consulting company's point of view, modeling the enterprise as a set of business events and the data that each event emanates was a more practical view. The communication to their clients was that they have modeled the GRC flow for each of these biz events. And the integration of the GRC solution involved the client's IT team "wiring" in these events into their existing solutions - wherever the said business activity occurs, and in any technology they may have, SOA or no SOA.  When they wire this, they will also need to pull out the data from their existing systems and "emit" the same as per the defined event data structure.

Pretty neat idea! Is anyone else seeing it this way? Is a GRC solution more about events and less about SOA? Is SOA primary and events incidental, or are events primary and the SOA infrastructure that could generate this be incidental?

The Rise of Events and Eventing

One of the key things I saw in 2008 was the rise of events and eventing through Event Driven Architecture (EDA).  While "events" have always existed, they have primarily been used in specific markets such as algorithmic trading where our Apama product is commonly used.  The shift in 2008, and moving forward, was that events became more mainstream and that meant that the business and their technical developers were looking to see how they could harness events.

In contrast to traditional SOA infrastructure that use service orchestration such as BPEL to control all aspects of the flow, event based architectures can be used to implicitly enable process to occur.  An event typically happens on a state change or document passing between organizational boundaries or similar.  In order for the event to be useful, it needs to be published to everyone who could possibly be interested and as such, an event consumer can correlate multiple event instances to define new logic and/or to monitor the business event in action.  The key benefit for the enterprise is that the events and their correlation do not have to occur at the initial development time, they need to be captured in a single flow. Handling business events in this way allows a much more organic growth to the logic to be implemented.

Now, any changes in practice will always create new challenges you will need to overcome, not least of which will be the ability for people to assume that business will implicitly happen when there is no end-to-end picture, such as what people expect from BPEL or UML. With guidance, we can overcome these issues because typically the number of processes that are discretely modeled within an organization are few.  One of the key things that has to happen is the idea of event modeling. While there are elements of this in UML through state change diagrams, other tools may have to be used.  Another thing to consider here is that the dynamic aspects of the business will need to be captured and displayed in real-time to allow business decisions to occur when an event really occurs rather than hours or days later. 

In these challenging and opportunistic times that currently exist, I see the arrival of event based architecture as arriving in the mainstream at a very convenient time.  Businesses are restructuring through mergers and acquisitions, so IT professionals need to be able to satisfy this change, plus forces in our market are causing new approaches to be taken, including the idea of clouds that become more than mere hosting platforms.

SOA What? I too agree with Dan Foody that the conventional SOA is not dead by any means. It will actually continue to grow and serve more needs. However, I believe you will see that EDAs will rapidly augment existing projects to initially provide real time visibility and then gradually it will help lift some of the burden on the business as your EDAs evolve and grow.  When that happens, you will start to see more business users being able to implement business logic in real-time rather than having to go back to IT.

What Makes a Good Policy?

Anyone ever see David Letterman's stupid pet tricks? I think we should have a tech show called stupid IT tricks. I recently heard that one CIO's strategy for meeting user requirements was to forbid users to submit feature requests. Software would be installed, and the way it works out of the box is what the users get. His policy of "no more user requirements" ensures a 100% success rate of meeting his IT policy. I applaud his brilliance.

As silly as that sounds, check these two policies out...

• Please don't ride the subway if you have a cold. These signs are posted throughout the NYC subway system.
• Don't drink and drive.

These sound like pretty good policies, right? The former benefits everyone by preventing sick people from getting others sick. The latter, well, it saves lives.

But let's take a closer look... This first policy is totally unenforceable and really doesn't reflect user's realities. Whoever thought of this "marketing campaign" did nothing but waste tax-payer dollars. Most people would love to not come into work when they're sick, but that's just not a reality. If we wanted that policy to work, we'd have to make a realistic alternative for sick people. After all, what's their motivation for following a policy if it's not enforceable, especically when they may not have any direct benefit from following the policy?

On the subject of enforcement, let's look at the second policy. Clearly, this is an enforceable policy. Why then do people continue to drink and drive? There are realistic alternatives to drinking an driving, and would-be drunk drivers benefit directly. A sick person is already sick so not riding the subway doesn't help them, but a drunk who doesn't get behind the wheel is stopped from hurting themselves, or worse, others.

Unfortunately, as Mayor Rudy used to say, "common sense is, unfortunately, not all that common." I think, in part, it's because enforcement is subject to "user interpretation."

But apparently, people are more worried about being embarrassed by having their names/photos published if they are caught drunk driving, than they are about hurting someone, losing their license, or jail time. A new policy started on Memorial Day in Long Island that stated that people arrested with blood alcohol levels over the limit will have their names and photos published on Monday mornings. It's an effective enforcement procedure but has been called "controversial" - probably because it's a government program that works.

SOA What? I commend the person at the MTA who is trying to instill manners on subway riders (though I still think that they should be fired for wasting money/time). But it's a noble cause doomed to fail. And, for those officials on LI willing to risk controversy and continue to publish drunk's names and photos, you set a good example for those of us trying to police our SOA infrastructure.

What can we learn here? Again, it's about the carrot and the stick. Both need to be in place organizationally in order to make policing the SOA practical. You've heard it before, SOA (and SOA Governance) is as much about people and process as it is about technology.

1. Carrot. Enforcement depends upon user behaviors. You can threaten to cut off access to a service if an application doesn't follow policies, but... if that application is mission critical, you probably won't be able to do that. What you'll have to do is adjust your governance policies! It may not be obvious, but there are probably low cost ways to encourage user behaviors without making them political battles. How about posting a list of insecure applications? You may not be able to cut them off, but perhaps users will think twice before depending upon the application. Put the onus back on the developer rather than being the bad guy yourself.
2. Stick. Policies need to be enforceable. And that simply means watching what's happening. Let's compare our SOA to the subway... When we create a list of policies stored in a registry and then test our application against those policies once, when it comes time to place the application into production, it's like making sure we only sell metrocards* to healthy people. Once you have a metrocard (once you're app is in production), you can get sick and ride the subway (you can violate your governance policies), but we have no way to tell because you have no visibility into your run-time environment. If you laughed when you read the policy above, and do this in your SOA, I suggest you rethink your approach to SOA governance.

Business visibility should be a driver for SOA

I've just read a Gartner research note by Massimo Pezzini published on the 16 November. It's titled Greater Business Process Insight Is an Unexpected Benefit of SOA. It argues that an unexpected and unplanned consequence of SOA projects is that organisations gain better visibility of their business processes and business data. It recommends that organisations think and plan about things like Business Activity Monitoring (BAM) from the outset.

I was surprised by this and also disappointed. Not because I don't agree that better business visibility is a consequence of SOA – it is. What surprises me is that this is still seen as noteworthy. It shows how far the industry still has to go toward understanding the merits of enterprise architectures which give them more open, simpler, more flexible access to real-time business information. service-Oriented Architecture (SOA) is about service reuse, open standards, etc. etc. But you're doing this for a reason, not for its own sake. Surely one of the benefits of a more open, flexible architecture is that you can get at your business information better so you can drive efficiencies in your processes, identify threats and opportunities more quickly, and better communicate with customers, etc?

Some organisations do recognise this. Only last week we responded to an RFI which not only included "usual suspect" SOA requirements satisfied by such things as an Enterprise Service Bus (ESB), but also requirements for event processing and BAM. The organisation recognised that the building of a SOA infrastructure gives them access to business event streams that they can obtain value from immediately. They had indeed expected this consequence and were planning for it.

SOA What? These kinds of requirements will become the norm. Real-time access to enterprise information is becoming more relevant in a whole range of industries – we're seeing this directly in some of the engagements we're involved in. Furthermore, event processing and BAM are at the more business end of the SOA universe. One of the things that the industry harps on about is that the business rarely sees the relevance of a SOA initiative. By taking the advice of the article and thinking about these things apriori then perhaps not only can organisations get more out of their initiatives but also CIOs can gain a little more credit from the wider organisation.

I see events. They’re everywhere.

After having been involved in the study and implementation of advance event processing techniques and technology since the mid 1990s, it’s gratifying to see this field getting much deserved attention. In a recent presentation, Yefim Natis of Gartner, predicted that Complex Event Processing (CEP) will be one of the five most significant IT trends over the next few years. Yefim also noted that CEP will increasingly become the foundation for the next generation of Business Activity Monitoring (BAM) applications. This is very good to hear as this is a conclusion that we have also reached.

One reason the market is getting excited about CEP is that event-capable middleware—most significantly the Enterprise Service Bus (ESB)—is now becoming mainstream in corporate IT. ESBs, most popularly regarded as the critical messaging-based backbone for SOA implementations, are just as capable for event-driven architecture (EDA) implementations because of their messaging heritage. EDA takes the SOA principle of loosely coupled services one step further – to decoupled services. Whilst the interaction model is a little different, the underlying technology remains the same so ESBs provide an important driver for CEP adoption. Along with the adoption of these technologies also comes familiarity with the whole event processing model, and the technical skill and best practices that go with it. The result—in short—is the sudden availability of business event on a much more pervasive scale. Here, the event processing version of Metcalfe's law kicks in—the value of CEP is increases exponentially as more events become available to process. That is, as long as your event processing infrastructure can handle it.

Another reason interest in CEP is rising is the proven success of the technology within capital markets. Not only have most of the leading investment banks adopted CEP to implement their algorithmic trading and risk management systems but now the financial markets and regulators are adopting the technology to monitor the market in real time to detect compliance violations or patterns of fraud, like insider trading. For example, the Financial Services Authority (FSA) is adopting CEP technology to help monitor the UK markets and enforce MiFID compliance in an increasingly fragmented trading environment. The system called SABRE II is being developed by Detica. You can read about it here.

But it’s not just capital markets that will benefit from CEP. It is also being adopted on the manufacturing floor for process control and improvement, in logistics applications for early detection of supply chain issues, in retail RFID for inventory management and security, by military branches for situational awareness of the battlefield, and even by casinos to help their monitoring of suspicious gaming behaviour. It seems events are everywhere these days.

I was recently at an academically hosted event in Germany. Many vendors and practitioners attended and many themes in software were discussed – BPM, BAM, SOA, CEP amongst others. What came over loud and clear in the sessions was that CEP was increasingly seen as a major new and important capability that fitted well into the SOA world. An ESB for example was important in giving the foundation for the gathering and analysis of events from around the enterprise.

As more and more organizations embrace SOA infrastructure that supports event processing—and the event processing skill set that goes with it, the value of CEP technology will continue to rise. In fact, expect interest in CEP-powered applications (BAM and otherwise) to be a real driver in infrastructure investment over the next few years. Soon you’ll hear many more CIOs saying: “I see events. They’re everywhere.”