Business Making Progress

CEP in Transport and Logistics

Now maybe I’m biased since my focus is CEP outside of capital markets but, the announcement of Royal Dirkzwager’s (Koninklijke Dirkzwager  if you’re keeping score in Dutch) use of Apama sets an important tone for CEP. Working with the team that pulled Royal Dirkzwager’s project together, it was easy to see that the business need was there to harness the value of their free-flowing events.

Transport and Logistics (T&L) has a vast fabric of events that range from all the management “systems” to supply chain events and telematics. Without CEP, analysis of those events is either siloed (with spotty real-time abilities) or aggregated a day late and a dollar short. The T&L industry is being squeezed for efficiency and chasing new revenue opportunities is challenging. In the case of Royal Dirkzwager, it didn’t take long for them to survey their domain and recognize that real-time visibility and analysis of events in their fabric could significantly enhance customer satisfaction and the bottom line. Royal Dirkzwager also appreciates that CEP projects aren’t protracted lifetime engagements. They’re looking forward to rapid results.

Royal Dirkzwager’s use of GPS, GIS events as well as information from Automatic Identification Systems (AIS) and Long Range Identification and Tracking (LRIT) systems, when correlated with their other infrastructure events, is where the magic is. Yes, they had all those events before but CEP pulls them together and makes them much more meaningful. They’re also going to leverage their Sonic ESB as a convenient on- and off-ramp for many of their events. And while Royal Dirkzwager’s a maritime logistics provider, it’s not a far stretch to see how other land- or sea-based LSPs can put themselves in a similar position. Whether they own, manage or contribute to the supply chain, the events are there – harnessing them should be an imperative. Royal Dirkzwager’s not our first in this space and it won’t be the last.

I think we’ve done the right thing by having a laser beam focus on capital markets. Truly, a demanding environment when it comes to performance and usability. All that experience gives us the ability to jump into other event-oriented markets with the tools and experience to take on any challenge.

To Royal Dirkzwager, thank you for making my day. Who’s next?

-DO

Cloud Computing Impact on Enterprise Integration Recap

Well, that was fun. Spoke at 7PM this evening, at the end of a long day of presentations at the Cloud Computing Expo. I was a little afraid of a 7PM speaking slot because I thought people would rather be at the bar rather than listen to me talk. Boy, was I wrong. Apparently, people were standing in the hallway. Had I noticed, I certainly would have rearranged furniture to get more people in - even if it meant more people sitting on the floor!

First, as promised, here are the presentation materials:

• Download Presentation (please view this in presentation mode for best effect)
• Download Cloud Computing Impact on Enterprise Integration Paper

The presentation was targeted at the Enterprise Architect and hopefully left people in the room with thoughts about how to approach (and explain) cloud computing, and how to get started in the enterprise with a few best practices. Based on the nodding-heads in the audience, I think I touched a few key points.

Summarizing in my own words:

• Cloud computing is about turning a commodity into a utility.
• Integration is way too hard, non-technical people don't understand, so discount the efforts.
• Moving what we can to a utility model, will let us focus on what's important and elevate our game.
• Integration, the easy stuff is way too hard, so we never get to do the cool stuff; just do it. Mistakes with cloud computing are much less expensive and irrevocable than they are with typical enterprise software purchases/deployments.

And as for the best practices... well, just download the paper or the presentation.

For a full "live blog" summary, check out Brenda Michelson's post.

Great feedback that I heard afterwards was that I was the "first presenter of the day who knew how to present." That was the best feedback of all.

Thanks to all my friends, real and virtual, who a ttended. It was great to see so much support.

Download ExpoPreso

SOA Governance in SMEs == Cart Before the Horses

I have always held that SOA governance is premature yet. While we are struggling to just get SOA adoption in the right spirit, governance is the least of our problems. If we cannot identify a mechanism to first re-engineer "service orientation" into IT solutions and have the IT teams in an organization be "SOA aware" ground up and not just deliver the POCs that the CIO or CTO demands, where is the need for operational processes and governance? In this context I found this article that professes a SOA governance approach for SMEs to be quite out of touch with reality.

While the thoughts proposed, such as agile approach to governance and the business users defining the need and IT groups quickly implementing the same, a managed governance process is not necessarily bad. It's just that we are dealing with more fundamental issues right now. Issues that are preventing the widely touted gains of SOA. That led to the drastic predictions of SOAs death!

In the present stage of SOA adoption, where we have just crossed the hype-curve's chasm, and are just now pondering over the serious practical use cases with a full grip on reality (of issues with SOA adoption), all that is immediately needed is anything but SOA Governance. Good tools and utilities to help quickly create services, test services, deploy and manage the same in production. The need for complete SOA lifecycle governance is still a few years away. At least, and surely not for SMEs, their SOA problems will be much more fundamental. SOA Management is what they will need now.

 

Actional Business Process Visibility (Definition)

I haven't been writing much (here) lately but, well, I've been thinking of writing.

The process is weird. Topics come into my head often enough but I talk myself out of sharing. I'm not sure how useful they are or how biased I'd sound.

I'm working on the outline for training materials integrating Sonic and Actional, and I had to explain Actional Business Processes (tracking of)  to some new people.

As I did that, I remembered hearing a prospect earlier this week say "Actional sounds good but really functions more at an IT level, we're interested in monitoring at a business level."

I must admit my bias but Actional not only monitors at a business level, but we tie it back to the IT level, to achieve two great benefits:

1. Business people and IT people can communicate because they are working off the same vocabulary of processes, services, KBI's, Dimensions, etc;
2. Staff can look at the same picture (automatically drawn and updated in real time) and see the business impact of a technology change/failure, and vice versa.

My definition to the training team earlier today went something like...

"Remember how Actional automatically learns flows. Well, Actional business processes are really just named sub-flows. Meaning, without any orchestration or BPM we can take a 'free form sub-flow,' name it, and we have something we call a business process that we can track as a unit. We can track it that way even in a shared environment, so shared components report in only their statistics for that process, when implementing policy or service level agreements. If other processes or services use those shared components, we can separate all the different performance statistics (for policy based management) so that things aren't lost in the average statistics as they are with so many other management systems."

You saw our press release earlier this week about Q1's results. Actional rocked, again.

It's because we do some really unique, creative and relevant stuff, like business process visibility.

NYC Cloud Computing Expo Preparation Almost Done

Well, we're just a few days away and preparation is winding down. We've got an article written and ready to go, and I've got my presentation mostly complete and just need to work on layout, notes and review. Sounds like I'm almost done... just hope I don't get sidetracked by some fire drill.

The media alert went out today and it got me thinking that it might be interesting to share the creative process.

Personally, I don't get nervous with public speaking. It doesn't mean I always do well, its just that I quite enjoy the opportunity to get in front of a crowd of strangers full of adoration for the gems spewing from my mouth. In fact, I tend to not "rehearse" for presentations as much as "immerse" myself in the content and visualize how it will go, the thoughts I'd have, the questions that might come up, and feed that back into the presentation in a way that makes it more robust.

This presentation started with work on a white paper I began working on about two months ago. It was a bit of a free form set of ideas I have about cloud computing. I had only the guideline of my submission - The Impact of Cloud Computing on Enterprise Architecture - and I ended up with content in four areas:

1. The difficulties of enterprise integration
2. The corporate challenges of enterprise software
3. A definition of cloud computing
4. Some best practices

The level that I speak at is relatively high, and I worry that it's not enough detail for people. I guess we'll see on Monday.

It took me a week of writing, with some interruptions I'm sure, to get down about 3,000 words on paper. That's about 3 1/12 pages of writing, and it was handed over the PR team for editing.

It may be totally behind the scenes but the people that have edited my stuff in the past, this time included, do wonderful work. It's always important to double check the content and the meaning, and to make sure that the important points stayed, but they really tighten it up and focus it down.

They turned my 3,000 words into 1,120. The final piece has just 20 additional words, though I had to do about a day of editing to bring some of the meaning back, and highlight some points that were lost originally. It'll be posted on Monday so keep an eye out for it. As always, let me know what you think.

I had been thinking about this cloud stuff for some time, and tweeting ideas to see how people would respond. I've also been vigorously reading the tweets and blogs of others to get a sense for where the conversation is. Hopefully, my presentation on Monday adds something to the conversation, in a way that constructively builds on what others are saying.

The difficulty with creating the presentation, of course, is that I didn't just want to put the paper in presentation form.

That would be boring. (Boring is bad.)

I came across a presentation titled "Charts are Cheap" by Major Dan Ward (who unfortunately, didn't include a link to his blog in his presentation, so I can't link you to him!). The idea was 1/2 - 1 idea per slide/chart. Simplify and people will understand more.

So, I'm trying something new with my presentation. It's really lightweight and a lot of what I'll say is not written on the slides. It is, mostly, written in the paper. And, what's not, would be great to discuss online after the event.

Part of the reason to do something different is my "middle of the night" speaking slot. And, I believe I'm also last for the day, so anyone that does come, and stays awake, has me between them and the bar. Not an enviable position.

Though, if they need a drink after my presentation, I believe Mike is organizing a Tweetup at ESPN Zone in Times Square after. Hope to see you there.

Five Dirty Little Secrets of Highly Available Integration Infrastructure

Even some of the world’s largest, most demanding companies suffer from a hardware, software, or network failure. Learn more about how a continuous availability solution will protect you from lost transactions, and guarantees in-order delivery without the cost and management complexity of the 3rd party hardware or software add-ons.

Register to download our new white paper, Five Dirty Little Secrets of Highly Available Integration Infrastructure. Topics presented in the white paper include:

• Recovery Time
• Data Corruption
• Hidden Cost and Complexity

... and those are just three secrets. Download the paper >

We're going on Twitter

Louis Lovas and myself, Giles Nelson, have started using Twitter to comment and respond to exciting things happening in the world of CEP (and perhaps beyond occasionally!).

The intent is to complement this blog. We'll be using Twitter to, perhaps, more impulsively report our thinking. We see Twitter as another good way to communicate thoughts and ideas.

We would be delighted if you chose to follow our "twitterings" (to use the lingo), and we'll be happy to follow you too.

Click here to follow Louis and here to follow Giles (you'll need to signup for a Twitter account).

Scalable Performance - a sneak preview of what's ahead for Apama

We will soon be formally announcing the next major version of the Apama product, but I thought I would give a bit of a sneak preview here. We've added a number of significant enhancements to the product, one in particular I want to highlight is a new parallel execution model. Our CEP engine's new parallelism allows massive vertical scalability on multi-core hardware. We can leverage 8-way, 16-way even 32-way processors with ease.  We've enabled this capability by extending the Apama EPL with four new keywords. To explain requires a bit of background ...

In the Apama EPL, we have the notion of a “sub-monitor”, which can be thought of as a separate parameterized “instance” of a set of business logic that encompasses a scenario, a trading strategy for example. Each sub-monitor can load and maintain its own data (e.g. consider a sub-monitor as corresponding to a different occurence, with different timeouts, key indicators, client data, etc.) and can set up its own listeners to key into a (different or overlapping) subset of the event stream. This allows us to easily model factory like behavior, with each sub-monitor maintaining it's own state and possibly listening for different (sequences of) events – but applying the same basic logic including the actions to take when an event condition becomes true.  We call this our micro threading or mThread architecture.

In the latest Apama release, v4.1 we extended this and introduced the notion of contexts. These are silos of execution which take the factory model to the next level. ”context" is a reserved word in Apama 4.1 – it defines a collection of sub-monitors which can inter-operate in a protected address space, with strong semantics for inter-context communication (via event passing) similar in concept to the Erlang message passing programming model. Most importantly, it is also the unit of parallelization – allowing the same CEP engine to spawn multiple “contexts” which key into the event flow but execute in parallel on multi-core architectures. 

Contexts in Apama's EPL adhere to our language's basic event paradigm, providing a safe semantic for concurrency yet avoiding the typical race conditions common in multi-threaded programming in other languages (i.e. java) which require the use of mutexes, semaphores, etc.

"The world IS concurrent. It IS parallel. Things happen all over the place at the same time. I could not drive my car on the highway if I did not intuitively understand the notion of concurrency; pure message-passing concurrency is what we do all the time."

A great quote, one that we've taken to heart in the design of the parallelism now available in our EPL. Our approach is based on a deep understanding of the types of applications being built with the Apama platform. Our broad customer base provided us that advantage. For example, we took our Smart Order Router Solution Accelerator, enhanced it to use the context model and did a performance benchmark and achieved a 6.5 times increase in overall capacity on an 8-core server while holding to a steady low-latency threshold (notice we also improved overall performance in the v4.1version over previous versions as well).

This is a graph that compares the Capacity (number of concurrent open orders that can be processed in a specific timescale) of the Equities Smart Order Router for an increasing number of symbols. The comparison was on three versions of the Apama product. In the parallel version we have modified the SOR to partition symbols across contexts (each context goes on a processor core). The machine used for the experiment was an 8-core Intel Server. 

Apama's new enhanced parallel execution model is a reflection of how our customers use CEP technology to build out real world applications. The competitive landscape of CEP dwells on performance with wild claims of speed often without substance. It reminds me of a teenager revving their car's engine at a traffic light. You can see the needle on the tachometer race up, the engine makes a lot of noise, but to what purpose? With the new release of the Apama CEP platform it shows that we know how and when to engage the transmission.

SOA Next- Just a New Skin for the Same Problems

After the recent (and dramatic) proclamation that SOA is dead, is it a question of time until we see the rising of the proverbial Phoenix? Seems to be starting... at least going by the interview with Miko Matsumura in Information Week. Referring to the Gartner hype Cycle, he says, "I think we're on the 'Slope of Enlightenment' before we reach the 'Plateau of Productivity'. Anne Thomas Manes' pronouncement about the death of SOA was made in the depths of the 'Trough of Disillusionment'." I particularly liked the reference to "coarse grained cost-cutting", where he says when a whole division is rendered obsolete, what happens to the systems that the division was responsible for; that may still remain operational? SOA infrastructure is supposed to solve this!

Miko's solution is that if the services are well defined - with dependencies clearly modeled and governance practices in place - then it is easy for the organization to deal with such extreme situations as a whole division not existing anymore! This seems like Utopia. Really!

To me, the scenario is more the promise of SOA. The issue though (leading to the proclamation of SOAs death) is the reality, which is far from the promise. None of the anticipated flexibility and value is realized beyond the POCs and pilots! Not much mainstream gains yet. But, like I mentioned in an earlier post, the issue is more about the adoption approaches rather than any inherent deficiencies in the SOA design approaches or the platforms/technologies selected.

So even if we were to invent a "new SOA" or "SOA Next" or "SOA Whatever", the problem will remain because it never was about the technology, per se. It is more about the practical difficulties in getting an organization (its management, architects and developers) to understand and appreciate the SOA principles even half as much as they yap about the next minor revision to a WS* standard as the necessary consideration in their present SOA projects.

At the end of a successful SOA initiative, it's not just about the individual IT groups/teams understanding the need for defining a services layer, it's also about their well defined UI and DB layer which have to be ready for enabling coarse-grained services based on sufficient cross-functional business analysis of the organization. What's more these should be XML-aware layers, beyond just the fringes ("wrappers"). Now, all of these need to be in a single solution, regardless of who needs it now, and what SOA platform is in place. In this bottom-up approach to SOA adoption, the principles are adopted and ingrained first, and then comes the platforms, services, orchestration and governance.

Simply put, successful "service orientation" is the single most significant key to successful SOA in any organization!

An(other) industry award for Apama

I’m very pleased to announce that Apama has won the 2009 Technical Analyst award for “Best Automated Trading product”. The awards ceremony took place last night at the Sheraton Park Lane Hotel in London.

The judges not only considered the product itself and the way it was being used in electronic trading but also considered the customers who were using it too, so this is a really fantastic validation of Apama.

Here are some of the key attributes of Apama that were considered by the judges:

• Apama is (of course) a CEP platform – one thing that that provides is flexibility. Once deployed to run an application such as FX aggregation, Apama can be applied equally well to algorithmically trade equity futures or any other type of asset.
• Connectivity to a vast range of general software and trading specific systems.
• Availability of Solution Accelerators for FX aggregation, order routing, algorithmic trading and others providing 90% of an end-user application.
• Backtesting of trading strategies and ongoing strategy evolution.

For the record, the other finalists were Alphacet, Berkeley Futures (IQ Trader), Patsystems (Pro-Mark), QuantHouse (QuantFACTORY) and TradeStation.

Now, from a CEP technology perspective, this is the significant thing for me. Apama was up against, in quite a focussed industry category, a number of other vendors who do nothing else except build their products for use in capital markets. And yet, Apama won, thus demonstrating the fact that a general purpose CEP platform provides the first-class choice for organisations who want to flexibly deploy a whole range of event-driven applications. That’s great news for all CEP aficionados.

Now, where’s the rest of that bottle of bubbly?

Chapter 5: The Data Integration Dialogues (Part 2) is Now Available

Chapter 5 of The Semantic Dialogues, The Data Integration Dialogues (Part II), is available now. Here's an excerpt: "Sanjiev, Patrice and Jerry are reaching consensus on the best approach to their data integration problem. And Cliff is showing some enthusiasm about using the data subdivisions of the SID as a guideline for internal projects. But it will be in vain if the team can’t find a way to put their theories into practice. Will their agreement last long enough to find a technical solution? "

Click here to read Chapter 5, The Data Integration Dialogues (Part II), today. And if you’re enjoying The Semantic Dialogues, be sure to share it with a colleague.

The Semantic Dialogues tells the story of how National Networks, a fictional telecommunications service provider, pursued and achieved data interoperability within their SOA infrastructure. The story begins with an urgent request from Operations for a fix to a problem that's keeping them from billing for new services. Assured that extensions can be made to the common data model (the SID) for the COTS billing system that's causing the headache, the problem is resolved in short order. The story then goes back a year to a time when the lack of a data interoperability layer made similar problems seem insurmountable and a tiger team is formed to put in place an OSS/BSS architecture that will change the way the CSP does business.

The instance design pattern in the Apama EPL

In this my third installment on a series devoted to the Apama Event Processing Language (EPL), I'll continue where I left off last time in which I described the basic design of event passing for a consumer/producer model.  For this revision I've extended the model to support multiple consumers.  This introduces the instance management feature of the Apama EPL.  Instances or 'sub-monitors' as they're often referred to define a discrete unit of work. The unit of work can be very small (an analytic calculation) or very large (a whole trading algo). Each instance gets spawned with it's own parameter set, listens for it's own event streams and operates in a very singleton manner. To which I mean, within the semantics of the application an instance need only be concerned about managing itself not other instances. Overall, it is a  factory behavioral model extended to include an execution model.  This is a key aspect of the Apama EPL, making a common application requirement simple to implement, robust in design, and highly performant in the CEP model.

The Apama CEP engine manages the execution of these sub-monitors (also known as mThreads internally). In a typical production deployment, there would be 100's or 1000's of sub-monitors running. The spawn operator is the single statement in the language that accomplishes this unique feature. Spawn is basically a self-replicating scheme with certain behavioral rules. The main rule: the cloned instance does not get the active listeners (i.e. on all someEvent...) of the parent. It must establish it's own. Actually it's the perfect model for that factory-like behavior.  The clone does not want it's parents listeners, but would create it's own based on the parameters passed such as the symbol name in a trading algo or the subscriberId in our Producer example below. Speaking of our example ...

For the sake of brevity, I've just listed the extended Producer side of my consumer/producer example below. For the complete example, you can download it here.

The (extended) Item Producer.

package com.apamax.sample; monitor ItemService {         event ClearUserID {         integer id;     }         UnsubscribeFromItems u;              integer count := 0;     float price := 0.0;     listener l;         action onload {         // list of subscriber (user) identifiers         sequence<integer> ids := [];                 SubscribeToItems s;         on all SubscribeToItems():s {             if ids.indexOf(s.subscriberId) = -1 then {                 ids.append(s.subscriberId);                             spawn startSubscriptions(s.subscriberId, s.item_name);                   }         }                 ClearUserID c;         on all ClearUserID():c {             log "in " + c.toString();                integer index := ids.indexOf(u.subscriberId);                 if index != -1 then {                     ids.remove(index);                 }         }     }     action startSubscriptions(integer this_subscriberId, string name) {         log "in startSubscriptions";                 l := on all wait(0.1) {             route Item(this_subscriberId, name, count, price);             count := count + 1;             price := price + 0.1;         }         on UnsubscribeFromItems(subscriberId = this_subscriberId):u {             stopSubscriptions(u.subscriberId, u.item_name);         }            }         action stopSubscriptions(integer subscriberId, string name) {         // do something to stop routing events         log "in stopSubscriptions";         l.quit();         route ClearUserID(subscriberId);     } }

To get a general sense of what this bit of code is intended to do, I suggest a quick scan of my previous installment where I introduced this example.

The extended Item Producer is expected to manage multiple uniquely identified consumers. For that it must maintain a list of identifiers, one for each consumer.  It does that by appending and removing entries from an array (sequence<integer> ids).  his is a common idiom for tracking identifiers, syntactically it's similar in many imperative languages.

This example uses a single-cast event passing scheme where the Producer routes Item events uniquely tagged to the consumer (route Item(this_subscriberId, name, count, price)).

 On the consumer side, Item events are listened for based on a subscriberId (on all Item(subscriberId = myId)). It's the uniqueness of subscriberId (one per consumer) that defines this as a single-cast design. A common twist to this a multi-cast event passed scheme (not be be confused with the UDP multicast) where multiple consumers might be requesting the same information (i.e. the item_name in our example).  A well understood example of this would be a market data adapter providing trade data for the same symbol to multiple consumers.  The Item Producer would change very little to support a multi-cast event passing scheme.

In the listener "on all SubscribeToItems()", we spawn to the action startSubscriptions when we receive a  SubscribeToItems event from a consumer. We pass the parameters of  the consumer's identifier (s.subscriberId) and the item (s.item_name) to instantiate the new sub-monitor.  A new mThread of execution is created for the sub-monitor and it begins executing producing Item events. The parent monitor continues waiting (listening) for another SubscribeToItems event.

 You'll also notice the use of a private event ClearUserID, the purpose behind this is to communicate between the spawned sub-monitor(s) and main (parent) Monitor when an UnsubscribeFromItems request is received.  This is necesssary since the parent monitor manages the id's of connected consumers. A spawned sub-monitor uses this event to simply inform of termination.

The event paradigm in the Apama EPL extends far beyond the notion of processing data events. In one sense you could categorized events as data and control. Data events are consumed and processed by the CEP application. Control events direct the semantics of the application. 

 

This example is designed to illustrate a few powerful yet easy to use features of the Apama EPL:

1. To highlight that the notion that managing multiple consumers (clients) becomes a simple and safe programming task in the Apama EPL. Instance management is an intrinsic design pattern based on the commonly understood factory model. We did not reinvent the wheel, we simply refined it and made it approachable in the CEP paradigm.
2. Events are not just application data to be processes by monitors. They provide semantic control of an application as well.

Here's the complete example with the consumers, interface and producer. 

Once again thanks for reading,
Louie

This Should Make It Obvious to Anyone

Looking at things from the perspective of how software is evaluated and brought into the enterprise, it's no wonder cloud computing is dramatically changing the landscape. Combine that with the trend of software vendors looking to compliance to shore up revenues, and customers pushing back to reduce their software maintenance costs, and I think we're in for some interesting changes in selling software over the next couple of years*.

In any case, I thought it would be interesting to put together a quick list of how software is piloted in a "traditional" purchase cycle, and how it might be piloted if "purchased" from a cloud.

So, assume a thorough selection process has been completed on a product and its alternatives (I don't think we do that well today either... I think it's too feature-focused and we're not nearly evaluating the "right" things), and we're talking about getting the first small set of people up and running on the selected solution.

Traditional Software Purchase

1. Software is purchased.
2. Hardware is purchased.
3. Space is found in data centers for the production kit.
4. A plan is created for moving into production, often a lengthy process inserted around specific rules/restrictions for changing production systems.
5. Development and test environments are setup.
6. Computer room space is found for the permanent location of the development and test kit.
7. Any databases required to support the application are purchased and configured, including hardware if necessary.
8. Administrative and security staff support is required to configure all this new equipment to company standards, backup procedures, and security policies.

Cloudly Purchased Software

1. Get a login.

Even people who don't know anything about software should be able to figure out which is easier. And, while I shouldn't need to ask, but, how much longer do you think the first process takes than the second from the time a decision is made?

If you want to follow the doings of a new startup taking advantage of just this difference, follow Mike on Twitter. He's starting a new company and taking full advantage of open source and clouds to have some incredible infrastructure and development cost metrics - actually showing through his own experience what can be achieved.

Want to hear more? Remember, I'm speaking at the NY Cloud Computing Expo later this month.

* "Ray" Wang has been tweeting and blogging about this topic quite a bit. I find it very interesting to follow.

Who Needs DW? Welcome, Real-time BI !

Last week I got a last minute invite to speak at the Silicon India BI conference (last minute for a good reason I guess... me & BI? Hmmm... :-) My database background notwithstanding!). Initially I resisted as couldn't think of anything of value, especially because the participants were expected to be architects and senior tech folks. As Pradeep (heads Silicon India) nudged, a thought crossed. I recently also came across a BAM use case where the kind of questions they wanted the solution to answer bordered on BI. And, in this solution they needed both SOA and Complex Event Processing (CEP).

Isn't using CEP for BAM in some form real-time BI? I deliberated for a few minutes - the CEP SOA synergies seemed to really make sense, so I agreed to speak. When asked for the title, I submitted "Design Approaches for Real-time BI".

BAM (business activity monitoring) is an extension of traditional business intelligence; adding event monitoring to scheduled, batch-based reporting. As I see it, BI can be of a few different types (when seen from the tainted SOA lens :-) ):

• Simple BAM is watch ‘as-is’. Most BAM software supports simple aggregation and watches for thresholds, et al.
• Complex BAM can watch for more complex patterns.

At some level isn't complex BAM BI? It provides insight and information that traditional OLTP systems cannot give. For BAM or BI, the discrete steps in the decision cycle are the same: Track/monitor; Analyze; Hypothesize/ model; Decide; and Act . The only difference being that BAM (however sophisticated) may not be able to answer adhoc queries and what-ifs. It can only answer pre-defined and pre-modeled situations. Nevertheless, this is BI. And it is real-time!

SOA and CEP form a good basis for realizing this. Conventional (read 'simple') BAMs may not suffice. BI is more than just thresholds or SLA violation alerts. One may need to detect a pattern involving a series of biz failures that could potentially cause a much serious eventual failure, which if detected and alerted in time, can be averted. This is where more complex pattern recognition come into play. The CEP engine. Given that we are talking about business events from all over the enterprise as the inputs, SOA infrastructure fits right in as the potential vehicle to deliver these events to the CEP engine.

SOA+CEP for real-time BI! An interesting proposition! And surely something traditional DW solutions cannot handle. DW & real-time? Nah. Now, can we extend this basic real-time BI to also include adhoc querying and such? Something to ponder about.