Business Making Progress

Structured Programming in the Apama EPL

This is my second installment on a series devoted to the Apama Event Processing Language, MonitorScript. In my introduction to the Apama EPL I described the basic structure of the language, some concepts and terminology. I also included the obligatory Hello World sample. In this entry I'll continue that basic tutorial delving into the elemental structure of the EPL. I've decided to do this to highlight the very simple fact that the language adheres to the principles of structured programming.  Just so we're all on the same page, here's a definition I found on the web:

DEFINITION - Structured programming (sometimes known as modular programming) is a subset of procedural programming that enforces a logical structure on the program being written to make it more efficient and easier to understand and modify.

That's a definition that is clearly irrefutable. In order for a language to live up to that definition it must have a number fundamentals.  Code modularity, or developing code as separate modules. This allows for parallel development, improves maintainability and allows modules to be plug replaceable. Encapsulation is a means of hiding an implementation or inner workings of a calculation or algorithm. Lastly there are  Interfaces and instances. This architectural pattern  may be applied to the design and implementation of applications which transmit events between loosely coupled software components. These are just a few of those fundamentals. These essential elements of languages have been part of the scene for decades. They are present in the procedural programming languages like C and in object oriented languages like C++ and java.  

Lest we forget our roots as software engineers, with all the hype surrounding CEP and their constituent domain-specific languages, it's important to do a reality-check and not get swept up and forget the basic principles necessary for creating long-standing maintainable code. Last year I wrote a piece on readability as a criteria for a successful language this is primarily based on a development language having these basic fundamentals: modularity, encapsulation, interfaces and instances.

Apama's EPL, MonitorScript has all these capabilities, as I will describe below. It's what allows us to build reusable framework components, and solutions. Which we've done so with our Capital Markets Framework and Solution Accelerators for FX Aggregation, Smart Order Routing, Surveillance, Bond Pricing, etc.  These are components written in our EPL that have the plug'n'play modularity to be redeployed in multiple customers. 

To illustrate this idea of structure - modularity, encapsulation and interfaces I'll use a short example of a producer and consumer. This is a common design pattern or idiom we use extensively. The Apama EPL's event paradigm extends not only to the type of CEP applications we enable but also to the nature of the language itself. If you're familiar with message passing languages such as Erlang this will be a familiar concept. Different modules that make up an application communicate with one another by passing messages (or events as is the case). 

In this example I have a service or producer monitor that generates and sends Item events, a client or consumer monitor that consumes Item events and an interface for the interaction between the two. If the term Monitor seems strange, I've defined a set of terms and concepts in my introduction, I suggest a quick review of that get up to speed.

The interface to the monitor, defined below is the set of events it receives and transmits. The event definitions for these events are declared within a package name (i.e. com.apamax.sample). Apama's EPL supports java-like package names for name-space isolation to strengthen that modularity notion.

The Item Interface.

package com.apamax.sample; event SubscribeToItems {     string item_name; } event UnsubscribeFromItems {     string item_name; } event Item {    string  item_name;    integer item_count;    float   item_price; }

The Item Interface is simply a set of prescribed events. Two are to instruct the consumer to start/stop receiving Items and the Item definition itself. As I mentioned it uses a subscription idiom.   We use this notion extensively where a Monitor is a layer over some subscription-based service such as a market data adapter. A real market data interface would be much more extensive. Here, I've scaled it back for the sake of simplicity, but you can imagine a more robust interface including error handling and status events.

 

The Item Consumer.

package com.apamax.sample; monitor ItemClient {     SubscribeToItems subscribeToItems;     Item item;         action onload {         subscribeToItems.item_name := "sample";         route subscribeToItems;         on all Item():item {             log "Got an item: " +  item.toString();             if (item.item_count > 10) then {                route UnsubscribeFromItems("sample");                log "All done.";             }         }     } }

The Item consumer is also a Monitor in the com.apamax.sample namespace.  It is a user of our event interface to the Item service and as such is interested in receiving events of type Item.  The interface defines the means to do this by subscribing. The consumer simply has to create a SubscribeToItems event and forward it to producer service.

As I mentioned earlier, the Apama EPL adheres to an event paradigm as a fundamental characteristic of the language. The route statement is a means by which Monitors communicate. This is another precept and underlines the fundamentals of modularity and encapsulation.

Once a subscription request has been sent  (route subscribeToItems), the consumer listens for Items events, (on all Item()).  In this simple example, we're just looking to receive them all without any filtering or pattern matching.  I will explore event pattern matching - both the simple and complex in a follow-up blog.

To complete the picture, the sample tests a field in the Item event and terminates the subscription if it exceeds a constant value, (item_count > 10).

The Item Producer.

package com.apamax.sample; monitor ItemService {         SubscribeToItems subItems;     UnsubscribeFromItems unsubItems;     integer cnt := 0;     float price := 0.0;     listener l;         action onload {         on all SubscribeToItems():subItems {                 startItems(subItems.item_name);                  }         on all UnsubscribeFromItems():unsubItems {                 stopItems(unsubItems.item_name);         }     }     action startItems(string name) {         l := on all wait(0.1) {             route Item(name, cnt, price);             cnt := cnt + 1;             price := price + 0.1;         }     }         action stopItems(string name) {         // do something to stop routing events         l.quit();     } }

The Item producer is also in the com.apamax.sample namespace.  It defines listeners for SubscribeToItems and UnsubscribeFromItems.Two events from our interface. Typically, subscriptions would be managed on a per-user basis, thus allowing multiple consumers to subscribe to our Item service. This is a detail I will outline in a subsequent installment along with a few other related features such as instance management. 

Once a subscription request has been received, the startItems action (i.e. a method) is invoked to continuously route Item events to the consumer every 0.1 seconds (on all wait(0.1) ...) .   Again, in a real world scenario, this particular portion of a service would be more involved, such as possibly managing the interaction to an external adapter for example.

For terminating a subscription on behalf of the client (on all unSubscribeItems()), we simply terminate the wait listener (a looping construct) set up in startItems.

 

This example is designed to illustrate a few common principles in the Apama EPL:

1. To stress that the fundamentals of structured programming are ever present: modularity, encapsulation and interfaces. Two benefits of Apama's modularity that are noteworthy relate to the plug'n'play idea I mentioned earlier.  a) As typical with modular programming, revised modules can be introduced with little or no impact as long as interfaces are left intact. This is also a truism with the Apama EPL. b) Those revised modules (Monitors) can be replaced in a running system, a shutdown is not required. Furthermore, the modularity also extends beyond the boundary of a single CEP engine to multiple CEP engines in an Event Process Network (EPN) with no actual code change.
2. The event paradigm is inherent in the language not just for application events but for the entire application design center.

In future installments on the Apama EPL, I'll delve into a few more of the language constructs that extend this basic idiom (multiple consumers, spawning, and parallelism).

Once again thanks for reading,
Louie

Embedded Not-so-lightweight ESBs

David Linthicum touched upon a very prevalent problem of technology not living up to its expectations set during the hype curve! He says, "With cloud computing, SOA, or whatever we come up with next, we still need to figure out core issues within our enterprises that no set of technologies or emerging trends can solve." In the service-oriented architecture (SOA) case, we all know that the widely professed use cases of systems "automatically" looking up service registries on the web/intranet, locating the required services, getting the interfaces, picking the right one, and actually making a service request is just a pipe dream! The reality is nowhere close! It is difficult to even find enterprise wide service buses, much less the dynamic on-the-fly service-look up-and-use!

I recently came across a scenario where even a large international bank has SOA infrastructure solving regional problems - and not necessarily as a corporate backbone! I was at this tech discussion with the regional technology head of a large bank, exploring how Apama (Progress' Complex Event Processing (CEP) engine) fits into a biz requirement they have. Specifically, he wanted to understand how Apama works and how it is a good fit for their biz problem. They were trying to build a business performance tracking solution (a la GRC) that involves tracking key biz activities, ensure due process is followed, and then proactively detect potential service-level agreement (SLA) failures so they can be averted. A classic use case for Apama!

Now, when we started to see where the business events information came from and how they could be "wired" to emit the required events into Apama, we suggested that they could just tie into their enterprise service bus (ESB)—we knew they had standardized on one. As we dwell further, it turns out they are using an ESB to solve departmental integration requirements, and there are multiple instances and no central ESB that can help integrate these "departmental integrations".

Now that was a revelation. (I am not complaining because suddenly our opportunity now included Sonic ESB, along with Apama). While we have been reading (and I myself have been blogging and talking) about how SOA failed to live up to its expectations, I still expected at least a large number of banks to have standardized backbones. But not to be! I guess, like in many large enterprises, even here SOA adoption has started purely to solve departmental integration problems—more as localized islands of "integration". And its nothing like the enterprise-wide integration platforms or service bus that we have so much noise about since 2004. This reality check at this bank was a clear validation of this failure! It seemed clear that the technology promise was something, and the actual on the ground realization was something totally different!

Come to think of it, it does seem like a good model for use cases such as BAM or GRC - to have a complete platform that in addition to the CEP engine and dashboard, it also includes a "local/embedded" ESB. Now, embedded is not about being lightweight or smallas it is about being available out of the box when you get the platform. In this case, the ESB is available along with CEP, well integrated - so no surprises as you start using it. And the purpose is to "wire" exchanges from existing ESBs or applications in the enterprise to the CEP environment.

Not a bad idea at all! Embedded ESBs!

Chapter 4: The Data Integration Dialogues (Part 1) is Now Available

Chapter 4 of The Semantic Dialogues, The Data Integration Dialogues (Part I), is available now. Here's an excerpt: "There's a new kid on the block, and Patrice is shaking things up. She wants to write off National Networks' investment in an enterprise data model she hasn't even seen, and replace it with an industry standard that most of the team is unfamiliar with. Cliff is openly skeptical about the upside to standardization. Naturally he's attached to his previous work, but he's also worried about adopting a complex model that looks a lot different from his. Will Jerry's vote for a do-over prevail? "

Click here to read Chapter 4, The Data Integration Dialogues (Part I), today. And if you’re enjoying The Semantic Dialogues, be sure to share it with a colleague.

The Semantic Dialogues tells the story of how National Networks, a fictional telecommunications service provider, pursued and achieved data interoperability within their SOA infrastructure. The story begins with an urgent request from Operations for a fix to a problem that's keeping them from billing for new services. Assured that extensions can be made to the common data model (the SID) for the COTS billing system that's causing the headache, the problem is resolved in short order. The story then goes back a year to a time when the lack of a data interoperability layer made similar problems seem insurmountable and a tiger team is formed to put in place an OSS/BSS architecture that will change the way the CSP does business.

There's a Dilbert Cartoon in Here Somewhere...

I'm reviewing a customer case study, and there was a comment from a sales guy that I'd like to share. It brought up a funny visual for me. Did I mention, this came from a sales guy?

What about the benefit of knowing you have a problem at all? I think this is a big benefit. Finding out you have a problem from your users is embarrassing and not error proof. They will start reporting everything as your problem. [...rambling deleted...] having evidence to avoid the root cause effort altogether seems meaningful. It is also a point I have heard from other customers. It is rewarding to be in a position to say, "Mr. End User, the systems are performing just fine. Are you sure you actually hit the 'submit' button?"

I know, sounds like utopia. Just wait 'til you see the case study. Huge returns and cost improvements, all validated by a third party, and currently in production by the reference customer.

And, did I mention, it was a sales guy who pointed this out? A really smart one got through, call in the hounds. ;-)

If you are interested in seeing the case study once it's ready for distribution, email me and I'll email it to you.

Sun host Developers Workshop of complementary technology for low latency

Last month I was invited to speak at a Developers Workshop sponsored by Sun Microsystems on building low-latency trading applications. I had a 25 minute time slot to fill with the goal of educating an audience of 110 architects, engineers and technical managers from Wall Street on CEP and Apama's vision of it.  I'm usually able to sequester an audience for a lot longer than that to give them my perspective and since I tend to ramble just a bit, it was a tall order to me to whittle it down to this shorter time slot. 

To go one step further, I also did a demonstration of the Apama Algorithmic Trading Accelerator (ATA) and our graphical modeling tool, Event Modeler^TM. So I had to move fast to accomplish this.

Since this was a Sun sponsored event, there were a number of sessions devoted to Solaris and Sun hardware. Sun has done some great work with the Solaris operating system to leverage multi-core processors for scaling low-latency applications. Still, you need ample knowledge and expertise to be able to fine-tune the OS to unlock that capability. There were a few demonstrations of how, using a few of the Solaris OS command tools you can better apportion  processor cycles to starving applications to achieve tremendous gains in performance. One has to be quite scholarly in Solaris systems management so one does not shoot thyself in the foot.

Besides myself representing Apama and CEP, I thought Sun did a great job of bringing together a group of complementary vendors that touched upon the theme for the workshop - low latency. Just to highlight a few... Patrick May from Gigaspaces discussed distributed cache technology and  Jitesh Ghai from 29West described the benefits of low-latency messaging to the trading infrastructure.  I've always considered both of these technologies very complimentary to CEP.  Among many other uses, distributed caching engines provide a basis for a recoverability model for CEP applications. Low-latency messaging brings new possibilities for architecting CEP applications in a distributed model.  

As for me, I presented a number of CEP themes in my talk:

1) Drivers for CEP Adoption. 

Fitting with the theme of the workshop, the drivers for CEP adoption are the increasing demands for low-latency applications. It's the race to the micro-second on Wall Street whether we like it or not. Additionally, the need for rapid development and deployment of complex applications is pushing CEP technology into the hands of the sophisticated business users. Graphical modeling tools empower these astute users, the ill prepared will get left behind.

2) Evolution of CEP  in Capital Markets.

 From single asset broker algos to cross asset, cross border smart order routing to real-time risk and surveillance. CEP is growing and maturing in Cap Markets on numerous fronts.

3) Anatomy of  CEP Architecture.

I presented a macro-level anatomy of the Apama vision of that architecture. There are 4 key components: 1) the CEP engine itself. 2) Integration to the external world. 3) Tools for development and deployment. 4) Visualization across a wide spectrum from richly interactive desktop click trading to widely distributed web-based user interfaces.

Lastly, I want to thank my pal Gareth Smith the Apama Product Manager and slide designer extraordinaire for these slides on the architecture. He's a wiz at putting ideas into compelling visuals.

You can download the slides of my Apama CEP presentation here.

As always thanks for reading,
Louie

Algos Go Global - How Do I Get a Ticket to Brazil?

Carnival, or Carnaval as it's spelled in Portuguese, runs through the weekend of February 21^st this year in Brazil.  Who do you think might be there?  Turns out it’s U.S. President Obama, in the form of thousands of plastic masks.  Wish you could send a likeness of yourself to the festival too?  You might not have made it to Brazil quite yet, but Apama’s algorithms have.

This week, we announced that Ágora Corretora, the largest broker in Brazil, is deploying the Apama platform to support algorithmic trading in both futures and equities on the BM&FBovespa, the leading exchange in Latin America and the world’s third largest in terms of market value.  Ágora is the first broker to develop and deploy unique trading strategies within the Brazilian market.  In the first phase of the project, Ágora has deployed the Apama platform in support of its proprietary trading desk. In the second phase, Ágora will provide access to the Apama platform for its buy-side clients to create, execute, and monitor customized strategies that are unique to their specific trading goals. Ágora represents the first broker in Brazil to offer custom algorithmic trading strategies to its clients.

Developments like these are important outside of Brazil too, as the buy side worldwide increasingly participates in multi-asset, cross-asset, and cross-border trading.  Historically, bringing order flow from market participants in the long-standing algorithmic trading centers such as New York and London to the Latin American markets has not been easy.  Doing so requires exceptional local market expertise and leadership.  As the Brazilian capital markets continue to become more strategic on the world stage, it is important for funds with expansion goals to see alignments of leaders.  In this case, the leading Brazilian broker has chosen to trade on Latin America's leading futures and equities exchange with the world’s leading algorithmic trading platform.

Maybe next year around this time, my customer visits will take me to Sao Paolo.  It will be interesting to see how trading firms have been able to innovative over the course of time on a flexible, open platform.  And then, a quick detour to Rio for some rest and relaxation. Maybe I’ll watch a parade.  No Obama mask required.  One more pin on the world map, with customers from New York to the Bank of China International.  How far can your trading platform take you?

-Dan

The Enterprise Architecture Dialogues - Chapter 3 of The Semantic Dialogues

Chapter 3 of The Semantic Dialogues, The Enterprise Architecture Dialogues, is available now. Here's an excerpt: "Sanjay believes that enterprise architecture is the solution, and he takes a hard line on forcing it into the enterprise. In his view, all they need to do is complete the architecture with a data strategy. But Jerry convinces him to see things from the business users' perspective, and proposes some radical changes to the way they use applications. Mick sees an opportunity to make the business happy and fix IT's reputation. Are they all drinking Kool-aid?"

Click here to read Chapter 3, The Enterprise Architecture Dialogues, today. And if you’re enjoying The Semantic Dialogues, be sure to share it with a colleague.

The Semantic Dialogues tells the story of how National Networks, a fictional telecommunications service provider, pursued and achieved data interoperability within their SOA infrastructure. The story begins with an urgent request from Operations for a fix to a problem that's keeping them from billing for new services. Assured that extensions can be made to the common data model (the SID) for the COTS billing system that's causing the headache, the problem is resolved in short order. The story then goes back a year to a time when the lack of a data interoperability layer made similar problems seem insurmountable and a tiger team is formed to put in place an OSS/BSS architecture that will change the way the CSP does business.

Report from Lisbon

We have just wrapped up the latest TM Forum Team Action Week in Lisbon.  This semiannual event is unusual as conferences go – no exhibits, no keynote addresses, no marketing – just a couple of hundred people with a common interest in creating, improving and using standards and frameworks. 

It’s something that everyone in the ICE (information, communications, and entertainment) sector should experience at least once.  You can spend the whole week with a single framework or interface team, or sample the ongoing work of a variety of projects, including the key planning sessions for upcoming Catalyst projects that will be demonstrated at Management World a few months later. And it’s not for geeks only – there are as many debates about business requirements as there are about interface specifications. 

Given the state of the world economy, it was reassuring to see the high overall turnout and the record number of first-time attendees.  Even better, people were engaged and enthusiastic.  Meetings of the teams driving TM Forum business transformation initiatives like Cable, Content Encounter, Customer Experience, Revenue Management, and Service Quality attracted respectable crowds and made visible progress towards their goals.  I found similar energy when I participated in meetings of the Industry Group Advisory Council (IGAC) and the Defense Interest Group. The number of liaisons with other standards organizations under IGAC has increased steadily, and this week saw a full meeting table and more active projects than I can recall over the last eight years. The Defense Interest Group meeting was also packed, and the discussion for its catalyst project spilled over into the break periods.

I spent most of the week representing the Information Framework (SID) in cross-team meetings with the Architecture Harmonization, Application Framework (TAM), Process Framework (eTOM), Service Delivery Framework, and TM Forum Interface Program groups.  The meetings were productive, and reflected the Forum’s emphasis on getting value from the synergy that exists between the Frameworks and across standards organizations. As one example among many, inspired by the recent groundbreaking work by the Process Framework team in partnership with itSMF to map the eTOM to ITIL processes, the Information Framework team has begun a study to harmonize the SID with ITIL’s Configuration Management Database (CMDB). We plan to release a draft technical report by Management World in May, and I’m optimistic that the mapping we develop will find its way into demonstrations as well.

And if you’ve never attended, think about catching the next Team Action Week this summer – it’s already scheduled and well worth the investment.

From Boston to Davos to Cross-Asset Trading

Cold, snow, and crossing boundaries of all kinds turned out to be the theme for the week of January 26th.  One such boundary crossing is cross-asset trading, where counter-party risk can leave traders out in the cold, fast. Cross-asset trading, especially including multi-legged trades with multiple counterparties, produces very high risk and complicated risk calculations.  Multi-legged trading requires constant data updates to monitor each trade’s lifecycle and the firm’s corresponding risk exposure.

Dealing With Technology landed on subscriber’s desks on January 26^th, with a special report on “Laying the Cornerstone for Cross-Asset Trading.”  In the report, business leaders including John Bates, founder and general manager of Progress Apama, discussed the common hurdles in deploying cross-asset trading platforms.  I thought about those deployments as I crossed a number of boundaries myself.

I attended the annual Progress Software Analyst Day event in cold, snowy, Boston on the 29th.  A global representation of customers spoke to industry analysts about their use of Progress products like Apama to solve business problems.  We were far away from the World Economic Forum in Davos, Switzerland, also running the same week, where news coverage featured the cold, the snow, global economic crisis, and an admission price of over USD 50,000 per person.  CNBC interviewed Thomas L. Friedman, who talked about investment in technology and business innovating its way out of crisis. 

The DWT special report participants spoke more accessibly about how close capital markets firms are to deploying truly cross-asset trading platforms with advanced capabilities such as multi-leg trades.  But the Apama consultants operate where the rubber really meets the road.  Some of them had come to the cold, snowy, greater Boston area from around the world for a week’s work too.  I visited some of their meetings for a detailed discussion of how Apama is used in the field.  Here are some real world experiences that they have had, helping customers create cross-asset trading applications with Apama:

• Trading options against futures
• Trading cash treasuries against treasury futures
• Algorithmic FX spot trading against FX futures
• Trading cross-border listed equities with a spot FX leg
• Adding FX futures as an asset class to an existing FX aggregation deployment
• Trading exchange-traded spreads that consist of two or more legs

Friedman’s comments in the CNBC/Davos interview were more around broad notions of innovation - like new energy technologies - than cross-asset trading, but taken generally I have to agree.  There’s innovation happening in our industry every day, and this activity will be an essential part of economic recovery.  I just hope the jet-set attendees found Davos worth the price.

-Dan

Progress, A Technology Company's Company

Last Thursday was analyst day at Progress Software, and while I wasn't there, I followed some of the activity on Twitter. I'd like to share a bit about what it's like to be at a product company, and some of the challenges we face getting our message(s) out - instead my usual rant about SOA.

WRT the conference, the tweet of the day was @rwang0's tidbit: "Did you know that Epicor, QAD, Infor, Consona, Sage, and others are using Progress technology in the back end?"

@neilwd's answer was: "@djbressler re: your @rwang0 RT - no I didn't know that - and that's part of the problem, perhaps?"

Perhaps? Neil's apparently an understated kind of guy.

I'll add a little nugget of my own. Can you guess how many SaaS customers we have? A few thousand. (Sorry I can't find the exact number but I think my point is made.)

Interestingly, @neilwd was right on the ball earlier when Judith said, "progress software - no differentiation" and he responded "jhurwitz v interesting Progress comment - I think Progress is a bit like Unilever - top-level brand is vanilla, sub-brands have chops".

We'd love for the Progress brand to have some chops, and we're trying but it's not trivial. To illustrate this point, and the challenge we face bringing all our brands together, here's a little visual riddle:



How many times do you see Progress on this screen shot? (Answer below)

Coincidentally to analyst day, I was doing some research in preparation for my upcoming presentation at the Cloud Computing Expo in NYC. When I saw this page I just had to email it to our new CMO, Gary Conway, to illustrate the problem we have.

No one knows who or what Progress is. And in part, it's our fault, but only in part.

You see, Progress is mentioned four times on that page but you'd never know it. Granted, there's one double-mention but still four times on one page, you'd think people would of heard of us.

I'll pat myself on the back for a second. Last year we had some moderated management meetings to tease out what we all thought Progress' core strengths were. My belief is that Progress is a technology company's company - that our core competency is creating software that, as Neil pointed out on Twitter after analyst day, others embed in solutions of their own. Our established network of partners is more valuable than gold, and having traveled the world talking to them, they really like us. Oh, and they're committed to our technology across the board. The exciting thing is that we're starting to realize that core competency in the Actional team here with our relationship with Software AG and their release of WebMethods Insight. It's amazing how when you do what you do well, it works well!

I'll say one last thing before giving you the answer to the riddle, since I suspect there are some important people at my company who might read this.

I'm a bit disappointed that we could have an analyst day without a social presence. Not a blog post about it, not a tweet, not a single conversation. Granted a lot of what was said is confidential but we've gotta be part of the conversation and we missed a good opportunity to do so.

Answer: The four places we're mentioned on the screen shot...

1. "Progress Software to Present at Cloud Conference" (top center bullet #2)
2. "FUSE on Demand: Webinars on Open Source Integration" (sponsored bullet top center)
3. "SOA Presentation, SOA, WOA, and Cloud Computing: The Frontier for Data Services" (top right)
4. "FUSE on Demand: Webinars on Open Source Integration" (bottom left)

DataDirect is what some would say is the core of Progress, FUSE is our open source ESB acquired with IONA. And, as mentioned above, I'll be presenting at Cloud Computing Expo in NYC. I'd love to meet you and have a conversation. Of course, you don't have to wait until then.

2 April 2009, @NeilWD has blogged his response/feelings about this topic. It makes for a good read. - db