Business Making Progress

In the words of Mark Allen, founder of Corticon

Corticon founder Dr. Mark Allen shares the full story behind rules management innovation:  

The idea for Corticon started back in 1995, during my medical training at UCLA.  As a part of government-funded research projects, I built rule-based systems to automate clinical decisions, such as what diagnostic tests to order, or treatments to render.  My research proved that physicians using these systems practiced a much higher quality of care at a lower cost (they ordered more of the appropriate tests and treatments, but far fewer of the inappropriate ones).  And, they were twice as fast in the encounter.  I was sold.  This was the future of medicine.

The problem was that the systems were prohibitively difficult to build and maintain.  Using the best available software development technologies, and a great team of programmers, the systems took years to develop.  Worse still, as soon as we finished development, the guidelines would change.  Even simple changes would take weeks to code, and would often break our systems.  I became very interested in this problem, and how to solve it.

I discovered ways to more easily visualize the logic as sets of interrelated business rules.  This provided a common language between subject matter experts, who defined the business rules, and programmers, who implemented the rules.  Ultimately, this helped accelerate application development and change cycles, and ensure the accuracy of the logic.

I saw the opportunity to transform not just healthcare delivery, but also decision-making within other industries.  In early 2000, I joined forces with Pedram Abrari, an expert in Enterprise Java, XML and AI technologies. He brought deep expertise in developing rule-based systems in financial services and HR applications as well as innovative ideas to incorporate into our product. Our founding team of expert engineers began building the Corticon solution, and today, Corticon products are used to power billions of decisions every day in diverse industries and applications.

Now, as part of Progress’ extensive BPM, BI and CEP offerings, Corticon BRMS has the ability to expand into new areas of decision management and responsive actions. We are excited to begin the next chapter of business agility and responsiveness with Progress Software. 

If you have questions or comments, feel free to contact me at maallen@progress.com. If you’re an existing Corticon customer, you can continue to use your existing Corticon contacts for support, professional services and sales, and if you’d like more information about our solutions, please contact a Progress/Corticon sales person through www.corticon.com.

 

Our 2012 Predictions: What to expect in capital markets

What will we see for capital markets in 2012?  The countdown to 2012 has begun. On the capital markets horizon is a great deal of change – no surprise to those following this year’s rollercoaster of rogue algorithms and regulation tension. So with no further ado, here are our capital markets predictions for 2012:

 1. Billion Dollar Blunder. At least one financial institution will take a billion dollar (or more) hit when a rogue algorithm goes wild. The algo will go into an infinite loop, taking on an irreversible and un-hedged position, which cannot be shut down. Losses will challenge those by human rogue traders, which banks and financial institutions will prevent from happening next year.

2. Occupy HFT. The public, government and regulators will start the "Occupy HFT" movement -- a popular uprising against the ultimate elite of those making money in this climate. Despite immense financial industry pressure, regulators in both the US and the EU will be panicked by investor and political disapproval of HFT and will rein it in with draconian rules and controls.

3. SEFs Spur Splash Crash. Swaps execution facilities (SEFs) will revolutionize OTC derivatives trading, enabling them to be traded electronically. This, in turn, will lead to increased risk of a cross-asset class swaps "splash crash" which will confound regulators, who have little understanding of these markets.

4. Global Regulation Rocks. Countries will finally realize that regulatory harmonization is a good thing and that individual self-interest is not. Banks and financial services firms will realize that they need to think like regulators, taking control of internal surveillance and compliance before regulators make them do it.

5. RICs Get Smarter. The RICs in BRICs are getting smart order routing and gearing up for an increase in algorithmic trading. This, coupled with looser regulations, will begin to attract regulatory arbitrageurs and Volcker Rule escapees.

6. The Wild East. The West's supremacy in financial markets will further decline as new trading regulations - the Volcker Rule in the US and MiFID in Europe - create a surge of regulatory arbitrage favoring more lightly regulated geographies such as Russia and China. Wall Street and the City of London will lose human and financial capital as a result.

7. Financial Terrorism. An exchange or trading destination will be hacked by financial terrorists intent on manipulating markets for political gain. This will lead exchanges and ECNs to add more stringent monitoring and market surveillance capabilities.

8. Head in the Clouds. Explosive growth in foreign exchange trading and SEFs means that participating firms will require complex hosted solutions. Even the smallest FX broker needs aggregation and pricing services which require a big technology footprint. SEFs present new challenges as swaps markets attract algorithms and require surveillance.

9. Crime & Punishment. Regulators are cracking down hard on financial fraud and market manipulation and they will bring in some big fish in 2012. Prosecutions and punishments will increase in size and in impact.

There you have it – nine predictions for capital markets in 2012. What are your thoughts on these predictions, and have we missed any? Comment below, or tell us on Twitter at @DrJohnBates or @ProgressSW.

Can market surveillance help to keep traders on track?

"According to TABB Group new compliance costs are indicated at between 512 and 732 million euro, with ongoing costs between 312 and 586 million euros.  But while regulators are still determining what regulation will look like, the need for market surveillance is undiminished. Traders made about 13.3 billion euros ($18.2 billion) from market manipulation and insider dealing on EU equity markets in 2010, according to an EU commission study.  With some arguing that firms can only do so much to survey markets themselves as trades cross multiple brokers and gateways, the panel discussed the need for fragmented market data to be brought together in a consolidated tape and surveillance performed at an aggregate market-wide level."

This is just an excerpt from a recent post to our Event Processing Blog by Richard Bentley, VP Capital Markets, Progress Software. Can market surveillance help to keep traders on track? Read the entire post.

What is RPM?

This week Progress Software end users and partners worldwide gathered in Boston to get some actionable insights that will help them lower costs, raise efficiency, improve customer experiences, and drive revenue. Our attendees have the ability to chose from over 100 sessions designed specifically for business and IT professionals. One of our breakouts is dedicated to Responsive Process Management (RPM) which is a topic we’ve been talking about a lot over the past year. The introduction of RPM has stimulated discussions around our hallways and within the industry on what RPM exactly is…

• Is RPM the Next-generation BPM?
• Is RPM a totally new and different concept?

WHAT DO YOU THINK?

Below are two articles written by Daniel Schlosky. Dan is a technology writer with more than two decades of experience writing for companies like Sun Microsystems, Qualcomm, Broadcom, and Western Digital. He’s also written for publications such as Datamation, Silicon Strategies, and in-house publications for Hewlett-Packard and Texas Instruments.

Take a minute to look at both perspectives and TELL US WHAT YOU THINK!

To share your thoughts, click the Comments link below.

---

BPM Is Still Evolving

In the early 1990s, the first Business Process Management (BPM) systems could track processes with only limited scaling and relatively few users. Adequate for tracking accounting or claims processes, they were not yet up to handling an entire company’s purchasing system that services tens of thousands of people. Since then, BPM has evolved into much more powerful and mature systems. They all now include process engines, process modeling, asset repositories, etc. Current large, robust enterprise BPM systems have great complexity and functionality, while web-based departmental BPM systems are usually simpler but still highly proficient.

Today’s BPM systems are extremely powerful and capable. They focus mainly on cutting costs by making companies more efficient through iterative, continuously improving processes. While BPM systems have come quite far, they are still advancing. Several new trends are making them even more powerful and useful.  

One new trend that goes beyond cost cutting is finding new revenue sources. For example, Progress BPM enabled an airline to expand its revenue stream by building an application to connect to alliance partner airlines. The customer, after paying a small insurance fee, is assured that if a flight cancellation occurs, the airline will automatically rebook him/her on the next alliance partner’s flight. It would also automatically change hotel reservations, cancel and rebook rental cars, etc. Happy customers are now saved from the hassle of rescheduling everything themselves – and the airline adds to its bottom line.

A second important trend is including complex event processing in BPM systems. With this capability, managers can monitor streams of activities such as stock market transactions, airline scheduling, and communications ordering systems. One basically searches for patterns and anomalies within those patterns. Although some other BPM systems may also deal with events, Progress BPM, when combined with the company’s Apama business event processing (BEP) platform, sets it apart in both power and performance.

A third new trend is the mobility enablement of BPM systems. These systems can now interconnect with intelligent handheld devices to speed up handling tasks with greater user convenience. For a BPMS process tracking purchase requests and approvals, managers can now actually respond to them on their iPods or blackberries without having to fully log onto financial applications on their laptops. This new mobile capability both simplifies the approval process and saves time.

Yet another significant new trend is enabling greater collaboration, especially in process modeling. Coupled with the new mobility, this is an even stronger improvement. The objective is to have managers in different locations review and provide input for the process model and simulation. Having this collaborative capability in the process planning stages is a major benefit.

These four innovations of the latest BPM systems – finding new revenue sources, events processing, enabling mobility and facilitating collaboration – are the leading new trends in the ongoing evolution of BPM. 

 

The Progress® RPM Suite – Is It Really About BPM?

What exactly is Responsive Process Management (RPM)? Is it a natural extension of BPM, or a totally new and different concept? Analysts and customers generally agree that RPM is an evolution of high-end BPM Gartner talks about Intelligent Business Operations as a development from BPM for responsive operations. Forrester refers to Business Transformation and IDC to Decision Management as expansions of BPM. Others, mainly business rather than technology professionals, see RPM as a completely new domain, one with greater ROI and faster time to value.

A recent Vanson Bourne independent research study found that most businesses are unable to respond to market or customer changes quickly enough to be competitive. RPM triumphs over this challenge with a very high level of operational responsiveness – the ability to sense and respond to changing conditions and customer interactions as they occur. If not totally new, RPM raises its responses to a whole new level of power and possibility. Savvion BPM is an integral part of Progress’ RPM Suite, but so are the company’s Apama business event processing (BEP) platform and Actional business transaction management (BTM) solution. To cap these three systems, the Suite adds its new groundbreaking Progress Control Tower, a first-of-its kind interactive business control panel that ties the three systems together for unprecedented responsiveness capabilities.

The Control Tower is the central RPM control point for

• Modeling and simulating processes
• Automatic documenting of processes
• Drilling down and analyzing information
• Cross-collaborating with team members
• Taking corrective steps as needed
• Continuously improving processes
• Reducing risk

So with all this said, which is it – a natural evolution of BPM or a totally new concept? Convincing arguments can be made for each view. If it’s an extension of BPM, it goes much further by adding real-time, end-to-end visibility into events and applications; enabling immediate response to situations when they arise; allowing users to capitalize on opportunities; and empowering business users with greater control for continuous business improvement.

What do you think? To share your thoughts, click the Comments link below.

 

Do you know Dr. John Bates?

If you don’t, get to know him. Though not a native of Massachusetts, it’s safe to say, “he’s wicked smaaht”. Not only is he smart, he’s really a genuine, nice person.  Recently, Dr. John Bates was named as one of the Top 10 Innovators of the Decade for Capital Markets, in which he has an extensive background. He helped pioneer new techniques in algorithmic and high frequency trading, real-time risk, and market surveillance. He was also a co-founder of Apama, a complex event processing (CEP) technology provider that Progress Software acquired in 2005.

Last year John became a member of the newly established Technology Advisory Committee (TAC) for the US Commodity Futures Trading Commission (CFTC). The CFTC is an independent agency with the mandate to regulate commodity futures and options markets in the United States. Most recently, he joined the blog roll over at the Huffington Post – his most recent post is From Icebergs to Autos, Effects of the Japan Earthquake Are Long-Lasting.

If you haven’t already, get to know John. I promise you, he’s a man worth knowing professionally, and if you are lucky, personally.

• Follow him on Twitter (@drjohnbates)
• See if he’ll confirm you as a friend on Facebook
• Watch him on YouTube
• Read what he has to say on our Apama and Business Making Progress blogs
• Listen to his CNBC interview Is a 'Splash Crash' Next?
• Check out his Executive Profile on Boston Business Journal

The Travel Technology Revolution and What It Means For You

By Joshua Norrid, Industry VP, Travel and Leisure, Progress Software

It was a pleasure to attend the SITA Air Transport IT Summit in Brussels and to see more than 300 delegates taking part in a lively, energetic discussion on a wide range of issues affecting the Travel and Leisure industry. Interestingly, of all the topics of conversation we saw during the two-day summit, the one that delegates kept coming back to was the extent to which technology was helping to bring them closer to their customers. But how prevalent is the travel technology revolution in the airline industry – and can it really add value?

It’s clear that the industry today finds itself is in a period of significant transition. With more congestion on the ground, and more complicated systems and processes in place, there’s an increased likelihood of things going wrong, meaning that for many, irregular operations are increasingly becoming the new standard. As a result, more organizations are turning to technology to help them predict these irregularities, manage contingencies and add value to their customers.

It’s also worth bearing in mind that today’s customers are very different to those we saw five or ten years ago. Advances in consumer technology in recent years mean that we’ve seen sweeping changes in customer behaviour during the same period, which brings its own set of challenges. Today’s passengers are a new breed of technology-savvy consumer who are ready, willing and able to tell people what is happening at all stages of their journey, using their smartphone or other mobile devices – particularly if they feel something is going wrong.

This, in turn, means that airlines must work even harder to connect the dots for their customers, with many turning to complex event processing (CEP) software which allows them to keep track of everything from reservations to the status of baggage as they strive to be truly responsive to the needs of their passengers.

Another key theme at the summit was the willingness to embrace cloud computing as a solution. In the past, concerns over the feasibility of cloud solutions have dominated, but this year was different. Indeed, most of the people I spoke to seemed to agree that by owning 100% of their assets, they had been missing a significant number of opportunities. As a result of coming to this conclusion, many had been entering into serious discussions around investing in cloud solutions, with some building the infrastructure required to deliver reservation and departure information as well as enhanced customer service across this platform.

Of course, while concerns remain about speed and security of cloud services, it’s perhaps too much of a stretch to argue that widespread adoption of this technology is imminent.  But what is clear is the extent to which the industry is waking up to the fact that cloud and other technologies can benefit and enable their customers.

In the twenty-first century, the technology-enabled customer is king, and those in the industry are having to work harder than ever to give them the information they need, and to provide solutions which will increase the overall quality of service.  The role that technology can play in bridging this gap should not be overestimated – isn’t it time you asked how prepared your organization is for the technological revolution?

Comment to this post and tell us what you think!

How Software Can Help Airlines to Keep All Their Plates Spinning

By Joshua Norrid, Industry VP, Travel and Leisure, Progress Software

I often think that managing an airline in today’s climate can be a bit like trying to keep several plates spinning at once – one mistake, and all your hard work can result in a costly, embarrassing mess. Over the last 12 months, we’ve heard a great deal from airlines which are finding themselves having to make some difficult decisions about how they approach the road ahead. Increased costs across the board have seen airlines of all shapes and sizes slashing their budgets as they look for the best way to cut costs without reducing overall efficiency.

The situation has now become so drastic that traditional industry growth has been put on hold by the majority of airlines, which are instead looking for ways to keep their heads above water by releasing funds from areas such as maintenance and distribution. So what are the causes of these increased costs? As we begin to emerge from the shadow of the global economic downturn, why are so many airlines finding it necessary to tighten their belts?

Clearly, there are a number of conflicting economic and political factors driving this need. Perhaps the most significant of these is that the cost of fuel has rocketed to as much as $100 a barrel, and as much as 31 per cent year-on-year in some territories. This price increase has had an obvious knock-on effect for the airlines industry, many of which have been faced with a straight choice between passing this increase onto their customers or trimming overheads.

Other factors, including an increase in political hotspots around the globe, are reducing the number of flights leaving each day - adding to the pressure. Either way, it’s becoming increasingly clear that airlines are facing an uphill struggle, as they fight tooth and nail to hold on to market share, while trying to ensure that customer satisfaction is not affected by their drive to reduce overall operational costs.

Perhaps this is where software developers can play a part? By allowing airlines to view business-critical information in real time, complex event processing software can help to establish which areas require greater resources, and which can be trimmed as and when circumstances dictate. This will allow them to view the performance of their entire operation in real time, and make better, more informed decisions. This software could be one way for them to avoid making potentially costly mistakes – and keep all of their plates spinning!

An Algorithmic Trading and Market Surveillance Wrap Up

Our Capital Markets and Progress Apama teams have been BUSY! Today many of them are recovering from a busy week at the SIFMA Financial Services Technology Expo in New York City.

In addition to lots of greeting and Tweeting, Dan Hubscher even had some time to post a couple blog posts:

• Tickets Please: Technology to Keep You on the Train
• Keeping the Train on Track

And while Dan helped man the floor, our VP of Corporate Communications, John Stewart, worked to get 3 press releases out onto the BusinessWire, including:

• Progress Software and Activ Financial Introduce New Market Data Adapter for Apama Capital Markets Foundation
• Progress Software Survey Reveals the Realities of Regulation, Monitoring and Algorithms on the OTC Derivatives Market
• Progress Software Launches Enhanced Market Surveillance and Monitoring Solution

Not only that, our own Dr. John Bates was tagged by Wall Street & Technology as one of our "Top 10 Innovators of the Decade for Capital Markets”.

Across the pond and beyond, Dr. Richard Bentley was quoted in the Bobsguide article “Suspect movements in share price fall to an eight-year low”. Dr. Giles Nelson traveled to India to promote Progress’ business in Capital Markets. He shared his thoughts on our complex event processing (apama.typepad.com) blog:

• The rise of algo trading in Asia - first stop Mumbai
• Still big room for growth in Japan

Phew! And, just in case you missed it, we wrapped up posting a 4 Part video series on Financial Regulation and Market Surveillance. Here are the blog posts that provide a brief overview and link to the videos:

• PLUS Stock Exchange Making Progress Around Regulation
• Using technology to prepare for regulatory change: how can market surveillance technologies help?
• Preparing for changes in the regulatory approach
• Cutting through the confusion of financial regulation

 

Dr. Ketabchi Reveals Secret Sauce of BPM

Are you a business or IT person who wants to... Reduce costs? Improve quality? Manage exceptions? Increase revenue? Most likely, it's all of the above.

Dr. M.A. Ketabchi, Chief Strategist at Progress Software, presented at the Gartner BPM Summit in March on how you can increase your business operations IQ.  He explains how you can run your operations more intelligently, and he reveals:

• The “secret sauce” of business process management
• Well-known companies that have benefitted from BPM, and how they did so
• How business event processing (often referred to as Complex Event Processing (CEP)) enhances BPM
• Typical business challenges that require a combination of BPM and BEP
• How Progress® Responsive Process Management (RPM) brings it all together

If you are interested in the replay, click here. It will be available FREE FROM REGISTRATION for the next five business days only. What a deal! ;-)

Hope you enjoy it. And, as always, make sure to share your feedback and comments here.

Building Customer Loyalty in a Churning Market

All businesses suffer from churn – the moving of customers from one service provider to another. As new and innovative services become better understood and more widespread, more suppliers enter the market and so the opportunities for customers to change suppliers increases. Churn is expensive. Recruiting a new customer can cost 5-10 times that of retaining an existing one. So how can technology help in the constant battle to retain customers? 

I’m going to illustrate what can be done by talking about mobile telecommunications – an industry where innovation is rife but where churn is a significant problem.

Mobile communications continues to grow very quickly. According to a recent Cisco survey, mobile data volumes are nearly doubling each year. By 2015 it predicts there will be 7B personal mobile devices globally. Analyst firm Ovum recently reported that in 2010 revenues from mobile data for European mobile operators exceeded that for voice calls for the first time.

Smartphones are completely changing the way that people use the Internet. It’s worth reminding oneself that now, in a pocket device, one has a phone, a camera, email, PDA, mapping with GPS, in some countries a near field payment device and of course access to thousands of applications. Morgan Stanley has predicted that in 2012 shipments of smartphones will, for the first time, exceed those for personal computers. The whole landscape of computing itself is changing.

With all this growth mobile operators should be very happy. Subscriber bases and mobile data volumes are growing. And yet, mobile operators can’t rest easy. Yes, innovation is everywhere, but most of the innovation (at least that visible to end users) isn’t going on in the mobile operators – it’s going on in the phones and the applications. End users are becoming more and more divorced from the particular network they use and, certainly in developed markets, operators primarily compete on two things only – price and coverage. The growth of the mobile Internet is pushing operators to the bottom of the value pile and risks leaving them as faceless utilities. This, in turn, leads to churn, with rates for mobile operators range from 20-40%, meaning that between 20 and 40% of subscribers will, per year, leave a network for another.

Other industries are of course liable to churn too. Insurance is one example – I recently used an online insurance aggregator to find car insurance and, within a few minutes, obtained a rate 20% below that that my current provider was offering. Online retail is another – it’s very easy to move to another retailer that might be offering a lower price on the same product. In general, churn is present wherever a product is a commodity or near commodity and where customer relationships are weak.

Some wireless operators are fighting back by identifying more ways in which they can meaningfully interact with their customers. To take a concrete example, one European telco, a Progress customer, is now continuously monitoring calls from their 30M subscribers to identify patterns of usage that indicate a different tariff would be more suitable for that subscriber. This could be as simple as noticing that the number of bundled monthly minutes used had been exceeded. A text message is then sent to the subscriber suggesting that they move to another tariff that would reduce the cost of calls in future. Time is of the essence. If the subscriber receives an offer soon after placing one of these calls they are far more likely to accept it than if the offer came through many weeks later.

The way that marketing campaigns are run can become a lot more responsive. The mobile operator may decide to run a campaign to, for example, promote a particular tariff it thinks will be of interest to a subset of its subscriber base – those people, for example, who spend more than $100 per month and roam frequently. Sending out offers via text message requires great sensitivity, as no operator wants its customers to feel it’s receiving spam. As the campaign executes results can be monitored in real-time and the target demographic of the campaign can be tightened to achieve a better response rate. Not only does this make the campaign more successful but also those subscribers that, in the end, are not targeted can become the target of a future campaign.

To do this requires a number of things. Firstly, software needs to be in place to allow the millions of subscriber calls to be analysed in real-time – an ideal use case for event processing. Secondly, there need to be tools which allow a marketing team itself, working largely autonomously without IT support, to create, test and dynamically enhance the rules which dictate which subscribers will receive the offer. And finally, positive responses to the offer need to be processed systematically through an order management system.

There are many other examples where responding quickly to subscriber activity can enhance a user’s experience of using a particular mobile operator. As Internet use becomes dominated by mobile, it’s likely that variable costs for data access, particularly where large downloads are concerned, will be introduced. The cost of a download will be calculated dynamically, dependent upon the bandwidth available within a particular cell at a particular time. At initiation of a large download (let’s say greater than 1Mb) the user could be prompted to ask whether he would like to download it at twice the normal bandwidth for another 10c. This would be a dynamic rate, calculated in real-time in response to current activity in the wireless cell and the propensity of the user to accept the rate.

So, what’s the general lesson here? By becoming more responsive to subscribers, mobile networks are increasing their value to customers, improving customer service and so reducing the likelihood of churn. Existing information about customer behaviour is being used but by being able to act on that information immediately they are able to communicate in a much more contextually suitable way so improving response rates and strengthening the customer relationship. All businesses should be looking how to use their operational information to respond to and interact with customers better. Real-time responsiveness to customer behaviour is becoming vital.

Responsiveness in Retail Banking

A general theme for this blog is how organisations are having to become more responsive, whether to customers, competition, regulation or to squeeze further operational efficiencies out of processes. I want to talk about some of the likely areas of innovation in retail and wholesale banking that will make these organisations more responsive.

Regulation is one obvious area where banks, over a multi-year period, are going to have to become more adept at dealing with change. The implications of the Dodd-Frank legislation in the US is still being understood and, in the UK, banks may be facing an almost existential threat if one gives credence to the rhetoric currently coming out from government and regulators.

Regulation and compliance with regulation is only going to become more and more important. This is being recognised by non-banking institutions that work in financial services. Thomson Reuters, for example, recently announced the formation of a Governance, Risk and Compliance division to deliver information solutions to those needing to comply with regulation. Useful certainly, but regulation will always manifest itself in different ways in different banks. The processes that support regulation must be able to change on a frequent basis. Applications set in concrete only allow this to occur slowly and expensively. The method of building applications needs to be responsive itself and needs to focus around dealing with complex events, business rules, workflow and user interfaces, not around programming language coding.

As an example, a bank, now a customer of Progress Software, had to deal with complying with new regulatory requirements in a matter of weeks. This was not just a box-ticking exercise – the regulator wanted visible demonstration of the systems that had been put in place. The bank’s existing application was simply too inflexible to be changed for reasonable cost and in the time required. An alternative approach, based on Business Process Management and event processing technology was selected which allowed the compliance process flow to evolve and change over time in a much more dynamic way and which could be tailored to meet the bank’s own unique combination of circumstances.

Regulation isn’t a nice to have or something aspirational. It’s a must-have – the risk of legal sanction makes it so.

Another focus for this year will be customer experience. Generally speaking, banks have been behind other industries in getting more responsive to customers’ individual needs and how they’re actually interacting with their bank. A recent US survey, by the industry analyst firm AITE, found that only 22% of small businesses were “extremely” satisfied with their bank and that 65% of larger corporates believed that banks did a bad job of understanding their needs.

A personal anecdote will illustrate a wider point. Recently, I moved money from a savings account with my usual bank to one paying a higher rate of interest with a competitor. The first bank finally contacted me about why I wasn’t using my savings account, but only after all the money had exited the account. By then it was simply too late. Banks still haven’t got sophisticated enough about monitoring how their customers are interacting with their accounts, online, by telephone etc., and sensitively interpreting what these actions might mean and responding in a timely fashion. Again, a watchword here is responsiveness.

Customer onboarding, the process of dealing with a customer’s application for a loan, credit card etc., has got some attention, both to improve the customer’s experience but also to increase the operational efficiency of the bank’s processes.  Some banks have now a real-time view of these processes and can determine immediately whether a high-value business loan should be treated as a greater priority than a car loan or whether a number of card applications have been spending too long receiving credit checks. By knowing this kind of information now (rather than often days or weeks hence) is vital to spotting exceptions quickly and re-prioritizing resources.

One of the core services that a bank performs is payments, whether these payments are executed using cards, direct bank transfer, cheque or otherwise. This year might see some significant innovations in payments processing which will require banks to respond in a way they’ve not been forced to previously. A key tipping point may be Apple introducing a near field communications (NFC) chip into the next iPhone, expected to be released about mid-year. This will allow an iPhone to be swiped across a store terminal to pay for goods. This isn’t that new. Particularly in Japan, NFC chips have been built into mobile phones for some time. However, Apple has a current habit of energising markets and it doesn’t just have the iPhone. There are 160m iTunes accounts which already handle payments. Google and Paypal are also considering offering NFC services. The question is, will this really disrupt existing payment systems which still, even in Paypal’s case, rely upon the core payment infrastructure provided by, primarily, banks? We should hope so. Banks have been hopelessly slow at innovating in payments. It took a diktat from the UK regulator to introduce “faster payments” into the UK. Still too many payments take 3 days, which, in today’s world is very slow for any kind of transaction. Cross-border payments in Europe are still slow and expensive.

If new technology does force payments innovation, banks will be forced to keep up by increasing the flexibility and speed of their own infrastructures to deal with, potentially, many more smaller payments. Guess what? They will need to make them more responsive. If they’re not able to move quickly banks may end up simply being sending round the larger aggregated payments resulting from all the transactions occurring using more modern platforms. Cost pressures will also be present. A recent Boston Consulting Group report indicates that payments providers need to reduce costs by between 10% and 35% by 2012 to keep cost-income ratios stable.

Retail and wholesale banking may often move slowly, but, through regulation, customer pressure and new-entrant innovation, expect some significant changes this year. As Chris Skinner, omnipresent observer of all things banking has recently said on his blog, banks will have to learn “how to be real-time nimble in a world of change”. 

(This first appeared as an opinion column in cio.co.uk)

Sharing some winter sun with Progress' application partners

Earlier this week I participated in the 2011 Progress Global Partner Conference which was held in Florida.

This is only the second time the partner conference has been global – previously it was held regionally – and I’m delighted to say that hundreds of representatives from business partners attended from all over the world.

Most of the partners present were what Progress terms application partners – those that have used Progress products to build applications that are then sold to end-users. As always, the sheer diversity of the applications partners create and sell is mind-boggling – from healthcare apps specialising in kidney treatment to point-of-sale retail systems deployed in 25,000 outlets worldwide to location-based content delivery platforms. Progress recognises the many varied achievements of its partners with its very own awards ceremony. The winners can be found here. And, yes, it is a tiny bit like the Golden Globes, although without the acerbic wit of Ricky Gervais.

These partners continue to be incredibly important to Progress Software. Supporting them with product innovation as well as facilitating new ways for them to deploy their applications (for example by testing out their applications in the cloud with Progress Arcade) is a key pillar of Progress’ strategy. Another strategic pillar is Responsive Process Management (RPM), launched by Progress to the market in 2010, and several sessions in the conference were dedicated to explaining how RPM fitted into the partners’ world. Adoption of RPM in the partner community is happening. An example of this is Skyward, school management software supplier, recently announcing their use of Progress’ OpenEdge BPM platform. This puts them on the first step to full RPM adoption.

John Rymer from Forrester Research, the software analysts, also addressed the conference. Amongst other topics, he talked about four big “on-ramps” of new functionality – business process management, analytics, business events, and collaboration. These, he believed, were the most effective ways for software vendors (Progress’ partners in this case) to deliver new functionality fast and will be the key technologies behind many of the next generation software platforms. Forrester’s presence at a Progress partner conference was timely. Recently one of their analysts, Mike Gaultieri, blogged about Java, despite being more popular than ever, being a “dead end” for enterprise application development. He encouraged developers to consider alternatives, including Progress OpenEdge, that offer substantially higher productivity. It was a reminder that OpenEdge remains as relevant as ever and is a great aid in application modernization. Further output on this topic from Forrester is imminent.

All in all, it was a successful, high-energy conference. Many thanks to all the partners who came, and to those that didn’t, please try and make it in 2012!

 

 

Red Flags in Morning, Firms Take Warning

A pattern is emerging within new financial services regulations where regulators and financial services firms deploy monitoring technology to "red flag" potential issues such as risk, position limits, errors and manipulation. The "red flags" raised would then alert the relevant personnel or authorities.

In the case of the Volcker Rule, prohibiting banks from proprietary trading and investing in or sponsoring hedge funds or private equity funds, the authorities would use a three tiered approach (http://tinyurl.com/2bh9ot3).  First "tripwires", such as the length of time a trader holds a position, its size or riskiness, would alert banks’ compliance departments  who would (#2) quiz the trader on the nature of the position. And (#3)regulators that keep inspectors on banks’ premises would see the tripwires and monitor both traders and compliance departments.

Over at the CFTC, regulators are looking at a similar approach to monitoring and controlling position limits on products such as oil and metals with a "points" system that would give the CFTC monthly reports that it could use to red-flag traders with large positions (http://tinyurl.com/2ugbdh6).

The tracking and red flag approach is the latest step in increased monitoring of trading operations with the ability to take response before it’s too late. At Progress, we have been advocating using monitoring and surveillance technology to help catch inside trading and avoid fat fingered trading errors for years. With new regulations, monitoring becomes not only mandated but more complicated. Red flags are likely to be flying all over the place within as little as months, both inside and outside financial services firms, presenting a fine opportunity for our Responsive Process Management software solution.

As the financial services world becomes more compliant, the ability to manage red flags becomes more critical. Every process within a financial services firm must be scrutinized, from trade entry to risk management, to analyse and understand internal and external events. This take sophisticated technology. This is where Progress Software's RPM software fits in. According to technology research firm Ovum: "Unless an organization has already made a significant investment in creating an operational responsiveness solution around best of breed products, it will be worth seriously considering the competitive advantage and improved effectiveness that could be achieved by deploying RPM."

Ovum noted in a Technology Audit note that multiple technologies are required to gain a comprehensive insight and respond more rapidly to changes to the environment. These include: business process management (BPM) to model, implement, and execute the processes; business analytics to determine how effectively the processes are working; complex event processing (CEP) to understand the implications of many streams of internal and external events; business rules management to determine the appropriate actions for a given set of conditions and variables; and visibility into end-to-end transactions to track and audit their progress.

The interrelationships between all of these components and the vast amount of information that has become available must be understood before its impact on processes can be ascertained and appropriate tuning performed. In other words, RPM is the answer.

RPM can monitor an increasing number of information feeds, both within or external to the organization, then apply business policy and governance rules, then automatically tune the  established process or alert a human decision-maker (if necessary) and present him/her with current, relevant information on which to base the most appropriate response.

According to Ovum: "All of these individual capabilities already exist (at different levels of maturity), but the cost and complexity of integrating these into an effective business solution is beyond the means of most organizations. Hence Ovum believes that the requirement identified by Progress represents a genuine market opportunity." Well said. 

 

BPM is good. Continuous Process Improvement is even better.

As businesses strive to achieve operational responsiveness, they need to make sure that business stakeholders and IT have the ability to collectively define business processes and deploy those processes as applications accessible via the Internet. That’s where business process management (BPM) comes in. BPM is core to achieving operational efficiency but it doesn’t just stop at defining and deploying processes, it means continuously improving how those processes are operating. It’s about having the ability to continuously adjust to changes in your business operating environment and community, and to proactively act on changes in government regulations, performance requirements, and technology.

Dr. John Bates, Chief Technology Officer at Progress Software

In Part 6 of our 7 part video series, Continuous Process Improvement, John talks about the how business process management allows businesses to easily replace and/or automate existing – possibly manual – processes. With responsive process management (RPM), however, you can dynamically improve processes and apply event-driven rules that will allow you to respond to potential problems before they occur.

Interested in hearing what industry analysts are saying about operational responsiveness? Read a recent blog post by industry analyst Mike Gualtieri entitled "Java Is A Dead-End For Enterprise App Development". In it he writes, "Progress Software’s responsive process management (RPM) combines the best of BPM and business events to help businesses respond to real-time events and change business processes. This is just a small sampling of the next generation of business application development tools." You may also be interested in the on-demand webinar Building Responsive Enterprises: One Decision at a Time presented by industry analyst James Taylor.

Enjoy past videos in this seven part series:

What Is Operational Responsiveness?
Why Is Operational Responsiveness So Hard To Achieve?
Delivering Operational Responsiveness
Four Types of Business Process Visibility
Immediate Sense and Respond

 

Encouraging Eventful Entrepreneurs

Our Progress Software Summit kicked off this week at the Hudson Hotel in New York City with a keynote address from CNBC’s Larry Kudlow, “From Washington to Wall Street.” Kudlow dove right into the dominant headline-grabber of the day – the mid-term elections. Larry used the elections as a springboard to touch upon his thoughts on economic policy, politics and other current events.

He then hit on several hot button issues of today, including the emergence of the Tea Party and Sarah Palin, his opinions on the oil and banking industries, as well as his thoughts on corporate tax policies and how they impact a key audience – entrepreneurs.

An overarching theme that we took from his speech is that entrepreneurs can learn from successes as well as failures, and this idea is not that different from business processes and event processing. When entrepreneurs launch a new company and it fails, it is not necessarily a disappointment, just as long as that business-person evaluates and applies what occurred. Then, when they take a risk with another launch, they will have learned from the events and processes that they faced the first time around, such as choosing that first employee, purchasing office space or accepting outside funding. These business risk takers can discover new paths and options from these previous situations, analyze them based upon their actions in the past and use this knowledge to catapult their latest business into a success. If such activity is encouraged, entrepreneurs will become responsive to the steps it takes to jumpstart a business, and ultimately turn those letdowns into triumphs.

A big thank you to Larry for taking the time to speak with our attendees and answer their questions after his speech! Follow Larry Kudlow on Twitter.

Sense and Respond to Event Streams in Real Time

Last week we introduced one of the key benefits of  responsive process management (RPM) - real time visibility. We've heard what it means to be operationally responsiveness, why it's so hard to achieve and how we deliver it through the Progress RPM® suite. Today we'll look at another key benefit of RPM – the ability to immediately sense and respond to business events so that you can quickly reveal opportunities, threats or inefficiencies, and take action.

Dr. John Bates, Chief Technology Officer at Progress Software

In Part 5 of our 7 part video series, Immediate Sense and Respond, John talks about the how one of today’s smart technologies, complex event processing (CEP), allows businesses to process event feeds and have the ability to sense and respond to the opportunities, or threats, that occur in real time. A good example of how CEP benefits companies is in fraud prevention.

The best part? You drive. A business control panel will give you the ability to gain real-time visibility into business events, immediately sense and respond to changing conditions, and achieve continuous process improvement. Learn how Agent O applies RPM to tackle credit card fraud in real time.

Interested in hearing what industry analysts are saying about operational responsiveness? Watch the 3-minute teaser, Gain Efficiency. Avoid Risk. Seize Opportunity, by Gartner analyst Roy Schulte, and then download the entire video. You may also be interested in the paper Building Responsive Enterprises: One Decision at a Time written by industry analyst James Taylor.

Enjoy past videos of this seven part series:

What Is Operational Responsiveness?
Why Is Operational Responsiveness So Hard To Achieve?
Delivering Operational Responsiveness
Four Types of Business Process Visibility

 

Fighting payment fraud with Complex Event Processing

Today Progress Software announced that SEB bank in Estonia has deployed the Apama Complex Event Processing (CEP) platform to look for payment fraud.

This is exciting news and the fact that such a production customer has gone public is a great validation of SEB's belief in the system they've built. A lot of banks are somewhat reticent to talk publically about fraud. Everyone knows it goes on of course, but banks usually prefer that this is a quiet fight, going on behind the scenes. SEB on the other hand see their use of CEP to detect fraud as a positive demonstration of their commitment to customers and card issuers to ensure that banking with SEB is as safe as possible.

A few weeks ago I visited SEB in Estonia and was hosted by Ain Rasva, the head of technology, together with the head of the fraud team. One of the key reasons they chose to invest in CEP was rising transaction rates and a realisation that conventional data management architectures were simply not suitable to look for fraudulent patterns in a timely fashion. Yes, transactions could be captured in, say, a database management system, but then queries to determine whether a pattern of fraud was emerging had to be run and re-run at arbitrary time intervals. An event processing approach is simply a better way of doing this job, both performance-wise and conceptually. With CEP, SEB can now detect a potentially fraudulent pattern of card use immediately and then start managing the case, using tools to conduct further analysis and to manage communication with the customer. This can be done now in minutes, rather than the hours it took previously. Patterns of fraud detection are constantly changing and SEB needs to be responsive to this - the fraud detection rules need to change frequently and quickly. With Apama, these fraud rules can be created, modified and tested rapidly, and this can be done largely by the fraud department itself rather than relying upon IT to effect each change.

Card payments continue to change and increase. There were 82B payments in Europe in 2009 and card payments are growing at a CAGR of 12% per year. Europe wide regulation such as the Single European Payment Area will significantly change the types of fraud that banks need to look for. SEB has placed itself in a good position to ensure that payments are conducted as safely as possible.

Delivering Operational Responsiveness

We've already learned that operational responsiveness is more than agility and business process optimization, it’s about plugging decision makers into business events and giving them the tools and information they need to respond to the unexpected, thereby allowing them to capitalize on opportunities, drive greater efficiencies, and reduce risk. We've also learned why it's so hard to achieve. Now, how do we help deliver it?

Dr. John Bates, Senior Vice President, Chief Technology Officer and Head of Corporate Development at Progress Software

Watch Part 3 of our 7 part video series. In this video, Delivering Operational Responsiveness, John explains how we've brought together three proven technologies that help companies achieve operational responsiveness – business transaction assurance, complex event processing (CEP) and business process management (BPM). All these powerful technologies are further enhanced with business rules and analytics.

The best part? You drive. A business control panel will give you the ability to gain real-time visibility into business events, immediately sense and respond to changing conditions, and achieve continuous process improvement.

Read what Wikipedia has to say about Responsive Process Management (RPM) and learn how Agent O applies RPM to tackle credit card fraud in real time. You can also follow Dr. John Bates on Twitter.

Enjoy past videos of this seven part series:

What Is Operational Responsiveness?
Why Is Operational Responsiveness So Hard To Achieve?

 

Are you a sitting duck or one that will respond immediately to threats?

While many organisations are being ‘cautiously optimistic’ about what the future holds, the realities of today’s tough business environment could leave them as sitting ducks, according to Rick Reidy, CEO at Progress Software. They might take consolation that they’re in the same pond, but when interest rates in Japan hit near-zero, banks continue to fail and mistakes can lead to a ‘flash crash’, the pond is not a safe place to be. Businesses may have money, but fear and uncertainty is holding back decision-making – we await further regulation and want to know the consequences of recent government changes.

 
Listening to Rick’s keynote at our UK business summit (#progresswsummit, if you want to follow on twitter), in the impressive surrounding of Chelsea Football Club’s ground, London, it seems most of the audience agrees – it’s not good enough to sit around and wait to see if growth returns, and you cannot grow simply by cutting costs. You have to take control of your own ‘growth agenda’, as Rick put it. Businesses that want to survive the next five years need better visibility, through putting processes in place that enable them to react quickly to meet customer demands, adapt to market changes and take advantage of new opportunities. As Rick has advised, businesses need to act on up to the minute information so that leaders can make decisions based on foresight, not hindsight.
 
If you’re a regular reader of this blog you’ll already know that we call this ‘operational responsiveness’: the ability to sense and respond to customer and market changes so that organisations can move quickly to meet challenges and take advantage of new opportunities. 
 
Rick has talked about what this means in the airline industry: the notion of irregular operations has become a weekly reality as companies face intense market pressure, striking staff and disruption from natural phenomenon. ‘Swivel chair’ communication between operational areas is no longer good enough. To react quickly enough, they need responsive processes in place that can help them maintain services and inform customers, almost as-it-happens. If they don’t, they will face massive fines, lost custom and damaged reputation – risks no company can afford at present.
 
We’ll be hearing more from Gordon Penfold, CTO at British Airways, about their approach to becoming operationally responsive to meet the challenges of today and tomorrow. Watch this space for my take on his talk…

 

Stamford Bridge, here we come

Tomorrow sees Progress Software taking over Stamford Bridge, home ground to the world-famous Chelsea Football Club. We’re not just there to check out the players’ dressing rooms – we are being joined by James Caan, of Dragons' Den fame, as well as the great and the good of the UK business community, to discuss how businesses can start to make decisions based on foresight, not hindsight, in their operations.

Gordon Penfold, Chief Technology Officer at British Airways, will be sharing his insight on ‘operational foresight’, revealing how the organization has set itself up to better deal with the irregular operations that have become a fact of life in the last year. And Mike Gualtieri, senior analyst at Forrester Research, will be sharing his views on where the next wave of truly responsive business management is coming from, and which trends to watch for. And Progress' own Chief Executive Officer, Rick Reidy, will be giving a keynote too.

I'll be there, speaking in one session but also blogging and tweeting from the event. So watch this space for the latest updates.

For those of you attending, I look forward to seeing you there.

www.progresssoftwaresummit.com

 

How Being Complex Makes Transaction Assurance Simpler

I have been seeing an increasing amount on interest in the marriage between Business Transaction Management (BTM) and Complex Event Processing (CEP). On July 29^th Dr Dobbs Journal published an article called Complex Event Processing: IT Liberator or Over-Engineering Hell? This article was about the synergy of BTM and CEP (although I felt it was rather biased towards one company). Also, last week Jean Pierre Garbani at Forrester published this blog in which he discussed the evolution towards BTM and CEP working together.

Business Transaction Management is a rapidly growing area of Application Performance Management (APM). BTM enables users to look into the transaction flows within their business and ensure everything is running as expected. BTM enables problems in transaction flows to be discovered – such as a bottleneck in an important business process. The really appealing aspect of BTM is it can do this without the need to change the applications in the business; BTM can “discover the transaction flows” by tapping non-intrusively into the flows going through application servers, middleware buses, business process management systems and other systems within the environment. Over time, BTM can build up a picture of the environment’s business flows, look inside the transactions and flag up immediately problems that can really hurt the business.  Thus BTM works really well in legacy environments – not just modern SOA environments. And of course it appeals to business executives and operations users – not just IT users.

Complex Event Processing is the ability to correlate events flowing through a business  - to identify patterns in real-time. These patterns might indicate opportunities and/or threats to the business that have just happened, are in the process of happening or are likely to happen right now. Events are occurrences in the business, such as stock market quotes in trading, call data records being generated in communications or packages changing location in logistics. An example of a real-time opportunity is a trading “statistical arbitrage” opportunity – to sell one instrument and buy another at a micro profit; another is the ability to upsell something to a customer who has just purchased an item on their credit card – based on their spending and buying patterns, their location and context. Threats to be detected include risk exceeding a certain key level in a bank or gaming fraud occurring in a casino. This kind of business level visibility and immediate response also appeals to business users as well as IT.

Listening to the descriptions of BTM and CEP, does it sound like there is a little overlap? Well there is some. What BTM is really good at is non-intrusively discovering process endpoints and the events they exchange – and then tracking these events. What CEP is really good at is correlating complex real-time business events in real-time, including arbitrary user-defined patterns, which can evolve over time as the business evolves. So it makes perfect sense to put these capabilities of BTM and CEP together. For BTM this strengthens the real-time correlation and pattern detection capabilities. For CEP this enables discovery of services without the need to do expensive and time-consuming instrumentation of the environment.

At Progress we have two leading products in the BTM and CEP categories: Actional and Apama. We believe that BTM and CEP capabilities are converging for certain business use cases, so as part of our Responsive Process Management (RPM) suite we now provide seamless integration between these capabilities. Of course RPM does much more than that. More on that later!

Information Overload?

Last week I had the wonderful fortune to deliver a webinar with Robin Bloor on the role of events in decision management. This event, as a part of Information Management’s Espresso Shot webinar series, gave us a chance to dig into the market conditions and the potential ways that events can affect decision management by injecting real-time information into the process.

As many of you that might be familiar with my webinars and papers will recall, I’m a strong proponent of complex event processing (CEP) as a significant contributor to the changing landscape of decision management. Let’s face it; even in the intense world of Capital Markets where Apama’s event processing thrives in high-volume, high-velocity scenarios, the result is improved decision management, but in the end it’s all about who makes the right decision first. There is absolutely no reason why that benefit cannot be enjoyed outside of the capital markets space – and in fact it is. And, the good news is that it doesn’t have to be high-volume, high-velocity scenarios. Need to find the needle in the haystack? – event processing will do nicely.

Event Processing has hit the main stream. The recognition by business that events can be harnessed and used to influence decision management is gaining momentum. But, there two things that I think Robin and I eluded to, but probably needed to push a bit more.

1. First, event processing is architecture. Yes, it’s about data. But data that’s moving, transient and very little context. I see a time in the near future where there will be event architects with the responsibility of making sense of an enterprise’s event streams.
2. Second, event processing requires technology. There’s a reason why products such as Apama exist. It’s not easy or possible without them. There’s a lifetime of R&D, experience and, in the case of Apama, patented technology to simplify your job. You’ll spend more time picking off the events that matter most and leave the infrastructure to us.

Whoever makes the right decision first – wins. And, this is something you can plug into what you have now. Go for it.

Replay the webinar >

Evacuate the Dancefloor

I just posted a blog entry on how changes in capital markets regulations may lead to out of work traders joining hedge funds. It may be a culture shock without the big bank budgets - but advanced trading technology is not out of reach.

See the full article here: "Evacuate the Dancefloor".

Algorithmic Terrorism

I just posted a blog on the potential of algorithmic trading terrorism -- can a "denial of service" style attack cripple world markets? See the full posting here:

Blog Post: Algorithmic Terrorism >

Smart - But Is It Smart Enough?

Today Nasdaq group purchased Smarts - a provider of market surveillance. This is an interesting development . Read the full post on our Apama blog :

Smart- But is it Smart Enough >

Information Overload? Three Ways Real-time Information is Changing Decision Management

Upcoming webinar: 26-Aug-2010 at 11am ET
More Info and Registration: http://bit.ly/dyCu0c

JOIN US! In a recent survey by Vanson Bourne, 94% of respondents say that responding to information immediately is critical to their business, yet only 8% currently report information in real time. Why? Because many are unable to filter through the enormous deluge of information coming at them from every angle. This web seminar will discuss 3 ways that real-time information and business events affect your decision-making process. Through real-time visibility and immediate sense and respond, you get the right information at the right time. The result is decision management that is “in the moment” and operational responsiveness that meets the speed of your business.

Hope you can attend: http://bit.ly/dyCu0c

Beware the weight-challenged digits

Fat fingers (or weight-challenged digits - for the more politically correct :-) ) are trading errors that can have catastrophic consequences. And they've been happening a lot recently!!

Read my full blog post here.

India: Big Potential for Algorithmic Trading

I spent last week in India, a country that, by any standards, is growing fast.  Its population has doubled in the last 40 years to 1.2B and economic growth has averaged more than 7% per year since 1997.  It’s projected to grow at more than 8% in 2010. By some measures, India has the 4th biggest economy in the world. 

Progress Software has a significant presence in India. In fact, people-wise, it’s the biggest territory for Progress outside the US with over 350 people. Hyderabad is home to a big development centre and Mumbai (Bombay) has sales, marketing and a professional services team.

The primary purpose of my visit was to support an event Progress organised in Mumbai on Thursday of last week on the subject of algorithmic trading. It was also our first real launch of Progress and Apama, our Complex Event Processing (CEP) platform, into the Indian capital markets. We had a great turnout, with over 100 people turning up. I spoke about what we did in capital markets and then participated in a panel session where I was joined by the CTO of the National Stock Exchange, the biggest in India, a senior director of SEBI, the regulator, and representatives from Nomura and Citigroup. A lively debate ensued.

The use of algorithmic trading is still fairly nascent in India, but I believe it has a big future. I’ll explain why soon, but I’d like first to give some background on the Indian electronic trading market, particularly the equities market, which is the largest.

Watch my Interview on NDTV Profit >

The market: India has several, competing markets for equities, futures and options, commodities and foreign exchange too.  In equities, the biggest turnover markets are run by the National Stock Exchange (NSE) and the Bombay Stock Exchange (BSE), with market shares (in the number of trades) of 74% and 26% respectively. Two more equity exchanges are planning to go live soon – the Delhi Stock Exchange is planning to relaunch and MCX is also currently awaiting a licence to launch. This multi-market model, only recently adopted in Europe for example, has been in place in India for many years.

It was only two years ago that direct market access (DMA) to exchanges was allowed. Although official figures don’t exist, the consensus opinion is that about 5% of volume in equities is traded algorithmically and between 15% and 25% in futures and options. Regulation in India is strong - no exchange allows naked access and the BSE described to me some of the strongest pre-trade risk controls I’ve come across - collateral checks on every order before they are matched. The NSE has throttling controls which imposes a limit on the number of orders a member organisation can submit per second. Members can be suspended from trading intra-day if this is exceeded. The NSE also forces organisations who want to use algorithms to go through an approval process. I’ll say more about this later. Controversially, the NSE will not allow multi-exchange algorithmic strategies so cross-exchange arbitrage and smart-order routing cannot take place. Lastly, a securities transaction tax (STT) is levied on all securities sales.

So, with the above restrictions, why do I think that the Indian market for algorithmic trading has massive potential?

The potential: The Indian market is very big. Surprisingly so to many people. Taking figures from the World Federation of Stock Exchanges (thus I’m not counting trading on alternative equity venues such as European multi-lateral trading facilities), the Indian market, in dollar value, may still be relatively modest – it’s the 10th largest. However, when you look at the number of trades, India’s the 3rd largest market, only beaten by the US and China. The NSE, for example, processes 10 times the number of trades as the London Stock Exchange. So why isn’t more traded in dollar terms? That’s because trade sizes on Indian exchanges are very small. The median figure worldwide is about $10K per trade. The figure in India is about $500 per trade, a 20th of the size. In summary, surely the task of taming the complexity of this number of trades and the orders that go with them is ideal for algorithmic trading to give an edge? To compare to another emerging, “BRIC”, economy, that of Brazil, where the number of firms using Apama has gone from zero to over 20 in as many months, the dollar market size is fairly similar but the number of equity trades in India is 33 times more. The potential in India is therefore enormous.

India is already there in other ways. All exchanges are offering co-location facilities for their members and debate has already moved on to that common in more developed markets on whether this gives certain firms an unfair advantage or not and whether co-location provision should be regulated.

The challenges: There are some difficulties. The STT is seen by some as an inhibitor. However, its effect is offset somewhat by the fact that securities traded on exchange are not subject to capital gains tax.

The NSE process for approving algorithms is more controversial. Firms that want to algorithmically trade must show to the NSE that certain risk safeguards are in place and “demonstrate” the algorithm to the exchange. As the biggest exchange, the NSE wields considerable power and thus its decision to vet algorithms puts a brake on market development. I believe this process to be unsustainable for the following reasons:

1. As the market develops there will simply be too many algorithms for the NSE to deal with in any reasonable timeframe. Yes, India is a low-cost economy, but you need highly trained people to be able to analyse algorithmic trading systems. You can’t simply throw more people at this. Firms will want to change the way algorithms work on a regular basis. They can’t do this, with this process in place.
2. It raises intellectual property issues. Brokers will increasingly object to revealing parts of their algorithms and their clients, who may want to run their alpha seeking algorithms on a broker-supplied co-location facility, will most definitely object.
3. It puts the NSE in an invidious position. Eventually an algo will “pass” the process and then go wrong, perhaps adversely affecting the whole market. The NSE will have to take some of the blame.
4. Competition will force the NSE’s hand. The BSE is trying to aggressively take back market share and other exchanges are launching which will not have these restrictions.

It strikes me that the NSE should spend its efforts into ensuring that it protects itself better. Perhaps a reasonable comparison is a Web site protecting itself from hacking and denial of service attacks. If they can do it, so can an exchange. And it would offer much better protection for the exchange and the market in general.

In conclusion: I’m convinced of the growth potential in India for algorithmic trading. The market is large, the user base is still relatively small and many of the regulatory and technical prerequisites are in place. There are some inhibitors, outlined above, but I don’t think they’ll hold the market back significantly. And finally, why should India not adopt algo trading when so many other, and diverse, markets have?

Progress has its first customers already in India. I look forward to many more.

Defending Against the Algo Pirates

It was an honor to sit on the CFTC Technology Advisory Committee (TAC) last week. The CFTC is the US regulator for derivatives trading – which is a big area. The topic of the first meeting of the TAC was to understand Algorithmic and High Frequency Trading – and its impact on the market. The CFTC is sensibly trying to understand where they should regulate and where they should propose best practices. Obviously the memory of the "flash crash" is still fresh in everyone's mind. One of the topics of interest was that of pirate algorithms that attack other algorithms and try to manipulate the market. I’ve just written a blog posting on it if you’d like to read more:

Full Blog Post: Defending Against the Algo Pirates

CFTC Launches Technology Advisory Committee

Yesterday the CFTC, the regulator in charge of Futures and Options markets, announced a new Technology Advisory Committee (TAC), chaired by the very capable Commissioner Scott O’Malia. Read the complete article.

I am absolutely delighted to be included in the group of experts that the CFTC has called together to form the TAC. I am joined by an extraordinary group of some of the industry's top executives from banks, brokers, trading firms, exchanges and clearing firms as well as some very impressive academics. On Wednesday, July 14th (tomorrow as I write), we will meet to discuss the impact of high frequency and algorithmic trading on the markets, including whether algorithms may be implicated in the May 6th 'flash crash'. From this, we’ll discuss what recommendations we have for regulation of and/or best practices for algorithmic and high frequency trading.

High frequency and algorithmic trading are essential for efficient execution and alpha generation in a complex, multi-asset, fast-moving world. However, there are a number of accusations that have been made against these forms of trading, including that they may aggravate volatility and may even have caused the ‘flash crash’. I believe evidence from the TAC participants will exonerate the accused.

I am hoping that our meetings will result in solutions that not only head-off future ‘flash crashes’, but also help exchanges, banks and brokers to better monitor and police trades. The proactive use of real-time monitoring systems can alert regulators to problems before they become a crisis. Monitoring technology can 'see' major price and volume spikes in particular instruments, how often they happen and maybe even why, and whether a pattern in market behavior caused them. It can also tell how much trading is potentially market abuse, for example, insider trading might be detected by correlating unusual trading incidents with news releases and market movements. (The FSA, for example, thinks that 30% of trading around acquisitions is insider.)

It is now possible to apply high frequency techniques to not just trading – but also to market monitoring, surveillance and pre-trade risk checks – for regulators, exchanges and brokers. The technology is out there (with proven approaches built on next generation platforms such as complex event processing or CEP) and it needn't be expensive. The CFTC's TAC is a positive step in the right direction. I look forward to the meeting and will let you know how it goes! Follow me on Twitter @drjohnbates where I'll Tweet when possible.

Shipping logistics with event processing

On Friday last week I visited Progress customer, Royal Dirkzwager, in the Netherlands. It’s nice to be able to talk about a customer publically - Dirkzwager have been a generous public reference for some time. This was my first visit to their head office in Maassluis, near Rotterdam, and it gave me a new insight into their business. Dirkzwager’s offices are right on the main entrance to the port of Rotterdam and it was great to be in their board room and watch the container ships move by as Paul Wieland, head of IT, took me through their business and what they are using Progress’ products for.

Dirkzwager is in the business of supplying information about shipping movements to the maritime industry. Its customers include the Port of Rotterdam, shipping companies and the logistics companies involved in supporting the ships and handling the goods. Dirkzwager currently supplies information on shipping movements at sea and in and out of ports from Le Havre in France to Denmark. Rotterdam is its epicentre - the biggest port in Europe and one of the biggest in the world, handling around 30,000 ships per year.  

Dirkzwager are users of the Sonic Enterprise Eervice Bus (ESB) and the Apama event processing platform. Sonic is used for delivering information to customers reliably and Apama is used for processing and analyzing the events reporting the positions of ships, up to several thousand per second. Live data is augmented with data from a reference database on tonnage, flag, crew, and lots of other information. Applications range from simple to more complex. One example is: “alert when a ship crosses this line”, to give an official port entry time (important for calculating harbour fees). A more complex example is the calculation of an actual time of arrival, using course, speed, weather conditions and historical information about the ship’s movements. This latter application can bring very significant benefits: on average, 50 companies are involved servicing a ship when it’s in port, from loading or unloading goods to servicing the radar. If these companies can receive more accurate information about a ship’s arrival time, they can optimize their operations resulting in the whole supply chain becoming more efficient – capacity is increased and costs are lowered. Paul gave some insight too into how Dirkzwager had become more efficient by using Apama to automate tasks that were previously done manually and in the process saving substantial amounts of money (the details of which, unfortunately, I can’t share here).

Dirkzwager’s industry is changing. Currently, most ships identify themselves by a VHF-radio system. This is shore based and has a range of approximately 60km. This is changing to a satellite based system which will give more continual and accurate reporting and also give simpler access to ship movements anywhere in the world (approximately 80,000 ships are in transit at any one time worldwide). This gives business expansion opportunities and also challenges when dealing with the new quantities of data that available. Paul Wieland clearly believes that complex event processing (CEP) is an ideal way for Dirkzwager to analyze this data and provide the monitoring of logistics and supply chain processes that they support for customers. Paul’s obviously a visionary – the room was buzzing with ideas. Now his team is experienced with Apama, Paul reported that people’s approach to problem solving had changed – they were thinking naturally about things in an event based way, both architecturally and in terms of data processing.

One might not necessarily think of shipping logistics as being the most innovative of industries, but Dirkzwager has always been at the forefront of technological innovation. Shipping was, and continues to be, vital to the Dutch economy. When the company was founded, in 1872, people used to stand at the opening of the port with binoculars. As a ship was sighted and recognized, someone would mount a horse and rush back to the port to report. Dirkzwager had the first commercial telephone line in the Netherlands. Paul showed me a room where some of the computing and communications equipment used previously by Dirkzwager is stored – examples include a megaphone, semaphore flags, telephones, telex machines and a mini computer. It was a mini museum for the communications industry. The use of event processing software is simply another step in this historical evolution.

What do you do with the drunken trader?

The news that Steven Perkins, (former) oil futures broker in the London office of PVM Oil Futures, has been fined 72,000 pounds ($108,400) by the FSA and banned from working in the industry is no surprise, see article here.

It could have been worse given that the broker, after a few days of heavy drinking, took on a 7.0 million barrel long position on crude oil in the middle of the night. The fine seems miniscule since it cost PVM somewhere in the vicinity of $10 million - after unwinding the $500+ million position.

The surprising thing about this incident is that it happened at all. Perkins was a broker, not a trader. He acted on behalf of traders, placing orders on the Intercontinental Exchange among other places. That he could go into the trading system and sneak through 7.0 million barrels without a customer on the other side is unbelievable.

Heavy drinking is practically a job requirement in the oil industry, my sources tell me, so this kind of thing could be a real issue going forward. As algorithmic trading takes hold in the energy markets, trading may approach the ultra high speeds seen in equities markets.  This is a recipe for super high speed disaster, unless there are proper controls in place - especially if there were a way for the broker or trader in question to enrich himself in the process.

One powerful way to prevent this kind of accident or fraud is through the use of stringent pre-trade risk controls. The benefits of being able to pro-actively monitor trades include catching "fat fingered" errors, preventing trading limits from being breached, and even warning brokers and regulators of potential fraud - all of which cost brokers, traders and regulators money. PVM is a good example of this.

Ultra-low-latency pre-trade risk management can be achieved by brokers without compromising speed of access.  One solution is a low latency "risk firewall" utilizing complex event processing as its core, which can be benchmarked in the low microseconds.  Errors can be caught in real-time, before they can reach the exchange. Heaving that drunken trader right overboard, and his trades into the bin.

Banks & Bullets: Maybe They Dodged One - But Still Needs Some Silver Ones!

By now you’ve probably seen that a deal was reached this morning by the House and Senate on regulation. Some would say it waters down provisions from the tougher Senate bill, limiting rather than prohibiting banks to trade derivatives and invest in hedge funds. This articles describes it as banks “dodging a bullet” http://www.businessweek.com/news/2010-06-25/banks-dodged-a-bullet-as-u-s-congress-dilutes-trading-rules.html

We applaud the superhuman efforts put into the new financial regulation bill by the U.S. Congress and the House of Representatives. However, as I’ve said many times, transparency and consistency are critical to successful regulation in the capital markets. One could be forgiven for fearing that the watering down of the regulations, including the Volcker Rule, may create more havoc rather than increase transparency and consistency.

One thing is for sure – there’s going to be a raising of the priority on handling the complexity and requirement for real-time risk and surveillance within institutions. Risk managers and C-level executives concerned about minimizing risk and maximizing capital will need to view trading positions and limits across the firm, including, if permitted, derivatives that are 'spun out'. Ideally the risks should be aggregated and analyzed, in real-time, giving the ability to detect and prevent “accidents”. Pre-trade risk management will be increasingly important, as firms seek to maintain capital requirements at all times.

A top-down approach to risk where managers can see, in a single view on a dashboard, the risks across all asset class silos has gone from a “nice to have” to high on the wish list – but many still wonder if it actually possible. Continual monitoring of trades in real-time can help to prevent exceeding trading limits, prevent mistakes and catch market abuse.

p.s. Many thanks to all the comments from market practitioners who comment that technology can't solve all the problems for Regulators, Banks and Trading Venues. I completely agree! But we can go a lot further than what happens right now. But of course technology is only one of the approaches. Changes in regulation is another, as is increased transparency, improved reporting (e.g. from fax to real-time data!) etc. etc. 

Rogue Trading Below the Radar

Jerome Kerviel, the trader who allegedly lost Societe Generale nearly 5.0 million euros, went on trial in Paris on Tuesday, June 8th. The bank alleges that Kerviel took "massive fraudulent directional positions" in 2007 and 2008, which were far beyond his trading limits.

It is interesting to note that Kerviel was not only experienced on the trading floor, but he also had a background in middle office risk management technology. It may have been this knowledge that enabled him to manipulate the bank's risk controls and thus escape notice for so long.

Still, it is perplexing that fraud on such a scale can go on without detection for so long, even if Kerviel did have an insider's knowledge of the firm's risk management systems. Internal risk controls are not something that a financial firm can take for granted, left to run unchecked or unchanged for months or years.

The detection of criminal fraud or market abuse is something that must happen in real-time, before any suspicious behaviour has a chance to lose a firm money or to move the market. Pre-trade risk management is paramount, with trading limits specified and checked in real-time. Internal controls should be monitored for possible manipulation, again in real-time. The good news is that technology does exist in the form of real-time market surveillance software from companies can analyse data transactions by the millisecond.

Financial institutions need to start looking inward to improve standards, regardless of current regulation. Otherwise the culture of greed and financial gain at all costs will encourage more and more Kerviels.

HOW MATCH2BLUE MAKES PROGRESS

The challenge match2blue faced… they needed to scale their existing location-aware social networking platform with an event-driven architecture that could support millions of users. They accomplished this by using Progress® Apama® (http://bit.ly/apama-bep). Today match2blue can scale to meet the needs of any organization. They’ve enabled their customers to deliver real-time information to users, and they gained the flexibility to customize solutions – all with faster time-to-market. Learn more about match2blue: http://bit.ly/prgs-match2blue

Tell us how your company is achieving operational responsiveness using an event-driven architecture!

FSA Loses Insider Trading Case - but more to come...

Today’s insider trading cases acquittals in London are a big blow to the FSA, but their ability to detect and prosecute these market abusers cannot be overlooked. Without the technology to detect the trading anomalies, alleged white collar criminals cannot be prosecuted in the first place.

It’s also clear that the FSA is sending a message to the investment community: shape up or be prepared to pay. The £33.32 million ($48.8 million) fine for JPMorgan is the largest in the FSA’s history. 

As the SEC and CFTC in the US looks to adopt market surveillance technology, it will be interesting to see the potential rise in insider trading court cases and the size of fines in the US.

I think we're going to see a lot more of this type of prosecution around the world. The FSA is currently prosecuting 11 people for alleged market abuse.

As you may have read this week (link here -> http://bit.ly/aslFmV), the FSA is using new technology to crack down on potential market abusers. in the UK, the FSA receives 6m-8m transaction reports daily. The FSA will soon even have a system in place that will automatically alert staff to potential abuse in “real time”. Alexander Justham, the FSA’s director of markets, says the use of such “complex event processing” technology will give the FSA “a more proactive, machine-on-machine approach” to surveillance.

 
See the article "FSA Loses Insider-Trading Case" here
http://online.wsj.com/article/SB10001424052748704025304575284033106574578.html?mod=WSJ_hps_LEFTWhatsNews

Encouraging Noises from the SEC

Washington, D.C., May 26, 2010 — The Securities and Exchange Commission today proposed a new rule that would require the self-regulatory organizations (SROs) to establish a consolidated audit trail system that would enable regulators to track information related to trading orders received and executed across the securities markets. http://www.sec.gov/news/press/2010/2010-86.htm

I support the SEC's intention to improve real-time market surveillance and establish a consolidated audit trail system. It's always been a worry that regulators do not have real-time access to all securities market data, but it's good to see that they are about to address the issue.

The SEC estimates it would cost the trading industry $4 billion to implement and $2.1 billion in annual maintenance to establish this system. But the US taxpayer is already fed up with the government using 'their' dollars to bail out banks and shore up an ailing financial system. They are unlikely to cheerfully approve of another government agency getting billions more of their dollars.

However, I'm not sure the solution has to be so costly. The markets have come a long way with standard protocols like FIX but also ways of normalizing heterogeneous protocols in real-time. Time series database technology, to consolidate and retain all the information on quotes, orders and trades, has been scaling up for years to handle massive increases in volumes.  And low latency, high speed, monitoring technology exists and is already in place at some of these destination venues and trading participants.

I would love to see the SEC working together with other regulators, like CFTC to form a committee of market participants, exchanges, ECNs and market practitioners to hammer out a cost-effective and timely solution. May 6th could potentially happen again and the regulators must be prepared. I believe a solution could be achieved quickly and by using less taxpayer dollars than the SEC might initially think.

Is High Frequency Trading (HFT) Really Evil?

I was very interested in Tim Bass' response to my posting Regulation: Don’t Throw the Baby Out With the Bathwater. You can read Tim's post here but in summary he feels that "the US economy (read individual investors) would be much better off if financial services firms (or anyone) were not permitted to use computers next to stock exchanges to seek split second “get rich quick” opportunities that the normal, traditional investor does not have. The only “economy” that benefits are entities that participate in this type of activity, and the technology firms selling the hardware and software to make “all the madness” possible."

My response to Tim is as follows: I respect your views as always and understand where you're coming from. My view on this is that HFT is not driving the market big picture -- fundamentals will still do that. So people still invest in companies driven by good economics. HFT makes the market more efficient. And yes it does accentuate movements.

The point I was trying to make is that you can protect investors from some of the rapid movements that HFT can accentuate by having electronic safeguards -- like real-time pre-trade risk and real-time market surveillance.

Sure I want to sell people more software - but there's a bit more to it. If I didn't work for Progress I'd still feel the same way as I do.

On the other hand Colin Clark, who had a lot of experience in real-time surveillance with his company Kaskad, posts that he doesn't blame CEP and HFT for the Flash Crash. Instead he cites issues with NYSE and market fundamentals. See his post here.

This topic definitely insights strong opinions. A lot of bankers I know often don't feel safe even admitting they are bankers in public in case they get lynched for causing such suffering in the economy!! What do you think caused the Flash Crash? We set up a little vote which you can get to here.

PRGS Announces the RPM Suite - A Convergence of BTA, BEP, and BPM

It’s been raining here in Massachusetts for 3 days straight, but today I got to add a NEW acronym to my arsenal...

Introducing RPM - Responsive Process Management

In case you missed it, a Progress Software press release went out this morning that announced the launch of our NEW Progress Responsive Process Management (RPM) suite. The Progress RPM suite brings together our best-in-class solutions for Business Process Management (BPM), Business Transaction Assurance (BTA), and Business Event Processing (BEP) (most commonly referred to as Complex Event Processing (CEP)). The Progress RPM suite will enable enterprises to achieve a higher level of business performance than previously possible. It is scheduled to launch in late April, and the market opportunity for this type of solution is expected to be greater than $10 billion [based on IDC Research*]. The release includes a quote from Maureen Fleming, program director of IDC's business process management and middleware research service:

“Over the past two years, one of the fastest-growing areas of software investment by enterprises has been to improve their situational awareness. Logically, the next step is broadening the focus to not only gain visibility into problems or opportunities but to rapidly respond. Enterprises will increasingly look for vendors that offer a knowledgeable and comprehensive approach to building this next generation of critical business applications."

It may sound like a pretty complicated implementation but core to the Progress Responsive Process Management suite is the Progress Control Tower™, a unified product dashboard, or GUI, that displays real-time alerts, interactive interfaces and tools. The Control Tower will provide users with the ability to view what is happening within their business and to improve it from a single source - thereby gaining greater ROI. It’s fully configurable, feature-rich, interactive framework delivers a wealth of relevant, KPIs and business information. What’s more, a powerful modeling environment enables new business processes to be rapidly created, modeled, monitored, controlled and improved dynamically.

To learn more, read our 7 page brochure and visit our website. We've also written the white paper Achieving Operational Responsiveness Through Responsive Process Management that you can register for.

Over the coming months, we’ll introduce more collateral, white papers, and webinars so stay tuned.

TTYL!

Rumblings in the Cloud

Cloud computing... it's on everyone's mind these days. Personally I think it's a term that has attained such aggrandized acclaim that vendors, analysts, bloggers and anyone with marketing muscle has pulled and stretched its definition to such and extent that it could mean just about anything hosted. Cloud Computing Journal polled twenty-one experts to define Cloud Computing.  Just the fact they had to ask the question of twenty-one experts is rather telling in itself.  Well I read what the experts had to say.

So armed with my newly minted (yet fully stretched, but not of my own making) Cloud definition I happened upon this commentary about CEP in the Cloud or the lack thereof.  There's a great quote in the article: "I don’t care where a message is coming from and I don’t care where it’s going”. Correctly indicated, this in a sense defines a key aspect of CEP. Event-based applications should be transparent to messages (or events to which messages transform) origin and destination (sans a logical or virtual name).  However, unlike the author Colin Clark, I do believe the current crop of vendor products, most notably Progress Apama maintain this separation of the physical from the virtual.

The rationale behind the lack of CEP-based applications in the Cloud (ok, there's that word again) are found in other factors. To explain my reasoning I'll start by dividing CEP-based applications into two categories. Of course there are many ways to categorize CEP-based applications, but for the sake of this discussion, I'll use these two:

CEP-based Application Categories

1. Those that do things
2. Those that observe other applications doing things

Not sure I could make a simpler layman-like description, but needless to say it warrants further explanation (or definition in sticking with our theme)

CEP-based applications that do things

This category is best explained by example. Typical of event processing applications that do things are those in Capital Markets like algorthmic trading, pricing and market making. These applications perform some business function, often critcal in nature in their own right. Save connectivity to data sources and destinations, they are the key ingredient or the only ingredient to a business process.  In the algo world CEP systems tap into the firehose of data, and the data rates in these markets (Equities, Futures & Options, etc.) is increasing at a dizzying pace. CEP-based trading systems are focused on achieiving the lowest latency possible. Investment banks, hedge funds, and others in the arms race demand the very best in hardware and software platforms to shave microseconds off each trade. Anything that gets in the (latency) way is quickly shed.

In other verticals, an up and coming usage of CEP is location-based services. This is one that leveraging smart mobile devices (i.e "don't care where the message is going") to provide promotions and offers.  

• Algo Trading, Pricing, Market Aggregation
• Location Based Services (providing promotional offers and alerts)

CEP-based applications that observe other applications doing things

Conversely, event-based applications that observe other applications doing things are classified as providing visibility or greater insight into some existing business function. These event-based applications overlay business processes to take measures to improve their effectiveness. As is often the case critical business applications provide little visibility or the information is silo’ed. There is a need to provide a broader operational semantic across a heterogeneous mix of business applications and processes.  Here are a few typical examples of event-based visibility applications observing other business systems.

• Telco Revenue Assurance
• Click Stream Analysis
• Fraud Detection
  
• Surveillance
  

Of  course the demarcation line between these two classifications is not clear cut. Providing greater visibility is just a starting point, monitoring for opportunities to take action is just as important such as kicking-off a fraud watch if a suspected wash-trade occurred  (so in a sense they are doing things).

Where for art thou oh CEP
When considering the Cloud, an important point to consider is dependency. Specifically, there is a dependency that the underlying applications and business processes exist in the Cloud for (observing) CEP to overlay them.  I would offer that Enterprise business has not yet migrated their key business processes to the Cloud on a widespread scale just yet. Why not? What are the barriers? Security, regulatory compliance, DR, investment costs, limited skill sets are just a few of the challenges mentioned in this ITProPortal article.  I suspect these barriers are far reaching, keeping the pace of Cloud deployment in check to the point where it's not as yet strategic to many.
 
One of key things that makes the Cloud a reality is virtualization, it has clearly revolutionized PaaS as the Cloud. Virtualization does come at a cost, there is a latency penality for the conveinence, no matter how small for some use-cases that cost is too great.

Make no mistake, I am certain the Cloud with all it's twenty-one definitions is the future of computing. It's an imperative that will knock down the barriers and change the face of the Enterprise and when it reaches critical mass CEP will be there.

Once again thanks for reading, you can follow me at twitter, here.
Louie

. 

Peas and Carrots

In the words of the auspicious Forrest Gump some things go together like peas and carrots. Truer words were never spoken. Some things just do go together well, sometimes by design, often by accident. I don't think anyone actually planned milk and cookies or popcorn at the movies but nonetheless these things are made for each other.  When it comes to technology the same harmonious relationships exist.

In the recent Aite report on High Performance Databases (HPDB),  the market for specialized databases is surveyed along with a handful of vendors in this space.  This is a cottage industry where the big database vendors don't play. It's hard to imagine in this day and age where database technology is so standardized and mature and a multitude of choice abounds from commercial products to open source that any other database technology and a gang of vendors would have a chance. Yet it is happening and it's thriving.  

I believe it has to do with a synergistic relationship to event processing. If CEP is the "peas" then HPDB's are the "carrots". These two technologies share two fundamental precepts:

•  A focus on Extreme Performance
•  Temporal Awareness

I. Extreme Performance, Speeds and Feeds
These HPDB's which are often referred to as Tick databases, are found in the same playground as event processing technologies. In the Capital Markets industry they connect to the same market sources, consume the same data feeds. Both technologies are designed to leverage modern multi-core hardware to consume the ever-increasing firehose of data. By the same token, once that data is stored on disk, database query performance is equally important.  The massive amount of data collected and is only as good as the database's ability to query it efficiently thus creating another (historical) firehose of data which an event processing engine would be the consummate consumer.  

II. Temporal Awareness, when is the data
Time is a basic principle in event processing technology, applications typically have as a premise to analyze data-in-motion within a window of time. HPDB's design center is to store and query time series data. Some of the database vendors even bring time to a higher level business function. They understand the notion of a business Calendar, knowing business hours, business week, holidays, trading hours, etc.  Imagine the simplicity of a query where you want 'Business hours Mon-Fri for the month of February' and the database itself would know the third Monday was Presidents Day, skipping over that, thus preventing analytic calculations from skewing erroneously.

Leveraging the Synergy
These two fundamental shared principles provide the basis for a unique set of business cases that are only realized by leveraging Event Processing platforms and High Performance Databases

• Back testing algorithms across massive volumes of historical data compressing time

What if you could test new trading algorithms against the last 6 months or 1 - 2 years of historical market data but run that test in a matter of minutes? What if you could be assured that the temporal conditions of the strategies (i.e. timed limit orders) behaved correctly and deterministically matching the movement of time in complete synchronicity with the historical data? These are just a few of the characteristics that define the harmony between event processing and high performance (Tick) databases.

• Blending live and historical data in real-time

Querying historical data in-flight to obtain volume curves, moving averages, the latest VWAP and other analytics calculations are possible with these high performance databases. Leading edge trading algorithms are blending a historical context with the live market and even News. The winners will be those that can build these complex algo's and maintain ultra low-latency.

• Pre-Trade Risk Management

Managing positions, order limits and exposure is necessary, doing it in real-time to manage market risk is a mandate.  In addition to market data, these high performance databases can store pre and post trade activity to complement event-based trading systems and become the basis for trade reporting systems.

In the Trading LifeCycle, Event Processing and High Performance databases are partner technologies harmoniously bound together to form a union where the whole is greater than the sum of the parts. They are the peas and carrots that together create a host of real-world use-cases that would not be possible as individual technologies.

Myself along with my colleague Dan Hubsher we are doing a 3-part Webinar series entitled "Concept to Profit". The focus is on event processing in the trade lifecycle, but we include cases that touch upon high performance databases. You can still register for part 2: Building Trading Strategies in the Apama WorkBench where I will focus on the tools for strategy development aimed at the IT developer.

Once again thanks for reading, you can follow me at twitter, here.
Louie

Why BPM should be on the CIO’s agenda in 2010

New article by Giles Nelson published in CIO (http://bit.ly/biSKFo) online.

"In 2010, the business prerogative across all sectors is to use IT to drive efficiency and enable a business to react more quickly to customer and market changes. To do this, I believe we need to take a different view of BPM technology and try to see how it can be used to make knowledge-based business more ‘operationally responsive', reacting to customer needs and market changes instantly. This is already beginning to happen, and as it gains momentum, BPM will prove its usefulness in bringing ‘order to the chaos', and will make it onto the strategic agenda of every CIO."

The article does a great job at illustrating the synergy between business process management (BPM) and complex event processing; or in this case business event processing (BEP). Giles even provides examples of industry's already deploying these technologies. Most of us know the benefits of BPM but as he points out... "The next stage is to match it with the other side of the coin, where it can help an organisation respond to events and become truly operationally responsive - something worthy of the full attention of any CIO." Read the full article.

Brazil Embraces High Frequency Trading - Do You?

The trading business feels like a fight, now as ever, with the threat of sweeping regulations as the most pressing concern of the moment.  This business even sounds like a war, too - with algorithms names like "Raider" and "Sniper;" and with terms like "dark pools," and "low latency arms race" drawing focus from regulators and media alike.  But the war-like aspect remains because trading is a highly competitive business. 

The imperative to increase market share will remain a top priority this year along with risk management and regulatory compliance, and the technology required to compete is available to everyone.  As Apama has expressed before here, the markets are still driven by those with the flexibility to quickly adapt to new regulations, the insight to understand new market behaviors, and the imagination to conceive a trading strategy that can capitalize on the opportunity.  It's no wonder that the relatively small number of firms using high frequency trading strategies are responsible for over 70% of US equity trading volume.  These pressures push the rest of the capital markets in the same direction and the trend is unlikely to reverse. 

On February 8th 2010, Apama announced that Banco Fator Corretora, a Brazilian bank and brokerage firm, has deployed the Progress® Apama® Algorithmic Trading Accelerator. Apama plays a critical role in Banco Fator’s new electronic trading strategy, enabling it to more effectively develop high frequency, proprietary trading tactics, achieve rapid customization, and perform low latency execution of trades on behalf of its buy-side clients.  Banco Fator is also working with its clients to design customized algorithmic trading strategies that provide them significant competitive advantage, and the bank explicitly emphasized the importance of providing its clients with a fast method to enter the high frequency trading business.

Why so much emphasis on high frequency trading (HFT)?  Reasons will differ among traders and regions, but a short primer on HFT and some ideas are here.  The Brazilian market has expressed a strong opinion on the matter, with over 15 customers deploying Apama internally to automate execution and/or alpha-seeking strategies in the past year-and-a-half, with many further rolling out the platform to downstream clients. 

So, do you have an opinion on HFT as well?  What characteristics should a platform for HFT have to enable you to be more competitive?  Let us know - leave a comment, or take our poll.

-Dan

CEP consolidation continues

It’s been another remarkable week. I told my wife there would be a lot of traveling for me in the first part of the year and I was right. Last week was New Jersey and New York. This week was Dallas Fort Worth and Silicon Valley. I’ve been visiting key customers, journalists and analysts.

This week has also seen further consolidation in the CEP market. I have been predicting that there could not be a stand-alone CEP market and that CEP will either find a home in applications, databases, stacks or business application platforms.  In this case Sybase has snapped up Aleri to extend its database business into the CEP domain, as well as solutions in the risk space. Aleri are a good company with good people and good products. They come from the “in memory database” perspective but developed a high performing CEP engine and learned lessons from real customers that a SQL approach is not adequate to address real applications, and embedded actions statements are need within a CEP language. Also they learned that the best way to sell CEP is not as a technology but as solutions.

I think Sybase have made a smart move – for probably a bargain price – judging by the release that says they acquire the assets only. I wish my friends at Aleri all the best for the future.

From Concept to Profit in No Time Flat – High Frequency Trading

Colleagues Dan Hubscher and Louie Lovas have begun a great webinar series that outlines the “lifecycle” of an algorithmic trading strategy.  Illustrated with Apama’s Event Modeler, they use a Commodity Futures trading strategy to illustrate how trading firms can accelerate the delivery of trading strategies with a development tool that is accessible to the trading desk. This can help make significant cuts in development times, allowing firms to capitalize more quickly on opportunities.  Future sessions will explore some of the other aspects of the platform that target developers, recognizing that firms have different business models and development styles.

We’ll be posting the recorded version shortly for on-demand viewing, but if you have not registered, I’d encourage you to click here and get on board for parts 2 and 3.

A BEP On Our Radar...

Last year Progress Software started talking a lot about Business Event Processing (BEP)—more commonly known as complex event processing (CEP). It really kicked into gear when we commissioned an independent technology market research company, Vanson Bourne, to conduct a survey and report on their results. Vanson Bourne interviewed 400 companies representing energy generation, telecommunications,and logistics sectors in the US and Western Europe. Why these industries? Well, because of the volume and complexity of their "business events" (service delivery) - both through systems and processes, and customer, partner, and supplier interactions. The results of the survey, detailed in the paper Overtaken by Events? - The Quest for Operational Responsiveness, demonstrates that harnessing business events, smart interpretation, and fast response are clear objectives for these industries. And the need is immediate.

Below I've included the Executive Summary of Key Findings. If you'd like to get more detail on their findings, visit our website and download the complete paper. 

---

EXECUTIVE SUMMARY OF KEY FINDINGS

The Objective Is Operational Responsiveness

Operational responsiveness is the ability of business processes and systems to respond to changing conditions and customer interactions as they occur, enabling business leaders to capitalize on opportunities, drive greater efficiencies, and reduce risk. The survey identified a number of key pointers as to why businesses would be keener than ever to improve how they respond operationally, for example:

Customers

• 91% said they are trying to act in a more personal “one-to-one” way with the customer. That means paying more attention to specific, individual feedback.
• 74% reported that areas such as digital market channels, mobile platforms, and social channels have caused a significant increase in the flow of information into and through their business. That means paying more attention to events in the context of a blizzard of communication.

Competition

• 70% of the businesses surveyed said that it would it be an advantage to be able to price their products based upon dynamic factors, in response to intra-day changes, such as changes in competitor prices/activity.

Process efficiency

• Operational incidents can be costly: 82% of companies surveyed have to continuously monitor processes to try to prevent them happening.
• 72% said their business processes take too long, and they need to shorten them.

Businesses want to respond quickly and more accurately to business events at the operational and business planning level. Real-time information delivery is seen as an important contributor, seen as having a role in three key areas:

• Monitoring KPIs—overseeing pre-ordained service or business performance benchmarks.
• Automatically alerting end users when certain conditions occur—flagging exceptional circumstances or activity for colleagues to take action.
• Automating response processes—delegating conditional processes to the operational systems

Of the companies surveyed 82% are planning investments in real-time technology by mid-2010 in the hope of achieving the vision.

But the Road Is Long...

The survey reveals that most companies still have a long journey on the path to operational responsiveness as defined above. Here are a few stand-out numbers that underline the current situation:

Service delivery and process gaps

• 67% hear about problems in service from customers before they have identified those problems themselves.
• Only 8% report currently business information in real-time: indeed only 19% report on an intra- day basis.
• 72% think that their business processes take too long and they need to shorten them.
• 89% cannot get a single view of process performance because information on business processes is held in multiple different operational systems. 80% use middleware to try to bring data together but not to the satisfaction of those in charge of operations.

Business planning gaps

• 34% say that, by the time they are able to see a change or trend in one of their business processes, they have missed some if not all of the opportunity to react to it.
• 47% of companies surveyed report that business information is typically analyzed to identify patterns and trends historically and not in real time.
• 58% admit that they have significant gaps in the information they need to support their business decision making.

Real-time Information and Business Event Processing (BEP)

In fact, 94% of businesses said that real-time information is important to them, and 78% said immediacy of response to business events provides a competitive advantage. But where business information is incomplete and/or sits across a range of disparate, non-compatible operational systems (as is admitted by most of the companies surveyed here), then speed alone is not enough. Where BEP is being tried out, users are already witnessing the power of combining and correlating across platforms, as well as the desired advantages that real-time systems would provide:

BEP benefits experienced so far

• Filter and analyze lots of events quickly—66%
• Take automatic actions in response to certain sequences of events occurring—55%
• Better monitoring of existing operational systems—50%
• Normalizing and correlating events from multiple different sources—45%
• Providing real-time visibility into information for business end-users—42%
• Spotting time-sensitive event patterns—25%

If you'd like to get more detail on their findings, visit our website and download the complete paper.

Progress Software Announces Q4 results - Here are some highlights

Progress Software (NASDAQ: PRGS) just announced their Q4 earnings release. To summarize, it says "Earnings Up in Q4; Progress® Actional® Revenue Up with Triple-Digit Growth; Progress® Apama® Revenue Up with Double-Digit Growth." What I really thought was interesting was the Q4 highlights. The majority of wins involve building or enhancing an integrated infrastructure, and application modernization - both topics we cover in this blog. In case you missed the release, I've included these highlights below. Enjoy!

Q4 Highlights

• Progress Software announced that the Progress® Sonic® ESB (enterprise service bus) is deployed and operational at British Airport Authority’s (BAA) Heathrow Airport Terminal 5.
  
  The Progress solution enables BAA to provide airport integration capabilities using the Sonic ESB product. This includes the creation of reusable integration services for new Terminal 5 systems and of specialist adaptors for the integration of existing key operational BAA systems, such as the Airport Operational Database Integration. (Tag: Application Integration)
  
  
• Progress Software has successfully enabled more than 250 Independent Software Vendors (ISVs) to deploy thousands of on-demand, SaaS applications over the past five years.  These ISVs use the Progress® OpenEdge® SaaS platform to build applications that are used in some of the most demanding and diverse business environments in the world. (Tag: Cloud Computing)
  
  
• British Airways selected Progress Software SOA Solutions to upgrade their travel experience.  The UK’s largest international airline, British Airways (BA), will use the Progress portfolio of SOA solutions as a key part of its travel program to upgrade its IT systems by integrating over 600 different electronic systems and processes involved in getting BA passengers in the air. The flexibility of the Progress SOA portfolio allows BA to extend the features of its e-commerce site right through to its airports, by allowing greater self-service functionality and 'plug and play' capability. (Tag: SOA Success)
  
• match2blue stands out from the crowd with the Progress® Apama® Business Event Processing (BEP) platform by adding real-time capability to next-generation social networking. Enterprise platform enabler for mobile solutions, match2blue (www.match2blue.com), has selected the Apama platform to empower its social networking platform with real-time information on location, ideas, news and trends.  The Apama BEP platform will form a crucial part of match2blue’s back-end infrastructure, providing the performance and scalability needed, as well as supporting its business partners, who will be operating the location-based services to control and monitor their operations through dashboards. (Tag: Complex Event Processing)
  
  
• Alphameric Solutions Ltd, the leading solutions provider to the gaming industry, selected the Sonic ESB to revolutionize the way it handles content and messages across its network. Relying on highly complex and automated processes to deliver odds, prices, race information and documents across a distributed architecture – most needing to be handled in a sub-hundred millisecond timeframe – Alphameric needed a simpler way to incorporate new or updated information in real-time. (Tag: SOA Best Practices)
  
• West Bend Mutual Insurance Company has selected the Sonic ESB (enterprise service bus) and Actional products to underpin a service-oriented architecture (SOA) based IT infrastructure.   West Bend Mutual Insurance, a property and casualty insurance carrier, is pulling together dozens of disparate internal policy administration applications into a single integrated insurance portal. (Tag: Distributed SOA)
  
• Progress Software announced the availability of the Apama 4.2 Event Processing Platform.  The Apama 4.2 release extends the capabilities of the previously announced Apama Parallel Correlator, and introduces significant new developer productivity features that accelerate the deployment of event processing applications. The Apama Parallel Correlator leverages multi-core, multi-processor hardware to deliver high throughput, low latency execution that has achieved seven-fold performance improvements, as benchmarked with real-world customer applications. (Tag: Event Driven SOA)
  
• Slumberland, a leading furniture retailer, is now using standards-based data connectivity products from Progress® DataDirect® for reliable, high-performance support for all their major databases and 64-bit operating systems, for reliable connectivity to their Oracle applications, and streamlined reporting to improve fulfillment and customer satisfaction. (Tag: Semantic Data Integration)
  
• Progress unveiled the industry's first mainframe SQL engine for non-relational data, which can leverage zIIP specialty processors for lowering a mainframe’s total cost of ownership (TCO), with the announcement of its DataDirect Shadow Release 7.2.1.  The DataDirect Shadow release includes ANSI SQL-92 to Non-Relational Data with zIIP Offload and new capabilities that lower costs and attract new process-intense workloads to the mainframe.

CEP as XTP, in SOA Environments

I have been tracking caching based application platforms such as Gigaspaces for sometime now, and I was surprised to see a new spin on these - as XTP (Extreme Transaction Processing) complimenting SOA. They are extending the caching capabilities to SOA solutions. Labeling this stateful SOA environment as a SOA Grid, and projected as an enabler to build next generation applications are in the areas of fraud detection, trade resolution, and risk management calculations away from the mainframe to low cost commodity hardware. Now, this is exactly the pitch from complex event processing (CEP) environments as well. And even CEP is seen as extending SOA infrastructure to support the capabilities to handle large volume event streams such a the banking or credit-card transaction events, needed for fraud detection.

At some level, the notion of "stateful" SOA grids proposed in the above paper is a contradiction of sorts. SOA is by definition discrete and loosely coupled services environment that are brought together in an orchestration environment to realize required business functions. Products such as Progress Sonic ESB do extend the orchestration capabilities to a more distributed environment (itineraries). Here one can see some value for a distributed state information (that caching products such as Giga and Terracota provide). But expecting this to provide the required capabilities for large volume event stream processing is a real stretch!

For starters, CEP is more than (surely, different from) a fast data-access/query mechanism, which is what caching solutions are. Caching solutions essentially extend the database processing model by providing a faster access to data. While CEP is less about data and much much more about quickly 'reacting' to events and detecting temporal patterns that occur in and across these large volume event streams, combining this within an SOA that can emit the events (based on the biz processing occurring within), or provide the required biz processing after the complex-event is detected, is in itself an XTP environment. It is capable of processing a very large volume of event streams and it needs to detect complex patterns in these streams. And, largely due to the unique approach used in the Progress Apama Correlator engine, it flips the conventional store-and-query approach (which is where caching products come in) to a look-ahead-and-react mode. This has already been proven in some of our successful Apama CEP deployments, such as Algorithmic Trading and Fraud Detection (the very same cases referred in above blog post). See the diagram below to see how Apama CEP works:

Who Needs DW? Welcome, Real-time BI !

Last week I got a last minute invite to speak at the Silicon India BI conference (last minute for a good reason I guess... me & BI? Hmmm... :-) My database background notwithstanding!). Initially I resisted as couldn't think of anything of value, especially because the participants were expected to be architects and senior tech folks. As Pradeep (heads Silicon India) nudged, a thought crossed. I recently also came across a BAM use case where the kind of questions they wanted the solution to answer bordered on BI. And, in this solution they needed both SOA and Complex Event Processing (CEP).

Isn't using CEP for BAM in some form real-time BI? I deliberated for a few minutes - the CEP SOA synergies seemed to really make sense, so I agreed to speak. When asked for the title, I submitted "Design Approaches for Real-time BI".

BAM (business activity monitoring) is an extension of traditional business intelligence; adding event monitoring to scheduled, batch-based reporting. As I see it, BI can be of a few different types (when seen from the tainted SOA lens :-) ):

• Simple BAM is watch ‘as-is’. Most BAM software supports simple aggregation and watches for thresholds, et al.
• Complex BAM can watch for more complex patterns.

At some level isn't complex BAM BI? It provides insight and information that traditional OLTP systems cannot give. For BAM or BI, the discrete steps in the decision cycle are the same: Track/monitor; Analyze; Hypothesize/ model; Decide; and Act . The only difference being that BAM (however sophisticated) may not be able to answer adhoc queries and what-ifs. It can only answer pre-defined and pre-modeled situations. Nevertheless, this is BI. And it is real-time!

SOA and CEP form a good basis for realizing this. Conventional (read 'simple') BAMs may not suffice. BI is more than just thresholds or SLA violation alerts. One may need to detect a pattern involving a series of biz failures that could potentially cause a much serious eventual failure, which if detected and alerted in time, can be averted. This is where more complex pattern recognition come into play. The CEP engine. Given that we are talking about business events from all over the enterprise as the inputs, SOA infrastructure fits right in as the potential vehicle to deliver these events to the CEP engine.

SOA+CEP for real-time BI! An interesting proposition! And surely something traditional DW solutions cannot handle. DW & real-time? Nah. Now, can we extend this basic real-time BI to also include adhoc querying and such? Something to ponder about.

Embedded Not-so-lightweight ESBs

David Linthicum touched upon a very prevalent problem of technology not living up to its expectations set during the hype curve! He says, "With cloud computing, SOA, or whatever we come up with next, we still need to figure out core issues within our enterprises that no set of technologies or emerging trends can solve." In the service-oriented architecture (SOA) case, we all know that the widely professed use cases of systems "automatically" looking up service registries on the web/intranet, locating the required services, getting the interfaces, picking the right one, and actually making a service request is just a pipe dream! The reality is nowhere close! It is difficult to even find enterprise wide service buses, much less the dynamic on-the-fly service-look up-and-use!

I recently came across a scenario where even a large international bank has SOA infrastructure solving regional problems - and not necessarily as a corporate backbone! I was at this tech discussion with the regional technology head of a large bank, exploring how Apama (Progress' Complex Event Processing (CEP) engine) fits into a biz requirement they have. Specifically, he wanted to understand how Apama works and how it is a good fit for their biz problem. They were trying to build a business performance tracking solution (a la GRC) that involves tracking key biz activities, ensure due process is followed, and then proactively detect potential service-level agreement (SLA) failures so they can be averted. A classic use case for Apama!

Now, when we started to see where the business events information came from and how they could be "wired" to emit the required events into Apama, we suggested that they could just tie into their enterprise service bus (ESB)—we knew they had standardized on one. As we dwell further, it turns out they are using an ESB to solve departmental integration requirements, and there are multiple instances and no central ESB that can help integrate these "departmental integrations".

Now that was a revelation. (I am not complaining because suddenly our opportunity now included Sonic ESB, along with Apama). While we have been reading (and I myself have been blogging and talking) about how SOA failed to live up to its expectations, I still expected at least a large number of banks to have standardized backbones. But not to be! I guess, like in many large enterprises, even here SOA adoption has started purely to solve departmental integration problems—more as localized islands of "integration". And its nothing like the enterprise-wide integration platforms or service bus that we have so much noise about since 2004. This reality check at this bank was a clear validation of this failure! It seemed clear that the technology promise was something, and the actual on the ground realization was something totally different!

Come to think of it, it does seem like a good model for use cases such as BAM or GRC - to have a complete platform that in addition to the CEP engine and dashboard, it also includes a "local/embedded" ESB. Now, embedded is not about being lightweight or smallas it is about being available out of the box when you get the platform. In this case, the ESB is available along with CEP, well integrated - so no surprises as you start using it. And the purpose is to "wire" exchanges from existing ESBs or applications in the enterprise to the CEP environment.

Not a bad idea at all! Embedded ESBs!