Business Making Progress

Building Customer Loyalty in a Churning Market

All businesses suffer from churn – the moving of customers from one service provider to another. As new and innovative services become better understood and more widespread, more suppliers enter the market and so the opportunities for customers to change suppliers increases. Churn is expensive. Recruiting a new customer can cost 5-10 times that of retaining an existing one. So how can technology help in the constant battle to retain customers? 

I’m going to illustrate what can be done by talking about mobile telecommunications – an industry where innovation is rife but where churn is a significant problem.

Mobile communications continues to grow very quickly. According to a recent Cisco survey, mobile data volumes are nearly doubling each year. By 2015 it predicts there will be 7B personal mobile devices globally. Analyst firm Ovum recently reported that in 2010 revenues from mobile data for European mobile operators exceeded that for voice calls for the first time.

Smartphones are completely changing the way that people use the Internet. It’s worth reminding oneself that now, in a pocket device, one has a phone, a camera, email, PDA, mapping with GPS, in some countries a near field payment device and of course access to thousands of applications. Morgan Stanley has predicted that in 2012 shipments of smartphones will, for the first time, exceed those for personal computers. The whole landscape of computing itself is changing.

With all this growth mobile operators should be very happy. Subscriber bases and mobile data volumes are growing. And yet, mobile operators can’t rest easy. Yes, innovation is everywhere, but most of the innovation (at least that visible to end users) isn’t going on in the mobile operators – it’s going on in the phones and the applications. End users are becoming more and more divorced from the particular network they use and, certainly in developed markets, operators primarily compete on two things only – price and coverage. The growth of the mobile Internet is pushing operators to the bottom of the value pile and risks leaving them as faceless utilities. This, in turn, leads to churn, with rates for mobile operators range from 20-40%, meaning that between 20 and 40% of subscribers will, per year, leave a network for another.

Other industries are of course liable to churn too. Insurance is one example – I recently used an online insurance aggregator to find car insurance and, within a few minutes, obtained a rate 20% below that that my current provider was offering. Online retail is another – it’s very easy to move to another retailer that might be offering a lower price on the same product. In general, churn is present wherever a product is a commodity or near commodity and where customer relationships are weak.

Some wireless operators are fighting back by identifying more ways in which they can meaningfully interact with their customers. To take a concrete example, one European telco, a Progress customer, is now continuously monitoring calls from their 30M subscribers to identify patterns of usage that indicate a different tariff would be more suitable for that subscriber. This could be as simple as noticing that the number of bundled monthly minutes used had been exceeded. A text message is then sent to the subscriber suggesting that they move to another tariff that would reduce the cost of calls in future. Time is of the essence. If the subscriber receives an offer soon after placing one of these calls they are far more likely to accept it than if the offer came through many weeks later.

The way that marketing campaigns are run can become a lot more responsive. The mobile operator may decide to run a campaign to, for example, promote a particular tariff it thinks will be of interest to a subset of its subscriber base – those people, for example, who spend more than $100 per month and roam frequently. Sending out offers via text message requires great sensitivity, as no operator wants its customers to feel it’s receiving spam. As the campaign executes results can be monitored in real-time and the target demographic of the campaign can be tightened to achieve a better response rate. Not only does this make the campaign more successful but also those subscribers that, in the end, are not targeted can become the target of a future campaign.

To do this requires a number of things. Firstly, software needs to be in place to allow the millions of subscriber calls to be analysed in real-time – an ideal use case for event processing. Secondly, there need to be tools which allow a marketing team itself, working largely autonomously without IT support, to create, test and dynamically enhance the rules which dictate which subscribers will receive the offer. And finally, positive responses to the offer need to be processed systematically through an order management system.

There are many other examples where responding quickly to subscriber activity can enhance a user’s experience of using a particular mobile operator. As Internet use becomes dominated by mobile, it’s likely that variable costs for data access, particularly where large downloads are concerned, will be introduced. The cost of a download will be calculated dynamically, dependent upon the bandwidth available within a particular cell at a particular time. At initiation of a large download (let’s say greater than 1Mb) the user could be prompted to ask whether he would like to download it at twice the normal bandwidth for another 10c. This would be a dynamic rate, calculated in real-time in response to current activity in the wireless cell and the propensity of the user to accept the rate.

So, what’s the general lesson here? By becoming more responsive to subscribers, mobile networks are increasing their value to customers, improving customer service and so reducing the likelihood of churn. Existing information about customer behaviour is being used but by being able to act on that information immediately they are able to communicate in a much more contextually suitable way so improving response rates and strengthening the customer relationship. All businesses should be looking how to use their operational information to respond to and interact with customers better. Real-time responsiveness to customer behaviour is becoming vital.

Responsiveness in Retail Banking

A general theme for this blog is how organisations are having to become more responsive, whether to customers, competition, regulation or to squeeze further operational efficiencies out of processes. I want to talk about some of the likely areas of innovation in retail and wholesale banking that will make these organisations more responsive.

Regulation is one obvious area where banks, over a multi-year period, are going to have to become more adept at dealing with change. The implications of the Dodd-Frank legislation in the US is still being understood and, in the UK, banks may be facing an almost existential threat if one gives credence to the rhetoric currently coming out from government and regulators.

Regulation and compliance with regulation is only going to become more and more important. This is being recognised by non-banking institutions that work in financial services. Thomson Reuters, for example, recently announced the formation of a Governance, Risk and Compliance division to deliver information solutions to those needing to comply with regulation. Useful certainly, but regulation will always manifest itself in different ways in different banks. The processes that support regulation must be able to change on a frequent basis. Applications set in concrete only allow this to occur slowly and expensively. The method of building applications needs to be responsive itself and needs to focus around dealing with complex events, business rules, workflow and user interfaces, not around programming language coding.

As an example, a bank, now a customer of Progress Software, had to deal with complying with new regulatory requirements in a matter of weeks. This was not just a box-ticking exercise – the regulator wanted visible demonstration of the systems that had been put in place. The bank’s existing application was simply too inflexible to be changed for reasonable cost and in the time required. An alternative approach, based on Business Process Management and event processing technology was selected which allowed the compliance process flow to evolve and change over time in a much more dynamic way and which could be tailored to meet the bank’s own unique combination of circumstances.

Regulation isn’t a nice to have or something aspirational. It’s a must-have – the risk of legal sanction makes it so.

Another focus for this year will be customer experience. Generally speaking, banks have been behind other industries in getting more responsive to customers’ individual needs and how they’re actually interacting with their bank. A recent US survey, by the industry analyst firm AITE, found that only 22% of small businesses were “extremely” satisfied with their bank and that 65% of larger corporates believed that banks did a bad job of understanding their needs.

A personal anecdote will illustrate a wider point. Recently, I moved money from a savings account with my usual bank to one paying a higher rate of interest with a competitor. The first bank finally contacted me about why I wasn’t using my savings account, but only after all the money had exited the account. By then it was simply too late. Banks still haven’t got sophisticated enough about monitoring how their customers are interacting with their accounts, online, by telephone etc., and sensitively interpreting what these actions might mean and responding in a timely fashion. Again, a watchword here is responsiveness.

Customer onboarding, the process of dealing with a customer’s application for a loan, credit card etc., has got some attention, both to improve the customer’s experience but also to increase the operational efficiency of the bank’s processes.  Some banks have now a real-time view of these processes and can determine immediately whether a high-value business loan should be treated as a greater priority than a car loan or whether a number of card applications have been spending too long receiving credit checks. By knowing this kind of information now (rather than often days or weeks hence) is vital to spotting exceptions quickly and re-prioritizing resources.

One of the core services that a bank performs is payments, whether these payments are executed using cards, direct bank transfer, cheque or otherwise. This year might see some significant innovations in payments processing which will require banks to respond in a way they’ve not been forced to previously. A key tipping point may be Apple introducing a near field communications (NFC) chip into the next iPhone, expected to be released about mid-year. This will allow an iPhone to be swiped across a store terminal to pay for goods. This isn’t that new. Particularly in Japan, NFC chips have been built into mobile phones for some time. However, Apple has a current habit of energising markets and it doesn’t just have the iPhone. There are 160m iTunes accounts which already handle payments. Google and Paypal are also considering offering NFC services. The question is, will this really disrupt existing payment systems which still, even in Paypal’s case, rely upon the core payment infrastructure provided by, primarily, banks? We should hope so. Banks have been hopelessly slow at innovating in payments. It took a diktat from the UK regulator to introduce “faster payments” into the UK. Still too many payments take 3 days, which, in today’s world is very slow for any kind of transaction. Cross-border payments in Europe are still slow and expensive.

If new technology does force payments innovation, banks will be forced to keep up by increasing the flexibility and speed of their own infrastructures to deal with, potentially, many more smaller payments. Guess what? They will need to make them more responsive. If they’re not able to move quickly banks may end up simply being sending round the larger aggregated payments resulting from all the transactions occurring using more modern platforms. Cost pressures will also be present. A recent Boston Consulting Group report indicates that payments providers need to reduce costs by between 10% and 35% by 2012 to keep cost-income ratios stable.

Retail and wholesale banking may often move slowly, but, through regulation, customer pressure and new-entrant innovation, expect some significant changes this year. As Chris Skinner, omnipresent observer of all things banking has recently said on his blog, banks will have to learn “how to be real-time nimble in a world of change”. 

(This first appeared as an opinion column in cio.co.uk)

Sharing some winter sun with Progress' application partners

Earlier this week I participated in the 2011 Progress Global Partner Conference which was held in Florida.

This is only the second time the partner conference has been global – previously it was held regionally – and I’m delighted to say that hundreds of representatives from business partners attended from all over the world.

Most of the partners present were what Progress terms application partners – those that have used Progress products to build applications that are then sold to end-users. As always, the sheer diversity of the applications partners create and sell is mind-boggling – from healthcare apps specialising in kidney treatment to point-of-sale retail systems deployed in 25,000 outlets worldwide to location-based content delivery platforms. Progress recognises the many varied achievements of its partners with its very own awards ceremony. The winners can be found here. And, yes, it is a tiny bit like the Golden Globes, although without the acerbic wit of Ricky Gervais.

These partners continue to be incredibly important to Progress Software. Supporting them with product innovation as well as facilitating new ways for them to deploy their applications (for example by testing out their applications in the cloud with Progress Arcade) is a key pillar of Progress’ strategy. Another strategic pillar is Responsive Process Management (RPM), launched by Progress to the market in 2010, and several sessions in the conference were dedicated to explaining how RPM fitted into the partners’ world. Adoption of RPM in the partner community is happening. An example of this is Skyward, school management software supplier, recently announcing their use of Progress’ OpenEdge BPM platform. This puts them on the first step to full RPM adoption.

John Rymer from Forrester Research, the software analysts, also addressed the conference. Amongst other topics, he talked about four big “on-ramps” of new functionality – business process management, analytics, business events, and collaboration. These, he believed, were the most effective ways for software vendors (Progress’ partners in this case) to deliver new functionality fast and will be the key technologies behind many of the next generation software platforms. Forrester’s presence at a Progress partner conference was timely. Recently one of their analysts, Mike Gaultieri, blogged about Java, despite being more popular than ever, being a “dead end” for enterprise application development. He encouraged developers to consider alternatives, including Progress OpenEdge, that offer substantially higher productivity. It was a reminder that OpenEdge remains as relevant as ever and is a great aid in application modernization. Further output on this topic from Forrester is imminent.

All in all, it was a successful, high-energy conference. Many thanks to all the partners who came, and to those that didn’t, please try and make it in 2012!

 

 

Fighting payment fraud with Complex Event Processing

Today Progress Software announced that SEB bank in Estonia has deployed the Apama Complex Event Processing (CEP) platform to look for payment fraud.

This is exciting news and the fact that such a production customer has gone public is a great validation of SEB's belief in the system they've built. A lot of banks are somewhat reticent to talk publically about fraud. Everyone knows it goes on of course, but banks usually prefer that this is a quiet fight, going on behind the scenes. SEB on the other hand see their use of CEP to detect fraud as a positive demonstration of their commitment to customers and card issuers to ensure that banking with SEB is as safe as possible.

A few weeks ago I visited SEB in Estonia and was hosted by Ain Rasva, the head of technology, together with the head of the fraud team. One of the key reasons they chose to invest in CEP was rising transaction rates and a realisation that conventional data management architectures were simply not suitable to look for fraudulent patterns in a timely fashion. Yes, transactions could be captured in, say, a database management system, but then queries to determine whether a pattern of fraud was emerging had to be run and re-run at arbitrary time intervals. An event processing approach is simply a better way of doing this job, both performance-wise and conceptually. With CEP, SEB can now detect a potentially fraudulent pattern of card use immediately and then start managing the case, using tools to conduct further analysis and to manage communication with the customer. This can be done now in minutes, rather than the hours it took previously. Patterns of fraud detection are constantly changing and SEB needs to be responsive to this - the fraud detection rules need to change frequently and quickly. With Apama, these fraud rules can be created, modified and tested rapidly, and this can be done largely by the fraud department itself rather than relying upon IT to effect each change.

Card payments continue to change and increase. There were 82B payments in Europe in 2009 and card payments are growing at a CAGR of 12% per year. Europe wide regulation such as the Single European Payment Area will significantly change the types of fraud that banks need to look for. SEB has placed itself in a good position to ensure that payments are conducted as safely as possible.

Transformational IT change at British Airways

Gordon Penfold, Chief Technology Officer, at British Airways, started off by telling us that not only is the Boeing 747 40 years old this year, but so is the IT that supports the 747. Having been early to the world of real-time processes, the technology is now facing end of life at the end of 2010, and it's time for a major change.

Gordon shared with the audience at the Progress Software Summit (#progressswsummit) that to get to a SOA-based infrastructure that will deliver the vision of a single real-time infrastructure linking retail, customer, operational and corporate data and processes has required significant technical and strategic change. BA needs to use their enterprise architecture to improve quality and minimise cost, directly affecting BA's objective of being a global airline of the highest quality and providing exceptional customer service. And the term "global" isn't some platitude in this case as BA has operations and staff distributed all over the world. As Gordon said, BA is a peripatetic company.

BA's ‘common architecture’ will ensure consistent implementation, enabling BA to run existing services and implement new ones effectively. SOA is Gordon’s preferred option. The aim is to reduce complexity and increase agility, introducing a plug-and-play capability that can also manage ‘heritage’ applications, created by the industry and service providers – all to support over 15 million transactions a day.

Progress is supporting the change at BA with its Sonic, Actional and DXSI products, and is working with Gordon’s team on a proof-of-concept for Savvion and on how Apama can support with interactions with partners and other airlines. At the half-way point in the implementation, BA is now live with its Progress powered Service Integration Platform, which links the business to IT processes in a controlled way.

Gordon made some kind remarks on working with Progress - the best-in-class products and the added value provided by the Progress people who worked at BA on site. All great to hear from a business with very real mission critical requirements.

Are you a sitting duck or one that will respond immediately to threats?

While many organisations are being ‘cautiously optimistic’ about what the future holds, the realities of today’s tough business environment could leave them as sitting ducks, according to Rick Reidy, CEO at Progress Software. They might take consolation that they’re in the same pond, but when interest rates in Japan hit near-zero, banks continue to fail and mistakes can lead to a ‘flash crash’, the pond is not a safe place to be. Businesses may have money, but fear and uncertainty is holding back decision-making – we await further regulation and want to know the consequences of recent government changes.

 
Listening to Rick’s keynote at our UK business summit (#progresswsummit, if you want to follow on twitter), in the impressive surrounding of Chelsea Football Club’s ground, London, it seems most of the audience agrees – it’s not good enough to sit around and wait to see if growth returns, and you cannot grow simply by cutting costs. You have to take control of your own ‘growth agenda’, as Rick put it. Businesses that want to survive the next five years need better visibility, through putting processes in place that enable them to react quickly to meet customer demands, adapt to market changes and take advantage of new opportunities. As Rick has advised, businesses need to act on up to the minute information so that leaders can make decisions based on foresight, not hindsight.
 
If you’re a regular reader of this blog you’ll already know that we call this ‘operational responsiveness’: the ability to sense and respond to customer and market changes so that organisations can move quickly to meet challenges and take advantage of new opportunities. 
 
Rick has talked about what this means in the airline industry: the notion of irregular operations has become a weekly reality as companies face intense market pressure, striking staff and disruption from natural phenomenon. ‘Swivel chair’ communication between operational areas is no longer good enough. To react quickly enough, they need responsive processes in place that can help them maintain services and inform customers, almost as-it-happens. If they don’t, they will face massive fines, lost custom and damaged reputation – risks no company can afford at present.
 
We’ll be hearing more from Gordon Penfold, CTO at British Airways, about their approach to becoming operationally responsive to meet the challenges of today and tomorrow. Watch this space for my take on his talk…

 

Stamford Bridge, here we come

Tomorrow sees Progress Software taking over Stamford Bridge, home ground to the world-famous Chelsea Football Club. We’re not just there to check out the players’ dressing rooms – we are being joined by James Caan, of Dragons' Den fame, as well as the great and the good of the UK business community, to discuss how businesses can start to make decisions based on foresight, not hindsight, in their operations.

Gordon Penfold, Chief Technology Officer at British Airways, will be sharing his insight on ‘operational foresight’, revealing how the organization has set itself up to better deal with the irregular operations that have become a fact of life in the last year. And Mike Gualtieri, senior analyst at Forrester Research, will be sharing his views on where the next wave of truly responsive business management is coming from, and which trends to watch for. And Progress' own Chief Executive Officer, Rick Reidy, will be giving a keynote too.

I'll be there, speaking in one session but also blogging and tweeting from the event. So watch this space for the latest updates.

For those of you attending, I look forward to seeing you there.

www.progresssoftwaresummit.com

 

India: Big Potential for Algorithmic Trading

I spent last week in India, a country that, by any standards, is growing fast.  Its population has doubled in the last 40 years to 1.2B and economic growth has averaged more than 7% per year since 1997.  It’s projected to grow at more than 8% in 2010. By some measures, India has the 4th biggest economy in the world. 

Progress Software has a significant presence in India. In fact, people-wise, it’s the biggest territory for Progress outside the US with over 350 people. Hyderabad is home to a big development centre and Mumbai (Bombay) has sales, marketing and a professional services team.

The primary purpose of my visit was to support an event Progress organised in Mumbai on Thursday of last week on the subject of algorithmic trading. It was also our first real launch of Progress and Apama, our Complex Event Processing (CEP) platform, into the Indian capital markets. We had a great turnout, with over 100 people turning up. I spoke about what we did in capital markets and then participated in a panel session where I was joined by the CTO of the National Stock Exchange, the biggest in India, a senior director of SEBI, the regulator, and representatives from Nomura and Citigroup. A lively debate ensued.

The use of algorithmic trading is still fairly nascent in India, but I believe it has a big future. I’ll explain why soon, but I’d like first to give some background on the Indian electronic trading market, particularly the equities market, which is the largest.

Watch my Interview on NDTV Profit >

The market: India has several, competing markets for equities, futures and options, commodities and foreign exchange too.  In equities, the biggest turnover markets are run by the National Stock Exchange (NSE) and the Bombay Stock Exchange (BSE), with market shares (in the number of trades) of 74% and 26% respectively. Two more equity exchanges are planning to go live soon – the Delhi Stock Exchange is planning to relaunch and MCX is also currently awaiting a licence to launch. This multi-market model, only recently adopted in Europe for example, has been in place in India for many years.

It was only two years ago that direct market access (DMA) to exchanges was allowed. Although official figures don’t exist, the consensus opinion is that about 5% of volume in equities is traded algorithmically and between 15% and 25% in futures and options. Regulation in India is strong - no exchange allows naked access and the BSE described to me some of the strongest pre-trade risk controls I’ve come across - collateral checks on every order before they are matched. The NSE has throttling controls which imposes a limit on the number of orders a member organisation can submit per second. Members can be suspended from trading intra-day if this is exceeded. The NSE also forces organisations who want to use algorithms to go through an approval process. I’ll say more about this later. Controversially, the NSE will not allow multi-exchange algorithmic strategies so cross-exchange arbitrage and smart-order routing cannot take place. Lastly, a securities transaction tax (STT) is levied on all securities sales.

So, with the above restrictions, why do I think that the Indian market for algorithmic trading has massive potential?

The potential: The Indian market is very big. Surprisingly so to many people. Taking figures from the World Federation of Stock Exchanges (thus I’m not counting trading on alternative equity venues such as European multi-lateral trading facilities), the Indian market, in dollar value, may still be relatively modest – it’s the 10th largest. However, when you look at the number of trades, India’s the 3rd largest market, only beaten by the US and China. The NSE, for example, processes 10 times the number of trades as the London Stock Exchange. So why isn’t more traded in dollar terms? That’s because trade sizes on Indian exchanges are very small. The median figure worldwide is about $10K per trade. The figure in India is about $500 per trade, a 20th of the size. In summary, surely the task of taming the complexity of this number of trades and the orders that go with them is ideal for algorithmic trading to give an edge? To compare to another emerging, “BRIC”, economy, that of Brazil, where the number of firms using Apama has gone from zero to over 20 in as many months, the dollar market size is fairly similar but the number of equity trades in India is 33 times more. The potential in India is therefore enormous.

India is already there in other ways. All exchanges are offering co-location facilities for their members and debate has already moved on to that common in more developed markets on whether this gives certain firms an unfair advantage or not and whether co-location provision should be regulated.

The challenges: There are some difficulties. The STT is seen by some as an inhibitor. However, its effect is offset somewhat by the fact that securities traded on exchange are not subject to capital gains tax.

The NSE process for approving algorithms is more controversial. Firms that want to algorithmically trade must show to the NSE that certain risk safeguards are in place and “demonstrate” the algorithm to the exchange. As the biggest exchange, the NSE wields considerable power and thus its decision to vet algorithms puts a brake on market development. I believe this process to be unsustainable for the following reasons:

1. As the market develops there will simply be too many algorithms for the NSE to deal with in any reasonable timeframe. Yes, India is a low-cost economy, but you need highly trained people to be able to analyse algorithmic trading systems. You can’t simply throw more people at this. Firms will want to change the way algorithms work on a regular basis. They can’t do this, with this process in place.
2. It raises intellectual property issues. Brokers will increasingly object to revealing parts of their algorithms and their clients, who may want to run their alpha seeking algorithms on a broker-supplied co-location facility, will most definitely object.
3. It puts the NSE in an invidious position. Eventually an algo will “pass” the process and then go wrong, perhaps adversely affecting the whole market. The NSE will have to take some of the blame.
4. Competition will force the NSE’s hand. The BSE is trying to aggressively take back market share and other exchanges are launching which will not have these restrictions.

It strikes me that the NSE should spend its efforts into ensuring that it protects itself better. Perhaps a reasonable comparison is a Web site protecting itself from hacking and denial of service attacks. If they can do it, so can an exchange. And it would offer much better protection for the exchange and the market in general.

In conclusion: I’m convinced of the growth potential in India for algorithmic trading. The market is large, the user base is still relatively small and many of the regulatory and technical prerequisites are in place. There are some inhibitors, outlined above, but I don’t think they’ll hold the market back significantly. And finally, why should India not adopt algo trading when so many other, and diverse, markets have?

Progress has its first customers already in India. I look forward to many more.

Shipping logistics with event processing

On Friday last week I visited Progress customer, Royal Dirkzwager, in the Netherlands. It’s nice to be able to talk about a customer publically - Dirkzwager have been a generous public reference for some time. This was my first visit to their head office in Maassluis, near Rotterdam, and it gave me a new insight into their business. Dirkzwager’s offices are right on the main entrance to the port of Rotterdam and it was great to be in their board room and watch the container ships move by as Paul Wieland, head of IT, took me through their business and what they are using Progress’ products for.

Dirkzwager is in the business of supplying information about shipping movements to the maritime industry. Its customers include the Port of Rotterdam, shipping companies and the logistics companies involved in supporting the ships and handling the goods. Dirkzwager currently supplies information on shipping movements at sea and in and out of ports from Le Havre in France to Denmark. Rotterdam is its epicentre - the biggest port in Europe and one of the biggest in the world, handling around 30,000 ships per year.  

Dirkzwager are users of the Sonic Enterprise Eervice Bus (ESB) and the Apama event processing platform. Sonic is used for delivering information to customers reliably and Apama is used for processing and analyzing the events reporting the positions of ships, up to several thousand per second. Live data is augmented with data from a reference database on tonnage, flag, crew, and lots of other information. Applications range from simple to more complex. One example is: “alert when a ship crosses this line”, to give an official port entry time (important for calculating harbour fees). A more complex example is the calculation of an actual time of arrival, using course, speed, weather conditions and historical information about the ship’s movements. This latter application can bring very significant benefits: on average, 50 companies are involved servicing a ship when it’s in port, from loading or unloading goods to servicing the radar. If these companies can receive more accurate information about a ship’s arrival time, they can optimize their operations resulting in the whole supply chain becoming more efficient – capacity is increased and costs are lowered. Paul gave some insight too into how Dirkzwager had become more efficient by using Apama to automate tasks that were previously done manually and in the process saving substantial amounts of money (the details of which, unfortunately, I can’t share here).

Dirkzwager’s industry is changing. Currently, most ships identify themselves by a VHF-radio system. This is shore based and has a range of approximately 60km. This is changing to a satellite based system which will give more continual and accurate reporting and also give simpler access to ship movements anywhere in the world (approximately 80,000 ships are in transit at any one time worldwide). This gives business expansion opportunities and also challenges when dealing with the new quantities of data that available. Paul Wieland clearly believes that complex event processing (CEP) is an ideal way for Dirkzwager to analyze this data and provide the monitoring of logistics and supply chain processes that they support for customers. Paul’s obviously a visionary – the room was buzzing with ideas. Now his team is experienced with Apama, Paul reported that people’s approach to problem solving had changed – they were thinking naturally about things in an event based way, both architecturally and in terms of data processing.

One might not necessarily think of shipping logistics as being the most innovative of industries, but Dirkzwager has always been at the forefront of technological innovation. Shipping was, and continues to be, vital to the Dutch economy. When the company was founded, in 1872, people used to stand at the opening of the port with binoculars. As a ship was sighted and recognized, someone would mount a horse and rush back to the port to report. Dirkzwager had the first commercial telephone line in the Netherlands. Paul showed me a room where some of the computing and communications equipment used previously by Dirkzwager is stored – examples include a megaphone, semaphore flags, telephones, telex machines and a mini computer. It was a mini museum for the communications industry. The use of event processing software is simply another step in this historical evolution.

Optimism in the world of financial services regulation

It seems that we’re finally making some progress on making the financial markets function more safely. 

After the “flash-crash” of 6 May, US equity market operators have agreed to bring in coordinated circuit-breakers to avoid a repeat of this extreme event. There is widespread agreement on this. Industry leaders from brokers and exchanges yesterday made supportive statements as part of submissions to the SEC.

Regulators are going public with their use of real-time monitoring technology. Alexander Justham, director of markets at the Financial Services Authority, the UK regulator, told the Financial Times that the use of complex event processing technology will give the FSA “a more proactive machine-on–machine approach” to market surveillance (the FSA is a Progress customer). Other regulators are at least admitting they have a lot of work to do. Mary Schapiro, the SEC chair, believes that the technology used for monitoring markets is “as much as two decades behind the technology currently used by those we regulate”. Scott O’Malia, a commissioner at the Commodity Futures Trading Commission admitted that the CTFC continues to receive account data by fax which then has to be manually entered. 

The use of real-time pre-trade risk technology is likely to become much more widespread. “Naked” access, where customers of brokers submit orders directly to the market without any pre-trade checks, is likely to be banned. This is an important change as late last year Aite Group, an analyst firm, estimated that naked access accounted for 38% of the average daily volume in US stocks. The SEC is also proposing that regulation of sponsored access is shorn up – currently it has evidence that brokers rely upon oral assurances that the customer itself has pre-trade risk technology deployed. The mandated use of pre-trade risk technology will level the playing field and will prevent a rush to the bottom. Personally I’ve heard of several instances of buy-side customers insisting to brokers that pre-trade risk controls are turned off as they perceive that such controls add latency and therefore will adversely affect the success of their trading.

The idea of real-time market surveillance, particularly in complex, fragmented markets as exist in the US and Europe is gaining credence. The SEC has proposed bringing in a “consolidated audit trail” which would enable all orders in US equity markets to be tracked in real-time. As John Bates said in his previous blog post, it’s likely that the US tax-payer will not be happy paying the $4B the publically funded SEC estimates that such a system would need to get up and running. Perhaps the US could look at the way the UK’s FSA is funded. The FSA reports to government but is paid for by the firms it regulates.

As I mentioned in my last blog our polling in April at Tradetech, a European equities trading event, suggests that market participants are ready for better market monitoring. 75% of respondents to our survey believed that creating more transparency with real-time market monitoring was preferable to the introduction of restrictive new rules.

CESR, the Committee of European Securities Regulators, is currently consulting on issues such as algorithmic trading and high frequency trading. It will be interesting to see the results of their deliberations in the coming months.

I’m so pleased the argument has moved on. This time last year saw a protracted period of vilifying “high frequency trading” and “algo trading”. Now, there is recognition of the benefits as well as the challenges that high frequency trading has brought to equity markets and regulators seem to understand that to both prevent disastrous errors and deliberate market manipulation occurring it is better for them to get on board with new technology rather than try to turn the clock back to mediaeval times. 

New approaches are sorely needed. Yesterday saw the conclusion of another investigation into market manipulation when the FSA handed out a $150,000 fine and a five-year ban to a commodity futures broker.

Monitoring and surveillance: the route to market transparency

Again this week, capital markets is under the spotlight, with the SEC and Goldman standoff. Just a few weeks ago, the FSA and UK Serious Organised Crime Agency were making multiple arrests for insider trading. Earlier this year Credit Suisse were fined by the New York Stock Exchange for one of their algorithmic trading strategies damaging the market. Still, electronic trading topics such as dark pools, high frequency trading are being widely debated. The whole capital markets industry is under scrutiny like never before.

Technology can't solve all these problems, but one thing it can do is to help give much more market transparency. We're of the view that to restore confidence in capital markets, organisations involved in trading need to have a much more accurate, real-time view on what's going on. In this way, issues can be prevented or at least identified much more quickly.  I talked about this recently to the Financial Times, here. 

Last week at the Tradetech conference in London, Progress announced its release of a second generation Market Monitoring and Surveillance Solution Accelerator. This is aimed at trading organisations who want to monitor trading behaviour, whether to ensure compliance with risk limits for example, or to spot abusive patterns of trading behaviour. Brokers, exchanges and regulators are particularly relevant, but buy-side organisations can also benefit from it. Previously this solution accelerator just used Apama. Now it's been extended to use our Responsive Business Process (RPM) suite, which includes not only Apama, but Savvion Business Process Management, which extends the accelerator to give it powerful alert and case management capabilities. We know that monitoring and surveillance in capital markets is important now, and believe it will become more so, which is exactly why we've invested in building out product. You can read the take on this from the financial services analyst Adam Honore here and more from Progress about the accelerator and RPM. A video on the surveillance accelerator is here. 

As all this is so relevant at the moment and Tradetech is the largest trading event of its kind in Europe (although very equity focused), we thought we'd conduct some research with the participants. We got exactly 100 responses on one day (which made calculating the percentages rather a breeze) to a survey which asked about attitudes to European regulation, high frequency and algorithmic trading and dark pools. Some of the responses relating to market monitoring and surveillance are worth stating here. 75% of respondents agreed to the premise that creating more transparency with real-time trading monitoring systems was preferable to the introduction of new rules and regulations. 65% of respondents believe that European regulators should be sharing equity trading information in real-time. And more than half believe that their own organisation would support regulators having open, real-time access to information about the firm's trading activity. To me, that's a pretty strong sign that the industry wants to open up, rather than be subjected to draconian new rules.

There will be substantial changes to the European equity trading landscape in the coming year. There will be post MiFID regulation change by the European Commission acting on recommendations by the Committee of European Securities Regulators who are taking industry evidence at the moment. Their mantra, as chanted last week, is "transparency, transparency, transparency". Let's hope that this transparency argument is expressed in opening up markets to more monitoring rather than taking a, perhaps politically expedient, route of outlawing certain practices and restricting others.

Observations from Tradetech 2010

Day one of Tradetech Europe 2010 has nearly finished. I won't be here tomorrow, so here are some thoughts and take-aways from today's event.

It's fair to say that Tradetech is the premier European equities trading and technology event, and thus very relevant for Progress' business in capital markets, particularly customers using Apama. Progress has a substantial presence as always. It's a good event to meet brokers, hedge funds, exchanges and pretty much every one within the industry. Lots of old friends are here every year. Regarding the event itself, it's pretty well attended considering the recent issues with volcanic ash. It usually takes place in Paris, but I'm sure the organisers were pleased that they chose London this year as the London contingent was able to attend without disruption.

This years big theme really seems to be market structure and regulation. In the third year after MiFID, an event which brought competition into European equity markets, and after the credit crunch, issues about how the market is working, the influence of alternative venues such as dark pool,  and how high-frequency trading is affecting the market are issues front of mind.

What's interesting is how some things stay the same. Richard Balarkas, old Tradetech hand and CEO of Instinet Europe, talked about trading liberalisation in the late 19th and early 20th century. Then, vested interests were complaining about the rise of "bucket shops", giving access to trading on the Chicago Board of Trade via telegraph to people that wouldn't previously have traded. In the view of some at the time, this lead to speculation and "gambling". Regulators were wrestling at the time with the fact that only 1% of CBOT trades resulted in actual delivery of goods - the rest were purely financial transactions and therefore arguably speculative. This reminds me of some of the current debate around the "social usefullness" of high frequency trading which is going on now.

European equities trading has changed a lot. Vodafone, a UK listed stock, has now only about 30% of its average European daily volume traded on the London Stock Exchange (LSE). The rest is traded on alternative trading venues across Europe. However, Xavier Rolet, CEO of the LSE, believes that there's a long way to go. He stated  that "the European equities market remains anaemic when compared to the US". Volumes, adjusted for relative market capitalisation, are about 15% of that in the US.

Regulation of European markets is a thorny issue. Regulation is fragmented, together with the market itself. CESR - the Committee of European Securities Regulators, the nearest Europe has to a single regulator - is taking evidence on a whole range of issues and will recommend a set of reforms to the European Commission in July this year. These recommendations will relate to post-trade transparency and information quality and enhanced information about systematic internalisers and broker crossing systems. CESR is also looking at other issues such as algorithmic trading and co-location. Legislation will follow towards the end of 2010.

Equity markets are in a sensitive place. There's still more deregulation to do, more competition to be encouraged and yet, with sentiment as it is, regulators may decide to introduce more rules and regulations to prevent this taking place. The CESR proposals will be about "transparency, transparency, transparency" - as part of this we believe that more real-time market monitoring and surveillance by all participants is key to bringing back confidence in the markets and ensuring that draconian rules don't have to be introduced.

Emerging markets were talked about in one session, and Cathryn Lyall from BM&FBovespa in the UK, talked about Brazil in particular. We've seen Brazil become a pretty significant market recently. Not only have demand grown for all Progress products substantially but Apama is now being used by 18 clients for algorithmic trading of both equities and derivatives. Brazil is the gorilla in the Latin American region. It accounts for 90% of cash equities and 95% of derivatives business in Latin America. 90% of Brazilian trading is on exchange. Brazil emerged largely unscathed from the credit crunch and it's taken only 2-3 years to achieve the level of trading infrastructure that took perhaps 10-15 years to evolve in the US and Europe. More still needs to happen. Although the regulatory regime has an enviable reputation, it is moving slowly. Concerns regarding naked and sponsored access are holding up liberalisation that would lead to DMA and co-located access to the equities market, something which is place already for derivatives.

So, that's what I saw as highlights from the day. Tradetech seems, still, to be the place the whole industry gathers.

Predictions for increased transparency in Capital Markets

It is my view that one of the most significant causes of the global financial crisis was a lack of transparency in financial markets.  Put simply, that means no one, not regulators or market participants, knew what the size of certain derivatives markets (like credit default swaps) was, who held what positions, or what the consequences of holding positions could be.  If financial reform brings nothing else, it should at least hold banks accountable for the business they conduct, and that means full disclosure and constant monitoring by responsible regulators.  

This action would help provide the basis for preventing future crises. No matter how inventive financial products may become, if regulators have complete and detailed information about financial markets and banks’ activities there, better assessments of risk can be made. This means that if necessary, banks’ activities can be reigned in through higher capital requirements or similar measures.  Simply limiting banks’ ability to conduct certain business is a blunt instrument that does not resolve the lack of transparency and likely will hamper economic growth.

Market transparency exhibits itself in many forms. Particularly relevant is that related to electronic trading. Therefore, I predict that regulators will require banks to implement relevant stronger pre-trade risk mechanisms. Regulators, such as the FSA & SEC, will ultimately bring in new rules to mitigate against, for example, the risk of algorithms ‘going mad’. This is exemplified by Credit Suisse, which was fined $150,000 by the NYSE earlier this year for “failing to adequately supervise development, deployment and operation of proprietary algorithms.”

Furthermore, volumes traded via high frequency trading will increase, although at a much slower pace than last year, and at the same time the emotive debates about high frequency trading creating a two-tier system and an unfair market will die down.

In addition, with regards to mid market MiFID monitoring, greater responsibility for compliance will be extended from exchanges to the banks themselves. Banks and brokers will soon be mandated to implement more trade monitoring and surveillance technology. There will also be no leeway on Dark Pools; they just simply have to change and be mandated to show they have adequate surveillance processes and technology in place. They will also have to expose more pricing information to the market and regulators.

This year will see a definite shift to an increasingly transparent – and therefore improved – working environment within capital markets. The ongoing development of market surveillance technologies and changes in attitudes to compliance will drive this forward, creating a more open and fairer marketplace for all.

Why BPM should be on the CIO’s agenda in 2010

New article by Giles Nelson published in CIO (http://bit.ly/biSKFo) online.

"In 2010, the business prerogative across all sectors is to use IT to drive efficiency and enable a business to react more quickly to customer and market changes. To do this, I believe we need to take a different view of BPM technology and try to see how it can be used to make knowledge-based business more ‘operationally responsive', reacting to customer needs and market changes instantly. This is already beginning to happen, and as it gains momentum, BPM will prove its usefulness in bringing ‘order to the chaos', and will make it onto the strategic agenda of every CIO."

The article does a great job at illustrating the synergy between business process management (BPM) and complex event processing; or in this case business event processing (BEP). Giles even provides examples of industry's already deploying these technologies. Most of us know the benefits of BPM but as he points out... "The next stage is to match it with the other side of the coin, where it can help an organisation respond to events and become truly operationally responsive - something worthy of the full attention of any CIO." Read the full article.

Open Source Software Powers the Biggest Physics Project in History

Today Progress Software announced that the European Organization for Nuclear Research (CERN*) is using Progress® FUSE™, to run its operational grid activities of the Large Hadron Collider (LHC) re-launch which happened this month. FUSE is an open source product line based on several Apache projects for which IONA (acquired by Progress in June 2008) provided leadership and Progress today continues to be a significant contributor. There are many skeptics that believe open source software isn’t meant for large-scale projects but CERN has proven that wrong. Not only has FUSE will underpin all grid monitoring systems used in CERN’s quest to find the Higgs Boson—known as 'The God Particle', but CERN welcomes the opportunity to contribute back to the open source project and deploy it freely across all their sites.

James Casey, Technical Architect at CERN, sites “We needed to find a partner that could help us bring agility and reliability to our IT infrastructure.” He added, “We have a pipeline of projects that we need to deliver over the coming years, so this first step lays the foundation for change.”

In addition to using FUSE, CERN also deployed Progress® SonicMQ® to form the communications backbone of its Technical Infrastructure Monitoring (TIM) system, designed to alert researchers in the event of an emergency. The use of open and “closed” source software creates a true open integration environment that re-enforces the fact that every organization has the power to choose the solutions that best fit their integrated infrastructure requirements.

The relationship between CEP, EDA and SOA

I published a post on our sister CEP blog last week. The topic has wider relevance than CEP so here's a link to it and I include the original text below. Enjoy.

I’d like to add my voice to the debate this week (here, here and here) on how Complex Event Processing (CEP) fits into the wider software architectural themes of Service Oriented Architectures (SOA) and Event Driven Architectures (EDA). Although I think I know how these three areas relate to one another fairly well, I was able to further clarify my thinking this week by spending some time with Neil Macehiter of Macehiter Ward-Dutton Advisors, a UK based software analyst. I found our discussion enlightening as Neil had a slightly different way of looking at these things than I had heard expressed previously. So let me try and express my own view on this in as clear a way as possible.

• CEP is a technology. SOA and EDA are not technologies. SOA and EDA are philosophies for the design and build of modern distributed computing architectures.
• A SOA is a loosely coupled set of services, the functionality of which closely reflects an organisation’s business functions and processes. A SOA will typically use modern, Web services technology and standards for implementation, but is not required to. Building SOA infrastructure requires much thinking about the services that the SOA will use.
• An EDA is a loosely coupled architecture, the endpoints of which interact with one another in an event-driven fashion. Information flows around the EDA as events. An EDA will have endpoints which produce events and endpoints which consume events. An EDA works in a “sense and respond” fashion. Building an EDA requires much thinking on the event-types that the EDA will use.
• An EDA may use business focussed services as endpoints. An EDA may therefore also be a SOA but it does not have to be.
• CEP is a capability within an EDA, providing analysis and matching of multiple events being sent between endpoints. You can have an EDA without CEP.
• If you’re building your architecture and focussing on defining event-types, it’s very likely you’re building an EDA.
• If you are using CEP then you have at least the beginnings of an EDA because you will have been focussing on event-types. Your EDA may a simple one, with one event producer and consumer, but it’s still an EDA.

Comments welcome!

Is transformational IT still locked out of the board room?

I recently took part in a roundtable discussion in London. Other participants included John Murdoch, Head of Consultancy from BT Global Services, a Progress Software customer and partner, Ruediger Spies, Independent Vice President Enterprise Applications from IDC, and members of the media.

The topic of discussion was a recent research project commissioned by Progress Software, which surveyed Chief Information Officers (CIOs) and heads of business across 500 European enterprises, with the purpose of examining the requirement and delivery of information in large and complex businesses. A lot of questions were asked in the research but the area we concentrated our discussion on was the role of the CIO. Here are a few of the more surprising results. The research found that in only 13% of cases are CIOs required to sign-off on a new business strategy. CIOs predict their budgets will increase less than eurozone inflation over the next few years whilst they are still expected to deliver more. Finally, only 51% of CIOs sit on the board of their companies.

All this paints a picture of IT being under pressure, having to respond to business strategies where it has no major influence in defining them and not being strategically valued in the way that one imagines IT should be.

Taking a closer look, there are some variations around Europe and between vertical sectors. In France, 69% of CIOs are at board level, whereas the UK only has 48%. This perhaps reflects France's increased faith in technology and the more sceptical attitude of the UK.  Telecommunications leads the way as a vertical with 64% of CIOs at board level. Manufacturing comes in last with just 42% in that position. Perhaps surprisingly, in financial services, a sector with IT very much at its heart, only has 49% of CIOs on the board.

But enough of statistics. The conclusion from the discussion was that many organisations risk missing out on opportunities if they underestimate the importance of IT to business strategy (anyone who doubts this only has to look at the way the Internet has influenced businesses in every vertical industry). In some industries, it is certainly more central than others - it's not easy to imagine how a telco operator would exist without IT for example - but it is always dangerous to simply view IT as providing efficiency gains and support to the business without looking for strategic opportunities to transform the way that the organisation works.

It seems that CIOs have work to do to prove their strategic impact and to better communicate the transformational value of technology. Likewise boards need to wise up. If IT isn't central to their business thinking, they risk being leapfrogged by more innovative competitors.

Event-driven SOA vs CEP. What is the difference?

Technology blogs and online communities have been chattering about event-driven SOA and many of our customers have been asking "How does event-driven SOA differ from CEP?" In this podcast, Giles Nelson gives an example of an event-driven style of business, and how today’s SOA infrastructure benefits from an event processing backbone is.

Listen to the entire interview below:

Subscribe to the SOA Infrastructure iTunes channel.

Gain Better Enterprise Visibility of Your SOA

What if you could gain better visibility of business systems and services across your SOA infrastructure? At Progress Software, we believe that you can achieve the promise of SOA without compromising your architecture or getting locked into a single-vendor approach. Listen as Giles Nelson, Director of Technology at Progress Software, presents his observations on the challenges that you and other enterprises face when trying to identify business requirements, deploy smart solutions that will support these requirements, and how - at the end of the day - you'll be able to monitor and analyze operations so that you can make smart decisions that will improve ROI and reduce risk.

Subscribe to our iTunes channel for more SOA Infrastructure podcasts from Progress Software.

Learn more about SOA Infrastructure products from Progress...

• Enterprise Service Bus - Sonic ESB
• SOA Management - Actional
• Complex Event Processing - Apama
• Data Interoperability - DataXtend
• Mainframe Integration - Shadow

SOA Governance and Management - the Difference

Joe McKendrick's recent post comments that many analysts are now putting SOA governance and SOA management tools into a single category. He cites arguments that they are in fact very different, stating that while governance is about managing the development and the life cycle of services, management is about the technical monitoring of such. I couldn’t agree more.

Lets spell it out: in simple terms SOA governance is focused mainly on the development process, while SOA management is focused on the 'running' aspects of SOA infrastructure. Management systems must ensure that that the business services in the SOA remain reliable and live up to the expectations of the business partners, customers, suppliers, internal users and regulators.

I believe people are confused by the various contexts in which the term "SOA governance is used." In the context of a product category it is indeed confusing to lump together products with very different uses and characteristics. However, if an organisation is designing a "SOA governance strategy" then tools to aid in the development life cycle of services as well as tools to allow those services to be monitored at run-time may be required. Some organisations may describe this latter capability as "runtime governance."

Developmental governance and run-time management tools are distinct but closely related. They both aid to make SOA successful. Increasingly they work together to form a link across the entire life cycle of development and production deployment, in terms of enforcing and sharing enterprise and business policy.

So, feel free to think about the governance of your SOA in holistic, wide-ranging terms. But when it comes to thinking about which products to use, remember the distinctions. The fog of confusion will only arise if you don't.

The Emergence of CEP

There's good evidence that Complex Event Processing is transitioning from interesting but niche software technology into a more powerful theme within the software industry.

Let me present some evidence for this from a number of angles. Firstly, the analysts. Although Gartner is being somewhat reticent about publishing anything explicitly on CEP, they did hold the first analyst-hosted dedicated event processing summit in September 2007. Earlier this year, Forrester Research published a report on CEP. One of the surprising results of their research was that 71% of the decision makers they surveyed were aware of CEP (particularly surprising was that there is reportedly greater awareness of CEP than another software TLA, Enterprise Service Bus (ESB), a term that has been around for some years). Forrester sees that the market is currently small but, without putting any figures on it, has much potential for growth. IDC has been talking about Event Processing and Correlation software (which, for the sake of argument, we can treat as being equivalent to CEP) having a compound annual growth rate of 70% for the next few years. AITE Research, specialists in financial services, highlight the tremendous growth potential of event processing in financial services, arguing the market may be shortly worth $1B. Bloor research have also been publishing in the event processing area. The point about this is that not only are analyst firms now talking and making predictions about the size of the market, but that they've done pretty much all of this in the last year. Rewind the clock 12 months... the amount of analyst coverage around event processing was virtually zero.

Much vendor activity has also taken place in the last year. In addition to those that have been around a while with offerings - Progress, Tibco, Streambase, Coral8 and some others - other established vendors are getting on board. BEA launched their Event Server product last year. Oracle has now got a CEP product (in addition to that acquired from BEA), and IBM acquired Aptsoft in January. Vendors are also increasingly seeing CEP as fitting into their wider enterprise offerings, not just as a niche. Usually this will be in the context of an event-driven SOA, sometimes termed an Event Driven Architecture or EDA.

This wider context for CEP is being seen by potential users of it. The best known area of event processing applications is in the finance domain, in particular in relation to high-frequency trading applications. This is still really the only domain with significant commercial traction. From our perspective at Progress, we're now seeing a broader set of application domains take an interest in it - logistics, telco, insurance and retail. Also we're seeing requests for information from end-users which have CEP as one of several enterprise architecture capabilities that they're looking for - a reflection again of the beginnings of CEP maturity. Forrester observes that CEP is very closely aligned to the business, because "it's the business who understands events." This is the case now, but I predict that increasingly the acquisition of a CEP capability will move from the business to architects and CIOs as its appeal broadens and the understanding of it in a wider enterprise architecture context becomes better understood.

On a closing point, we're very fond of TLAs in the software industry, CEP being one of them. CEP will probably stick - there's a fair amount of momentum behind it now, but I reckon that plain old EP, or Event Processing, would be better. "Complex" is rather a regrettable word to describe something which is hoped, by vendors at least, to have widespread market acceptance. Business Intelligence has done alright after all. Perhaps there is time to change - according to Google, the Communciations, Energy and Paperworkers Union of Canada is presently a far better known user of the acronym CEP.

Business visibility should be a driver for SOA

I've just read a Gartner research note by Massimo Pezzini published on the 16 November. It's titled Greater Business Process Insight Is an Unexpected Benefit of SOA. It argues that an unexpected and unplanned consequence of SOA projects is that organisations gain better visibility of their business processes and business data. It recommends that organisations think and plan about things like Business Activity Monitoring (BAM) from the outset.

I was surprised by this and also disappointed. Not because I don't agree that better business visibility is a consequence of SOA – it is. What surprises me is that this is still seen as noteworthy. It shows how far the industry still has to go toward understanding the merits of enterprise architectures which give them more open, simpler, more flexible access to real-time business information. service-Oriented Architecture (SOA) is about service reuse, open standards, etc. etc. But you're doing this for a reason, not for its own sake. Surely one of the benefits of a more open, flexible architecture is that you can get at your business information better so you can drive efficiencies in your processes, identify threats and opportunities more quickly, and better communicate with customers, etc?

Some organisations do recognise this. Only last week we responded to an RFI which not only included "usual suspect" SOA requirements satisfied by such things as an Enterprise Service Bus (ESB), but also requirements for event processing and BAM. The organisation recognised that the building of a SOA infrastructure gives them access to business event streams that they can obtain value from immediately. They had indeed expected this consequence and were planning for it.

SOA What? These kinds of requirements will become the norm. Real-time access to enterprise information is becoming more relevant in a whole range of industries – we're seeing this directly in some of the engagements we're involved in. Furthermore, event processing and BAM are at the more business end of the SOA universe. One of the things that the industry harps on about is that the business rarely sees the relevance of a SOA initiative. By taking the advice of the article and thinking about these things apriori then perhaps not only can organisations get more out of their initiatives but also CIOs can gain a little more credit from the wider organisation.

Selling Oriented Architecture

I participated in a SOA roundtable the other week. End-user organisations were present – in banking, automotive, logistics and other sectors. It was chaired by Butler Group, the analysts. The topic of the roundtable was how SOA technology and principles were being adopted in end-user organisations. The participants were invited to talk about what had worked and had not worked in their own organisations and how they saw their initiatives evolving. As usual many topics were covered including the usual yawn-inducing argument about whether “it’s about the technology” or “it’s about organisational change” (why can’t people just accept that it’s about both). Fortunately, this led us onto the much more interesting territory about how to incent members of the IT organisation to share the services they build. After all, two of the usual arguments trotted out to support SOA are reuse and the help SOA gives to cut across organisational silos to achieve greater efficiencies.

Heaven can wait

Too often organisations consider acquiring IT with an ambitious programme of IT infrastructure change and the belief that a radical, multi-year re-architecting of the organisation’s systems will lead to a more capable and modern IT architecture that is better able to respond to the requirements of the business. While this "IT heaven" may be a laudable objective, it is a very difficult one to achieve and, in our opinion, is unsuitable for most organisations.

IT heaven can wait. We encourage clients to take an approach which is focussed on definable business projects each delivering value, typically in a 9-12 month timeframe. It sounds obvious doesn't it? A recent experience with a large investment bank shows that to many it isn't.

The bank had gone out to tender for the capabilities it wanted to procure in the IT organisation. These capabilities were ambitious. The programme's intent was to transform the way that IT projects were delivered - all very exciting. Naturally I imagined that they had a whole set of business projects lined up just waiting for the right infrastructure in order to deliver them. When asked about this though, they appeared to be rather taken aback - "Projects in mind? No, we don't have any specific projects in mind." I was surprised, and disappointed. They were operating in a vacuum. They certainly had vision, but they had no justification for actually implementing it and for spending any money with us. They just hadn't got other parts of the organisation onboard with this programme of activity. As I write, they have yet to make their planned investment…

Maintain your vision to be sure, but make sure you deliver some demonstrable business projects along the way.

SOA What? SOA is all about enabling incremental change. It's about modern and legacy systems working together. It’s about working in a heterogeneous world and evolving your SOA infrastructure in non-disruptive way.

I see events. They’re everywhere.

After having been involved in the study and implementation of advance event processing techniques and technology since the mid 1990s, it’s gratifying to see this field getting much deserved attention. In a recent presentation, Yefim Natis of Gartner, predicted that Complex Event Processing (CEP) will be one of the five most significant IT trends over the next few years. Yefim also noted that CEP will increasingly become the foundation for the next generation of Business Activity Monitoring (BAM) applications. This is very good to hear as this is a conclusion that we have also reached.

One reason the market is getting excited about CEP is that event-capable middleware—most significantly the Enterprise Service Bus (ESB)—is now becoming mainstream in corporate IT. ESBs, most popularly regarded as the critical messaging-based backbone for SOA implementations, are just as capable for event-driven architecture (EDA) implementations because of their messaging heritage. EDA takes the SOA principle of loosely coupled services one step further – to decoupled services. Whilst the interaction model is a little different, the underlying technology remains the same so ESBs provide an important driver for CEP adoption. Along with the adoption of these technologies also comes familiarity with the whole event processing model, and the technical skill and best practices that go with it. The result—in short—is the sudden availability of business event on a much more pervasive scale. Here, the event processing version of Metcalfe's law kicks in—the value of CEP is increases exponentially as more events become available to process. That is, as long as your event processing infrastructure can handle it.

Another reason interest in CEP is rising is the proven success of the technology within capital markets. Not only have most of the leading investment banks adopted CEP to implement their algorithmic trading and risk management systems but now the financial markets and regulators are adopting the technology to monitor the market in real time to detect compliance violations or patterns of fraud, like insider trading. For example, the Financial Services Authority (FSA) is adopting CEP technology to help monitor the UK markets and enforce MiFID compliance in an increasingly fragmented trading environment. The system called SABRE II is being developed by Detica. You can read about it here.

But it’s not just capital markets that will benefit from CEP. It is also being adopted on the manufacturing floor for process control and improvement, in logistics applications for early detection of supply chain issues, in retail RFID for inventory management and security, by military branches for situational awareness of the battlefield, and even by casinos to help their monitoring of suspicious gaming behaviour. It seems events are everywhere these days.

I was recently at an academically hosted event in Germany. Many vendors and practitioners attended and many themes in software were discussed – BPM, BAM, SOA, CEP amongst others. What came over loud and clear in the sessions was that CEP was increasingly seen as a major new and important capability that fitted well into the SOA world. An ESB for example was important in giving the foundation for the gathering and analysis of events from around the enterprise.

As more and more organizations embrace SOA infrastructure that supports event processing—and the event processing skill set that goes with it, the value of CEP technology will continue to rise. In fact, expect interest in CEP-powered applications (BAM and otherwise) to be a real driver in infrastructure investment over the next few years. Soon you’ll hear many more CIOs saying: “I see events. They’re everywhere.”