Integrated Infrastructure Blog

Open Source Software Powers the Biggest Physics Project in History

Today Progress Software announced that the European Organization for Nuclear Research (CERN*) is using Progress® FUSE™, to run its operational grid activities of the Large Hadron Collider (LHC) re-launch which happened this month. FUSE is an open source product line based on several Apache projects for which IONA (acquired by Progress in June 2008) provided leadership and Progress today continues to be a significant contributor. There are many skeptics that believe open source software isn’t meant for large-scale projects but CERN has proven that wrong. Not only has FUSE will underpin all grid monitoring systems used in CERN’s quest to find the Higgs Boson—known as 'The God Particle', but CERN welcomes the opportunity to contribute back to the open source project and deploy it freely across all their sites.

James Casey, Technical Architect at CERN, sites “We needed to find a partner that could help us bring agility and reliability to our IT infrastructure.” He added, “We have a pipeline of projects that we need to deliver over the coming years, so this first step lays the foundation for change.”

In addition to using FUSE, CERN also deployed Progress® SonicMQ® to form the communications backbone of its Technical Infrastructure Monitoring (TIM) system, designed to alert researchers in the event of an emergency. The use of open and “closed” source software creates a true open integration environment that re-enforces the fact that every organization has the power to choose the solutions that best fit their integrated infrastructure requirements.

The relationship between CEP, EDA and SOA

I published a post on our sister CEP blog last week. The topic has wider relevance than CEP so here's a link to it and I include the original text below. Enjoy.

I’d like to add my voice to the debate this week (here, here and here) on how Complex Event Processing (CEP) fits into the wider software architectural themes of Service Oriented Architectures (SOA) and Event Driven Architectures (EDA). Although I think I know how these three areas relate to one another fairly well, I was able to further clarify my thinking this week by spending some time with Neil Macehiter of Macehiter Ward-Dutton Advisors, a UK based software analyst. I found our discussion enlightening as Neil had a slightly different way of looking at these things than I had heard expressed previously. So let me try and express my own view on this in as clear a way as possible.

• CEP is a technology. SOA and EDA are not technologies. SOA and EDA are philosophies for the design and build of modern distributed computing architectures.
• A SOA is a loosely coupled set of services, the functionality of which closely reflects an organisation’s business functions and processes. A SOA will typically use modern, Web services technology and standards for implementation, but is not required to. Building SOA infrastructure requires much thinking about the services that the SOA will use.
• An EDA is a loosely coupled architecture, the endpoints of which interact with one another in an event-driven fashion. Information flows around the EDA as events. An EDA will have endpoints which produce events and endpoints which consume events. An EDA works in a “sense and respond” fashion. Building an EDA requires much thinking on the event-types that the EDA will use.
• An EDA may use business focussed services as endpoints. An EDA may therefore also be a SOA but it does not have to be.
• CEP is a capability within an EDA, providing analysis and matching of multiple events being sent between endpoints. You can have an EDA without CEP.
• If you’re building your architecture and focussing on defining event-types, it’s very likely you’re building an EDA.
• If you are using CEP then you have at least the beginnings of an EDA because you will have been focussing on event-types. Your EDA may a simple one, with one event producer and consumer, but it’s still an EDA.

Comments welcome!

Is transformational IT still locked out of the board room?

I recently took part in a roundtable discussion in London. Other participants included John Murdoch, Head of Consultancy from BT Global Services, a Progress Software customer and partner, Ruediger Spies, Independent Vice President Enterprise Applications from IDC, and members of the media.

The topic of discussion was a recent research project commissioned by Progress Software, which surveyed Chief Information Officers (CIOs) and heads of business across 500 European enterprises, with the purpose of examining the requirement and delivery of information in large and complex businesses. A lot of questions were asked in the research but the area we concentrated our discussion on was the role of the CIO. Here are a few of the more surprising results. The research found that in only 13% of cases are CIOs required to sign-off on a new business strategy. CIOs predict their budgets will increase less than eurozone inflation over the next few years whilst they are still expected to deliver more. Finally, only 51% of CIOs sit on the board of their companies.

All this paints a picture of IT being under pressure, having to respond to business strategies where it has no major influence in defining them and not being strategically valued in the way that one imagines IT should be.

Taking a closer look, there are some variations around Europe and between vertical sectors. In France, 69% of CIOs are at board level, whereas the UK only has 48%. This perhaps reflects France's increased faith in technology and the more sceptical attitude of the UK.  Telecommunications leads the way as a vertical with 64% of CIOs at board level. Manufacturing comes in last with just 42% in that position. Perhaps surprisingly, in financial services, a sector with IT very much at its heart, only has 49% of CIOs on the board.

But enough of statistics. The conclusion from the discussion was that many organisations risk missing out on opportunities if they underestimate the importance of IT to business strategy (anyone who doubts this only has to look at the way the Internet has influenced businesses in every vertical industry). In some industries, it is certainly more central than others - it's not easy to imagine how a telco operator would exist without IT for example - but it is always dangerous to simply view IT as providing efficiency gains and support to the business without looking for strategic opportunities to transform the way that the organisation works.

It seems that CIOs have work to do to prove their strategic impact and to better communicate the transformational value of technology. Likewise boards need to wise up. If IT isn't central to their business thinking, they risk being leapfrogged by more innovative competitors.

Event-driven SOA vs CEP. What is the difference?

Technology blogs and online communities have been chattering about event-driven SOA and many of our customers have been asking "How does event-driven SOA differ from CEP?" In this podcast, Giles Nelson gives an example of an event-driven style of business, and how today’s SOA infrastructure benefits from an event processing backbone is.

Listen to the entire interview below:

Subscribe to the SOA Infrastructure iTunes channel.

Gain Better Enterprise Visibility of Your SOA

What if you could gain better visibility of business systems and services across your SOA infrastructure? At Progress Software, we believe that you can achieve the promise of SOA without compromising your architecture or getting locked into a single-vendor approach. Listen as Giles Nelson, Director of Technology at Progress Software, presents his observations on the challenges that you and other enterprises face when trying to identify business requirements, deploy smart solutions that will support these requirements, and how - at the end of the day - you'll be able to monitor and analyze operations so that you can make smart decisions that will improve ROI and reduce risk.

Subscribe to our iTunes channel for more SOA Infrastructure podcasts from Progress Software.

Learn more about SOA Infrastructure products from Progress...

• Enterprise Service Bus - Sonic ESB
• SOA Management - Actional
• Complex Event Processing - Apama
• Data Interoperability - DataXtend
• Mainframe Integration - Shadow

SOA Governance and Management - the Difference

Joe McKendrick's recent post comments that many analysts are now putting SOA governance and SOA management tools into a single category. He cites arguments that they are in fact very different, stating that while governance is about managing the development and the life cycle of services, management is about the technical monitoring of such. I couldn’t agree more.

Lets spell it out: in simple terms SOA governance is focused mainly on the development process, while SOA management is focused on the 'running' aspects of SOA infrastructure. Management systems must ensure that that the business services in the SOA remain reliable and live up to the expectations of the business partners, customers, suppliers, internal users and regulators.

I believe people are confused by the various contexts in which the term "SOA governance is used." In the context of a product category it is indeed confusing to lump together products with very different uses and characteristics. However, if an organisation is designing a "SOA governance strategy" then tools to aid in the development life cycle of services as well as tools to allow those services to be monitored at run-time may be required. Some organisations may describe this latter capability as "runtime governance."

Developmental governance and run-time management tools are distinct but closely related. They both aid to make SOA successful. Increasingly they work together to form a link across the entire life cycle of development and production deployment, in terms of enforcing and sharing enterprise and business policy.

So, feel free to think about the governance of your SOA in holistic, wide-ranging terms. But when it comes to thinking about which products to use, remember the distinctions. The fog of confusion will only arise if you don't.

The Emergence of CEP

There's good evidence that Complex Event Processing is transitioning from interesting but niche software technology into a more powerful theme within the software industry.

Let me present some evidence for this from a number of angles. Firstly, the analysts. Although Gartner is being somewhat reticent about publishing anything explicitly on CEP, they did hold the first analyst-hosted dedicated event processing summit in September 2007. Earlier this year, Forrester Research published a report on CEP. One of the surprising results of their research was that 71% of the decision makers they surveyed were aware of CEP (particularly surprising was that there is reportedly greater awareness of CEP than another software TLA, Enterprise Service Bus (ESB), a term that has been around for some years). Forrester sees that the market is currently small but, without putting any figures on it, has much potential for growth. IDC has been talking about Event Processing and Correlation software (which, for the sake of argument, we can treat as being equivalent to CEP) having a compound annual growth rate of 70% for the next few years. AITE Research, specialists in financial services, highlight the tremendous growth potential of event processing in financial services, arguing the market may be shortly worth $1B. Bloor research have also been publishing in the event processing area. The point about this is that not only are analyst firms now talking and making predictions about the size of the market, but that they've done pretty much all of this in the last year. Rewind the clock 12 months... the amount of analyst coverage around event processing was virtually zero.

Much vendor activity has also taken place in the last year. In addition to those that have been around a while with offerings - Progress, Tibco, Streambase, Coral8 and some others - other established vendors are getting on board. BEA launched their Event Server product last year. Oracle has now got a CEP product (in addition to that acquired from BEA), and IBM acquired Aptsoft in January. Vendors are also increasingly seeing CEP as fitting into their wider enterprise offerings, not just as a niche. Usually this will be in the context of an event-driven SOA, sometimes termed an Event Driven Architecture or EDA.

This wider context for CEP is being seen by potential users of it. The best known area of event processing applications is in the finance domain, in particular in relation to high-frequency trading applications. This is still really the only domain with significant commercial traction. From our perspective at Progress, we're now seeing a broader set of application domains take an interest in it - logistics, telco, insurance and retail. Also we're seeing requests for information from end-users which have CEP as one of several enterprise architecture capabilities that they're looking for - a reflection again of the beginnings of CEP maturity. Forrester observes that CEP is very closely aligned to the business, because "it's the business who understands events." This is the case now, but I predict that increasingly the acquisition of a CEP capability will move from the business to architects and CIOs as its appeal broadens and the understanding of it in a wider enterprise architecture context becomes better understood.

On a closing point, we're very fond of TLAs in the software industry, CEP being one of them. CEP will probably stick - there's a fair amount of momentum behind it now, but I reckon that plain old EP, or Event Processing, would be better. "Complex" is rather a regrettable word to describe something which is hoped, by vendors at least, to have widespread market acceptance. Business Intelligence has done alright after all. Perhaps there is time to change - according to Google, the Communciations, Energy and Paperworkers Union of Canada is presently a far better known user of the acronym CEP.

Business visibility should be a driver for SOA

I've just read a Gartner research note by Massimo Pezzini published on the 16 November. It's titled Greater Business Process Insight Is an Unexpected Benefit of SOA. It argues that an unexpected and unplanned consequence of SOA projects is that organisations gain better visibility of their business processes and business data. It recommends that organisations think and plan about things like Business Activity Monitoring (BAM) from the outset.

I was surprised by this and also disappointed. Not because I don't agree that better business visibility is a consequence of SOA – it is. What surprises me is that this is still seen as noteworthy. It shows how far the industry still has to go toward understanding the merits of enterprise architectures which give them more open, simpler, more flexible access to real-time business information. service-Oriented Architecture (SOA) is about service reuse, open standards, etc. etc. But you're doing this for a reason, not for its own sake. Surely one of the benefits of a more open, flexible architecture is that you can get at your business information better so you can drive efficiencies in your processes, identify threats and opportunities more quickly, and better communicate with customers, etc?

Some organisations do recognise this. Only last week we responded to an RFI which not only included "usual suspect" SOA requirements satisfied by such things as an Enterprise Service Bus (ESB), but also requirements for event processing and BAM. The organisation recognised that the building of a SOA infrastructure gives them access to business event streams that they can obtain value from immediately. They had indeed expected this consequence and were planning for it.

SOA What? These kinds of requirements will become the norm. Real-time access to enterprise information is becoming more relevant in a whole range of industries – we're seeing this directly in some of the engagements we're involved in. Furthermore, event processing and BAM are at the more business end of the SOA universe. One of the things that the industry harps on about is that the business rarely sees the relevance of a SOA initiative. By taking the advice of the article and thinking about these things apriori then perhaps not only can organisations get more out of their initiatives but also CIOs can gain a little more credit from the wider organisation.

Selling Oriented Architecture

I participated in a SOA roundtable the other week. End-user organisations were present – in banking, automotive, logistics and other sectors. It was chaired by Butler Group, the analysts. The topic of the roundtable was how SOA technology and principles were being adopted in end-user organisations. The participants were invited to talk about what had worked and had not worked in their own organisations and how they saw their initiatives evolving. As usual many topics were covered including the usual yawn-inducing argument about whether “it’s about the technology” or “it’s about organisational change” (why can’t people just accept that it’s about both). Fortunately, this led us onto the much more interesting territory about how to incent members of the IT organisation to share the services they build. After all, two of the usual arguments trotted out to support SOA are reuse and the help SOA gives to cut across organisational silos to achieve greater efficiencies.

Heaven can wait

Too often organisations consider acquiring IT with an ambitious programme of IT infrastructure change and the belief that a radical, multi-year re-architecting of the organisation’s systems will lead to a more capable and modern IT architecture that is better able to respond to the requirements of the business. While this "IT heaven" may be a laudable objective, it is a very difficult one to achieve and, in our opinion, is unsuitable for most organisations.

IT heaven can wait. We encourage clients to take an approach which is focussed on definable business projects each delivering value, typically in a 9-12 month timeframe. It sounds obvious doesn't it? A recent experience with a large investment bank shows that to many it isn't.

The bank had gone out to tender for the capabilities it wanted to procure in the IT organisation. These capabilities were ambitious. The programme's intent was to transform the way that IT projects were delivered - all very exciting. Naturally I imagined that they had a whole set of business projects lined up just waiting for the right infrastructure in order to deliver them. When asked about this though, they appeared to be rather taken aback - "Projects in mind? No, we don't have any specific projects in mind." I was surprised, and disappointed. They were operating in a vacuum. They certainly had vision, but they had no justification for actually implementing it and for spending any money with us. They just hadn't got other parts of the organisation onboard with this programme of activity. As I write, they have yet to make their planned investment…

Maintain your vision to be sure, but make sure you deliver some demonstrable business projects along the way.

SOA What? SOA is all about enabling incremental change. It's about modern and legacy systems working together. It’s about working in a heterogeneous world and evolving your SOA infrastructure in non-disruptive way.

I see events. They’re everywhere.

After having been involved in the study and implementation of advance event processing techniques and technology since the mid 1990s, it’s gratifying to see this field getting much deserved attention. In a recent presentation, Yefim Natis of Gartner, predicted that Complex Event Processing (CEP) will be one of the five most significant IT trends over the next few years. Yefim also noted that CEP will increasingly become the foundation for the next generation of Business Activity Monitoring (BAM) applications. This is very good to hear as this is a conclusion that we have also reached.

One reason the market is getting excited about CEP is that event-capable middleware—most significantly the Enterprise Service Bus (ESB)—is now becoming mainstream in corporate IT. ESBs, most popularly regarded as the critical messaging-based backbone for SOA implementations, are just as capable for event-driven architecture (EDA) implementations because of their messaging heritage. EDA takes the SOA principle of loosely coupled services one step further – to decoupled services. Whilst the interaction model is a little different, the underlying technology remains the same so ESBs provide an important driver for CEP adoption. Along with the adoption of these technologies also comes familiarity with the whole event processing model, and the technical skill and best practices that go with it. The result—in short—is the sudden availability of business event on a much more pervasive scale. Here, the event processing version of Metcalfe's law kicks in—the value of CEP is increases exponentially as more events become available to process. That is, as long as your event processing infrastructure can handle it.

Another reason interest in CEP is rising is the proven success of the technology within capital markets. Not only have most of the leading investment banks adopted CEP to implement their algorithmic trading and risk management systems but now the financial markets and regulators are adopting the technology to monitor the market in real time to detect compliance violations or patterns of fraud, like insider trading. For example, the Financial Services Authority (FSA) is adopting CEP technology to help monitor the UK markets and enforce MiFID compliance in an increasingly fragmented trading environment. The system called SABRE II is being developed by Detica. You can read about it here.

But it’s not just capital markets that will benefit from CEP. It is also being adopted on the manufacturing floor for process control and improvement, in logistics applications for early detection of supply chain issues, in retail RFID for inventory management and security, by military branches for situational awareness of the battlefield, and even by casinos to help their monitoring of suspicious gaming behaviour. It seems events are everywhere these days.

I was recently at an academically hosted event in Germany. Many vendors and practitioners attended and many themes in software were discussed – BPM, BAM, SOA, CEP amongst others. What came over loud and clear in the sessions was that CEP was increasingly seen as a major new and important capability that fitted well into the SOA world. An ESB for example was important in giving the foundation for the gathering and analysis of events from around the enterprise.

As more and more organizations embrace SOA infrastructure that supports event processing—and the event processing skill set that goes with it, the value of CEP technology will continue to rise. In fact, expect interest in CEP-powered applications (BAM and otherwise) to be a real driver in infrastructure investment over the next few years. Soon you’ll hear many more CIOs saying: “I see events. They’re everywhere.”