Business Making Progress

Progress Software Introduces Actional Diagnostics

Presented by Dan Foody, VP of Products for Actional at Progress Software

With Actional Diagnostics, you can accelerate Web service development, testing and validation with tools to improve service quality, performance, and compliance, and simplify time consuming XML-oriented tasks. Listen to this podcast and hear Dan talk talk about the features and benefits of introducing a service quality and validation tool into your SOA infrastructure initiative. Register to be alerted when the FREE download is available.

For more SOA podcasts, subscribe to our iTunes channel.

Calling all VARs

Greetings from Raleigh on January 26, 2009

I suppose an introduction is in order. My name is David Olson. Long time Progress person and I tend to float between technical and business roles in the company. Recently crossed over into the Apama world to focus on opportunities other than Capital Markets, which brings me to an interesting observation…

I've spent a lot of time working with our Application Partners (formerly known as VARs) and over the past few years I've made an effort to push CEP concepts to key partners. Many are in industries that have a few event streams that are ripe for the picking. Manufacturing, logistics, telco, and retail certainly rise to the top of the list. All tend to agree that there are events streaming through their customer's infrastructure and there may be a potential use case there but, few know what to do about it - and I suppose that's understandable.

For 50 years or so, we've concentrated our business application development on transforming traditional business processes into software. MRP/ERP has certainly led the call – but for the most part, ERP is done. And like the Holy Grail in the Monty Python movie it's getting harder to sell ERP because, you see, we’ve already got one.

Operational Effectiveness through CEP is the next frontier for business application software and I believe that it will shake up business processing the same way ERP did in the past. This isn't to say that ERP is not necessary, it's just that underlying platforms are not conducive for the "in the moment" nature of CEP solutions. In fact, the two should live comfortably side-by-side.

So VARs, this is your call. The infrastructure is mature, the technology is available, and your customers are ready. There's a new seed you can plant that can bring additional opportunity to your existing solutions. Take a look at what Manuvis has done with their FactoryMRI solution set. Real-time dashboards - right on their home page. CEP's a big part of what they do and they're making waves.

You should be next.

-DO

SOA and Events - Two Sides of BAM?

I came across a prospect recently (new markets so engineering help is always welcomed :-) ) - a domain consulting company now building a Governance Risk and Compliance (GRC) solution. Now, this very much being in the area of BAM, one would expect that traditional Business Activity Monitoring (BAM) would be used. But as we were discussing it more, it turns out that more often than not, GRC is about triggering the risk assessment and compliance/governance checks as business events occur. Here again, the trite assumption (that even I made) is that the organization has SOA imbibed all over and so it can easily tie in BAM and use that BAM environment to 'detect' and trigger the necessary checks/rules. And here again, I was proven wrong because SOA is not as widely prevalent as we all would like to believe! The consulting company, a group with a strong domain expertise in Governance & Risk/Performance, who is building the solution said they wanted an "extremely loose" integration, but their prospects were unlikely to have a mature SOA infrastructure. Which means, they cannot really expect BAM to be practical.

After looking at the biz analysis they had already performed, and taking the time to breakdown the GRC system into a bunch of flow-charts (one for each scenario), they found a pattern:

• a business event occurs as part of ongoing business processing
• the event includes the relevant business data
• a series of GRC checks and rules are executed
• any violation is recorded and an alert raised

This consulting company has gone a step further and also modeled the whole biz domain (they were targeting a specific sector in the financial space) - in a very interesting manner. As a set of key business events, anyone in the domain can immediately understand what each of these events mean. Each business event includes a clear definition of the accompanying data. In effect, they flipped the complete domain business ops as a set of business events! Hmmm...

So far, we have seen models that show the enterprise as a bunch of business processes that invoke a set of services. With the above flipped view, the enterprise modeled as a bunch of biz events is very unique. The former is a view of defining the business flows (processes) that form the enterprise's biz processing. The latter view is more of a snapshot of the enterprise post the fact - as business events that "occur" in the enterprise. Both very clearly model the purpose and the data involved, but in the former method, it is the data that flows through. And with the latter approach, the data that gets "generated" post a business activity.

From this consulting company's point of view, modeling the enterprise as a set of business events and the data that each event emanates was a more practical view. The communication to their clients was that they have modeled the GRC flow for each of these biz events. And the integration of the GRC solution involved the client's IT team "wiring" in these events into their existing solutions - wherever the said business activity occurs, and in any technology they may have, SOA or no SOA.  When they wire this, they will also need to pull out the data from their existing systems and "emit" the same as per the defined event data structure.

Pretty neat idea! Is anyone else seeing it this way? Is a GRC solution more about events and less about SOA? Is SOA primary and events incidental, or are events primary and the SOA infrastructure that could generate this be incidental?

The World According to a 4-Year-Old, and the Real Business of Foreign Exchange

The following conversation with my 4-year-old daughter happened just a few days ago:

Her:    I really like China, it sounds like a great place.

Me:     Why?

Her:    They have the best toys in the world there.

Me:     How do you know?

Her:    All my favorite toys are "Made in China."

I stopped asking her questions.  But my mind quickly went from wondering about the origins of toys to wondering about the origins of the capital markets business applications I encounter these days.  These applications rely heavily on functions like alpha-seeking and execution algorithms, and foreign exchange aggregation, to name a few.  I’ll elaborate on others in future posts, but the point is that this is a high risk environment, not child’s play.  So where do the ideas for those functions come from? 

A few Apama customers I have met lately note that the markets seem to be acting like a bunch of 4-year-olds having temper tantrums.  They noted wild swings in the FX marketplace, for instance, since the credit crunch became a crisis a few months ago.  FX trading will continue to be frantic this year. But necessity is indeed the mother of invention, which is a good thing with the markets the way they are.

This week, we announced that Standard Chartered Bank has gone live with the Progress Apama FX Market Aggregation Accelerator.  We’ve talked about FX aggregation before, and have done a number of similar Tier 1 deployments.  As you would expect, some great ideas in these projects have been customer-driven.  And when I look behind the scenes at Apama, I see how these projects provide a springboard for people to think of new things and try them out.  

On my very first day at Apama, a new colleague walked up to me and said, “I solved a problem that was bugging me.  Do you think customers might like this too?”  These ideas tend to grow into a demo, which may become the basis for a collaborative deployment, and eventually a customer has built on the idea and taken it in a new direction – a similar Apama story can be found here. 

Yesterday someone asked me if I think that Capital Markets firms are using technology to save cost or to grow.  Cost savings?  Yes, at least, and for some that may be the end of the story and the mind is closed.  But a former manager of mine, now a colleague and adviser, taught me to listen to the markets by telling me, “You have 2 ears and 1 mouth, use them in that proportion.”   A similar philosophy serves software firms, as well.

Funny that this week, one of my Wall St. friends at a very alive firm wrote to tell me his opinions about the state of software evolution, and the importance of innovation.  I think the crisis will force patches of growth and innovation.  Do you?  Comments welcome.

 

-Dan

The Data Architecture Dialogues - Chapter 2 of The Semantic Dialogues

Chapter 2 of The Semantic Dialogues, The Data Architecture Dialogues, is available now. "With too many different data stores, too much bad data, too many incompatible definitions of the data (and forget about managing change) it doesn’t seem like a bad idea to create an Enterprise Data Platform. But some members of the team push back. Cliff Chen, database designer, stands his ground as the debate continues."

Read Chapter 2, The Data Architecture Dialogues, today. And if you’re enjoying The Semantic Dialogues, be sure to share it with a colleague.

The Semantic Dialogues tells the story of how National Networks, a fictional telecommunications service provider, pursued and achieved data interoperability within their SOA infrastructure. The story begins with an urgent request from Operations for a fix to a problem that's keeping them from billing for new services. Assured that extensions can be made to the common data model (the SID) for the COTS billing system that's causing the headache, the problem is resolved in short order. The story then goes back a year to a time when the lack of a data interoperability layer made similar problems seem insurmountable and a tiger team is formed to put in place an OSS/BSS architecture that will change the way the CSP does business.

SOA - Wanted! Dead or Alive

Burton group's Anne Thomas wrote an obituary  for SOA, and she writes: "SOA met its demise on January 1, 2009, when it was wiped out by the catastrophic impact of the economic recession. SOA is survived by its offspring: mashups, BPM, SaaS, Cloud Computing, and all other architectural approaches that depend on “services”."

While this is dramatizing a bit, it is not entirely without merit. I have always wondered where exactly are the SOA use cases. In the real world (beyond the POCs, CTO proposals and pilots), time and time again I come across cases where someone in an organization bemoans that the promise of SOA was never delivered! A very rosy picture was shown during the POC and pilots, but post that, in the real world, the gains realized are nowhere close!

Yesterday, I was at a "SOA - past, present & future" panel discussion in Bangalore, along with SOA Matrix CEO and Head of the Java and middleware horizontal at Satyam. We spent most part of the hour talking about precisely this! Recognizing that mainstream acceptance is not as wide as expected, we were deliberating on the possible reasons. One consensus that emerged is that the complexity is not so much in the paradigms, technology nor the products. The biggest challenge is getting the required functionality at the right granularity, and this is more about the existing solutions in the enterprise than it is about SOA. Re-engineering the solution to "expose" the functionality with a coarse grain is never easy. Often the incumbent applications would be those with just a UI interface. In such applications, as we all know, the tier separations are not that rigidly maintained. A good amount of functionality spans the presentation and functional tiers. Trying to re-engineer this and create a single coarse grained function for the required service is not going to be easy at all! If this is the case for a single service, imagine the magnitude if the problem is one must get hundreds of key business functions "exposed" as services. (And without these hundreds, one cannot even begin to realize the dynamic enterprise promised by SOA, with easy to compose business processes, BAM on-demand, and more.)

Now, I can't imagine how any amount of technology can solve the above problem?

In this context it is interesting to see a pragmatic narrower view of the cloud as proposed by David Linthicum - SOA morphing into private clouds, or as a subset of the whole enterprise or the larger public cloud, as put forth by Mikael Ricknäs of IDG News service. Thankfully, ZapThink MD quantifies the reality realistically; as quoted in ITWeb: "People are asking the right questions about governance, loose coupling, best practices and business agility, one of the major benefits SOA confers," says Bloomberg. "And they're asking the right questions up-front, seeking to solve a business problem rather than adopting SOA for SOA's sake."

For more perspective on Anne's post, read my colleague Dan Foody's post, Goodbye SOA, we hardly knew you., or listen to his podcast, Cloud Computing - The New SOA?

A Transparent New Year

It's been a while since I've put a few thoughts in print on these pages on event processing, but that's not to say I've not been busy. We've wrapped up quite a successful year in the Apama division within Progress software. For this year, 2009 we've got quite a number of new features and capabilities in store for the Apama CEP platform. We will be extending the breadth and depth our event processing language (EPL) to meet the challenges we've seen in the market and a vision we have for the future. Of course you will see announcements in our standard marketing press releases but I (and others) will explore the technical merits of those new features in these blog pages in much more detail. Something we've not done that much of in the past. There are many historical reasons for that not really worth explaining. Suffice to say our intention is to be more transparent in the coming year.

To kick that off, it's worth starting a bit of a tutorial on the Apama EPL, a language we call MonitorScript.  I'll begin with the basics here and in subsequent blogs build upon these main concepts, providing insight into the power and flexibility of our EPL. And as we release new extensions and capabilities it will be easier to explain the benefits of those new features. So without further ado, here is the first installment.

First a few basic concepts...

• Apama MonitorScript is an imperative programming language with a handful of declarative statements. This is an important consideration and one we highlight as a distinction in our platform against competitive platforms that are based on a declarative programming model. The imperative model provides a more natural style of development similar to traditional languages like java and C++.
• The language is executed at runtime by our CEP engine called the Correlator.  

Second a few basic terms...

• A monitor defines the outer most block scope, similar to a class in java or C++. It is the basic building block of Apama applications. A typical application is made up of many monitors. As you might imagine monitors need to interact with one another, I'll explore that capability in a later blog.
• A event defines a well ordered structure of  data. The EPTS Glossary definition has a handful of great examples of Events
• A listener, defined by the on statement, declares or registers interest in an event or event pattern. In a large application it's the Correlator runtime engine that will typically process 1,000's or even 10,000's of registered event patterns. In our example below we just have one.
• An action defines a method or function similar to java or C++. 
• The onload() action is a reserved name (along with with a few others) that is analogous to main() in a java program. 

To put it all together... "A monitor typically listens for events and takes one or more actions when an event pattern is triggered."

The language sports a number of capabilities that will be familiar to anyone schooled in java, C++ or any traditional imperative programming language. I won't bore with all the nuances of data types and such obvious basics, those are well articulated in our product documentation and customer training courses. I will however, focus on the unique paradigm of event processing.

Apama MonitorScript, a basic monitor.

event StockTrade {   string symbol;   float price;   integer quantity; } monitor StockTradeWatch {   StockTrade Trade;     action onload {       on all StockTrade():Trade processTick;     }     action processTick {       log "StockTrade event received" +       " Symbol = " + Trade.symbol +       " Price = "  + Trade.price.toString() +       " Quantity = " + Trade.quantity.toString() at INFO;     } }

This monitor called StockTradeWatch defines an event of type StockTrade. Events can originate from many sources in the Apama platform, the most obvious would be an adapter connected to a source of streaming data (i.e. stock trades as example shows), but they can come from files, databases, other monitors, even monitors in other Correlators.

The onload action declares a listener for events of type StockTrade (i.e. on all StockTrade). When the monitor StockTradeWatch receives StockTrade events, the action processTick is invoked. As you can see in this example we simply log a message to indicate that it occurred. The obvious intent is that this monitor will receive a continuous stream of StockTrade events, each one will be processed in-turn by the processTick action.  One can be more selective in the event pattern with a listener, such as on all StockTrade(symbol="IBM"). I will explore the details of event patterns and complex listeners later on.

As I mentioned, I've started with a simple example that shows the basics of the Apama EPL, MonitorScript. It demonstrates the simplicity by which one can declare interest in a particular event pattern, receive matching events and act on them in your application (i.e. action).

 In subsequent installments I will demonstrate more complex features highlighting the power of the language.  That's all for now.

You can find the second installment here.
Regards,
Louie

Cloud Computing... It Depends on Who You Ask

What exactly is cloud computing? Lately, this is one of the most often repeated question in any discussion about the Cloud. And the answer depends on who you ask! (Personally, I think that if there are not at least three different popular definitions of a new technology, then the technology is not in its "hype" phase of the hype cycle :-) ).

Cloud is often confused to be just one of:

• Utility Computing - The on-demand resources with dynamic provision offered by virtualization platforms such as VMware, EC2 etc.
• Grid computing - Loosely coupled discreet computing nodes, that come together to form a larger computing-grid.
• SaaS - Just the notion of a software abstracted on the web is cloud to some.
• PaaS - To some, cloud is if one were to "build" the solution and then deploy it on a virtualized platform on the web.
• User centric cloud - RIA desktops that provide seamless and integrated access to information and services on the web, while still retaining a "sane" visual front to the user.
• Even Wikipedia defines it as: The majority of cloud computing infrastructure as of 2009 consists of reliable services delivered through data centers and built on servers with different levels of virtualizationtechnologies. The services are accessible anywhere in the world, with The Cloud appearing as a single point of access for all the computing needs of consumers.
• And, I am sure there are more!

Syscon attempted to reconcile the definitions by getting no less than twenty one (!!) industry experts to define cloud. Their definitions had wide ranging defining attributes: elastic, virtualized, services over the web, multi-tenanted, pay-as-you-use, massively scalable technology enabled services, SaaS is the consumer-face of cloud computing, distributed utility grid (wow!), on-demand resources with APIs, and more!

What I see missing though is the end enterprise view. It is this view that brings all of the above (and more) together. As I see it, this is "cloud computing". A fairly abstract IT enterprise landscape. Where the biz requirement is real—with its solution made available over the web, with simple web based user access, with guaranteed SLAs on service availability and performance, with seamless integration to other information sources/services over the landscape. The exact location, platform, configuration, size and such are all abstracted from the enterprise by the solution provider. Further, the solution provider also takes care of maintaining the application (enhancements/bug fixes), managing the production environment, and managing any backend integration requirements that the client-enterprise may have - offering a complete "managed service". The enterprise needs to focus just on the biz at hand, and in defining the IT solution that is needed. The solution-service providers takes care of the rest. All of them coming together to form the enterprise "cloud" - accessed over the web, managed over the web and probably even orchestrated over the web.

The enterprise Cloud will also evolve over time. When there is just one ISV making a solution available in the above managed model, that is SaaS. If the same ISV also "builds" it on the web before making it available on the web, then that is PaaS. If the ISV just uses storage on the web, then that is cloud storage. If the same solution is deployed within the enterprise network on a virtualized platform such as VMware, then that is virtualization. If the solution is distributed across discreet computing "nodes" on the web, then that will be the computing grids. Now fast-forward a few years... If the IT landscape in an enterprise architecture is a combination of all of the above types of solution environments, then there is an opportunity for a loose "biz grid" that can bring all of these together, which in many ways is already prevalent today - the SOA infrastructure! SOA is all about discreet services that are location agnostic, with standards based description and easy access. Today's SOA environments are very very vendor dependent (that is assuming the emphasis is on the reliability and performance). Soon we may see this a reality. A vendor neutral SOA environment that will bring together all the disparate IT solutions in the enterprise, both within and on the cloud. Likewise, Web 2.0 will define the user experience that brings all of the same solutions in a seamless integrated and very-functional UI.

Below is a visual of what I'm thinking... use Comments to let me know what you think.

You Can Look, But You Better Not Touch

Actually, touching would be great, but be warned... it costs extra! Did I ever tell you about that time in South Africa where I presented at a conference during one of their open sessions? Afterward, a guy actually came up and hugged me for my performance. True, I swear.

Come see yours truly at Sys-Con's Cloud Computing Expo in NYC March 31st and April 1st. I'll be talking about the impact of cloud computing on enterprise application architecture. I'm excited that they accepted my abstract, but now I need to flesh it out.

For a preview on my thoughts on the topic, check out the podcast I did recently about the impact of cloud computing on enterprise architecture.

Let me know what you think, but I'm on the fence about wearing shoes while I present.

I'm looking forward to seeing you there!

And the winner for the best CEP product goes to....

IDC, one of the "big three" software analyst firms has just published a report on CEP - Complex Event Processing Opportunity Analysis and Assessment of Key Products (more information about it is available here).

The report has a few major themes. Firstly, IDC look at the market for CEP applications. They state that trading applications in capital markets makes up the biggest single CEP application segment but go on to note that it represents only a minority of CEP applications. Other areas mentioned where CEP is in production are risk analysis, fraud identification and general transaction & business process monitoring in a range of different industries. Transportation and logistics is cited as one particular industry where CEP is gaining significant traction. IDC also mention use cases in a whole range of different verticals where deployments of CEP are live: healthcare, government, leisure, banking, media and manufacturing; this breadth of application areas is excellent validation that CEP is not a technology only useful within the capital markets trading arena.

In terms of market growth IDC believe CEP will continue to grow strongly, despite the general market downturn. In their view, one which I share, the increased focus on risk management and on running existing operations more effectively and efficiently will actually further stimulate the CEP market. IDC estimate that CEP is the biggest growing segment of, what in their categorisation is, the middleware market.

Thirdly, from a parochial Progress point of view, here's the best bit: as well as looking at the CEP market and where CEP is being used, IDC also did a vendor evaluation using various criteria covering correlation sophistication, integration with event sources, the presentation of results and control of event processing scenarios by business users etc. All the usual suspect vendors participated. I am very pleased to say that Apama came top which, of course, is very gratifying.

My final point is that what's most important here is not the views of any one analyst firm, but that the publishing of such a report validates that CEP is becoming more important to end-users. Analyst firms are driven by end-users and thus the fact that an increasing number of analyst firms are investing in conducting CEP research demonstrates that they believe there's an end-user market for it. And end-users are the most important people around.

Convergence of SaaS, Cloud and SOA - Use Case

Was at the HeadStart (innovation showcase) and Compute (ACM Bangalore chapter) co-event this weekend at Bangalore. Was chairing the panel discussion on "Delivering SaaS from the Clouds". The panel included the Founder/CEO of Computational research Labs (part of TATA goup), Head of Cisco/Webex in India (they have a large dev center and equally large India sales ops), Lead Architect at Honeywell, and the Head of the RIA tools development at Adobe. There were an interesting mix of perspectives ranging from the glue and access paradigms from the fringes of the cloud (as in the RIA & webex rich desktops), to hard core grid paradigms and application/service abstractions on the cloud (bright by CRL), to cloud backends (CRL and Webex) and the usage perspective brought on by Honeywell exploring cloud for many of its field initiatives. We also had great discussions ranging from the market opportunities that each one sees to their take on the solution architectures and to the cloud trends that we can expect in 2009.

In particular, I found two aspects very interesting:

1. Build-on-desktop or Build-on-cloud? Unlike the more popular expectations of using the cloud to "host" applications that are built on desktops and 'uploaded' to the clouds to be accessed from browsers, there are also models (in industrial automation) where the application is built on the cloud and 'downloaded' to the devices. Essentially, they are engineered on cloud and downloaded to the devices in the enterprise - examples include energy audits and automation optimizations.
2. The serious production use cases are still elusive. Most people dabbling in cloud are still experimenting and playing around with it. Serious use models are yet to evolve. I believe that this is probably a manifestation of the hype curve. Hopefully the dust will settle down and serious usage emerges in 2009.

Talking about use cases, I came across another interesting use case in the same week. This is in the Governance risk and Compliance (GRC) space. Here the value proposition seen from the cloud is unique—beyond just SaaS. In terms of the agility and the arms-length distance:

The GRC space is extremely regulation driven. And is bound to change often. Today the model for implementing GRC is to "build it in" the existing operational solutions, which we all know is cumbersome and extremely difficult to manage changes. Given this, the proposition from this  company was unique. This company that specializes in governance and consults in best practices, has evolved a "canned" GRC model, and is offering the same as SaaS on the cloud. The compelling case to the clients is that this implicitly allows an arms-length distance between operational systems, easy verifiability of the GRC rules in effect, and is also able to easily modify the GRC rules as they evolve and change. The nature of this scenario is implicitly that of convergence. The value prop is SaaS - the runtime is the cloud and the solution involves extensive access to data and services in the enterprise which is best done using an SOA infrastructure.

Come to think of it, cloud in the enterprise will be deeply entwined with SOA. We are not talking about simple utilities like rate calculators or converters. If serious enterprise solutions are made available on the cloud by 3rd parties, these will always involve access to other enterprise information and services running either in the Intranet or possibly elsewhere in the web-cloud as well. So far, any SaaS solution provides for custom APIs to enable this integration. As the more generic cloud platform evolves, it becomes a question of time and the standards and generic approaches to integrate the enterprise into the cloud will emerge.

This will be yet another space to watch in 2009, even as we already are tracking aspects such as cloud performance, and the cloud monitoring and management (an extension of the SOA Management problem that Progress Software already solves very well with Actional).

Rumors of My Death Have Been Greatly Exaggerated

There was a time when I was diving intensively and I'd call my boss at the end of vacation to let him know I was still alive and would be back in the office as planned. I even left my passwords and his information with my brother so my bro could send him my laptop if a fatal event occurred. Have you ever written a "here's what to do if I die" letter to anyone before? I highly recommend it. It's good for the soul.

But, I digress... We're not here to talk about my death but the death of service-oriented architecture (SOA). It's all the rage since Anne's post the other day.

A good summary of the chatter it's stirred up can be found on Twitter, so I won't head there. But, I will point out that the second comment on Anne's post was mine - and I think my one line summarizes most people's feelings - whether they believe SOA is dead or not:

"SOA must die, so we can move on with, well... SOA"

From our perspective here at Actional, the death of SOA from a marketing perspective (but not from a technical one!) has been going on for some time. So, I wanted to give some insight (I'm usually not punny but couldn't resist - go ahead, click on the "insight link" to see what I mean) into our messaging and positioning of the Actional platform in advance of some releases you'll see later this month.

Clearly, we're defocusing our SOA message. There's been a bit of disillusionment with SOA, and associating with it doesn't seem to be in vogue. Now, we're not the only ones to do this. One of our competitors has decided there are many "hot topics" to grab onto and has a flash animation scrolling through them all to make sure people realize their solution is as good for cloud, SaaS, Web 2.0, and composite apps, as it is for SOA.

Our perspective is different. From our perspective, a SOA message is limiting. Fundamentally, we're about managing/controlling services, and anything built upon services, regardless of whether a SOA architecture is used or not. We (and I) believe that by messaging around SOA, organizations that are disillusioned will "throw the baby out with the bathwater" when, in fact, huge ROI* can be achieved with Actional.

The Buyers' Guide I wrote recently mentions SOA only once, and talks more about composite applications and business processes (both based upon services) and the benefits of a performant solution like Actional providing visibility and control. (Email me directly if you want a copy without registering.)

For the record, I believe fully in SOA, SOA principles and the use of SOA to drive business and IT alignment. Done properly, SOA infrastructure adds agility, accountability, and makes IT's contribution more visible to executives from a business perspective. I'll be very happy when the SOA paparazzi move onto a younger celebrity and let us get on with our SOA lives.

* Later this quarter - frankly, I'm just not sure of the date - we will be releasing some independent third party ROI analysis and case studies of some of our production customers. The idea being that these customers have adopted Actional for some time. What did they achieve? What was their ROI? What is their future expected ROI? And, how can you use this information (discounted for risk) to justify a decision on Actional? Stay tuned.

The problem with good governance

I saw an interesting story in the news two days ago.  It's about a company that has received multiple "good governance" awards.  Well, it turns out that their "good governance" has led to the chairman having to admit that their finances were falsified.  Know who I'm talking about?  Satyam.  Yes, that good governance award winner - the one that turned out to have over $1B of "falsified" cash on their balance sheet.

Sure, this is corporate governance, not IT governance - but it raises an interesting question for IT and SOA governance: How do you know your governance is not being bypassed? Because this is exactly what happened in the Satyam case.

The thing that likely got Satyam into the problem was likely one key thing: Their governance checks were manual, not automated.  This meant that they could be easily bypassed or avoided.  They were people processes, not automated processes.

SOA What? The moral of the story is that governance that's not automated (with the checks in the right, unavoidable, places) is governance that doesn't work.

The Semantic Dialogues Have Begun!

Are you thinking about OSS/BSS Integration?

The Semantic Dialogues tells the story of how National Networks, a fictional telecommunications service provider, pursued and achieved data interoperability within their SOA infrastructure. When the VP of Systems Architecture, Eric Golden, welcomed Mick Dundon, his new director of Data Architecture, he didn't pull any punches. "The top priority for your new team is solving our data problem. National Networks is under pressure from competition and regulators and needs to become more efficient. We can't afford another round of failures in our attempt to control OSS/BSS sprawl." And with those instructions ringing in his ears, Mick built the team that could make it happen.

How does this dialogue evolve? Who's involved and what decisions are being made? Are they the right ones? Does the story sound familiar? Read the prologue and first chapter of The Semantic Dialogues today. Then revisit every two weeks when we post the next chapter. Follow The Semantic Dialogues... they have begun.

Goodbye SOA, we hardly knew you.

Anne Thomas Manes of Burton Group posted how SOA is Dead; Long Live Services.  This one is too juicy not to comment on!

Firstly, I totally agree with Anne.  I think "SOA" is definitely dead in 2009.  And I also agree that the need for services will grow.  But, we still wrap too many things into the same terms.

At a simple level, I'll break it into four points, I'd like to talk about where I think things are going:

Service-oriented applications.  BPM, cloud computing, SaaS, PaaS, composites, mashups. All of these are doing well and will continue to do well even in the 2009 economy.  Why?  1) They can be leveraged as needed; 2) They can be done fast; and 3) Using them typically provides measurable benefit from day one (if it doesn't, then you should do something else shouldn't you!).  It's precisely the laser focus solution aspect of these that is their greatest strength.  Get in and get out while solving a problem that someone really cares about.

Service-oriented architecture (SOA).  This is where you look from a higher level above all your service-oriented applications to create a cohesive architecture that all of these follow.  Dead in 2009.  Why?  Because approaching this from an architectural perspective will always fail.  Is it a noble goal?  Sure.  But without other changes in your organization (see below), it just can't be successful.  Said another way, before you can start on a successful SOA infrastructure initiative you must finish the items below.

Service Delivery.  By service I don't mean what you're probably thinking (a thing you create).  I mean service in the most pure sense: as in "what value do you deliver to others" (with the follow up of "how do I continue to improve the value I deliver to others").  This requires a transformation for most IT organizations... and most haven't even started yet. To be fair, there are organizations that have figured this out (take S&P for example: their whole business is service delivery, so of course they understand it) - but the vast majority don't fit into this category - and if you do not pass go, you can't collect your $200: So this is the next step for most IT organizations that want to transform.  Think: "How do I create my first true service?" Here's a hint: you can't do it by creating a project, task force or cross-functional team (more on this in a subsequent post since this is a richer topic than I can cover here).

Service-oriented IT.  Once you've mastered service delivery on a small scale, then you can tackle it on a larger scale: turning IT from being project and application oriented to being service oriented.  Warning: this is a major transformation that can't be done without the CIO leading the charge.  Sorry, but no enterprise architect can make this happen.  And it can't happen until you've figured out service delivery.  This is not a 2009 project for most organizations.  It is a 2010+ transformation.

Most organizations that went down the SOA route tried to do SOA without first figuring out what it means to deliver a service, and without realizing they needed to get this right (and begin the service-oriented IT transformation) before they could never be successful at service-oriented architecture.

Service-oriented architecture is a technical and architectural transformation.  But it can't exist without first focusing on the organizational and cultural transformation of service delivery and service-oriented IT.

This is why SOA is dead in 2009 and why services will live on. But who knows... maybe once an organization figures out service-oriented IT, SOA will be resurrected - but don't wait up.

Do we really need this complexity?

My first post on the Progress SOA Blog! This gives me jitters... All the writing till now was somehow different. There was a carefree attitude to that writing. When I wrote my book (nearly 40% of the book on SOA approach to Integration), or when I write my personal blog posts, I never had to worry about the impact. Now as informal as this may be, at the end of the day it does have a non-personal tone. Even so, I am excited about this. :-)

My views essentially are reflections on the software services and SI ecosystem and what/how they use technologies in the integration solutions space. Lately, I have been also quite caught by the SaaS/PaaS promise in the SOA context - even if I am quite skeptical about the hyped up value.

Recently, I was speaking on this topic at the IndicThreads conference on Java at Pune. I included a slide on the complexity of SOA. While in my flow I was using it to talk about some of the SOA enablers in our SOA portfolio in simplifying this complexity, the presentation and some following questions triggered a thought... How much of this does the industry actually need? Aren’t these more of a fringe scenario that needs all of this? Isn’t the REST vs. SOAP argument a good case in point? 80+% of the world needs only basic remote service mechanism. While REST more than suffices here, this basic need from the world is but just a small part of SOAP! So, who needs the rest?

The simplicity of REST is stark - at least when compared to the extremely unwieldy SOAP and its numerous accompanying specifications. And it is further compounded by all the WS* standards. Making things worse are frameworks for frameworks such as WS-Policy. WS-policy is a framework that layers on SOAP. And there are other standards that use WS-Policy as the basis for definition. In effect, we have XML over HTTP, on which is defined the SOAP standard, on SOA is the WS-Policy. And on WS-Policy are standards such as WS-S & WS-T. Phew!

Now compare this with REST - that actually has no other specification other than HTTP itself! Granted, this seriously limits what we can do with REST. But lets accept it - the world needs mostly ONLY the likes of just REST and only a very small fraction needs the multiple layers of specifications such as SOAP! Even in the SOAP land this is very well captured in how AXIS evolved - trying to keep pace with the rather haphazard offshoots of SOAP. And now there is CXF whose basic premise is to simplify SOAP and Web Services. From the word go, it has embraced the simplicity and is also trying to be more about services and less about SOAP. In the spirit of the latter, CXF also supports REST right off the bat.

I see 2009 as year where this simplicity is bound to get consolidated. The more simpler and easier the SOA framework becomes, the more it will get accepted. FUSE open sources SOA framework will be one very serious contender in this space. Functionally as complete as any other SOA solution, yet lean and mean with a good set of tools. Watch out for more in the months to come.

The Rise of Events and Eventing

One of the key things I saw in 2008 was the rise of events and eventing through Event Driven Architecture (EDA).  While "events" have always existed, they have primarily been used in specific markets such as algorithmic trading where our Apama product is commonly used.  The shift in 2008, and moving forward, was that events became more mainstream and that meant that the business and their technical developers were looking to see how they could harness events.

In contrast to traditional SOA infrastructure that use service orchestration such as BPEL to control all aspects of the flow, event based architectures can be used to implicitly enable process to occur.  An event typically happens on a state change or document passing between organizational boundaries or similar.  In order for the event to be useful, it needs to be published to everyone who could possibly be interested and as such, an event consumer can correlate multiple event instances to define new logic and/or to monitor the business event in action.  The key benefit for the enterprise is that the events and their correlation do not have to occur at the initial development time, they need to be captured in a single flow. Handling business events in this way allows a much more organic growth to the logic to be implemented.

Now, any changes in practice will always create new challenges you will need to overcome, not least of which will be the ability for people to assume that business will implicitly happen when there is no end-to-end picture, such as what people expect from BPEL or UML. With guidance, we can overcome these issues because typically the number of processes that are discretely modeled within an organization are few.  One of the key things that has to happen is the idea of event modeling. While there are elements of this in UML through state change diagrams, other tools may have to be used.  Another thing to consider here is that the dynamic aspects of the business will need to be captured and displayed in real-time to allow business decisions to occur when an event really occurs rather than hours or days later. 

In these challenging and opportunistic times that currently exist, I see the arrival of event based architecture as arriving in the mainstream at a very convenient time.  Businesses are restructuring through mergers and acquisitions, so IT professionals need to be able to satisfy this change, plus forces in our market are causing new approaches to be taken, including the idea of clouds that become more than mere hosting platforms.

SOA What? I too agree with Dan Foody that the conventional SOA is not dead by any means. It will actually continue to grow and serve more needs. However, I believe you will see that EDAs will rapidly augment existing projects to initially provide real time visibility and then gradually it will help lift some of the burden on the business as your EDAs evolve and grow.  When that happens, you will start to see more business users being able to implement business logic in real-time rather than having to go back to IT.