Integrated Infrastructure Blog

Business Transaction Management with SAP

If you follow Actional, you may have seen that we announced Actional 8.1 today.  The most interesting part of the release is our new support for SAP.  Yes, we already supported SAP NetWeaver (like most of the other people in our space) so unless you're an SAP aficionado, you probably won't recognize the importance of natively supporting SAP ABAP - which is what we've announced.

SAP supports two main application server environments: one based on Java and one based on ABAP, SAP's own programming language (you might also hear the term "basis" which is another name for the ABAP application server stack).  OK, with me so far?  While SAP support both, almost all of the SAP packaged applications are written using the ABAP stack - Java is primarily used for infrastructure services (things like their portal).

With this new version, we've added the ability to trace business transactions into and through ABAP - so we can detect problems even within the ABAP portion of a business transaction.  This is critical for many SAP customers because - without the ability to do this - SAP packaged application logic is seen as a black box silo.  And, with 100's of millions of lines of code in the SAP packaged applications, that a pretty big area to have a blind spot.

While there are a lot of business transaction management vendors out there that support SAP, they are usually referring to the Java side of SAP. As a result, once a transaction hits ABAP (and almost all of them do) it enters the black box and can't be seen again until it leaves ABAP.

As you might have guessed, we're really excited to extend Actional's patented transaction tracing to SAP's core platform.  If you're an SAP user, and need to ensure the success of business transactions that span SAP and other applications, platforms, and middleware then hopefully you'll get a chance to see whether this unique Actional capability can help you.

Telecom Standards and SOA - Spreading further

The telecommunications industry has been one of the early adopters of SOA infrastructure - given their much higher level of need for integration in general. The complexities are unique with the vibrant M&A activity and a very large number of seriously inter-dependent solutions, aggravated by the hardware solutions that form the solution space. Over the years and as SOA models have evolved, they are now faced with multiple layers of integration, including:

• Services: The primary integration backbone for business functions across the enterprise solution/apps space. (Progress Sonic ESB fits in here very well.)
• Data Integration: By itself for MDM and such, and also for services integration when sending data as inputs for or as returned data from services.  (Progress DataXtend SI is a unique, and probably the only, product that helps here.)
• Management/Governance: Extends problems such as Business Transaction Assurance. (Progress Actional addresses these challenges.)
• Events: Complex BTA and BAM analytics. (Using business events intercepted by Actional, Prorgess Apama can perform more complex correlation and analysis.)

Consistent with the above, Telecom Management Forum (or TM Forum) has defined a plethora of standards that enable telecom Software ISVs, Telecom operators, and SIs to simplify the integration solution space. The Global SIs based out of Asia and the Telecom operators in this region are seeing a rapid uptake of these standards (specifically the SID model).

Reflecting this, NASSCOM (the Indian National software industry association) is organizing a half-day NASSCOM Symposium that will discuss and deliberate on the three facets of emerging telecom industry trends - Standards, Key Business Problems (Transformation and Migration) and case studies. Progress Software (India) is helping organize this event and Michael Aubin, VP focused on Telecom at Progress Software, is delivering a key session on the telecom standards and trends. For more details, visit www.nasscom.in.

Buysides Finally Getting Some OTC Derivatives Love!

The ISITC OTC Derivatives Working Group, of which I am a member, recently announced its Market Practice for Contract Notification. I've been involved with OTC Derivatives trading operations and technology for almost 20 years now, the last 10 primarily on the buyside.  When I first joined an FpML working group, allocations weren't even part of the spec, and we are only now beginning to see vendors who provide buyside OTC Derivative trading solutions.

The industry itself has lagged in recognizing the buyside's role in the OTC Derivatives market.  The first movement around helping the buyside deal with the complexity of these instruments started about 5 years ago with the Swaps Best Practices Initiative jointly sponsored by ISITC and the Asset Managers Forum within the old Bond Market Association.  The BMA is now part of SIFMA and the AMF's Swaps initiative evolved into the Derivatives Operations Committee and the formation of the ISITC OTC Derivatives Working Group.

While custodians and other service providers including DTCC, Markit and SWIFT have stepped in to assist their buyside customers to process OTC Derivative trades, there are few off-the-shelf solutions. The majority of sales of Progress’s OTC Derivatives solutions for FpML, DTCC Deriv/SERV, Markit Wire and SWIFTNet FpML still go to buysides and custodians who are adding support for buyside functionality to systems designed for sellside. I am happy to report that Progress now has software vendors as customers looking to address this market. I’m optimistic as buysides spend more to meet new regulations we’ll see more complete vendor solutions.

Putting SID to work

On July 21st at 10am (ET) John Wilmes, our Chief Technical Architect for the Communications sector, will be speaking along with John Reilly, the TM Forum’s Senior Program Manager for the Information Framework, on the Information Framework (SID).

You can register here

Everytime I attend one of the webinars that John Wilmes does with the Forum, I’m always impressed by the TM Forum’s completeness of thought. Unlike a lot of other industry and architectural frameworks out there – the Forum didn’t focus on just part of the puzzle.  What was understood, and what the Solution Frameworks (NGOSS) highlight, is that business transformation requires addressing elements that live both within and between business process, information models, logical applications and technical architecture. Part of the joy of hearing John talk about the SID is to hear again how the data, process and applications all interrelate to each other.

For those of you who might be new to the TM Forum – or even those of you who are looking for a solid conceptual framework for thinking about transformation problems - in addition to this webinar there have also been several other webinars from the Forum that John has done over the years, including:

• What is NGOSS and What Can it Do for You: If you’re looking for a great introduction to the Solution Frameworks
• NGOSS Contracts: The Key to Improved Interoperability:How Business Services (NGOSS Contracts) can improve interoperability across multiple business silos / stovepipes
• SOA and NGOSS Contracts: A Practical Approach to Integration: How SOA and the frameworks fit together

For an alternate, non-telco example of a similarly complete framework that spans process, data and applications—there was also the recent webinar by Boris Bulanov and Frank Neugabauer about the Insurance industry’s approach to transformation the ACORD Framework. Many of the concepts are similar and it's another proof point that one must consider the whole problem and all its requirements when initiating an integration or transformation project.

REST maybe part of the answer

In the last few weeks/months I have got more immersed in the world of REST and what it means to build and have a REST application.  While at first glance it may seem that REST is a lighter weight version of a web-service implementation, it is by no means the complete answer and I do not believe REST is simply a replacement of Web Services because of how they operate is different.

When developing Object applications, in whatever language, you hit a reality that if your application is successful, then it will grow beyond the confines of the machine/JMV that it runs on, and as such you have to refer to objects that are beyond a single instance of the application's memory space.  This can be done using a number of different technologies such as object databases that can map the same object into different application instances, but the locks are maintained by the central server so consistency of the objects are maintained, even if the object is distributed in 100s of applications at a time.

Now technologies such as IIOP (Corba), RMI, DCOM and Web Services have been used to access objects and invoke operations on the instances to retrieve data.  This essentially solves the long-wire problem of accessing an object and then returning an instance of the object, the issue here being is that typically a representation of the object that you want is returned, i.e. using Java serialization or XML and the reference to the actual instance is lost, unless you have a known object identity that you are working with.

REST adds the concept of the pointer to the SOA infrastructure in a way that any business object can be referenced by a URL and thus referenced anywhere within the internet/cloud or a single program space. This is something that is not readily available in Web Services and now makes any RESTful resource available to anyone (given the correct permissions).  Now this sparks a whole debate on to how to identity a particular object and to detect duplicates etc, this is beyond the scope of this blog but will have the same techniques that are used in the object-oriented world, but now working at the cloud scale.

So now that REST gives us some new capabilities, how does this change the current implementations that you are working on?  Interestingly, I think many of the challenges existing in the Web Services world still need to be addressed in the REST world, I just believe that REST is probably better able to address some of the immediate ones and in fact just like every Progress customer has another application to connect to Web Services will always be required.

The other cool thing about REST is that I can typically use the services directly from an HTML page on a browser, something that was impossible for Web Services because HTML and WSDL are not compatible.  However, there is still the need to mediate REST in the same way that applied to Web-Services, for instance to be able to turn a switch to expose the same business entity as REST and Web-Service and in fact any possible protocol.  Allowing external access to your business objects will require caching and a level of indirection to provide for maintenance, performance, etc., and the virtualization of the underlying technology to ensure that an object is not a manifestation of the technology that stores it, i.e. database tables and rows are not part of the object serialization.  Security, Quality of Service (QoS) and Quality of Protection (QoP) are observed along with SLA commitments and the ability to monetize the object and charge accordingly.  This last point maybe very interesting as it would allow organizations to potentially charge for access to their data for different reasons and allow IT to start to become a profit-center rather than just a cost-center - something that I think people will look at as they invest in the cloud.

Now I think that the promise of REST is very interesting, in more ways than Web Services was, but whether the ideal is purely based around REST or a hybrid of many technologies is going to depend on the goal of what is to be achieved. In any event, it is very exciting.

The ACORD Information Model

Congratulations are in order for our hard working data services and engineering teams. Progress was recently awarded two ACORD Accomplishment Awards for our work with the insurance standards organization, ACORD.

We won an Innovative Implementation Award for the Progress DataXtend Browsers for ACORD, in P&C and Life flavors. These browsers will enable ACORD members to more easily use and understand the existing ACORD standards for P&C and Life.   We also won an Early Adopter Award for our work supporting the unambiguous mappings between the existing ACORD XML standards and the new ACORD Information Model. 

If you’d like a seven minute summary, you should watch our interview with Frank Neugebauer, the Assistant VP of Technology at ACORD. John Petrie, Bill Gino and Boris Bulanov discuss with Frank our work with the Browsers and the Information Model. While the interview is a great summary, I'm sure that many of you would like to learn more about the new ACORD Framework, its core components and the business value it offers. 

At 3:00pm ET on June 25th Frank Neugebauer of ACORD and Boris Bulanov gave a webinar on ACORD Information model. An archive of the event will be posted here.  It’s a great opportunity to join your colleagues to learn more about the ACORD Framework and how the ACORD Information Model provides the foundation for information architecture in insurance.

The Enterprise Architecture Dialogues - Chapter 3 of The Semantic Dialogues

Chapter 3 of The Semantic Dialogues, The Enterprise Architecture Dialogues, is available now. Here's an excerpt: "Sanjay believes that enterprise architecture is the solution, and he takes a hard line on forcing it into the enterprise. In his view, all they need to do is complete the architecture with a data strategy. But Jerry convinces him to see things from the business users' perspective, and proposes some radical changes to the way they use applications. Mick sees an opportunity to make the business happy and fix IT's reputation. Are they all drinking Kool-aid?"

Click here to read Chapter 3, The Enterprise Architecture Dialogues, today. And if you’re enjoying The Semantic Dialogues, be sure to share it with a colleague.

The Semantic Dialogues tells the story of how National Networks, a fictional telecommunications service provider, pursued and achieved data interoperability within their SOA infrastructure. The story begins with an urgent request from Operations for a fix to a problem that's keeping them from billing for new services. Assured that extensions can be made to the common data model (the SID) for the COTS billing system that's causing the headache, the problem is resolved in short order. The story then goes back a year to a time when the lack of a data interoperability layer made similar problems seem insurmountable and a tiger team is formed to put in place an OSS/BSS architecture that will change the way the CSP does business.

Do we really need this complexity?

My first post on the Progress SOA Blog! This gives me jitters... All the writing till now was somehow different. There was a carefree attitude to that writing. When I wrote my book (nearly 40% of the book on SOA approach to Integration), or when I write my personal blog posts, I never had to worry about the impact. Now as informal as this may be, at the end of the day it does have a non-personal tone. Even so, I am excited about this. :-)

My views essentially are reflections on the software services and SI ecosystem and what/how they use technologies in the integration solutions space. Lately, I have been also quite caught by the SaaS/PaaS promise in the SOA context - even if I am quite skeptical about the hyped up value.

Recently, I was speaking on this topic at the IndicThreads conference on Java at Pune. I included a slide on the complexity of SOA. While in my flow I was using it to talk about some of the SOA enablers in our SOA portfolio in simplifying this complexity, the presentation and some following questions triggered a thought... How much of this does the industry actually need? Aren’t these more of a fringe scenario that needs all of this? Isn’t the REST vs. SOAP argument a good case in point? 80+% of the world needs only basic remote service mechanism. While REST more than suffices here, this basic need from the world is but just a small part of SOAP! So, who needs the rest?

The simplicity of REST is stark - at least when compared to the extremely unwieldy SOAP and its numerous accompanying specifications. And it is further compounded by all the WS* standards. Making things worse are frameworks for frameworks such as WS-Policy. WS-policy is a framework that layers on SOAP. And there are other standards that use WS-Policy as the basis for definition. In effect, we have XML over HTTP, on which is defined the SOAP standard, on SOA is the WS-Policy. And on WS-Policy are standards such as WS-S & WS-T. Phew!

Now compare this with REST - that actually has no other specification other than HTTP itself! Granted, this seriously limits what we can do with REST. But lets accept it - the world needs mostly ONLY the likes of just REST and only a very small fraction needs the multiple layers of specifications such as SOAP! Even in the SOAP land this is very well captured in how AXIS evolved - trying to keep pace with the rather haphazard offshoots of SOAP. And now there is CXF whose basic premise is to simplify SOAP and Web Services. From the word go, it has embraced the simplicity and is also trying to be more about services and less about SOAP. In the spirit of the latter, CXF also supports REST right off the bat.

I see 2009 as year where this simplicity is bound to get consolidated. The more simpler and easier the SOA framework becomes, the more it will get accepted. FUSE open sources SOA framework will be one very serious contender in this space. Functionally as complete as any other SOA solution, yet lean and mean with a good set of tools. Watch out for more in the months to come.

One data model, many approaches to integration

The use of a common data model in SOA has been recommended in definitive posts - from the viewpoints of integration and of governance.  The common model architecture facilitates integration by providing a common ground for data transformation, and supports governance by bridging the business and implementation views of enterprise information.  When a common model is properly used, these principles generally apply regardless of the runtime environment.  But a common model supports several quite different styles of integration.  We could look at those integration styles as a kind of spectrum, with extremes at each end and a blend in between.

One end of the integration style spectrum is represented by what I call the common interface style.  OSS/J and MTOSI are examples of this style in the communications sector, and there are comparable examples in other sectors.  The common interface style promotes a universal, standard interface between applications.  It requires that all participating applications use the standard interface, reducing some common integration problems and costs.  The level of adoption of standardized interfaces varies among domains and among application vendors.  A common model can be used to implement adapters over non-compliant applications in order to conform to the standard interface.  Although the common model and its mappings may resemble a hub and spoke model at design time, the runtime view of this style typically requires that the adapters be highly distributed.

In contrast to the common interface style, the other end of the spectrum is what I call the data integration layer style.  Several OSS transformation projects are examples of this style.  It starts with each application as-is, and adapts their existing interfaces to a set of "canonical" interfaces based on a common data model that supports mapping between all participating applications.  It also adapts canonical interfaces back into application-specific ones.  This allows process management infrastructure to intercept and redirect canonical requests, providing access to business process that would otherwise have been hidden in (and locked into) the "wiring" between applications.  For this style, in contrast to the hub and spoke design view, the runtime view depends largely on the location of the mapping services, but it usually inserts process management between requesters and responders.

Other integration projects fall somewhere between the ends of the integration style spectrum, with a mix of adapters, canonical interfaces, and direct translation between applications.  It will be interesting to see how these and other styles evolve, as SOA best practices and SOA-supporting technology improve.

NGOSS Contracts Gain Momentum

One of the most enjoyable aspects of my job at Progress is my daily contact with standards organizations like the TM Forum and OASIS. I work with many other industry-supported contributors to develop and extend the frameworks and guidelines that address the challenges they face in application development and systems integration.

Within the last month the TM Forum has issued for member evaluation new versions of three NGOSS frameworks: the Business Process Framework (eTOM) 7.5, the Applications Map (TAM) 3.0, and the Information Framework (SID) 8.0. These releases continue the evolution of the frameworks in both breadth and depth, and also support the upcoming release of the new NGOSS Contracts guidebook.

NGOSS Contracts extend the reach of the NGOSS frameworks into the system and implementation views. These contracts provide standards for application interaction that complement the existing SOA view of services and operations by defining the bigger picture within which those services run, such as conditions and constraints, message exchange patterns, and service level agreements. NGOSS Contracts also represent a meeting place between top-down service design based on the interplay of process, data and application models, and bottom-up service design based on real-world experience with successful service function and granularity.

Having seen firsthand both the promises and disappointments of client-server, CORBA and now SOA, I believe that contracts like those promoted by the TM Forum will make a critical difference in the long-term success of SOA infrastructure compared to earlier architectures, because contracts can cleanly separate the process implementation from the data model. This opens up the possibility of using a data integration layer, along with a robust business process management layer, to achieve real separation of business process from application interface details. The process layer can then select from alternative applications at a fine-grained level without the need for application-specific process changes, allowing an enterprise to reduce or eliminate expensive customization and to encourage competition between application vendors. Successful large-scale OSS transformation projects have demonstrated the technical and economic validity of this approach.

The importance of the data layer is also emphasized in a new white paper by Dave Hollander, a co-inventor of XML and a veteran of changing data standards. Dave examines the disparity between SOA's success in large-scale service interconnection, and its failure to address operational differences between applications, resulting in disappointment and failed projects. He concludes that the common model architecture is the key to achieving that semantic integration. Dave's paper, Common Models in SOA: Tackling the Data Integration Problem, will be available soon and I'll send a link to it when it's finalized.

A Note On Policy Portability

Today, I write like my colleagues. No humor. No funny title. No family stories. Just boring technology evangelizing.

I apologize.

But, I have something to say, and it can't wait until I think of a funny, but related personal story.

People often ask about policy portability. In particular, something along the lines of:

"Is there a way to create an XML representation of the policies configured/applied to services so that we can store them in the registry and share them with other policy enforcement points?"

Policy would only be truly portable if there were a standard that defined how policy should (could) be written. It would have to define a lot of things that we take for granted, like how to define semantics of policy constructs (defining something as simple as response time can vary greatly from vendor-to-vendor today). That in itself would be difficult, but perhaps not impossible.

The larger challenge becomes the "value added features" that vendors enable in their products that differentiate them from one another. From Progress® Actional's perspective, these value added features include Service Stabilizers, Dimensions, and Message Fields - all aspects of our product that add to the flexibility, scalability, and overall power of the platform. These are powerful features that provide detailed insight into service usage and control in a shared environment.

Realize, that this might sound like I'm trying to defend my product's value-add, but in fact, this is one of the real differentiations - how powerful can a governance solution write policy? How flexible can the policy be? How can it be applied creatively, to make sure everything is managed, even services you don't know exist? (Yeah-  you got that. We can write policy, and have it be inherited by services we don't even know exist. And, the policy automatically kicks in, when appropriate, without having to restart services, app server, or management platform).

I'm not sure I think these things could never be standardized. But, perhaps it might happen. And, we'd be all for it for those of us who work on and with Actional products. But in an emerging market, in many ways, the market is still trying to figure out what works and what doesn't. That's why many of the WS-* standards efforts (including WSDM, which I think in some ways might have indirectly led to policy standardization) have faltered. I think the standards got a bit ahead of the market and actual implementations.

Back to my point. It's very useful to visualize, in a way that scales to the enterprise, performance by customer, business process, or whatever arbitrary business characteristic a business manager wants. It's powerful to see this information without any a priori knowledge of the set of information to view things by (e.g. you don't need to pre-configure the list of customers, we can discover them live, and as the list changes, dynamically adapt and instantly start collecting new customer stats). None of that would be possible if policy were delivered as the lowest common denominator across vendors. Delivered as a standard, we'd all have to agree on how to define these things first... and in my opinion, that's too much to ask of a standards organization at this point in the maturity of SOA Governance.

SOA What? From another point of view, it seems to be a growing need. That is, for a policy written in one place to be stored centrally and then pushed to various (policy) enforcement points.

Of course, that's expressed as a feature, not as a benefit. The benefit is to be able to govern policy better, and have consistent policy deployment across the SOA.

One way to solve this problem is to only deploy solutions that embed Actional agents! Then, you can use our policies everywhere. Policies can be attached to the WSDL and stored in registries using WS-Policy standard (we do this today). Before you discount this as a "sale pitch," take a look at the breadth of platforms supported.

Actional Agents are available for a broad range of platforms, including:

1. BEA
2. Microsoft
3. IBM
4. Oracle
5. Sonic ESB & SonicMQ (Progress Software)
6. Tomcat
7. IBM DataPower
8. TIBCO BusinessWorks
9. JBoss
10. Lombardi Teamworks (see my recent webinar on this topic)
11. Cisco ACE XML Gateway (formerly known as Reactivity)

And, there are platforms that we support that have not yet been announced (and probably some I've overlooked) so ask your account manager for the complete list.

Why have these partners chosen to embed the Actional technology? Simple. We perform and scale orders of magnitude beyond any other solution out there. And, the Stabilizers, Message Fields, and Dimensions mentioned above add an abstraction layer that makes cost of ownership really low through a really flexible policy implementation.

In other words, our Agents can be embedded in places where performance and scalability are critical. Places where other technologies simply couldn't live because of their footprint or intrusiveness. Imagine how the very powerful XML accelerators that embed our technology would behave if each time an XML payload needed to be processed, the management solution had to send the message back to the central management server?

But, I digress into hitting my competitors for poor enterprise scalability. We've just been removing so much of their stuff lately...

The downside of putting Actional everywhere would be the perception of our policies being proprietary and being locked into Actional. Well, I tell you what... any solution you choose, that'll be the case. The advantage to Actional is that if you choose one platform, say BEA, and all of a sudden, they get acquired by another vendor, say Oracle, you can easily migrate to the new platform without having to redo your policies. You can easily plug-replace, or amend, you infrastructure with an instrumented platform, and not have to re-design your policy infrastructure. At that point, you need to decide three things to choose the right policy infrastructure (Actional!):

1. Does the solutions scale to my needs?
2. Is the policy authoring power sufficiently robust?
3. How many partners consistently support the policy infrastructure?

In fact, the argument against us being proprietary is to look at the alternatives (considering that any solution today is really proprietary)... one alternative is to have weak policy authoring (through a lowest common denominator policy language) or, point-to-point vendor integration to work around the lack of standards. Point-to-point integration has the weakness that usually each integration is done differently, and not all features are available everywhere.

I think a single vendor solution that doesn't compromise policy flexibility or power, or solution scalability or performance, while enabling a broad selection of policy enforcement points (and for that matter, policy repositories via WS-Policy) and technologies/protocols, is a desirable, and in fact, preferred alternative.

If necessity is the mother of invention, standards are...

...her bastard child.

I was once relaxing with some friends and was asked, "what's the one thing you do better than anyone else?" I thought for a minute, and answered "I'm stupid. Too stupid to give up. Long after normal people give up, I just can't let things go."

Last night, after two scratched and beat up DVD's refused to play in a new DVD player, I hooked my laptop up to my TV and played an on-demand movie from Netflix. It was pretty cool and I never would have done it had I not had the "misfortune" of getting two bum DVD's at the same time. (To give Netflix credit, it's the first, and second, time I ever had trouble in a couple of years of service.)

Six and a half hours after starting, I had successfully watched a 100 minute movie. I probably should have read a book.

SOA What? In reflecting on last evening... I realize that I could compare the evening's activities to an enterprise integration project. And, using the lessons learned perhaps make integration-life a bit easier in the future.

Reacting to a market situation (failed DVDs), I needed to rapidly implement a "business solution" - connect my laptop to my TV and watch a streaming movie.

Remarkably, reading ANY vendor material made this look easy. I was dealing with standards (1080p, S-video, DVI, HDMI, HTTP, WIFI, Browser) that were supported by all my products.

I was under the gun... spending my whole evening setting it up defeated the purpose. I wanted to watch a movie (and ironically, relax), not just muck with components. And, I have to admit, one of the voices in my head wanted me to test the new DVD player. After all, what's the likelihood of getting two bum DVDs at the same time when I've never even gotten one before - coincidental to installing a new player?

So, my project plan looked something like:

1. Test existing solution to evaluate if I can meet the business objective using the existing infrastructure. (Is it really broken?)
2. Research business terms for the new solution. (Would I have to pay extra to watch on demand with my current Netflix subscription?)
3. Research infrastructure requirements. (Basically, I pulled out a flashlight and crawled around to see what connectors were on which equipment and then dug around in a closet to see what cables I had.)
4. Come up with a plan of action simultaneous with acting.

Well, I hacked something together. In the end, it worked - quite well, actually. But... I was missing a neat feature of my existing setup that optimized sound for dialogue. Everytime there was a loud action scene, I jumped out off the couch (and out of my skin) to turn the volume down so I wouldn't bother the neighbors. Oh, and my wireless mouse left quite a bit to be desired as a remote control.

What's my point?