Integrated Infrastructure Blog

Buysides Finally Getting Some OTC Derivatives Love!

The ISITC OTC Derivatives Working Group, of which I am a member, recently announced its Market Practice for Contract Notification. I've been involved with OTC Derivatives trading operations and technology for almost 20 years now, the last 10 primarily on the buyside.  When I first joined an FpML working group, allocations weren't even part of the spec, and we are only now beginning to see vendors who provide buyside OTC Derivative trading solutions.

The industry itself has lagged in recognizing the buyside's role in the OTC Derivatives market.  The first movement around helping the buyside deal with the complexity of these instruments started about 5 years ago with the Swaps Best Practices Initiative jointly sponsored by ISITC and the Asset Managers Forum within the old Bond Market Association.  The BMA is now part of SIFMA and the AMF's Swaps initiative evolved into the Derivatives Operations Committee and the formation of the ISITC OTC Derivatives Working Group.

While custodians and other service providers including DTCC, Markit and SWIFT have stepped in to assist their buyside customers to process OTC Derivative trades, there are few off-the-shelf solutions. The majority of sales of Progress’s OTC Derivatives solutions for FpML, DTCC Deriv/SERV, Markit Wire and SWIFTNet FpML still go to buysides and custodians who are adding support for buyside functionality to systems designed for sellside. I am happy to report that Progress now has software vendors as customers looking to address this market. I’m optimistic as buysides spend more to meet new regulations we’ll see more complete vendor solutions.

Recap: Putting the Information Framework to Work for You

Nearly every time John Wilmes and John Reilly get together for a webinar, not only do they deliver valuable insight and content, they spend substantial amounts of time fielding questions from the audience. Personally, I find the live Q&A to be one of the best parts of the webinars.

The webinar we hosted on the 21st, Putting the Information Framework (SID) to Work for You, was true to form. John Wilmes and John Reilly spent nearly 30 minutes after the webinar finished answering questions.

What I’ve noticed over the past several webinars is that the tenor of the questions are shifting.  We’re seeing a shift from educational/informational questions towards questions about actual implementation.  During last week's webinar we received several questions about designing enterprise information models, developing business cases, and even the existence of pro-forma RFI/RFPs.

The community is coming to the realization that managing and maintaining information exchange is clearly within the remit of integration—leave it unaddressed at your peril.  Your responses to the TM Forum webinar survey bears this out. For example, in response to the question “What percentage of your projects are spent on integrating data between your applications and services?” - 70% of you told us that dealing with data integration consumed more than 40% of the development effort. And dealing with data integration is not easy work. 

When we asked about the most difficult challenge, your top three responses were:

1. Keeping documentation up to date (19%)
2. Not enough resources to keep up with development (30%)
3. Managing change (43%)

These challenges are interrelated and what’s more interesting is that all three challenges can be directly addressed by the use of exchange modeling technology for data interoperability.  The inability to keep documentation up to date makes it difficult to manage change.  When changes comes in, your teams are forced to manually ascertain the impact and test everything to ensure success in production.  For the small selection of you who are starting from a green field addressing change will not be an issue... the first time. However, with every subsequent change caused by new developments, new requirements, new business initiatives... you will need the means to understand and manage the effects in your environment. If you don’t have that capability, you don’t have agility.

So given the resource constrained world, the choice is etiher: to continue to muddle along with an allocation of resources heavily skewed towards these maintenance tasks; or to raise the effectiveness of your teams in maintenance and use those savings to address more mission critical tasks. Perhaps more succinctly: do more with less by doing what you must do more efficiently.  

This is the conceptual basis on which we have created many successful business cases.  If you’d like to learn more, or if you would like assistance in justifying the need for exchange modeling technology and framework implementation, please do not hesitate to contact us.

A New Renaissance for ODBMS? Part 2

I’ve been posting comments from an email discussion I had with Luis Ramos, one of our object database experts here at Progress Software. In our last exchange, Luis commented on the ideal use cases for object databases. Today Luis answers a few questions about switching from RDBMS to ODBMS, the market, and where to get more information.

Me: Why would someone want to switch from RDBMS to ODBMS?

Luis Ramos: There are two reasons - Performance and TCO.

There is significant overhead in accessing object-oriented data (in Java, C++, or C#) from a relational database.

If "complex" or object oriented data is stored in an object-oriented database, then the data is stored in the same format as the object. It is very straight-forward. But if you try to store the same object data in a relational database, then you would need to decompose the data into rows and columns. This procedure can be quite difficult and imposes quite a bit of overhead specially if the corresponding object model is complex in terms if its inheritance hierarchy and interconnections between objects.

There will be a magnitude-times improvement in query performance using an object database. In a relational database, queries are performed on the server-side and results are brought over to the client and materialized. In an object-oriented database, such as ObjectStore, the objects and indexes are cached at the client (via its cache-forward architecture). Consequently, performing a query is very efficient since it is carried out at the client by simply de-referencing pointers or references of the corresponding index structures.

For those interested in cost, using an object-oriented database instead of a relational database can reduce the TCO significantly because it avoids the development and maintenance of object relational mapping (OR mapping) code.

Using a relational database from an application that is implemented using an object-oriented language such as Java or C++ requires the use of OR Mapping technology to transform data between objects and relational tables. The amount of code and effort to undertake (and maintain!) this transformation can be significant. With an object-oriented database, there is no OR mapping code thereby reducing the TCO.

Me: Why is the market so small?

Luis Ramos: Relational databases such as Oracle and SQL Server are very well known in the industry and are common place. People generally tend to use what they know, and so developers usually start out using relational databases by default. Relational databases can be made to work and developers would go through all sorts of contortions writing a significant amount of OR mapping code to force complex data into two-dimensional tables and live with up to 100x slower performance. But then pain is relative. If you have not heard of object-oriented databases and these benefits, how would you know better? Object database vendors need to do a better job at marketing their products.

Very few people really know about them.

Me: Where can I learn more?

Luis Ramos: You can always email me if you have questions about ODBMS or log a question here on the blog in the comments section.  My email is lramos@progress.com.

There’s also a wide range of information on the ObjectStore and PSE Pro websites. There are a couple of white papers that are worth reading: here’s one for Real-Time Data Caching, and another a white paper on Coding Java Applications on Cache Forward Architecture.

Stay tuned! Our next (and final) post in this special series on object databases is where we will talk more about the question "A New Renaissance for ODBMS?"

Why on-premise SOA cannot offer cloud pricing

David Linthicum suggests that SOA infrastructure vendors must switch to a cloud pricing model and get paid only on delivering value, further taunting the vendors to put their money behind their products. Interesting proposition... But I don't believe it will work. And the comparison to cloud or SaaS may not even be valid. For SaaS, its easy to identify the value delivered - as that of a business solution being made available as a service. For cloud, the SLAs are at the platform level and can only ensure availability - they cannot ensure business value. On-premise SOA is even worse because the operating platform is not even in the SOA vendors control, so guaranteeing even the availability at the same level as cloud is not even possible. At best the vendor can assure that the SOA platform will not fail, but they definitely cannot guarantee against failure from the hardware, OS or other layers in the solution stack that reside with or on the on-premise SOA.

The business value that David refers to is way beyond just platform availability, and that surely cannot be assured by any single component in the overall solution stack - unlike in the SaaS model where the SaaS provider controls the complete solution stack and can provide the guarantees.

A New Renaissance for ODBMS? Part 1

Recently a team of our object database experts - Adrian Marriott and Luis Ramos - attended the 2009 International Conference on Object Databases. Not only did they present on design patterns and discuss the resurgence of object oriented databases, but Adrian won an award (and a netbook!).

Adrian Marriott was the first place winner of the award for best Common Persistent Model Patterns for Performance and/or Scalability Optimization. He beat out 25 other compelling patterns with his Query Visitor pattern which allows one "to define new result set formats without changing the underlying persistent object model."

Roberto V. Zicari, Editor of ODBMS.ORG, said of Adrian’s pattern...

"It is common practice that some database designers treat an Object Database (ODB) like a Relational Database (RDB). That is they are very query intensive rather than model intensive in their design. Some designers start with a “relational” model, and then adjust it to a model that is more "ODB-oriented", or closer to their problem domain, in order to get better results. This  task is difficult.  Marriott`s pattern, Query Visitor, can speed up the database  development process by providing a tested, proven development paradigm."

Luis Ramos contributed to the lively discussion in one of the more intriguing panel discussions at the conference: A New Renaissance for ODBMSs?  As part of our post-ICOODB2009 coverage, I asked Dr. Ramos to share some thoughts about object databases and their use.

Me: What are some use cases that benefit from an Object Oriented Database?

Luis Ramos: There's three main uses cases: complex (multi-dimensional) data, transactional caching, cloud-databases—and given today’s SaaS-world, you can see why ODBMSs are becoming more and more relevant.

By complex, multi-dimensional data, I mean data that is hard to render into rows and columns.  For an easy example of non-complex information consider the current roster for the Boston Red Sox Baseball team; the roster lists each player and their name, number, position and statistics–it looks like a rectangle with two dimensions.  This formulation makes it easy for me to ask the question – what is the current batting average of the 1^st baseman for the Red Sox?

But consider a slightly different question – how are individuals in a family tree related?  Consider Barack Obama’s family tree. If you visualize the data, it is not a matrix at all. It looks more like a tree of nodes. This would be very natural to store as objects with links to other objects. Consider asking a question like, is George W. Bush related to Barack Obama? Answering this question is quite easy in an object database. You simply follow pointers from the node representing Barack Obama and see if you can reach George W. Bush (and apparently they’re 11^th cousins). Following pointers or de-referencing references is certainly a lot more efficient than doing an arbitrary number of joins.

On the transactional caching use case. Our clients have selected object databases over other data caching technologies such as Memcache and Tangosol for a these important reasons: transactional access, durability, and automatic cache replacement. With object databases, there’s transactional access to the cache thereby preserving the data integrity of the cache – you don’t find this in many caching solutions.  Also, the cache is durable. If the application is terminated intentionally or otherwise, recreating the cache is fast and efficient because it is being populated from an object database.  In this case, there is no overhead for running SQL queries to find to the objects to bring into the cache and no overhead to transform between relational data and objects. Third, ObjectStore automatically manages the cache if the amount of data being accessed in the cache exceeds the amount of memory.

On cloud-databases, object-oriented databases are a more natural fit for persisting cloud data, which is inherently tuple-based, for a number of reasons. First, storing tuple-based data, whose values are arbitrary (strings, integers, double, boolean, etc) and that are automatically indexed could be challenging to do in a relational database. This type of problem is reminiscent of the issues related to formulating a relational model to store an object model that has an inheritance hierarchy. Second, scaling a relational database is not easy. The usual practice to scale a relational database in order to support more load is to use more powerful hardware. In an object-oriented database such as ObjectStore, it is easier and simpler. Since the queries are performed at the client (the cloud node or service), scaling the database can be accomplished by simply launching more services.

Stay tuned, the next post will cover additional Dr. Ramos’ comments about RDBMS to ODBMS, the market and where to get more information.

Secret No. 2 - The Myth of Five Nines

Lets continue to pull back the covers on highly available integration infrastructure.

Listen now to Part 2 of the Five Dirty Little Secrets of Highly Available Integration Infrastructure podcast series: The Myth of Five Nines.

Presented by Hub Vandervoort, CTO, and Ken Rugg, VP and General Manager, Integration Infrastructure, Progress Software, this is the second of five new interview-style podcasts that reveal the software industry’s five "dirty little secrets" or misconceptions of a so-called highly available integration infrastructure.

Three minutes of downtime may not seem detrimental now but if that failure occurs during peak transaction times, it will result in lost revenue and poor customer service. In this podcast, Hub and Ken present ideas on how to deliver a highly available enterprise infrastructure.

> Secret No. 2 - The Myth of Five Nines

Tune in next time for Secret No. 3 - Recovery Time.

Putting SID to work

On July 21st at 10am (ET) John Wilmes, our Chief Technical Architect for the Communications sector, will be speaking along with John Reilly, the TM Forum’s Senior Program Manager for the Information Framework, on the Information Framework (SID).

You can register here

Everytime I attend one of the webinars that John Wilmes does with the Forum, I’m always impressed by the TM Forum’s completeness of thought. Unlike a lot of other industry and architectural frameworks out there – the Forum didn’t focus on just part of the puzzle.  What was understood, and what the Solution Frameworks (NGOSS) highlight, is that business transformation requires addressing elements that live both within and between business process, information models, logical applications and technical architecture. Part of the joy of hearing John talk about the SID is to hear again how the data, process and applications all interrelate to each other.

For those of you who might be new to the TM Forum – or even those of you who are looking for a solid conceptual framework for thinking about transformation problems - in addition to this webinar there have also been several other webinars from the Forum that John has done over the years, including:

• What is NGOSS and What Can it Do for You: If you’re looking for a great introduction to the Solution Frameworks
• NGOSS Contracts: The Key to Improved Interoperability:How Business Services (NGOSS Contracts) can improve interoperability across multiple business silos / stovepipes
• SOA and NGOSS Contracts: A Practical Approach to Integration: How SOA and the frameworks fit together

For an alternate, non-telco example of a similarly complete framework that spans process, data and applications—there was also the recent webinar by Boris Bulanov and Frank Neugabauer about the Insurance industry’s approach to transformation the ACORD Framework. Many of the concepts are similar and it's another proof point that one must consider the whole problem and all its requirements when initiating an integration or transformation project.

Simple SOA. Not-so-simple SOA.

Open Source SOA has arrived! On a single day we have one platform announce that they have simplified basic SOA, and another has announced that SOA is never simple - and if it is simple, just use web services. The question is... is there really a simple SOA and a not-so-simple SOA?

As I look at it, there is a very fine line between simple SOA and the more complex ones that require an enterprise service bus (ESB). Lines are blurred today. Now, ESB itself can be in a single container or more massively distributed ESBs with a strong messaging backbone (like Sonic ESB). But messaging backbone or not is more about the underlying integration infrastructure. So, what does it mean for the application itself? Is the model/paradigm different for the two cases? This is what gets lost out in the melee - the application models and the underlying infrastructure are quite decoupled. From an application standpoint many of the ESB mediation requirements may always be needed functionally. Can one access services without getting the data in the right format? To get it in the right format, won't we need data transformation? And security is a given in all cases. Sometimes one can look at business rules based routing to services. All of these regardless of whether it is a simple SOA or not-so-simple (more distributed multi-data-center) SOA. 

This (common treatment of all SOA use cases regardless of complexity) is very evident with the way our open source platform, FUSE, works because it has an embeddable transport layer. One could use the containers with or without Active MQ. But with or without distribution, the same functionality is available. Simple SOA? Or not?

REST maybe part of the answer

In the last few weeks/months I have got more immersed in the world of REST and what it means to build and have a REST application.  While at first glance it may seem that REST is a lighter weight version of a web-service implementation, it is by no means the complete answer and I do not believe REST is simply a replacement of Web Services because of how they operate is different.

When developing Object applications, in whatever language, you hit a reality that if your application is successful, then it will grow beyond the confines of the machine/JMV that it runs on, and as such you have to refer to objects that are beyond a single instance of the application's memory space.  This can be done using a number of different technologies such as object databases that can map the same object into different application instances, but the locks are maintained by the central server so consistency of the objects are maintained, even if the object is distributed in 100s of applications at a time.

Now technologies such as IIOP (Corba), RMI, DCOM and Web Services have been used to access objects and invoke operations on the instances to retrieve data.  This essentially solves the long-wire problem of accessing an object and then returning an instance of the object, the issue here being is that typically a representation of the object that you want is returned, i.e. using Java serialization or XML and the reference to the actual instance is lost, unless you have a known object identity that you are working with.

REST adds the concept of the pointer to the SOA infrastructure in a way that any business object can be referenced by a URL and thus referenced anywhere within the internet/cloud or a single program space. This is something that is not readily available in Web Services and now makes any RESTful resource available to anyone (given the correct permissions).  Now this sparks a whole debate on to how to identity a particular object and to detect duplicates etc, this is beyond the scope of this blog but will have the same techniques that are used in the object-oriented world, but now working at the cloud scale.

So now that REST gives us some new capabilities, how does this change the current implementations that you are working on?  Interestingly, I think many of the challenges existing in the Web Services world still need to be addressed in the REST world, I just believe that REST is probably better able to address some of the immediate ones and in fact just like every Progress customer has another application to connect to Web Services will always be required.

The other cool thing about REST is that I can typically use the services directly from an HTML page on a browser, something that was impossible for Web Services because HTML and WSDL are not compatible.  However, there is still the need to mediate REST in the same way that applied to Web-Services, for instance to be able to turn a switch to expose the same business entity as REST and Web-Service and in fact any possible protocol.  Allowing external access to your business objects will require caching and a level of indirection to provide for maintenance, performance, etc., and the virtualization of the underlying technology to ensure that an object is not a manifestation of the technology that stores it, i.e. database tables and rows are not part of the object serialization.  Security, Quality of Service (QoS) and Quality of Protection (QoP) are observed along with SLA commitments and the ability to monetize the object and charge accordingly.  This last point maybe very interesting as it would allow organizations to potentially charge for access to their data for different reasons and allow IT to start to become a profit-center rather than just a cost-center - something that I think people will look at as they invest in the cloud.

Now I think that the promise of REST is very interesting, in more ways than Web Services was, but whether the ideal is purely based around REST or a hybrid of many technologies is going to depend on the goal of what is to be achieved. In any event, it is very exciting.