Integrated Infrastructure Blog

REST maybe part of the answer

In the last few weeks/months I have got more immersed in the world of REST and what it means to build and have a REST application.  While at first glance it may seem that REST is a lighter weight version of a web-service implementation, it is by no means the complete answer and I do not believe REST is simply a replacement of Web Services because of how they operate is different.

When developing Object applications, in whatever language, you hit a reality that if your application is successful, then it will grow beyond the confines of the machine/JMV that it runs on, and as such you have to refer to objects that are beyond a single instance of the application's memory space.  This can be done using a number of different technologies such as object databases that can map the same object into different application instances, but the locks are maintained by the central server so consistency of the objects are maintained, even if the object is distributed in 100s of applications at a time.

Now technologies such as IIOP (Corba), RMI, DCOM and Web Services have been used to access objects and invoke operations on the instances to retrieve data.  This essentially solves the long-wire problem of accessing an object and then returning an instance of the object, the issue here being is that typically a representation of the object that you want is returned, i.e. using Java serialization or XML and the reference to the actual instance is lost, unless you have a known object identity that you are working with.

REST adds the concept of the pointer to the SOA infrastructure in a way that any business object can be referenced by a URL and thus referenced anywhere within the internet/cloud or a single program space. This is something that is not readily available in Web Services and now makes any RESTful resource available to anyone (given the correct permissions).  Now this sparks a whole debate on to how to identity a particular object and to detect duplicates etc, this is beyond the scope of this blog but will have the same techniques that are used in the object-oriented world, but now working at the cloud scale.

So now that REST gives us some new capabilities, how does this change the current implementations that you are working on?  Interestingly, I think many of the challenges existing in the Web Services world still need to be addressed in the REST world, I just believe that REST is probably better able to address some of the immediate ones and in fact just like every Progress customer has another application to connect to Web Services will always be required.

The other cool thing about REST is that I can typically use the services directly from an HTML page on a browser, something that was impossible for Web Services because HTML and WSDL are not compatible.  However, there is still the need to mediate REST in the same way that applied to Web-Services, for instance to be able to turn a switch to expose the same business entity as REST and Web-Service and in fact any possible protocol.  Allowing external access to your business objects will require caching and a level of indirection to provide for maintenance, performance, etc., and the virtualization of the underlying technology to ensure that an object is not a manifestation of the technology that stores it, i.e. database tables and rows are not part of the object serialization.  Security, Quality of Service (QoS) and Quality of Protection (QoP) are observed along with SLA commitments and the ability to monetize the object and charge accordingly.  This last point maybe very interesting as it would allow organizations to potentially charge for access to their data for different reasons and allow IT to start to become a profit-center rather than just a cost-center - something that I think people will look at as they invest in the cloud.

Now I think that the promise of REST is very interesting, in more ways than Web Services was, but whether the ideal is purely based around REST or a hybrid of many technologies is going to depend on the goal of what is to be achieved. In any event, it is very exciting.

The Rise of Events and Eventing

One of the key things I saw in 2008 was the rise of events and eventing through Event Driven Architecture (EDA).  While "events" have always existed, they have primarily been used in specific markets such as algorithmic trading where our Apama product is commonly used.  The shift in 2008, and moving forward, was that events became more mainstream and that meant that the business and their technical developers were looking to see how they could harness events.

In contrast to traditional SOA infrastructure that use service orchestration such as BPEL to control all aspects of the flow, event based architectures can be used to implicitly enable process to occur.  An event typically happens on a state change or document passing between organizational boundaries or similar.  In order for the event to be useful, it needs to be published to everyone who could possibly be interested and as such, an event consumer can correlate multiple event instances to define new logic and/or to monitor the business event in action.  The key benefit for the enterprise is that the events and their correlation do not have to occur at the initial development time, they need to be captured in a single flow. Handling business events in this way allows a much more organic growth to the logic to be implemented.

Now, any changes in practice will always create new challenges you will need to overcome, not least of which will be the ability for people to assume that business will implicitly happen when there is no end-to-end picture, such as what people expect from BPEL or UML. With guidance, we can overcome these issues because typically the number of processes that are discretely modeled within an organization are few.  One of the key things that has to happen is the idea of event modeling. While there are elements of this in UML through state change diagrams, other tools may have to be used.  Another thing to consider here is that the dynamic aspects of the business will need to be captured and displayed in real-time to allow business decisions to occur when an event really occurs rather than hours or days later. 

In these challenging and opportunistic times that currently exist, I see the arrival of event based architecture as arriving in the mainstream at a very convenient time.  Businesses are restructuring through mergers and acquisitions, so IT professionals need to be able to satisfy this change, plus forces in our market are causing new approaches to be taken, including the idea of clouds that become more than mere hosting platforms.

SOA What? I too agree with Dan Foody that the conventional SOA is not dead by any means. It will actually continue to grow and serve more needs. However, I believe you will see that EDAs will rapidly augment existing projects to initially provide real time visibility and then gradually it will help lift some of the burden on the business as your EDAs evolve and grow.  When that happens, you will start to see more business users being able to implement business logic in real-time rather than having to go back to IT.

The past will predict the future

As you have probably read in today's press and blogs, Progress is going to acquire IONA.  This is great news and I welcome my new colleagues from IONA.  But what does this mean to our combined customers?  While I am not going to directly comment on what the Progress and IONA products will look like going forward; I will review some of the past acquisitions to understand the benefits for everyone involved.

If you look at the Actional acquisition, you will see how these products have evolved over the last couple of years to not only stand on their own and compliment other vendors' stacks, but also how they have enhanced products like the Progress ESB.  The same also goes for products from Pantero, now DataXtend SI, where we support full life-cycle of canonical models and the transformations in and out to save our ESB customers many hours of work, but again also provide the same to the application server vendors out there to ensure that we never have to go into XSLT hell again.

As I said earlier I am not going to comment on combining roadmaps, etc., but you can see from the past that not only is Progress Software adding more capability to our SOA infrastructure portfolio by allowing us to help our customers solve larger and more complex problems, but we can equally play as well in combined vendor solutions.

How the Wii Will Impact SOA

I am eagerly awaiting delivery of my Nintendo Wii Fit so that I can snowboard and ski in the comfort of my living room.  The Wii has changed the way that people play electronic games.  No longer do I have to contort five of my fingers in a specific manner to cut open a patient or have an 8 year old tell me how to play soccer; a game that I have played for many years.

Here is a new way for humans and computers to interact and it is available for the masses. Even my sister, a self confessed hater of anything with a keyboard and screen attached, is telling me how she has completed the latest run on the snowboard. (She is in England where for the first time in living memory the electronic toys have been released before us in the US.)

Anyway, the way that computers and humans have interacted has had a large impact on new possibilities and the increasing the demands on processing power, i.e. you can do a lot more with DOS than with punch cards, and more with RIA than DOS. But imagine what you can do when you are wearing the interface and how many more users will now want to access the system... (See how many Wiis are sold compared to its competitors.)

SOA What? You are creating and designing your SOA infrastructure with web services, portals and RIA applications in mind, but can it scale up and provide the ability to allow the masses to access your systems and provide services that we have not thought of at this time?  The vision of the future is here. Can you and your SOA adapt???

Does your organization shape your SOA or is it the other way around?

I had the privilege to speak at the IDC Event in Buenos Aries last week on Socially Oriented Architecture and then visit some companies who have or are about to invest in SOA infrastructure - in one form or another.  While the speaking engagement was fun and there were loads of questions, it was the visits to the companies that were enlightening because the questions in Argentina are very similar to the ones that I hear in the US.

As I was visiting the various companies, I noticed how many questions were NOT about the details of the technology - in this case the ESB - but about how the deployment maps into their existing organizational structure.  This means that many people use ESBs/SOA to essentially optimize what they are doing today, and as such the benefits of this must be limited by the organization that owns and implements the SOA. What I did not hear is how SOA changed their organization(s); in fact I only heard this question asked by a handful of people in all my years working with SOA technology.

This got me thinking...  When will we really know that a successful SOA implementation is achieved?  Sure, if the SOA matches the current organization, then essentially the existing systems and processes can be optimized, but the maximum potential appears to end there.  If, however, SOA changes an organization in a bigger, more fundamental way, the chance for even greater benefits and success can be achieved.

Perhaps most successful SOA implementations come as part of an organizational shift as well as a technology shift. This obviously means that the biggest reward comes with the biggest risk, and I guess more chance of failure is also possible.

SOA What?  Does your SOA project change the world or just your immediate surroundings?  And, if the organizational scope creep happens, can you and your SOA cope? Because, lets face it, scope creep will happen...

Where is the next opportunity?

Whatever disaster occurs next, there is always someone able to turn it into an opportunity.  Look no further than the last dip in the economy and the associated Enron scandal.  Here was a company that lost millions of dollars overnight, causing distress to investors and employees alike.  However, from Enron came Sarbanes Oxley and as such a number of companies were created or changed direction to support SOX compliance and then went on to make money on the back of that disaster.

Now as we watch the latest downturn happening, we should be wondering what opportunities there will be and whether IT is ready for these changes.  Personally I can see a couple of things that are going to happen...  In the banking sector there is going to be new products that are going to appeal to buyers that are being cut off now.  For this, look no further than the 100% mortgages that were common in the UK just a few months ago and that have gone the way of the dinosaur. What is going to replace them when these banks must look to open up other revenue streams?  Governance is probably going to be one of the main areas of opportunity so that repeats of Bear Stearns are avoided at all costs.

From a technology perspective, I believe the following will be important: For those that have invested in SOA infrastructure, this is your moment to shine and to continue seeing the benefits.  Events will become important, and the ability to extract and view events, and then correlate them to monitor and also to drive new business processes will have a leading edge.  The other thing that is probably going to happen is M&A activity as values of companies go up and down. And as such those that have invested in agile, distributed and heterogeneous SOAs are going to show the wisdom of their investment as they give the business an opportunity to recoup their investment many times over.

SOA What? You should ensure that your SOA is ready for the new opportunities that will come.  What does that mean?  It means that the future is going to be a test of how quickly your SOA can change. It means that the rules may need be rewritten every month to support the constant change that is going to happen.  Will your SOA cope?

Optimizing the SOA Assembly Line

We have all been there, performing the same task over and over again - this is no different when implementing a SOA.  There are many patterns, see www.integrationpatterns.com, that can be implemented plus many techniques to develop the particular functionality that is required, and as sure as god made green apples you will have to do it again.

In the latest version of Sonic ESB 7.6 (being formally announced soon) we've incorporated templates that any developer can create by simply copying any assembly/service that has been used before and dragging it into the templates palette.  Now any time that I want to use that pattern, it just needs to be dragged onto the canvas and all the dependent files will be generated, e.g. stylesheets, etc., and you are ready to go on knowing that the pattern is good.  Note you can export from one developer to another to rapidly build up a set of templates in a team very quickly.

The cool thing is that anyone can do this, no knowledge of Eclipse is required, no programming knowledge ,etc.  With features like this we can really start to increase productivity and solve the simple very quickly, the complicated may still take a little longer.

SOA What?  Now is the time that the business is going to start viewing all SOA infrastructure projects with a fine toothcomb, so providing the ability to deliver more functionality can only be good.  Solve once repeat many, is now truly realized by the developer.

Polymorphic SOA

There's lots of industry noise about Service-oriented architecture (SOA) and the expected benefits, but in many cases SOA is an implementation of procedural programming that may be implemented in Object Oriented (OO) technologies such as Java or .NET. That said, I cannot really see anything OO in a basic BPEL or Web Service implementation; have we all forgotten that OO was promising the benefits of reuse, etc., as well? What if we could improve our SOAs by implementing the OO paradigm in them and using techniques such as polymorphism?

What would a polymorphic service be in SOA? And more importantly, what would be the advantages? One of the most common tasks that occurs in any SOA project is the ability to transform data, e.g. convert a document from one format to another, for which many technologies are currently used, such as XSLT and java binding technologies like Castor and XML Beans (to name a few). While each of these technologies provide the ability to transform one document to another, they do not provide the ability to dynamically implement transforms without further hand coding.

Now what is a polymorphic transformation going to actually implement? Well, unlike an object model, I have no base implementation that I can derive from and no guarantee that my transforms are going to execute in a single environment. And what I don't want to do is deploy a specific XSLT wherever a transformation is required because if the format or the data in the input document changes for some unforeseen reason, then the XSLT is essentially invalid. Therefore, the service that I want to deploy will take in any document and transform it into the target type without having to understand how it is implemented, and if for some reason the transformation service cannot convert the inbound target, then an exception is thrown.

Now I am sure that some of you may see this as being a good idea, but what are the benefits of this? If one of the significant costs of a SOA is the time and money of transformation, being able to insert a single polymorphic transformation service in multiple places could severely reduce TCO of any implementation and reduce the number of errors; how many developers do you have working on essentially the same transformation? For a more tangible example, in the world of Software-as-a-Service (SaaS), what if adding a new partner was no harder than ensuring the partner’s request document can be managed by the OO transformation service. Suddenly the ability to get a return on investment on smaller traffic customers who are making money on fractions of a penny per transaction allows a new market to be tapped.

Is this possible today? Absolutely. Products such as DataXtend SI provide this polymorphic transform services, especially when coupled with ESBs that do not have to be strongly typed.

SOA What? Developers can spend more time developing business logic and less time writing complex transformations, which I am sure they like. But also your SOA infrastructure becomes more agile, allowing you to react to customer and market demands. And if your looking into SaaS to provide the ability to make money on smaller customers, imagine the benefits if all the points of mediation were polymorphic?

Distributed vs Federated

In a previous post, Distributed SOA, I have got a couple of comments about how technologies, such as SAML etc., fit in.  I would like to say that the title was Distributed Not Federated SOA, but Distributed and Federated mean two different things - although in many deployments, both are used.

Distributed SOA is where a deployment is performed over a geographical area. A good example is in Retail. In the case of Retail, there maybe 1000+ stores that are 100s of miles away from the head office.  All of these stores will typically be in the same organizational unit.  Requirements of a distributed SOA include (but are not limited to) the ability to manage, deploy and upgrade any location with no organizational overhead, i.e. there should be no extra IT resources required at the deployment site.  Within the SOA infrastructure the following may also required:

• The ability to go through firewalls using standard and accepted practices.
• The ability to provide high availability without the need for expensive IT infrastructure such as shared disks.
• The ability to define processes that will essentially travel the network and perform the processing on the network where it makes sense.
• The ability to deploy services on machines, either owned by the same or another organizational unit's data center again, without any extra manpower requirements. Note: This is common in state and local government use-cases.

A Federated deployment is very different because the idea is that different organizations will work together and as such some technologies, such as identity management, are key concerns.  A federated deployment does not necessarily need to be distributed. For instance, there are manufacturers that have different divisions that operate in the same building.  A federated environment allows the different organizational units to work together through a defined contract that allows the ability to invoke and share public services.  However, each organizational unit still will want to keep many things private.  Agreed upon security, SLAs, and other contractual definitions are required to make this infrastructure work.

Then there are Federated and Distributed deployments, such as large global manufacturers that have manufacturing plants in many countries and many lines of products managed by different organizational units.

SOA What? Before jumping in the technology toy box and trying to make sure that you have all possible TLAs in the solution, make sure you listen to what is being asked and bring the appropriate technology and no more; only then will you really be in tune with what your customer wants.

SOA - We Got the Name Wrong

For years those of us with a technical mind have found great technologies and looked to solve issues (normally technical with them).  But the merging of the Business and Technical mind are coming together more and more.  We see this especially with the use of SOA infrastructure where services can be plugged together to form business oriented solutions.

Within the last few weeks I have seen various blogs, such as Top 10 mistakes when implementing SOA projects by TaRiQ, that notes that at least one of the reasons that SOA implementations fail (see point 5) is because we forget that it is a business problem that is ultimately being solved - otherwise, lets face it, we are just performing a science experiment.

So to shift the focus, I am renaming SOA from Service Oriented Architecture to Solution Oriented Architecture, this moves the problem front and center not the technology.  Yes, we still need the various technologies to solve the solution: ESB for connection, mediation and control of the services, governance and visibility to manage and control the complete solution, and data management to ensure that all the information we use is correct. But we are now also using these and others in a combined manner to reach a complete solution.

Hopefully with this I will no longer have to argue with developers why making every Java object (I am sure there are .NET developers who feel the same - I just tend not to meet as many of them) a web-service is not a good idea (these are mere implementation issues).  And that you can still create a hub and spoke architecture or point to point deployment with Web-services even if they are open standards.  Note that I did find one reference to "Solution Oriented Architecture" when I Googled it, see blog entry, so maybe I am second here.   But what if we take this to the next stage and SaaS now becomes Solutions as a Service, or maybe Software as a Solution is better, IaaS becomes Information as a Solution or Integration as a Solution; depending on the definition that you use.

Or, perhaps, we should look at changing the acronyms completely and SOA becomes SOB - Service Oriented Business (or what you do if it fails), but my new favorite is BaaS - Business as a Service where companies would now be a service or services, but I guess nothing is that new because I can Google BaaS and get references to that too.

Maybe some of these work better than others what do you think?  Are there any other logical combinations of Business, Service, Architecture, etc. that have not been used?

Distributed SOA

Last week I attended the Forrester Enterprise Architect forum and heard a pretty consistent cry from people walking around and stopping at the booth; "How do I implement SOA in a distributed environment?"   This question is being raised by people talking about 3 or 4 locations distributed worldwide to 500 locations in the US.  Most of the people have evaluated the competition and realize their service orchestration engines are great in a hub and spoke model but do not really support global processing.  More to the point, even if they supported the idea of distributed applications, they do not support the idea of management/control of that environment, i.e. I can deploy the application to a remote location for version 1 of the solution, but version 2 will require me to re-deploy a whole new application.  Worse than this, there is management overhead that I must place in the remote location, e.g. someone to start/stop and monitor the application server or similar.

Well now I can see these problems are solved and have been for a number of years with Progress Sonic ESB.  The enterprise service bus (ESB) provides the ability to not only distribute the service/event processing to where it makes sense - e.g. if I am a retailer and need to allow my in store POS devices to communicate - but at the same time allows them to communicate through firewalls with the various applications distributed in other stores and the corporate headquarters.

This whole SOA deployment can be managed from a central location to ensure that no IT resources are required within the distributed locations - other than to turn the hardware on/off occasionally.  Metrics and Notifications are fed up to existing management software, and I can view log files of remote applications from the central console.  Also, new versions of the services can be deployed centrally so updates can be deployed to 500+ sites at the speed of the network, and if the network is down, the ESB will hold the request until the network returns.

SOA What? If you have a distributed SOA/event-based requirement, look no further than Progress Sonic ESB. OK, so as a principle architect, I'm probably not supposed to be "marketing" on this blog, but not only will Sonic ESB allow you to create your application with minimal code in a distributed environment, but you will also be able to manage it, and reduce the time and cost it takes to make your SOA a success.