By Jeff Davies-Oracle on Oct 05, 2008
Last week at Oracle Open World I had the opportunity to meet a number of folks interested in Oracle products in general and SOA in particular. One gentleman that I spoke with was very technical, but asked me to explain what I meant by, “point-to-point integration”. At first, the question surprised me. Then I realized that I use terms like “point-to-point” and, ‘EAI” so often that I’ve simply come to assume that everyone knew what they meant. This is a hazard of specialization in any field: you assume that everyone you speak with not only recognizes your particular jargon, but also that they have the same definition for those terms that you do.
So I am starting a new section for the blog, The Architect’s Dictionary, where I define some commonly used (and misused) terms from SOA and enterprise architecture. As always, I invite alternate opinions and well-thought argument.
Point-to-Point (abbreviated to P2P for the purposes of this dictionary) occurs when the IT systems are directly integrated with every other IT system from which they either consume information or provide information. Here is an area where a picture is worth a thousand words (see Figure 1).
Figure 1 shows a typical P2P integration. The OrderPortal application needs information from the following applications:
|ProductCatalog||Direct call to the product catalog’s database|
|InventoryManagement||Uses the SAP BAPI interface|
|CreditCheck||Invokes a web service to get the credit information from the externally hosted credit service bureau|
|CRM||Uses a Java Connectivity Adapter (JCA) to access the CRM (Customer Relationship Management) system|
|OrderManagement||Sends a file containing the submitted order information via FTP.|
|BillingSystem||Uses the VT100 wire protocol to request information from the billing system|
|FullfillmentM||A typical EJB application|
This is typical of any P2P. Systems are directly integrated with each other. There is virtually no abstraction of information between systems and for good reason: since you need to know so much about the systems with which you are integrating (i.e. authentication, physical location, specific API version and transport technology) why would you waste any time trying to create a higher level of abstraction between the systems?
Contrary to popular belief, P2P is not the work of the devil. There are times when P2P is the right approach to your integration challenges. Integrating systems using this approach is usually very fast because you don’t have to do much thinking ahead of time. Just connect the systems as specified by their various APIs and you are done. Because the integration time is generally quick, the initial integration projects cost less (time = money). Furthermore, the data exchange speeds between the systems is usually very fast because there are no mediation layers between the systems to add any overhead (other than the usual machine and process boundaries). It is because of these benefits that P2P is the most common integration patter in use today.
From the diagram you can begin to see how the P2P approach begins to introduce complex interdependencies to your IT system. The above diagram only shows 8 systems and their common relationships. Odds are that your IT environment has anywhere between a dozen applications to many thousands. A direct result of P2P is a rat’s nest of tightly coupled systems. Every time you upgrade any application, the interfaces of that application are likely to have changed. This requires you to modify all of the applications that depend on the upgraded application. So, while P2P is fast and cheap to start, it is slow and expensive to maintain over any length of time. The more your applications change over time (and the more applications that you have), the more time and money you spend on just maintaining what you have. That means you have less time and money to innovate, and to meet rapidly changing business demands. It is this lack of agility in the IT shops of the world
Enterprise Application Integration (EAI)
In the 1990’s, Enterprise Resource Planning (ERP) systems were becoming increasingly common in large enterprises. Integrating with these ERPs became imperative. People were beginning to understand that data and processes belonged to the enterprise as a whole, and not specific applications. Many vendors recognized this pattern and set about to create tooling that would ease the integration problems by providing a platform for integration. This was the birth of EAI.
One of the easiest ways to understand EAI is to think of it as the mother of all integrations. All integration logic lives in the EAI platform. EAI is also known as, “spoke-and-hub architecture” and you can see why from the following diagram.
EAI does provide a number of benefits over the P2P approach. First, it reduces the number of system integration points to anywhere from n to n * 2, where n is the number applications you are integrating. At its worst case, P2P might give you n * (n-1) integrations. Reducing the number of integration points saves time and money, especially when making changes to an existing system.
EAI also centralizes the inter-application business logic. This provides for greater control of how business logic is implemented in the IT environment. If the business logic changed, you only had to make the necessary changes in the EAI system and you could be pretty darned sure that you had covered your bases. In a P2P environment, business logic was scattered across all applications.
A third and final benefit of EAI is that an enterprise normally only needed a single EAI tool, and therefore a single EAI vendor. This provided the enterprise with a, “single throat to choke” since there was only one integration point, the various application vendors could not as easily point fingers at another vendor: it all came down to the EAI vendor.
The problems with the EAI approach manifested themselves over time, as EAI gained popularity throughout the ‘90s. The first problem is that the architecture itself requires centralized control. By that I mean, if you have a single IT organization that controls all of the implementations and integration in your company, EAI was fine. However, as companies globalized throughout the ‘90s and their IT organizations began to distribute their functions across the planet (IT outsourcing and off shoring), centralized IT control became impossible. You could no longer maintain a core of developers that understood the entire enterprise and how it was wired in the EAI tool. Geographically distinct departments of the global enterprise wanted to control their own IT assets and a centralized IT strategy was no longer viable.
A second drawback of the EAI approach is that it did not simplify integration logic. Basically all it did was to take the P2P rat’s nest of integration logic and put it all into the EAI tool itself. The only thing EAI simplified was the number of inter-system dependencies. The complex system integration logic remained. This centralized approach further complicated matters because a single mistake within the EAI integration logic could bring down the entire enterprise: all of your business logic “eggs” were now in one “basket”.
Along with these drawbacks came a new one, the overhead of the EAI tools acting as a mediation layer between the systems. In a pre-fiber networking era, the more messaging you did “over the wire” the slower the entire system ran. And you were paying this price in execution speed without gaining any benefits of abstraction between the systems. Messages sent from one system to another were still formatted for the specific destination system.
Finally, the cost to implement EAI was enormous. It forced companies to re-integrate all of their system using this new tool. The re-integration effort cost enterprises millions of dollars and provided very little benefit.
SOA is an architectural approach with three major objectives:
- Provide comprehensive access to the functionality of the entire enterprise in a repeatable, reliable manner.
- Leverage existing IT assets, modifying them (if needed) to plug into the new service-oriented enterprise.
- Loosely coupling IT system, not only from each other, but even from the business logic that those systems ultimately support. IT systems should be able to change without affecting the high level business processes and services, and those same high level business processes and services should be able to change independently from the low level IT systems that provide functionality. This requires abstraction layers between IT systems and business processes.
SOA is not about:
- Re-writing the enterprise
- Service-enabling every application
SOA varies from both P2P and EAI in some subtle but very important ways. SOA embraces a “separation of concerns” approach when it comes to the integration of systems. Whereas with P2P and EAI they put all of their integration eggs into one basket, SOA separates these concerns into discrete, manageable pieces. In addition, as its name implies, SOA is a marriage of enterprise architecture and “service orientation”. Finally, SOA is a part of the overall enterprise architecture (EA), whereas P2P and EAI attempted to be a complete approach to EA. As with the previous approaches, let’s take a look at a picture of a logical SOA implementation and then describe its attributes.
Probably the biggest difference between SOA and the previous architectures is that SOA does not seek to replace P2P or EAI. That statement may come as a shock to many people. In practice, you will see SOA incorporating P2P and EAI subsystems. SOA is an evolutionary result of everything that we have learned in enterprise computing to date. It not only accounts for the problems we’ve discovered with P2P and EAI, but also what we have learned from early distributed computing technologies, namely CORBA, RMI, EJB, DCOM and JINI.
Separation of Concerns
As I stated earlier, SOA is part of Enterprise Architecture. As a result it co-exists perfectly well with P2P, EAI, Event Driven Architecture (EDA), and BPM. Figure 3 shows all of these approaches working together in a single enterprise architecture. In fact, SOA enables all of these disparate architectures to co-exist because of SOAs separation of concerns. It allows EDA, BPM specifically to do what they do best (process and event management) without having to directly connect those systems to every IT application in the enterprise. The key enabling technology for this is mediation shown in the very center of the figure.
Mediation is the key enabler of agility. Without mediation in your architecture, you will never achieve either business or IT agility. Mediation allows you to loosely connect systems with regard to:
- Physical Location
- Message & Data Formats
- Transport Protocols
- Security Schemes
- Operating System or Platform
Mediation also provides an ideal architectural location to introduce the following capabilities to your enterprise architecture:
- Service Level Agreements
- Routing Logic
All of these capabilities (and more) exist on Oracle’s Service Bus. However, not all ESBs are created equal and you will find many of these features missing from other ESBs on the market. If you don’t have these capabilities built into your mediation layer, you still won’t have the agility needed, whether your vendors uses a single product for mediation, or 5 products to achieve the same results.
Service Orientation is really a design philosophy, not a technology issue. To understand service orientation we must first understand what a service is. For the purposes of this blog I’ll skip over the supporting documentation that I use in my book (The Definitive Guide to SOA: Oracle Service Bus, Apress) and just come to the definition of a service that I use:
Service: When I (the service consumer) ask you (the service provider or proxy) to perform some work on my behalf, and you do it in the manner I expect, within the timeframe I expect, with the quality I expect and without involving me in the work itself.
The word, “expect” is important in that definition because it implies that there is a contract involved between the service consumer and the service provider. The contract (defined in the WSDL of a SOAP message and the SLA defined in the mediation layer) covers issues like timeliness (the response will return within 1 second or less), content (the message format and data types) and quality (the response is complete, or partial).
Service orientation (SO) requires the right mindset. Just as we needed to make a philosophical jump when we moved from procedural programming to object oriented programming, we need to make a similar mental leap to SO. It takes practice and discipline to make that jump, but it can be done.
SOA provides exponentially more agility to both the business and the IT department because of its separation of concerns. Not only are individual applications more loosely coupled through a combination of separation of concerns and mediation, but clusters of applications that are pre-integrated using P2P or EAI approaches can simply be services enabled and plugged into the mediation layer, reducing costs by leveraging existing systems. You can see how this is done in Figure 3 if you examine the EAI integration (lower-left part of the diagram) you can see how that entire subsystem is plugged into the SOA through a single adapter. It is much faster and cheaper to leverage your existing EAI integration this way.
SOA simplifies (and therefore reduces development and maintenance costs) service consumption. If you compare Figure 1 and Figure 3 with regard to the OrderPortal component, you can see that in Figure 3the OrderPortal component only has to integrate with the mediation layer using an industry standard like Web Services. In figure 1 the portal developer would have to know the transport protocols and APIs for all 6 IT systems to which it was connected. In Figure 3 the portal developer only needs to know 1 transport protocol and API. If you make the same comparison with figure 2, you can see that the OrderPortal developer would have to know EJB and its associated API to the EAI system. In all fairness, the EAI approach is also superior to the P2P approach, yet not as robust as the SOA approach because EAI would still force the portal developer to understand the API aspect of the other systems he needed to invoke, even though those other APIs may be wrapped up with an EJB interface.
Another advantage that is not visible in the diagram is the fact that the SOA mediation layer can perform message translations and transformations, allowing each client to make a “logical” (i.e. not application specific) to the enterprise and receive logical information in return. This is another form of loose coupling and agility: it allows the company to change low-level applications (either through version upgrades or installing a completely different application that performs similar functionality) without having to modify any of the service clients.
Finally, you can see in Figure 3 how both BPM and EDA (in the form of Complex Event Processing) are incorporated into the overall enterprise architecture: the service mediation layer again acts as an information bus into which these other capabilities are plugged in.
Everything in engineering is a trade-off, and SOA does not break that principle. There are areas and applications that should not be service enabled. For example, extreme real-time systems (like missile tracking and navigation) are not well suited for service enablement. This is because service mediation and message translation and transformation may add sufficient overhead to adversely impact the time requirements of these extreme real-time systems. This is not to say that the same holds true for all real-time systems. There are many systems with sub second requirements for which service enablement is still the right approach.
Even in the case of extreme real-time systems, it is usually preferable to service enable a portion of it. In the case of missile tracking, the various events that naturally occur (launch detection, entering national airspace and detonation) are certainly candidates for triggering services in the SOA that are designed to respond to these events. Real-time systems an SOA are not mutually exclusive.
Information streaming is another area where SOA is not a good choice for integration. Extract-Transform & Load (ETL) functions are still better left at the lower levels of the IT environment and should not b e run through a service mediation layer. However, just as is true with the real time systems, there are natural events that occur in ETL processes that may invoke a service in the service mediation layer. And certainly, the resulting information from an ETL process is an ideal candidate for service enablement once the ETL function has completed.
Finally, I would caution not to service enable every application in your enterprise. I think of a service mediation layer as my API to the enterprise. If the enterprise is my television set, then the service mediation layer is my remote control to that TV. You should only service enable applications and data that need to be connected to the service mediation layer. Avoid the temptation to use web services or SOA as an integration approach for 100% of your IT systems, or as the only integration approach within your enterprise. Leverage what already exists as much as possible and change only what you need to.
That’s all for this entry. Until next time, keep you code clean and your hands dirty!