I'm presenting a keynote at the next International Association os Software Architects (IASA) IT Architect conferences http://www.iasahome.org/web/itarc/nyc/topics#Keynotes on May 22 - 23 in New York City.

I was looking through the agenda and I came across this - Interesting Real-world Architectures and the Handbook of Software Architecture presented by Grady Booch via Second Life

I checked with the conference organizers, and sure enough, Grady is going to be at his home base (which is usually Hawaii), and broadcasting the presentation via 2nd Life, and conference goers will be witnessing his avatar giving the presentation on the big screen.

How cool is that!?  I want some!

Dave

Following a recent posting on a SOA success story http://blogs.oracle.com/davidchappell/2008/04/soa_helps_you_move.html , I got some important feedback http://tech.groups.yahoo.com/group/service-orientated-architecture/message/10092 on how the writeup was a bit slanted towards highlighting the use of the tools and infrastructure such as BPEL and ESB, and didn't talk enough about the services themselves.   The point being made that Service Oriented Architecture is about service orientation, and all too often the folks who live and breath the supporting infrastructure tend to gloss over the part about the services themselves.

I plead guilty.

So as part of my retribution, I would like to provide some nuggets of wisdom on service design that came out of the Move project -

The Prospect to Cash business process includes a number of very large applications (PeopleSoft, Siebel, Oracle Customer Data Hub, and a variety of fulfillment applications).  CSC used the following criteria to identify service candidates:

- Is there similarity in the type of data that needs to be passed to various applications?  If so, then the business logic in each respective application that operates on that data is a candidate for service enablement.  The resulting collection of related services are also candidates for participating in a canonical data model.

- Does the business require part of an application to be used by both internal as well as external customers and partners - what are those parts and can they be broken into smaller atomic functions?

- Here is one that is bound to be a subject of discussion - A key service design consideration is that there is no direct interaction between two services.  They are self sufficient atomic functions that can be used to build orchestrations by the business e.g. - customer data needs to be created in Siebel, PeopleSoft, Oracle Customer Data Hub and the fulfillment system. Each one needs a different component of the data.  This raises a question - is "Create Customer" a service or should it be broken into "Create Customer, Create Address, Create Contact, Create Fulfillment Profile"?  The answer largely depends on whether each of these services is complete in itself, and depends on the nuances of the business.  On first thought, for instance, it may appear that an address without a customer may not be very useful. But if you are someone like Move the rules are different in this case. When they display a house on their website, they don't display the name of the person, they only display the address and the fulfillment profile. So for Move it makes sense to have them as separate services and business can decide how to string them together via BPEL to achieve business results they are intending to achieve.

A similar example can be given about the external customer who needs to consume these services.  If you want to update some information about your house, there is only specific information you can update.   So that would help determine what the service interface would look like.

The overall experience from the team about building the SOA at Move, Inc is that it is very easy to get carried away into JBOWS (Just a Bunch Of Web Services) mentality, without having proper checkpoints in place in order to ensure that the right services are being built for the right reasons.  To do it right really requires proper analysis and a "business case" for each logical function to make it candidate for being Service. If you follow this practice judiciously, you will not end up with JBOWS.

Dave

For the past several years, I have been involved in many healthy discussions centered around the benefits of adopting technology and its supporting tools and infrastructure.  Never once had I ever thought of measuring the benefits in terms of tonnage of hardware or in kilowatts per hour (kW/h).

Until now.

In David Linthicum's recent blog on "Is SOA Green" http://weblog.infoworld.com/realworldsoa/archives/2008/03/is_soa_green.html?source=NLC-SOA&cgd=2008-03-25, he makes the point that "the more efficient a business is, the fewer resources, natural and otherwise it will consume. Thus, the better the business is automated, and business processes optimized, the more efficient it will be. The primary of objective of an SOA is both efficiency and agility through architectural rejuvenation"

I couldn't agree more.  BTW today is Earth Day.  This week is also the Web2.0 expo in San Francisco http://en.oreilly.com/webexsf2008/public/content/home, and the Green Expo in New York http://gogreenexpo.com

In light of all these things coming together, and in the spirit of "going green", I would like to tell you about a concrete example in an exciting account of one of Oracle's customers, Verizon Wireless, who is in the process of going green by rewriting their fraud detection application using SOA, EDA, and Web 2.0, and as a result plan to eliminate 6 tons of hardware from their datacenter, and reduce power consumption by 99.5% from 200 kW/h down to 1100 W/h.

From a software and architecture perspective, this is an exciting story about how Verizon Wireless is building a new Fraud Detection application which uses Event-Driven Architecture (EDA), BPEL process management, a business rules engine together with a Web 2.0 style Flash/Flex UI to build what would be referred to in financial services as a straight-through processing application. 

In the process of doing this, they are dramatically reducing the amount of code that has been written, and the amount of data that needs to be stored locally.  As such, they plan to eliminate 6 E-class Sun boxes using ~192 processors and replace them with a single 8 core processor on a Sun UltraSPARC T1 using the Niagara chip architecture.

The old application is based on J2EE using what was available in the mid to late 90's.  It happened to replicate the entire data warehouse of call detail records for use by the fraud detection application. It also had a lot of procedural custom code that was hand written by 5 FTE over 2 years, some stuff that was ported from Forte to J2EE, and 100s of JSPs feeding (circa 1995) html 3.0 pages with data.

I plan to provide more detail about their service architecture in the future, but here is the basic outline of the new application structure:

• 
  
  Data coming from the switches is analyzed and checked for business exceptions.  Examples of business exceptions include detection of excessive data thresholds, which could be an indicator of a customer using their laptop to run a streaming video website, or the practice of phone cloning to make phone calls using someone else's account, or more ambitious fraud such as masquerading as a third party roaming partner and attempting to charge back to Verizon. (this is the EDA part of the architecture)
  
  
• 
  
  When such an exception is detected, an event is generated and sent to a BPEL process
  
  
• 
  
  The BPEL process invokes a number of services, which includes going out directly to the source of the call detail records to get the information necessary to enrich the event data.  It is then fed into a rules engine to check for violations, make decisions based on policy, and then on to generate more detailed reports.  (This is the SOA part of it)
  
  
• 
  
  The supporting SOA technologies involved include Oracle BPEL Process Manager, Oracle's rules engine, and Oracle AQ.
  
  
• 
  
  The UI interface is RIA/Flex based, and runs mostly in the browser (this is the RIA/Web 2.0 part of it)

So, back to the "going green" part - They are working with a third party energy consultant to produce the final number on energy reduction once they go into production (they are in UAT stage now), but here is their best estimate given what is known today. 

• 
  
  6 TONS of equipment reduced to ~200 pounds (production, qa, dev-test).  I just love those numbers!
  
  
• 
  
  192 processors reduced to a single, 8-core processor (production)
  
  
• 
  
  20+ Terabytes reduced to < 64 Gigabytes storage
  
  
• 
  
  100 kW/h reduced to 540 W/h (production)

I'm no expert at cooling system requirements, but I have heard the rough calculation for power consumption of cooling systems range from 1.3 to 2 X the amount of power consumption of the hardware, so the numbers should really be 200 kW/h reduced to 1080 W/h.  Its still a 99.5% reduction in power usage.

Of course I can't attribute 100% of the impressive results to adopting SOA.  There are lots of things that contribute to the reduction in footprint, such as the massive rewrite and the introduction of more powerful hardware and chip technology (I'll leave that writeup for  SUN to have their own day in the...sun).  However, I can point out some things that are obvious and significant -

It is often said that the best line of code is one that you never have to write.  The new implementation is 0.5% of its original size.  This is directly attributable to using Oracle  BPEL Process Manager and the rules engine instead of their custom code.   They also went from 5 FTE down to 1 consultant who built 40 - 50 BPEL processes over the course of a year.  The result is more flexible and resilient to change as it is declarative and UI driven rather than being hard wired into custom code.  Their presentation tier code also went down to 0.5% of its original size by moving to 1 JSP and 1 SWF for rich UI. 

The other significant contributor to the reduction in footprint is due to the nature of modern SOA standards and supporting infrastructure such as BPEL and ESB, which allows Verizon to extend the reach of their systems so that they don't have to store their "golden record" call detail data locally.  They can get the data remotely and on the fly, and use enrichment services along the way to get the data in the proper form to be make decisions about fraud and overage, and generate detailed reports about the business exceptions that are identified.  This is huge for them in that they no longer have to replicate the data warehouse, but are able to pull call records and other information directly from external heterogeneous systems.

BTW, in David's blog he also points out - "Thus, with gas approaching four bucks a gallon, I made the shift from an SUV that get's 11 MPG to a crossover that gets 30 MPG".   I'm still in the process of shopping around to make that shift...but I have some time since summer is here and my Harley gets 44mpg!

Dave

At this week's RSA Conference 2008, Thomas Kurian laid out Oracle's vision for what is being coined Service-Oriented Security.  Service-Oriented Security is a way of using a service orientation to enable applications with a complete set of the common security capabilities and security processes such as authentication, authorization, user administration, role management, identity virtualization and governance, and entitlement management, as well as audit and control.  Tony Baer also discusses the topic in his blog: http://www.onstrategies.com/CURRENT-NEWS/Oracle-Releases-Role-Manager-Pushes-Service-Oriented-Security-Strategy.html. 

Typically an application may hard wire security capabilities, which makes for a brittle architecture that is not conducive to change.  With Service-Oriented Security, security capabilities and processes are decoupled from applications and more centralized via SOA, making them more available, manageable and consistent across an enterprise.  So in a sense, we are using SOA to enable and strengthen your SOA projects by making them more agile.

To back this up, there have been some key deliverables to date which enable this vision, which cover security issues spanning deployment, governance, administration, and development

• 
  
  GA of Oracle Role Manager, which provides a service for roles-based access control, provisioning and approvals across business applications.
  
  
• 
  
  GA of Oracle Application Access Controls Governor 8.0, which is control monitoring software that provides segregation of duties analysis and enforcement for heterogeneous enterprise application environments
  
  
• 
  
   A controlled beta preview release of Oracle Fine Grained Authorization.  This is software designed to externalize hard-coded authorization policies from heterogeneous enterprise applications, and nicely complements Oracle's Identity and Access Management offering.
  
  
• 
  
  Identity Governance Framework - a multi-vendor standard proposal, spearheaded by Oracle, that provides a service-oriented privacy-aware architecture for developers to access identity data while adhering to usage policies.  Oracle, in conjunction with the Liberty Alliance, has delivered the first open source component of the proposed standard.  

For more information, check out these links -

White paper on Service Oriented Security  -

http://www.oracle.com/technology/products/id_mgmt/pdf/serv_oriented_sec.pdf

Other related announcements -

Oracle Releases Oracle Application Access Controls Governor 8.0

Liberty Alliance and Oracle Team to Advance Identity Governance Framework

Dave

It's always great to hear about successful adoption of SOA, and its particularly exciting when its being used for what you'd think. :)

MOVE, Inc. has recently finished up a really cool SOA project to integrate together and streamline a set of processes a variety of externally facing web sites with internal CRM solutions.  The solution involves the use of BPEL, Web Services, ESB Mediation, application adapters, and canonical data models.

The business benefit to all this from a high level is that MOVE, Inc. was able to improve customer service by improving visibility of customer information across all their systems and improved processes while helping reduce the cost of ownership for its existing applications including Oracle's Siebel CRM and Oracle's PeopleSoft Enterprise Financial Management.

Before I get into how they did it, first a little bit about MOVE, Inc. and the challenges they faced -

Move Inc., formerly Homestore, Inc., is the leader in online real estate.  Through it's network of sites (below) Move, Inc provides information on real estate property listings (homes and apartments), moving, home and garden and home finance.  . 

Move's network of websites:

• 
  
  REALTOR                                www.realtor.com
  
  
• 
  
  Welcome Wagon                    www.welcomewagon.com
  
  
• 
  
  Moving                                    www.moving.com
  
  
• 
  
  Senior Housing Net                www.seniorhousingnet.com
  
  
• 
  
  Home Plans                            www.homeplans.com
  
  
• 
  
  Home Insight                         www.homeinsight.com
  
  
• 
  
  Factory Built Housing             www.factorybuilthousing.com

In the midst of rapid growth, their challenge was they needed to consolidate disparate applications across two of its business units into an enterprise standard CRM solution that could scale to meet its evolving needs and provide a single, accurate view of customer data to improve efficiencies and automate auditing, billing and fulfillment processes. 

Move's business units had custom applications built on the .Net framework for supporting their customer relationship functions and order creation. There were numerous disparate systems and manual touchpoints built to support various business functions for customer relationship management. In an effort to consolidate these applications into an enterprise standard CRM solution and also provide an application which could scale to the growing needs of the business, a CRM solution consisting of Oracle Customer Data Hub, Siebel and PeopleSoft integrated with Oracle Fusion Middleware was proposed to enable the following.

• 
  
  Consolidated Customer Database
  
  
• 
  
  Support for Existing Sales Activities
  
  
• 
  
  Order Capture
  
  
• 
  
  Integration with Fulfillment Systems
  
  
• 
  
  Integration of customer usage from web sites into Siebel
  
  
• 
  
  Integration to PeopleSoft Billing

It sounds like a pretty tall order.

As part of the implementation MOVE worked with CSC (Computer Sciences Corp).  CSC built a SOA that focused on re-usability of services. CSC also ensured that the business processes were at least as efficient if not better than what the current systems offered.  Using standard WSDL interfaces, Oracle BPEL PM and Oracle ESB were used to extend the reach and normalize the integration across Siebel CRM, PeopleSoft Enterprise Financial Management and other systems both internal and external (Figure 1).

Figure 1: Using BPEL, ESB, and Web Services to help implement their SOA, Move extends the visibility of customer data.<o:p></o:p>

How They Got There

The first step in the design process was to model the future business process and identify the functionality that would be enabled through Siebel and the key integration points with the various systems. Move and CSC worked together to define canonical data models for customer, order, listing, usage and billing information and created service contracts for each of the Web services that were used in the integration.  CSC then build new streamlined processes as BPEL orchestrations using Oracle BPEL Process and Oracle Enterprise Service Bus (Figure 2) -

Figure 2: Canonical data models allow processes to be constructed that generically orchestrate a common set of canonical services to without the need for specialized point-to-point data transforms

Service contracts were defined for each of the web services that would be used in the integration.  The customer information was then consolidated from the multiple systems that the two business units had into the Oracle Customer Data Hub. Customer Data Hub also serves as the hub of customer information and provides visibility of customer information across the various systems in real-time including Siebel and PeopleSoft.

Integrations were developed as BPEL orchestrations for processes such as Order Fulfillment and Account/Customer Information creation across the various systems. Oracle Customer Data Hub is used to maintain customers for other business units. Oracle ESB is used to receive real-time outbound business events from Oracle Customer Data Hub and synchronize the customer information with PeopleSoft Enterprise (Figure 3). 

Figure 3: ESB combines Event Driven Architecture (EDA) to synchronize customer information between Customer Data Hub and PeopleSoft

So, What's the ROI?

BTW, this all took them 6 months to implement and deploy.  The net results are that Move was able to achieve the following benefits of this CRM Solution.

• 
  
  Real-time integration across these systems
  
  
• 
  
  Process Improvement and improved efficiency
  
  
• 
  
  Visibility of Customer Information across all the systems
  
  
• 
  
  Reduction in cost of ownership on existing applications
  
  
• 
  
  Quicker integration of new products into back office applications

From a long term perspective, this solution allows Move to

• 
  
  Leverage and Protect their existing investments with their .Net applications
  
  
• 
  
  Integrate back office applications from future acquisitions into this CRM Solution efficiently
  
  
• 
  
  Extend and Evolve this CRM Solution and mitigate the need to enhance their existing applications to support business needs

Dave

This is part 2 of an interview I recently did with Rich Seeley for SearchSOA/TechTarget on the relationship between eXtreme Transaction Processing (XTP) and SOA, CEP, BPEL and BAM.  This one focuses mostly on some of the details of Oracle Coherence and how it helps enable the "Not Your MOM's Bus" concept.  Here's an excerpt -

...Of the SOA grid, he says, "It's based on an architecture that combines horizontally scaleable database independent middle tier data caching with intelligent parallelization and affinity of business logic with cached data. What this enables is more efficient models for highly scalable SOA-based applications that can take full advantage of event-driven architectures." It also changes message-oriented middleware (MOM) into something Chappell dubs "not your MOM's enterprise service bus...." [read more]

Dave

I recently did an interview with Rich Seeley for SearchSOA/TechTarget on the relationship between eXtreme Transaction Processing (XTP) and SOA, CEP, BPEL and BAM.  Here's an excerpt -

Oracle mixes extreme transaction processing with SOA
SearchSOA.com
Feb. 12, 2008
Rich Seeley

Extreme transaction processing (XTP) is being added to complex event processing (CEP) in service-oriented architecture (SOA) implementations for the financial services industry, explains David Chappell, vice president and chief technologist for SOA at Oracle Corp. Matching XTP technology is acquired from Tangosol Coherence in March 2007 with the Oracle SOA Suite, companies can sort through massive transaction data streams and flag exceptions that may indicate crimes such as credit card fraud. Alerts in the SOA application can go through business process engineering language (BPEL) processors to business activity monitoring (BAM) dashboards. Chappell, who has more than 20 years of experience in the software industry and has written and lectured on SOA, the enterprise service bus (ESB) and message oriented middleware (MOM), calls XTP the "future for financial services infrastructure." [read more]

Dave

I recently helped Khanderao Kand, one of Oracle's Fusion Middleware lead architects, co-author an article on the current state of OSGi.  Here's an excerpt -

....The Open Services Gateway Initiative (OSGi) Alliance is working to realize the vision of a "universal middleware" that will address issues such as application packaging, versioning, deployment, publication, and discovery.

 In this article we'll examine the need for the kind of container model provided by the OSGi, outline the capabilities it would provide, and discuss its relationship to complementary technologies such as SOA, SCA, and Spring....[read more]

Across financial services firms we have been seeing a new set of business priorities.  There are the "grow the business" priorities that are primarily centered around things like improving customer intimacy and increasing competitive differentiation.  There are also ongoing issues of compliance to regulation and risk mitigation while also keeping an eye towards improving cost efficiency.  The thing that hasn't changed is that IT is viewed as the enabler to overcome these challenges. 

Financial institutions are pushing the envelope and require more processing capability, but without requiring exponential increase in hardware costs.   The growth of extreme transaction processing (XTP) in areas such as fraud detection, risk computation, and stock trade resolution are pushing current solutions such as those based on the mainframe to the limit.  These new applications require a new computing paradigm. 

We're seeing that SOA coupled with XTP  (eXtreme Transaction Processing) is the future for financial services infrastructure as the means to achieve these goals that are often perceived as conflicting.  XTP pertains to a certain class of applications that need to handle large volumes of data that needs to be absorbed, correlated, and acted upon.  Typically the data that is processed by XTP applications comes in the form of large numbers of events, and represents data that changes frequently.  XTP style applications require that transactions and computations occur in memory and do not incur a heavy reliance on conventional back end systems due to the need for extremely fast response rates while still maintaining transactional integrity. 

XTP, btw, is a concept that Gartner's Massimo Pezzini has been writing and speaking about as well.   In a Gartner research note Gartner Research ID # G00151768 he predicts that over the next 4 years the XTPP (eXtreme Transaction Processing Platform) technology category will evolve as the platform vendors work to embrace XTP and implement infrastructure to support it.  In a recent presentation that Massimo did on the subject, he also stated that there are vendors today that are already implementing important pieces of it. 

In addition, the Next Generation Grid Enabled SOA and Not Your MOM's Bus  (a.k.a SOA Grid) subjects that I have been writing about a great deal lately are conceptually something I would like to see lots of commercial vendors and open source projects embrace.  These are variations on the XTPP theme.  But we don't have to wait 4 years for it. The SOA Grid is a new approach to thinking about SOA infrastructure, which provides state-aware, continuous availability for service infrastructure, application data, and processing logic. It's based on architecture that combines horizontally scalable, database-independent, middle-tier data caching with intelligent parallelization and an affinity of business logic with cache data. This enables newer, simpler, and more-efficient models for highly scalable SOA-based applications that take full advantage of event-driven architectures.

SOA Grid based XTP applications can be built today using what Oracle has already in shipping products.  Oracle Coherence coupled with Oracle SOA Suite deliver the industry's first grid-enabled SOA platform for Extreme Transaction Processing.   

Typical use cases for using SOA Grid to build next generation applications are in the areas of fraud detection, trade resolution, and risk management calculations away from the mainframe to low cost commodity hardware.

When talking about this concept with folks I often get the question, or the observation, that Complex Event Processing (CEP) is positioned to solve some of the same thorny problems of tracking business exceptions such as what is required for fraud detection.  Actually XTP is tied to CEP in that they are both about consuming and correlating large amounts of event data and doing something meaningful with it.  However, often the amount of event data that needs to be captured and processed far exceeds the capacity for conventional storage mechanisms.  In the words of some of our customers, sometimes "there just isn't a disk that can spin fast enough" for the volumes of data that needs to be processed.  For these cases, the Oracle Coherence grid becomes the place where the large volumes of event data are collected, stored, examined for out of the ordinary events, etc.  In the financial services area, ATM banking transactions that occur across different geographical areas within a short time window can be considered as out of the ordinary events, or business exceptions.  Those business exceptions can be detected by rules that are coded into the application, or the CEP engine can be used more generically to identify patterns with temporal awareness and raise business exceptions appropriately.

The tie-ins to SOA are many, but the most obvious one is that once the exceptions are detected, there are actions to be taken.  Typical actions to be taken are to send alerts to a BAM dashboard and to invoke a business process or human workflow, such as what can be defined and executed using the Oracle BPEL process manager.   

Dave

There has been a fair amount of chatter lately about defending the value of SOA projects or justifying such projects to the "C-level".  Many of these discussions will point at the business value of doing more with less and achieving IT cost reduction by reducing redundant systems and reuse of services in a SOA.  Also streamlining processes in order to run the business more efficiently is a popular opinion, which I agree with. 

David Linthicum in his blog summarizes a few of the points - http://weblog.infoworld.com/realworldsoa/archives/2008/01/more_on_defendi.html

Delivering business value from SOA is not a new concept.  I have written a fair amount about the topic over the years: http://news.zdnet.com/2100-3513_22-6108621.html and http://www.oreillynet.com/xml/blog/2005/02/extracting_business_value_from.html. 

Today, similar to a few years ago, extracting business value from SOA and using SOA initiatives as a means for aligning IT with the business is critical.  Many of the same points are also relevant to "defending SOA" today.

To sum it up, I have always believed that the real business value of SOA is not so much about IT cost reduction but really about being able to react quickly to change.

• 
  
  Enterprises need the business agility to react to ever-changing business requirements, and continually implement new programs to attract and retain customers.


• 
  
  In support of this, business processes need to be automated, streamlined, refined, and measured.


• 
  
  The underlying IT infrastructure which supports those business processes need to be flexible and capable of adapting to change. Continued measurement of success means that the change needs to happen in real time and results need to be measured in real time.

Overall, the main business value of building a SOA is the ability to react quickly to changing business conditions including changing economic conditions.    During such times executives need to be at the helm more than ever.  Competitors will take drastic measures and you must be able to quickly alert, measure, and react, and measure again in order to quickly assess the effectiveness of your course corrections.  Having flexible business processes, which can be changed without requiring a considerable engineering effort, tied into realtime analysis and metrics made available through things like BAM dashboards can help tremendously in that type of dynamic environment.

I have always been a strong proponent of approaching SOA by identifying projects that can achieve demonstrable ROI within a matter of months, and that this needs to be accompanied by the longer term vision of where you need to get to in order to be successful.  This still holds true regardless of economic conditions or  IT budgets.

One could even argue that if you weren't already building SOA projects with these things in mind, then perhaps they were misguided to begin with.

Dave

Oracle SOA Suite, a component of Oracle Fusion Middleware, has been selected in the InfoWorld's Technology of the Year Award  for "Best Enterprise Service Bus"

These products were selected from hundreds of IT products evaluated by InfoWorld's Test Center over the past year.  According to the award main page InfoWorld's annual Technology of the Year awards "recognize the most capable, most polished, most groundbreaking, and most valuable products on the IT landscape."

One of the cool things about these awards is that they downloaded the product, installed it and actually kicked the tires hands-on.  As a result, I view their evaluation as carrying a lot more weight than those who just look at a bunch of ppt slides and demos and don't try anything themselves.

In the press release InfoWorld's executive editor Doug Dineley is quoted as saying -

"The Oracle SOA Suite pulls together top-notch governance, business rules, security, and business activity monitoring into an ESB package replete with native BPEL orchestration and human workflow integration.  Ease of implementation and affordability make it a standout consideration,"

According to the writeup  that was done earlier in the year by InfoWorld's James Borck they also seemed to like what we had done to bring together the acquired technologies to round out the SOA functionality, as noted in the article - "Oracle SOA Suite culminates strategic merger-and-acquisition execution into a well-integrated product that is at once effective, usable, and highly extensible, making it a sure shot at reducing initial integration costs and benefiting management of your SOA infrastructure going forward."

Dave

This month I have once again teamed up with Dave Berry, Oracle's ESB Product Manager, to publish part 2 of an article exploring the "Next Generation Grid Enabled SOA".  This one is sub-titled "Not Your MOM's Bus".

Abstract: In our previous article we discussed how SOA grids can be used to break the convention of stateless-only services for scalability and high availability (HA) by allowing stateful conversations to occur across multiple service requests, whether between disparate service boundaries or load-balanced groups of cloned service instances.

In this article we will challenge traditional applications of message-oriented middleware (MOM) for achieving high levels of quality of service (QoS) when sharing data between services in an enterprise service bus (ESB). We will further compare and contrast a state-based, in-memory storage and notification model, and investigate the intelligent co-location of processing logic with or near its grid data in large payload scenarios. Finally, we will also explain when to substitute an SOA Grid for existing MOM technologies as driven by the following question: "If you have an SOA grid that can reliably hold application state data and the necessary systems can access it, why continue to utilize conventional messaging?"

Read More..

Cheers,
Dave

• 
  
  Grid computing will grip the attention of enterprise IT leaders, although given the various concepts of hardware grids, compute grids, and data grids, and different approaches taken by vendors, the definition of grid will be as fuzzy as ESB.  This is likely to happen at the end of 2008.
  
  
• 
  
  At least one application in the area of what Gartner calls "eXtreme Transaction Processing" (XTP) will become the poster child for grid computing.  (see Gartner Research ID # G00151768 - Massimo Pezzini).  This "killer app" for grid computing will most likely be in the financial services industry or the travel industry.  Scalable, fault tolerant, grid enabled middle tier caching will be a key component of such applications.
  
  
• 
  
  Event-Driven Architectures (EDA) will finally become a well understood ingredient for achieving realtime insight into business process, business metrics, and business exceptions.  New offerings from platform vendors and startups will begin to feverishly compete in this area.
  
  
• 
  
  Service Component Architecture (SCA) will become the new way for SOA applications to be defined as support from all the major platform vendors (sans Microsoft) will be rolled out.
  
  
• 
  
  New lightweight containers models will become more prevalent, with continuing adoption of Spring, OSGi, and EJB3.
  
  
• 
  
  Web 2.0 applications will start to become more prevalent in enterprises.  Many initial successes of quick and dirty, Enterprise Mashups with attractive looking RIA style applications that include Flex, IM, and threaded discussion groups will be developed for useful business applications such as collaborative workflows.  The associated hype will leave some wondering why they are embarking down the path of long term SOA projects with similar goals in mind.
  
  
• 
  
  Some of those Web 2.0 applications will fail, with an important piece of the user interface just not showing up one day because one of the supporting feeds just isn't there any more.   The guy who mashed it up in the first place doesn't work there anymore, and many will be scrambling to figure out who is supposed to fix it, and who to point the finger at.
  
  
• 
  
  As a result, a renewed focus on governance will emerge.  Industry pundits and thought leaders who had previously been strong proponents of SOA governance will passionately start evangelizing the antithetical concept of building process and governance into the creation of enterprise mashups.
  
  
• 
  
   By end of year it will be clear that an understanding of infrastructure requirements for common problems such as predictable scalability, reliability, security, (*-ilities) will be necessary in order to support any combination of SOA, REST, or Web 2.0 style applications.  However the exact architecture or even the list of requirements in support of such infrastructure will not be well understood or agreed upon.  Such a common understanding will not come to bear until at least 2010.  This will be the new frontier to explore in the coming years.
  
  
• 
  
  Oh yeah, and Dave Chappelle will make a huge comeback this year, but without the $50M contract.  :)

In recent articles and presentations I have been postulating that a concept called "next generation Grid Enabled SOA", a.k.a. "SOA Grid" and "Not your MOM's Bus", combines conventional SOA infrastructure technologies such as BPEL and ESB with middle tier data grid technology to provide a new level of predictable scalability and high availability for SOA based applications. 

I often get asked - "How much better is it? What's the ROI?"

In our case, a key component of the implementation of the SOA grid is the Tangosol Coherence product (acquired early this year, now Oracle Coherence).  A common use for this technology is as a horizontally scalable fault-tolerant shared memory cache, which can act as a staging area for application state.

Recently at Oracle Open World I had the opportunity to meet one of the Coherence customers, a VP of Architecture at a major investment banking firm, who talked about several use cases during a panel discussion in front of a live audience.  The most fascinating use case had to do with a dramatic improvement in an overnight credit risk analysis calculation, which went from 17 hours down to 20 minutes.

The situation is one of regulatory compliance.  The bank must keep enough cash on hand to cover their current credit risk, and every night they need to run a risk calculation to "prove" what this credit risk is.   The challenge with the previous risk calculation application was that even though it utilized compute grid technology, it still took 17 hours to run a risk calculation across a volume of 40,000 trades per day.  They needed to allow for growth of up to 150,000 trades per day.  And by the way the calculation needs to run at 2:00am every night so don't have 17 hours anyhow.

They determined that the problem was that even though they were using compute grid technology they were disk I/O bound.  They rewrote the app to use Coherence and the now the same calculation takes 20 minutes to run!

Why the dramatic improvement?  They weren't beating the crap out of the disk resources anymore.  They were using the Coherence data grid to stage their data, which provides near in-memory access speeds without sacrificing reliability.

Doing some more digging I discovered some more interesting factoids about the new solution -

• 
  
  Not only did they decrease the amount of time, but they also decreased the number of machines from a 200 node grid down to a 60 node grid.
  
  
• 
  
  They are able to handle a 3X increase in exotic instruments trading volume, generating up to $1M profit per deal.
  
  
• 
  
  They increased their ability to run simulations from 4000 to 100,000, which is a 25X improvement.

BTW, to be fair -- In order for the risk calculation to run in 20 minutes there is a prefetch of 60 gigs of data from a variety of sources including database and flat files into the Coherence grid--an operation that takes about 6 hours.  In this case it wasn't an Oracle db and I'm not making any judgements on whether 6 hours is good or bad.  Presumably the prefetch into Coherence can happen once and the 20 minute calculations can occur as many times as needed. 

Even so...whether you look at it as 20 minutes or 6 hours 20 minutes that's a dramatic improvement over 17 hours.  And regardless of how you count the overall time it still using 60 machines instead of 200.

- Dave

Its been a pretty busy fall so far speaking events.  I have been out on the road across the US and Europe spreading the message of "Next Generation Grid Enabled SOA" and "Not Your MOM's Bus" to architects everywhere.  Its being well received as evidenced by this blog entry.

I'm just about to embark on another round the world tour (see below) which will bring me up into December. After which, I plan on hunkering down a bit and getting back to writing the next book.

If you wish to learn more on what all the buzz is about regarding SOA grid, come and see me at one of the upcoming conferences -

Speaking events of recent past -

• 
  
  9/12 - Sun Tech Days Bosto


• 
  
  9/20 ICSOC Vienna - Next Generation SOA Grid


• 
  
  9/28 - Sun Tech Days Milan - SOA Grid Keynote


• 
  
  ESB-Con IV - Next Generation SOA Grid - Listen to Replay - http://www.esbcon4.com/index.asp?type_content=programguide


• 
  
  10/9 - Belgium Java User Group (BeJUG) SOA Architect Forum - SOA Grid  (Just to prove to Oracle Marketing that I was actually there, here is a photo - http://www.flickr.com/photos/98726399@N00/1636703969/in/set-72157602528511134/) A replay of recorded session available soon.


• 
  
  10/15 - IASA Keynote - San Diego â?? SOA Grid - http://www.iasahome.org/web/socal/itarc2007


• 
  
  10/16 Oracle Fusion Middleware Forum Pittsburgh - Next Generation SOA Grid


• 
  
  10/17 Oracle Fusion Middleware Forum - Philadelphia PA - Next Generation SOA Grid


• 
  
  10/30 eBizQ "SOA In Action Webinar/Virtual Conference" - http://events.unisfair.com/rt/expoq06~soafall07/forward.jsp?id=330

Coming up starting this week -

• 
  
  2 sessions at InfoWorld SOA Executive Forum in NYC - http://www.infoworld.com/event/soa/07/november/soa_agenda.html


• 
  
  
  
  • 
    
    11/7 - InfoWorld SOA Exec Forum - Keynote on SOA Grid
    
    
  • 
    
    11/8 - InfoWorld SOA Exec Forum â?? Representing OASIS in a breakout session on SCA (co-presenting with Mike Rowley of BEA and Patrick Leonard of Rogue Wave)
    
  
  
• 
  
  11/12 - 11/16 3 sessions at Oracle Open World, San Francisco - http://www.oracle.com/openworld/2007/index.html
  
  
  
  
  • 
    
    11/11 - Oracle ACE Director Product Briefing - Hilton Hotel 11:30 -  5:00
    
    
  • 
    
    11/12 - S291557  Next-Generation Grid-Enabled SOA  Monday  11/12/07 12:30 PM Moscone South 300
    
    
  • 
    
    11/12 - Monday  11/12/07 03:15 PM - "No Slide Zone" white board chalk-talk presentation - Yerba Buena Theater Yerba Buena Theater 700
    
  
  
• 
  
  11/13 - Sys-Con SOA World - Next Generation SOA Grid -  Not Your MOM's Bus - San Francisco  Grand Hyatt - 9:15 - 10:00AM.
  
  
• 
  
  11/21 SOA India 2007 - Keynote Next Generation SOA - http://www.sda-india.com/conferences/soaindia/agenda.php
  
  
• 
  
  12/4 Gartner Gartner Application Architecture, Development & Integration Summit - Las Vegas -  http://agendabuilder.gartner.com/apn19/webpages/sessionlist.aspx?menuItem=2

See you out in the world!

Dave