David Chappell Blog

I'm presenting a keynote at the next International Association os Software Architects (IASA) IT Architect conferences http://www.iasahome.org/web/itarc/nyc/topics#Keynotes on May 22 - 23 in New York City.

I was looking through the agenda and I came across this - Interesting Real-world Architectures and the Handbook of Software Architecture presented by Grady Booch via Second Life

I checked with the conference organizers, and sure enough, Grady is going to be at his home base (which is usually Hawaii), and broadcasting the presentation via 2nd Life, and conference goers will be witnessing his avatar giving the presentation on the big screen.

How cool is that!?  I want some!

Dave

Following a recent posting on a SOA success story http://blogs.oracle.com/davidchappell/2008/04/soa_helps_you_move.html , I got some important feedback http://tech.groups.yahoo.com/group/service-orientated-architecture/message/10092 on how the writeup was a bit slanted towards highlighting the use of the tools and infrastructure such as BPEL and ESB, and didn't talk enough about the services themselves.   The point being made that Service Oriented Architecture is about service orientation, and all too often the folks who live and breath the supporting infrastructure tend to gloss over the part about the services themselves.

I plead guilty.

So as part of my retribution, I would like to provide some nuggets of wisdom on service design that came out of the Move project -

The Prospect to Cash business process includes a number of very large applications (PeopleSoft, Siebel, Oracle Customer Data Hub, and a variety of fulfillment applications).  CSC used the following criteria to identify service candidates:

- Is there similarity in the type of data that needs to be passed to various applications?  If so, then the business logic in each respective application that operates on that data is a candidate for service enablement.  The resulting collection of related services are also candidates for participating in a canonical data model.

- Does the business require part of an application to be used by both internal as well as external customers and partners - what are those parts and can they be broken into smaller atomic functions?

- Here is one that is bound to be a subject of discussion - A key service design consideration is that there is no direct interaction between two services.  They are self sufficient atomic functions that can be used to build orchestrations by the business e.g. - customer data needs to be created in Siebel, PeopleSoft, Oracle Customer Data Hub and the fulfillment system. Each one needs a different component of the data.  This raises a question - is "Create Customer" a service or should it be broken into "Create Customer, Create Address, Create Contact, Create Fulfillment Profile"?  The answer largely depends on whether each of these services is complete in itself, and depends on the nuances of the business.  On first thought, for instance, it may appear that an address without a customer may not be very useful. But if you are someone like Move the rules are different in this case. When they display a house on their website, they don't display the name of the person, they only display the address and the fulfillment profile. So for Move it makes sense to have them as separate services and business can decide how to string them together via BPEL to achieve business results they are intending to achieve.

A similar example can be given about the external customer who needs to consume these services.  If you want to update some information about your house, there is only specific information you can update.   So that would help determine what the service interface would look like.

The overall experience from the team about building the SOA at Move, Inc is that it is very easy to get carried away into JBOWS (Just a Bunch Of Web Services) mentality, without having proper checkpoints in place in order to ensure that the right services are being built for the right reasons.  To do it right really requires proper analysis and a "business case" for each logical function to make it candidate for being Service. If you follow this practice judiciously, you will not end up with JBOWS.

Dave

For the past several years, I have been involved in many healthy discussions centered around the benefits of adopting technology and its supporting tools and infrastructure.  Never once had I ever thought of measuring the benefits in terms of tonnage of hardware or in kilowatts per hour (kW/h).

Until now.

In David Linthicum's recent blog on "Is SOA Green" http://weblog.infoworld.com/realworldsoa/archives/2008/03/is_soa_green.html?source=NLC-SOA&cgd=2008-03-25, he makes the point that "the more efficient a business is, the fewer resources, natural and otherwise it will consume. Thus, the better the business is automated, and business processes optimized, the more efficient it will be. The primary of objective of an SOA is both efficiency and agility through architectural rejuvenation"

I couldn't agree more.  BTW today is Earth Day.  This week is also the Web2.0 expo in San Francisco http://en.oreilly.com/webexsf2008/public/content/home, and the Green Expo in New York http://gogreenexpo.com

In light of all these things coming together, and in the spirit of "going green", I would like to tell you about a concrete example in an exciting account of one of Oracle's customers, Verizon Wireless, who is in the process of going green by rewriting their fraud detection application using SOA, EDA, and Web 2.0, and as a result plan to eliminate 6 tons of hardware from their datacenter, and reduce power consumption by 99.5% from 200 kW/h down to 1100 W/h.

From a software and architecture perspective, this is an exciting story about how Verizon Wireless is building a new Fraud Detection application which uses Event-Driven Architecture (EDA), BPEL process management, a business rules engine together with a Web 2.0 style Flash/Flex UI to build what would be referred to in financial services as a straight-through processing application. 

In the process of doing this, they are dramatically reducing the amount of code that has been written, and the amount of data that needs to be stored locally.  As such, they plan to eliminate 6 E-class Sun boxes using ~192 processors and replace them with a single 8 core processor on a Sun UltraSPARC T1 using the Niagara chip architecture.

The old application is based on J2EE using what was available in the mid to late 90's.  It happened to replicate the entire data warehouse of call detail records for use by the fraud detection application. It also had a lot of procedural custom code that was hand written by 5 FTE over 2 years, some stuff that was ported from Forte to J2EE, and 100s of JSPs feeding (circa 1995) html 3.0 pages with data.

I plan to provide more detail about their service architecture in the future, but here is the basic outline of the new application structure:

• 
  
  Data coming from the switches is analyzed and checked for business exceptions.  Examples of business exceptions include detection of excessive data thresholds, which could be an indicator of a customer using their laptop to run a streaming video website, or the practice of phone cloning to make phone calls using someone else's account, or more ambitious fraud such as masquerading as a third party roaming partner and attempting to charge back to Verizon. (this is the EDA part of the architecture)
  
  
• 
  
  When such an exception is detected, an event is generated and sent to a BPEL process
  
  
• 
  
  The BPEL process invokes a number of services, which includes going out directly to the source of the call detail records to get the information necessary to enrich the event data.  It is then fed into a rules engine to check for violations, make decisions based on policy, and then on to generate more detailed reports.  (This is the SOA part of it)
  
  
• 
  
  The supporting SOA technologies involved include Oracle BPEL Process Manager, Oracle's rules engine, and Oracle AQ.
  
  
• 
  
  The UI interface is RIA/Flex based, and runs mostly in the browser (this is the RIA/Web 2.0 part of it)

So, back to the "going green" part - They are working with a third party energy consultant to produce the final number on energy reduction once they go into production (they are in UAT stage now), but here is their best estimate given what is known today. 

• 
  
  6 TONS of equipment reduced to ~200 pounds (production, qa, dev-test).  I just love those numbers!
  
  
• 
  
  192 processors reduced to a single, 8-core processor (production)
  
  
• 
  
  20+ Terabytes reduced to < 64 Gigabytes storage
  
  
• 
  
  100 kW/h reduced to 540 W/h (production)

I'm no expert at cooling system requirements, but I have heard the rough calculation for power consumption of cooling systems range from 1.3 to 2 X the amount of power consumption of the hardware, so the numbers should really be 200 kW/h reduced to 1080 W/h.  Its still a 99.5% reduction in power usage.

Of course I can't attribute 100% of the impressive results to adopting SOA.  There are lots of things that contribute to the reduction in footprint, such as the massive rewrite and the introduction of more powerful hardware and chip technology (I'll leave that writeup for  SUN to have their own day in the...sun).  However, I can point out some things that are obvious and significant -

It is often said that the best line of code is one that you never have to write.  The new implementation is 0.5% of its original size.  This is directly attributable to using Oracle  BPEL Process Manager and the rules engine instead of their custom code.   They also went from 5 FTE down to 1 consultant who built 40 - 50 BPEL processes over the course of a year.  The result is more flexible and resilient to change as it is declarative and UI driven rather than being hard wired into custom code.  Their presentation tier code also went down to 0.5% of its original size by moving to 1 JSP and 1 SWF for rich UI. 

The other significant contributor to the reduction in footprint is due to the nature of modern SOA standards and supporting infrastructure such as BPEL and ESB, which allows Verizon to extend the reach of their systems so that they don't have to store their "golden record" call detail data locally.  They can get the data remotely and on the fly, and use enrichment services along the way to get the data in the proper form to be make decisions about fraud and overage, and generate detailed reports about the business exceptions that are identified.  This is huge for them in that they no longer have to replicate the data warehouse, but are able to pull call records and other information directly from external heterogeneous systems.

BTW, in David's blog he also points out - "Thus, with gas approaching four bucks a gallon, I made the shift from an SUV that get's 11 MPG to a crossover that gets 30 MPG".   I'm still in the process of shopping around to make that shift...but I have some time since summer is here and my Harley gets 44mpg!

Dave

At this week's RSA Conference 2008, Thomas Kurian laid out Oracle's vision for what is being coined Service-Oriented Security.  Service-Oriented Security is a way of using a service orientation to enable applications with a complete set of the common security capabilities and security processes such as authentication, authorization, user administration, role management, identity virtualization and governance, and entitlement management, as well as audit and control.  Tony Baer also discusses the topic in his blog: http://www.onstrategies.com/CURRENT-NEWS/Oracle-Releases-Role-Manager-Pushes-Service-Oriented-Security-Strategy.html. 

Typically an application may hard wire security capabilities, which makes for a brittle architecture that is not conducive to change.  With Service-Oriented Security, security capabilities and processes are decoupled from applications and more centralized via SOA, making them more available, manageable and consistent across an enterprise.  So in a sense, we are using SOA to enable and strengthen your SOA projects by making them more agile.

To back this up, there have been some key deliverables to date which enable this vision, which cover security issues spanning deployment, governance, administration, and development

• 
  
  GA of Oracle Role Manager, which provides a service for roles-based access control, provisioning and approvals across business applications.
  
  
• 
  
  GA of Oracle Application Access Controls Governor 8.0, which is control monitoring software that provides segregation of duties analysis and enforcement for heterogeneous enterprise application environments
  
  
• 
  
   A controlled beta preview release of Oracle Fine Grained Authorization.  This is software designed to externalize hard-coded authorization policies from heterogeneous enterprise applications, and nicely complements Oracle's Identity and Access Management offering.
  
  
• 
  
  Identity Governance Framework - a multi-vendor standard proposal, spearheaded by Oracle, that provides a service-oriented privacy-aware architecture for developers to access identity data while adhering to usage policies.  Oracle, in conjunction with the Liberty Alliance, has delivered the first open source component of the proposed standard.  

For more information, check out these links -

White paper on Service Oriented Security  -

http://www.oracle.com/technology/products/id_mgmt/pdf/serv_oriented_sec.pdf

Other related announcements -

Oracle Releases Oracle Application Access Controls Governor 8.0

Liberty Alliance and Oracle Team to Advance Identity Governance Framework

Dave

It's always great to hear about successful adoption of SOA, and its particularly exciting when its being used for what you'd think. :)

MOVE, Inc. has recently finished up a really cool SOA project to integrate together and streamline a set of processes a variety of externally facing web sites with internal CRM solutions.  The solution involves the use of BPEL, Web Services, ESB Mediation, application adapters, and canonical data models.

The business benefit to all this from a high level is that MOVE, Inc. was able to improve customer service by improving visibility of customer information across all their systems and improved processes while helping reduce the cost of ownership for its existing applications including Oracle's Siebel CRM and Oracle's PeopleSoft Enterprise Financial Management.

Before I get into how they did it, first a little bit about MOVE, Inc. and the challenges they faced -

Move Inc., formerly Homestore, Inc., is the leader in online real estate.  Through it's network of sites (below) Move, Inc provides information on real estate property listings (homes and apartments), moving, home and garden and home finance.  . 

Move's network of websites:

• 
  
  REALTOR                                www.realtor.com
  
  
• 
  
  Welcome Wagon                    www.welcomewagon.com
  
  
• 
  
  Moving                                    www.moving.com
  
  
• 
  
  Senior Housing Net                www.seniorhousingnet.com
  
  
• 
  
  Home Plans                            www.homeplans.com
  
  
• 
  
  Home Insight                         www.homeinsight.com
  
  
• 
  
  Factory Built Housing             www.factorybuilthousing.com

In the midst of rapid growth, their challenge was they needed to consolidate disparate applications across two of its business units into an enterprise standard CRM solution that could scale to meet its evolving needs and provide a single, accurate view of customer data to improve efficiencies and automate auditing, billing and fulfillment processes. 

Move's business units had custom applications built on the .Net framework for supporting their customer relationship functions and order creation. There were numerous disparate systems and manual touchpoints built to support various business functions for customer relationship management. In an effort to consolidate these applications into an enterprise standard CRM solution and also provide an application which could scale to the growing needs of the business, a CRM solution consisting of Oracle Customer Data Hub, Siebel and PeopleSoft integrated with Oracle Fusion Middleware was proposed to enable the following.

• 
  
  Consolidated Customer Database
  
  
• 
  
  Support for Existing Sales Activities
  
  
• 
  
  Order Capture
  
  
• 
  
  Integration with Fulfillment Systems
  
  
• 
  
  Integration of customer usage from web sites into Siebel
  
  
• 
  
  Integration to PeopleSoft Billing

It sounds like a pretty tall order.

As part of the implementation MOVE worked with CSC (Computer Sciences Corp).  CSC built a SOA that focused on re-usability of services. CSC also ensured that the business processes were at least as efficient if not better than what the current systems offered.  Using standard WSDL interfaces, Oracle BPEL PM and Oracle ESB were used to extend the reach and normalize the integration across Siebel CRM, PeopleSoft Enterprise Financial Management and other systems both internal and external (Figure 1).

Figure 1: Using BPEL, ESB, and Web Services to help implement their SOA, Move extends the visibility of customer data.<o:p></o:p>

How They Got There

The first step in the design process was to model the future business process and identify the functionality that would be enabled through Siebel and the key integration points with the various systems. Move and CSC worked together to define canonical data models for customer, order, listing, usage and billing information and created service contracts for each of the Web services that were used in the integration.  CSC then build new streamlined processes as BPEL orchestrations using Oracle BPEL Process and Oracle Enterprise Service Bus (Figure 2) -

Figure 2: Canonical data models allow processes to be constructed that generically orchestrate a common set of canonical services to without the need for specialized point-to-point data transforms

Service contracts were defined for each of the web services that would be used in the integration.  The customer information was then consolidated from the multiple systems that the two business units had into the Oracle Customer Data Hub. Customer Data Hub also serves as the hub of customer information and provides visibility of customer information across the various systems in real-time including Siebel and PeopleSoft.

Integrations were developed as BPEL orchestrations for processes such as Order Fulfillment and Account/Customer Information creation across the various systems. Oracle Customer Data Hub is used to maintain customers for other business units. Oracle ESB is used to receive real-time outbound business events from Oracle Customer Data Hub and synchronize the customer information with PeopleSoft Enterprise (Figure 3). 

Figure 3: ESB combines Event Driven Architecture (EDA) to synchronize customer information between Customer Data Hub and PeopleSoft

So, What's the ROI?

BTW, this all took them 6 months to implement and deploy.  The net results are that Move was able to achieve the following benefits of this CRM Solution.

• 
  
  Real-time integration across these systems
  
  
• 
  
  Process Improvement and improved efficiency
  
  
• 
  
  Visibility of Customer Information across all the systems
  
  
• 
  
  Reduction in cost of ownership on existing applications
  
  
• 
  
  Quicker integration of new products into back office applications

From a long term perspective, this solution allows Move to

• 
  
  Leverage and Protect their existing investments with their .Net applications
  
  
• 
  
  Integrate back office applications from future acquisitions into this CRM Solution efficiently
  
  
• 
  
  Extend and Evolve this CRM Solution and mitigate the need to enhance their existing applications to support business needs

Dave

This is part 2 of an interview I recently did with Rich Seeley for SearchSOA/TechTarget on the relationship between eXtreme Transaction Processing (XTP) and SOA, CEP, BPEL and BAM.  This one focuses mostly on some of the details of Oracle Coherence and how it helps enable the "Not Your MOM's Bus" concept.  Here's an excerpt -

...Of the SOA grid, he says, "It's based on an architecture that combines horizontally scaleable database independent middle tier data caching with intelligent parallelization and affinity of business logic with cached data. What this enables is more efficient models for highly scalable SOA-based applications that can take full advantage of event-driven architectures." It also changes message-oriented middleware (MOM) into something Chappell dubs "not your MOM's enterprise service bus...." [read more]

Dave

I recently did an interview with Rich Seeley for SearchSOA/TechTarget on the relationship between eXtreme Transaction Processing (XTP) and SOA, CEP, BPEL and BAM.  Here's an excerpt -

Oracle mixes extreme transaction processing with SOA
SearchSOA.com
Feb. 12, 2008
Rich Seeley

Extreme transaction processing (XTP) is being added to complex event processing (CEP) in service-oriented architecture (SOA) implementations for the financial services industry, explains David Chappell, vice president and chief technologist for SOA at Oracle Corp. Matching XTP technology is acquired from Tangosol Coherence in March 2007 with the Oracle SOA Suite, companies can sort through massive transaction data streams and flag exceptions that may indicate crimes such as credit card fraud. Alerts in the SOA application can go through business process engineering language (BPEL) processors to business activity monitoring (BAM) dashboards. Chappell, who has more than 20 years of experience in the software industry and has written and lectured on SOA, the enterprise service bus (ESB) and message oriented middleware (MOM), calls XTP the "future for financial services infrastructure." [read more]

Dave

I recently helped Khanderao Kand, one of Oracle's Fusion Middleware lead architects, co-author an article on the current state of OSGi.  Here's an excerpt -

....The Open Services Gateway Initiative (OSGi) Alliance is working to realize the vision of a "universal middleware" that will address issues such as application packaging, versioning, deployment, publication, and discovery.

 In this article we'll examine the need for the kind of container model provided by the OSGi, outline the capabilities it would provide, and discuss its relationship to complementary technologies such as SOA, SCA, and Spring....[read more]

Across financial services firms we have been seeing a new set of business priorities.  There are the "grow the business" priorities that are primarily centered around things like improving customer intimacy and increasing competitive differentiation.  There are also ongoing issues of compliance to regulation and risk mitigation while also keeping an eye towards improving cost efficiency.  The thing that hasn't changed is that IT is viewed as the enabler to overcome these challenges. 

Financial institutions are pushing the envelope and require more processing capability, but without requiring exponential increase in hardware costs.   The growth of extreme transaction processing (XTP) in areas such as fraud detection, risk computation, and stock trade resolution are pushing current solutions such as those based on the mainframe to the limit.  These new applications require a new computing paradigm. 

We're seeing that SOA coupled with XTP  (eXtreme Transaction Processing) is the future for financial services infrastructure as the means to achieve these goals that are often perceived as conflicting.  XTP pertains to a certain class of applications that need to handle large volumes of data that needs to be absorbed, correlated, and acted upon.  Typically the data that is processed by XTP applications comes in the form of large numbers of events, and represents data that changes frequently.  XTP style applications require that transactions and computations occur in memory and do not incur a heavy reliance on conventional back end systems due to the need for extremely fast response rates while still maintaining transactional integrity. 

XTP, btw, is a concept that Gartner's Massimo Pezzini has been writing and speaking about as well.   In a Gartner research note Gartner Research ID # G00151768 he predicts that over the next 4 years the XTPP (eXtreme Transaction Processing Platform) technology category will evolve as the platform vendors work to embrace XTP and implement infrastructure to support it.  In a recent presentation that Massimo did on the subject, he also stated that there are vendors today that are already implementing important pieces of it. 

In addition, the Next Generation Grid Enabled SOA and Not Your MOM's Bus  (a.k.a SOA Grid) subjects that I have been writing about a great deal lately are conceptually something I would like to see lots of commercial vendors and open source projects embrace.  These are variations on the XTPP theme.  But we don't have to wait 4 years for it. The SOA Grid is a new approach to thinking about SOA infrastructure, which provides state-aware, continuous availability for service infrastructure, application data, and processing logic. It's based on architecture that combines horizontally scalable, database-independent, middle-tier data caching with intelligent parallelization and an affinity of business logic with cache data. This enables newer, simpler, and more-efficient models for highly scalable SOA-based applications that take full advantage of event-driven architectures.

SOA Grid based XTP applications can be built today using what Oracle has already in shipping products.  Oracle Coherence coupled with Oracle SOA Suite deliver the industry's first grid-enabled SOA platform for Extreme Transaction Processing.   

Typical use cases for using SOA Grid to build next generation applications are in the areas of fraud detection, trade resolution, and risk management calculations away from the mainframe to low cost commodity hardware.

When talking about this concept with folks I often get the question, or the observation, that Complex Event Processing (CEP) is positioned to solve some of the same thorny problems of tracking business exceptions such as what is required for fraud detection.  Actually XTP is tied to CEP in that they are both about consuming and correlating large amounts of event data and doing something meaningful with it.  However, often the amount of event data that needs to be captured and processed far exceeds the capacity for conventional storage mechanisms.  In the words of some of our customers, sometimes "there just isn't a disk that can spin fast enough" for the volumes of data that needs to be processed.  For these cases, the Oracle Coherence grid becomes the place where the large volumes of event data are collected, stored, examined for out of the ordinary events, etc.  In the financial services area, ATM banking transactions that occur across different geographical areas within a short time window can be considered as out of the ordinary events, or business exceptions.  Those business exceptions can be detected by rules that are coded into the application, or the CEP engine can be used more generically to identify patterns with temporal awareness and raise business exceptions appropriately.

The tie-ins to SOA are many, but the most obvious one is that once the exceptions are detected, there are actions to be taken.  Typical actions to be taken are to send alerts to a BAM dashboard and to invoke a business process or human workflow, such as what can be defined and executed using the Oracle BPEL process manager.   

Dave

There has been a fair amount of chatter lately about defending the value of SOA projects or justifying such projects to the "C-level".  Many of these discussions will point at the business value of doing more with less and achieving IT cost reduction by reducing redundant systems and reuse of services in a SOA.  Also streamlining processes in order to run the business more efficiently is a popular opinion, which I agree with. 

David Linthicum in his blog summarizes a few of the points - http://weblog.infoworld.com/realworldsoa/archives/2008/01/more_on_defendi.html

Delivering business value from SOA is not a new concept.  I have written a fair amount about the topic over the years: http://news.zdnet.com/2100-3513_22-6108621.html and http://www.oreillynet.com/xml/blog/2005/02/extracting_business_value_from.html. 

Today, similar to a few years ago, extracting business value from SOA and using SOA initiatives as a means for aligning IT with the business is critical.  Many of the same points are also relevant to "defending SOA" today.

To sum it up, I have always believed that the real business value of SOA is not so much about IT cost reduction but really about being able to react quickly to change.

• 
  
  Enterprises need the business agility to react to ever-changing business requirements, and continually implement new programs to attract and retain customers.


• 
  
  In support of this, business processes need to be automated, streamlined, refined, and measured.


• 
  
  The underlying IT infrastructure which supports those business processes need to be flexible and capable of adapting to change. Continued measurement of success means that the change needs to happen in real time and results need to be measured in real time.

Overall, the main business value of building a SOA is the ability to react quickly to changing business conditions including changing economic conditions.    During such times executives need to be at the helm more than ever.  Competitors will take drastic measures and you must be able to quickly alert, measure, and react, and measure again in order to quickly assess the effectiveness of your course corrections.  Having flexible business processes, which can be changed without requiring a considerable engineering effort, tied into realtime analysis and metrics made available through things like BAM dashboards can help tremendously in that type of dynamic environment.

I have always been a strong proponent of approaching SOA by identifying projects that can achieve demonstrable ROI within a matter of months, and that this needs to be accompanied by the longer term vision of where you need to get to in order to be successful.  This still holds true regardless of economic conditions or  IT budgets.

One could even argue that if you weren't already building SOA projects with these things in mind, then perhaps they were misguided to begin with.

Dave