SOA Governance@work

Here is an interesting article by Bob Rhubart that talks about collaboration across stakeholders as key for SOA success.

"A service-oriented architecture (SOA) is a busy place, a complex clockwork of interconnected, interdependent, and widely distributed moving parts, including those that enjoy a nice cup of coffee in the morning. As such, SOA demands extraordinary collaboration across a broad spectrum of stakeholders. Getting those stakeholders to behave as a cohesive community can be about as easy as teaching bears to tap dance, but that cohesion is an important objective of SOA governance....."

Link to the article

Oracle is hosting 4 enterprise customers to discuss real-world SOA Governance best practices at Oracle Open World on Monday, 4 PM.

This session outlines the best practices for Architects to achieve maximum returns from SOA Governance initiatives. Hear from industry experts and Oracle customers such as Todd Biske, Mike Knecht, Goutham Nelluthla and Jules De Ruijter as they share their extensive experience and expertise in the area of SOA Governance.

Session time: 4 PM, Monday, October 12, 2009

Venue: Hilton Hotel, Golden Gate 3

Check out other SOA Governance sessions at Oracle Open World here

Streamline your SOA: Risks, Strategies and Payoffs

With growing adoption of SOA taking place across your enterprise, the time is right to replicate some of the early successes on a larger scale. If you want your organization to reach its destination on time and under budget, you need to know exactly where it is headed - while avoiding roadblocks and detours along the way.

Learn how you can chart your roadmap for SOA success in our upcoming live Webcasts.

In addition, you will see how companies are successfully using Oracle SOA technology to:

• Maximize returns on existing investments


• Boost customer satisfaction


• Improve application performance and customer SLAs


• Reduce downtime for new deployments and upgrades


• Shorten your project timelines, while governing your costs and success

Register now for the session that best fits your schedule.

While discussing organizational governance processes in the last entry, I realized that endpoint management, UDDI and service bus is a topic probably best described as a technical governance process and thus really more about tools than people-processes. In contrast, I want to discuss a very important process that focuses more on the people and communications that should be established before even the first service is conceived. Change Management. Those 2 words alone might be enough to make even the most disciplined IT managers scurry back to the comfort of their Linux prompts. Why is it that organizations implementing SOA often minimize the importance of people processes such as change management in favor technical tasks? Undoubtedly for many reasons but maybe because they are easier than organizing a group of stakeholders in a weekly meeting and can be performed in the dead of night without anyone else's input.

So why is change management so important anyway for SOA and isn't that why IT departments create all of these environments to test, stage, retest and deploy to production? As with many things including coding, the majority of time should not be spent managing the things that go right but instead managing the exception cases or errors when things go wrong. Below is a list of questions you should ask yourself when considering the role of change management.

-Do you have a well documented back out procedure when things go wrong with a production deployment?

-Is there an approval chain of command in place to resolve and decide when a back out would be required?

-Are all of the stakeholders aware of the change schedule and understand the full impact of a failed production upgrade or deployment?

-Has the risk to external organizations to which you may have legal and financial obligations been taken into account?

-Do you regularly evaluate all of the risk(s) associated with changes to production systems?

-Have you taken into account the system loads across different time zones of your users?

-Have end users received adequate notification indicating the schedule and benefits of major changes?

-Do external auditors review the procedures and provide feedback on your processes?

This is not an exhaustive list by any means but simply meant to make you think about what people processes you have in place to address unexpected technology issues. For many, a list like this may in fact be the exact reason that you have not engaged in proper change management because it's just seems to be so hard to gain consensus. Maybe it is best to step back and take a higher view of things. I found the following definition of change management on Wikipedia that I think encapsulates the goals of change management very well.

"Change management is the process during which the changes of a system are implemented in a controlled manner by following a pre-defined framework/model with, to some extent, reasonable modifications".

Now granted this is a pretty high level view but wow, it sure says a lot in that 1 sentence. I especially like the highlighted words such as "controlled manner", "pre-defined framework" and "reasonable modifications". An organization could do a lot worse than posting reminders or mission statements in plane view for all stakeholders to see and read it regularly. I'm sure many have implemented fancy operational dashboards on large screens that show real time monitoring of critical systems for all to see. Why not pepper in some organization goals or definitions such as this that foster communications and other people oriented best practices at the same time?

Ultimately, these sorts of critical governance people-processes should be ingrained into the DNA of everyone in your organization much as code reviews (you do code reviews, right?) and extreme programming practices might be. With workers distributed across different geographies, regular change management meetings may pose a bit more of a logistical problem but really, that is just an excuse given the volume of repositories, collaboration and notification tools at your disposal. While they seem unexciting to many, these sorts of governance processes are going to be major contributors to your future SOA successes. I can only imagine some of the horror stories that readers could share on this topic. Please, don't be shy about posting them here and by all means make them anonymous if you need to, but I suspect these stories would be a good validation of why these people-process best practices are so important.

I was discussing with a solutions consultant recently the perceived complexity of putting in place an initial governance solution. In following up on Mike van Alst’s recent blog on SOA governance processes, I wondered what tactical processes would be approachable for an organization trying to get a handle on SOA asset management. A couple things came to mind that I’ve heard from customers such as loading assets into their repository to discover dependencies with the benefit being more granular deployment bundles and reduced deployment time. But an even more mundane and common process every customer has some processes in place for is the management of endpoints in different environments. I’ll admit up front that this may not be a simple task but it is one that almost every organization has some practical experience with and thus a good starting point.

The basic use case is fairly straightforward and well known. SOA projects checked into source control systems contain files such as WSDL, XSD and BPEL deployment descriptors. These assets import URL’s pointing to hard coded hostnames and ports specific to a particular environment such as dev.company.com, qa.company.com, stage, prod etc. The environments are typically separated from each other using a firewall or physically isolated network to make sure there isn’t any accidental sharing of data across environments. This is a great example by the way of a business requirement driving a data quality and assurance governance process. Anyway, managing these assets as they are deployed to their respective environments and assuring they only interact with other services, assets or infrastructure servers within that environments is not a trivial task. In the JCA adapter world, JNDI is used to create consistent names across different environments for database, JMS connectors and connection pools but this technique is generally not applicable for web service based assets. A service bus proxy service or service registry providing a normalized UDDI serviceKey can help fill this gap.

For this discussion, we will focus on a specific type of URL associated with the location of the web services runtime implementation also known as the WSDL Service Port SOAP location as shown below from an example at http://www.w3.org/TR/wsdl#_soap-e.

<service name="StockQuoteService"\>
   <documentation>My first service</documentation>
   <port name="StockQuotePort" binding="tns:StockQuoteBinding">
     <soap:address location="http://example.com/stockquote"/>
   </port>
</service>

Assuming the external service has an location for each environment, then logically each business process or composite application referring to it needs to ensure it is only calling the service located in the correct environment such as shown below

Location=http://dev.example.com or
Location=http://qa.example.com

Endpoint management methodologies seem to come under 2 main camps, automated or manual and I think this also matches customer’s size and SOA adoption maturity level. Customers using a manual process might not be as large or sophisticated enough to worry about using an IDE to manually change the project and redeploy in each environment. Larger more mature shops with more specialized IT operations groups and a much larger service portfolio are looking to automate this process at every stage. They likely make use of Apache Ant deployment scripts with substitution variables to handle WSDL and XSD files that include or import dependent assets using hard coded host and port names. Automation scripts do help a lot but usually do not solve the often-requested request for the ability to change the endpoint location without redeploying the business process or application. This type of solution provides more flexibility and overall IT agility for responding to unpredictable changes in the business.

Two approaches for advanced endpoint management are
1) Virtualize the external endpoints using a service bus proxy service
2) Integrated lookup mechanism such as UDDI in the SOA infrastructure or fabric layer.

Both of these solutions externalize the management of endpoints from the SOA application and can support multiple endpoints for load balancing or high availability failover situations. A service bus implementation provides a true virtualization solution making use of content or context based routing and document transformations in the event an endpoint schema changes. It should also allow for managing and changing endpoint locations in an operations management console. The UDDI lookup mechanism is purely for endpoint management and less robust than a service bus virtualization solution but still effectively enables IT operations to change or add endpoint locations. It should achieve this without affecting runtime performance or redeploying the consuming applications such as business processes or composite applications.

Often customers want a single location to manage all service endpoint locations. To meet this requirement, a service bus should allow it’s own endpoints to be externally managed at runtime without redeployment by using service registry’s UDDI serviceKeys. This is a topic for a future discussion but hopefully it has provided fuel for thought on how to break down seemingly complex governance problems into smaller manageable processes to jump start your overall governance strategy.

Unrealized Savings Potential from Existing (SOA) Initiatives.

It is no secret that a large percentage of Oracle Fusion Middleware customers are using Oracle SOA technologies for integrating enterprise applications such as Oracle CRM or ERP. Many may view these projects as, (or maybe replacing), EAI like point-to-point integrations of legacy systems to satisfy immediate business needs while others have charted out a long-term SOA vision for service reuse. This illustrates some key differences in various customers SOA adoption patterns which became apparent to me during a recent discussion with a customer who is long term user of Oracle BPEL PM for integrating various enterprise back end systems. They were very familiar with the newer SOA tooling but when I quizzed them on how they managed their service portfolio to achieve shared reuse, I got blank stares which indicated to me they were not as far along the SOA maturity curve as what I would have expected for a long term Oracle BPEL PM customer. This lead to a detailed discussion about the many benefits and challenges of adopting SOA and shared services reuse strategy.

The benefits of this approach while evident to many were not clear to this customer and I suspect many others as well, especially those in the depths of complex enterprise system integration projects. Significant savings can and should be realized when adding additional consumers to that CRM endpoint. What SOA tooling delivers here is that classic ESB many to one mediation consumer pattern that allows multiple consumers to access the same endpoint by (re) using a SOAP or REST based proxy service. Beyond the benefits of de coupling the service endpoint provider from the consumer for ease of endpoint change management, this proxy service can also be reused by portals, enterprise or legacy systems for future integrations with other business units. In a simplistic view of savings, it is easy to envision a 50% savings on future integrations of that endpoint minus whatever overhead is associated with governing and extending the service to meet additional consumers needs. This infrastructure overhead should not be underestimated and may take time but is key for reaching the critical goal of building trust with new prospective consumers of these services. So what does this all have to do with governance you may ask.

One of the main values of SOA Governance is to deliver savings by enabling this reuse of existing services and underlying assets, by creating and managing a shared service and asset portfolio. The first goal is to deliver visibility of these services and assets since logically, if you don’t know what is available, then you cannot even begin to reuse it. Once there is adequate visibility, there needs to be some controls put in place to monitor and manage the usage of these services to guarantee meeting the requirements of the various business organizations using them. A hidden benefit of this approach is to promote communications among these various business stakeholders but that is for another topic. Good governance tooling can promote this Visibility, Control and Monitoring necessary to manage a shared service portfolio. This is loosely referred to closed loop governance. The 2 main components for implementing closed loop governance are:

1) Enterprise Repository which provides the visibility and is the single source of truth for consumption of assets such as WSDL, XML schema’s, BPEL processes, web service endpoints etc…

2) Enterprise Monitoring Manager enables IT staff and prospective consumers the ability to see how a service is performing and key to building that consumer trust we mentioned previously.

The Repository and Monitoring tools must be integrated such that when consumers browse the repository, they can see historical information to help them decide if this service will meet their requirements for performance, security and availability.

While all of this may seem like SOA Governance 101 for service reuse, it really is just meant to bring attention to the fact that many users of SOA Technologies especially those doing point to point like integrations with SOA tooling may not be thinking in terms of reusable service portfolios. Even though they have built some services that are potentially sharable, by adding some initial SOA reuse methodologies and governance tooling, organizations can head down the path to realize paybacks on their existing enterprise integration investments. As it turns out, it is not a stretch to draw the modern corollary that reuse is a very ecological, even green activity that is far more useful than it’s older sibling, recycling which requires additional investment for repurposing existing materials. More on this later.

While technology leaders in SOA Governance hammer out ways to get policy enforcement tools to interoperate (which are still long in coming), what's an architect to do in the meantime?

Why, resort to pre-SOA tactics, of course.

Remember how we used to solve silo problems in the past? We'd bridge them with processes that span silos, and involve stakeholders in each silo. Lest we forget our roots, processes and people are still the heart of governance.

By taking these lessons to heart, we can immediately and dramatically simplify policy management. With the availability of robust commercial off-the-shelf event management and notification options, we can even do it better. No waiting around for standards bodies and vendor integrations.

Here are some simple steps you can take:

1) Select a human administrator in each of the SOA policy domains you would like to bridge. (Some examples that come to mind are authorization or entitlements, service level management, and privacy.)

2) Each time a consumer group requests access to a service from the service provider, have both parties negotiate and document the terms of use in their native (human) language.

3) Auto-notify policy administrators when the new terms of use are established, and include a request that policy administrators create policies in their existing tools to enforce the terms.

4) Auto-notify again once the policy work is complete and the terms of use are ready to be enforced.

To be sure, progress is being made on centralized policy management approaches. But it is unlikely that de jure standards will keep up with the kinds of policies we need, or that vendors will change their policy enforcement products quickly. So an open, federated model of policy enforcement is preferable right now to homogeneous policy management strategies.

By focusing on people and process instead of technology, we can have policy management now that is efficient, heterogeneous, agile, repeatable, easy to delegate and even provides the traceability needed for regulated environments.

Is it all automatic, untouched by human hands? No, but that may be just a pipedream.

Oracle Enterprise Repository (OER) 10gR3 is now available.  This is the first new release of OER (formally known as AquaLogic Enterprise Repository) since the BEA acquisition.  This is a minor point release, but there are some of the new features that are noteworthy.

• Enhancements to standards-based artifact harvesting
• Synchronization between OER and UDDI registries are now controlled via workflows
• A worklist of assigned assets now appear in 'My Stuff' within OER console
• Additional platform support and certifications
• Use of BPM Studio for workflow modification

Asset Harvesting

OER 10gR3 provides some enhancements to asset harvesting, which expands the number of ways in which assets can be synchronized with OER.  Traditionally, assets could be automatically collected through plugins in VS .NET and Eclipse, OER API calls, introspection of WSDL artifacts, synchronization with UDDI, or of course, manually.  OER 10gR3 has enhanced the introspection capabilities to harvest individual files or entire directories of BPEL, WSDL, XSD, and XSLT artifacts.  This can be accomplished via command line or via ant script provided with OER (see examples here).  The ant script provided can be incorporated into any build process, including the build process within JDeveloper.  The result of the harvesting functionality is the creation of assets, relationships, and their associated metadata from the artifacts collected. 

OER/UDDI Synchronization

Synchronization between OER and UDDI is now accomplished utilizing the embedded BPM workflow engine under the covers.  The workflows are based on OER BPM processes, which can be easily modified out of the box to match your organization's needs, or you can simply utilize the workflow OER provides for you.  These workflows can be configured to be triggered in different ways.  Common triggers include repository events such as lifecycle stage or registration status change.  However, these can also be configured for a time-based trigger. 

Worklist Notifications

Processing of assets has been streamlined in OER 10gR3 to include a worklist of assigned tasks within the 'My Stuff' tab of the OER console.  Tasks that are assigned to the user logged in, such as approval of registration status or other 'gates' in the lifecycle process that require human interaction, can be easily accessed for rapid completion.

Additional Platform Support and Certifications

As part of OER 10gR3, certifications have been made for additional platform support.  Check here for a list of supported configurations. 

Workflow Customization Using BPM Studio

One of the best items of this release is the ability to use the Oracle BPM Studio (separately downloaded) to customize the BPM workflows under the covers of OER without having to purchase Oracle BPM separately.  This provides you with the most flexibility for customizing the automation built into OER to accommodate whatever lifecycle methodology you may be using to apply governance as rigidly or flexibly as suits your organization.

You can download OER 10gR3 here.

ebizQ analyst Dennis Byron published an article not too long ago asking

What is business process management (BPM) software's role in IT and SOA governance?

In his article, Dennis quotes Forrester's Larry Fulton as stating

IT and SOA design-time governance solutions provide more than just storage and cataloging of service information; they also automate the process of service life-cycle management (SLM). Automating the process of service life-cycle management is the job of technologies like SOA design-time repositories.

Dennis goes on to make the argument that BPM could be used for this kind of automation

because the combination of an explosion in unstructured data and the coming exponential growth in services...means IT could lose control of its mission

The rest of the article goes on to cover various BPM vendor products and how they could be used.  However, Dennis never really fully answers the question he stated up front on what BPM's role in SOA Governance was other than to state it could be important if BPM is important to the overall business strategy.  I don't disagree with Dennis, but I think he's approaching this from a BPM angle, not a SOA governance angle.

Technology offerings aligned with SOA governance, such as those referenced by Larry Fulton, are there for the purpose of automating, and providing visibility into, the governance process.  Governance doesn't happen simply by harvesting assets from the SOA environment and providing visibility.  It's what you do to those assets once you have that visibility that really defines governance.  Effective governance is essentially the application of business and IT policies and procedures to influence behavior and outcome.  It's for this reason that I think BPM can, and does, play an essential role in SOA governance as it provides a means to structure the governance process and automate enforcement of policies and procedures to ensure business/IT alignment and compliance.

What I find surprising in Dennis's article is that Oracle is not mentioned alongside the other BPM vendors.  Why I find this surprising is not because Oracle has a BPM offering but because the Oracle SOA Governance solution has BPM embedded.  Oracle uses a lightweight version of Oracle BPM workflow under the covers for automation of the lifecycle, from planning through retirement.  These workflows can either be configured and used as is, or you are provided with all the tooling necessary to modify those BPM processes to fit your methodology and needs.   

So when it comes to the question of what BPM's role in SOA Governance is, Oracle believes it's so essential that they've embedded it within the governance solution offering itself.  BPM doesn't need to be strategic to the rest of the enterprise to be applied to the automation of your governance processes, but governance is a poster child for how BPM can be properly leveraged for internal processes.

The rest of Dennis's article can be found here.

A post on Joe McKendrick's SOA in Action Blog includes a transcript of Gartner analyst Frank Kenney's remarks during a keynote address at last month's virtual SOA Governance conference presented by ebizQ.  Among Kenney's observations: 

• "If you are moving towards an SOA initiative, if you are embarking on an SOA project, that project will fail without governance."
  
• "SOA...is evolving from a technical architecture to a governing framework. It is not so much about the applications that you build, or the services you compose, or the processes that you orchestrate and coordinate...It’s really about understanding the lifecycle [of] the artifacts that make up your service-oriented architecture."
  
• "Runtime and design time [don't] mean anything when it comes to SOA. It's not runtime or design time, it's all the time."
  
• "At the center of any good SOA governance technology is going to be a registry repository."

According to the transcript, Kenney pointed out that the lifecycle of a SOA artifact includes overlapping creation, usage, deployment, and governance cycles. He advised against focusing on what he described as the "artificial distinction between design time and runtime," and stressed the importance of focusing on the overlapping lifecycles, and on how those lifecycles affect each other.

Little of what Kenney talked about is all that new, especially given where we are in the evolution of SOA. But the ideas he presented are nevertheless critical and worth repeating until they are fully absorbed by every stakeholder in any SOA initiative.

Read the entire post: Transcript: No SOA Governance Strategy? No Problem- Prepare For Failure!