SOA Governance@work

I was discussing with a solutions consultant recently the perceived complexity of putting in place an initial governance solution. In following up on Mike van Alst’s recent blog on SOA governance processes, I wondered what tactical processes would be approachable for an organization trying to get a handle on SOA asset management. A couple things came to mind that I’ve heard from customers such as loading assets into their repository to discover dependencies with the benefit being more granular deployment bundles and reduced deployment time. But an even more mundane and common process every customer has some processes in place for is the management of endpoints in different environments. I’ll admit up front that this may not be a simple task but it is one that almost every organization has some practical experience with and thus a good starting point.

The basic use case is fairly straightforward and well known. SOA projects checked into source control systems contain files such as WSDL, XSD and BPEL deployment descriptors. These assets import URL’s pointing to hard coded hostnames and ports specific to a particular environment such as dev.company.com, qa.company.com, stage, prod etc. The environments are typically separated from each other using a firewall or physically isolated network to make sure there isn’t any accidental sharing of data across environments. This is a great example by the way of a business requirement driving a data quality and assurance governance process. Anyway, managing these assets as they are deployed to their respective environments and assuring they only interact with other services, assets or infrastructure servers within that environments is not a trivial task. In the JCA adapter world, JNDI is used to create consistent names across different environments for database, JMS connectors and connection pools but this technique is generally not applicable for web service based assets. A service bus proxy service or service registry providing a normalized UDDI serviceKey can help fill this gap.

For this discussion, we will focus on a specific type of URL associated with the location of the web services runtime implementation also known as the WSDL Service Port SOAP location as shown below from an example at http://www.w3.org/TR/wsdl#_soap-e.

<service name="StockQuoteService"\>
   <documentation>My first service</documentation>
   <port name="StockQuotePort" binding="tns:StockQuoteBinding">
     <soap:address location="http://example.com/stockquote"/>
   </port>
</service>

Assuming the external service has an location for each environment, then logically each business process or composite application referring to it needs to ensure it is only calling the service located in the correct environment such as shown below

Location=http://dev.example.com or
Location=http://qa.example.com

Endpoint management methodologies seem to come under 2 main camps, automated or manual and I think this also matches customer’s size and SOA adoption maturity level. Customers using a manual process might not be as large or sophisticated enough to worry about using an IDE to manually change the project and redeploy in each environment. Larger more mature shops with more specialized IT operations groups and a much larger service portfolio are looking to automate this process at every stage. They likely make use of Apache Ant deployment scripts with substitution variables to handle WSDL and XSD files that include or import dependent assets using hard coded host and port names. Automation scripts do help a lot but usually do not solve the often-requested request for the ability to change the endpoint location without redeploying the business process or application. This type of solution provides more flexibility and overall IT agility for responding to unpredictable changes in the business.

Two approaches for advanced endpoint management are
1) Virtualize the external endpoints using a service bus proxy service
2) Integrated lookup mechanism such as UDDI in the SOA infrastructure or fabric layer.

Both of these solutions externalize the management of endpoints from the SOA application and can support multiple endpoints for load balancing or high availability failover situations. A service bus implementation provides a true virtualization solution making use of content or context based routing and document transformations in the event an endpoint schema changes. It should also allow for managing and changing endpoint locations in an operations management console. The UDDI lookup mechanism is purely for endpoint management and less robust than a service bus virtualization solution but still effectively enables IT operations to change or add endpoint locations. It should achieve this without affecting runtime performance or redeploying the consuming applications such as business processes or composite applications.

Often customers want a single location to manage all service endpoint locations. To meet this requirement, a service bus should allow it’s own endpoints to be externally managed at runtime without redeployment by using service registry’s UDDI serviceKeys. This is a topic for a future discussion but hopefully it has provided fuel for thought on how to break down seemingly complex governance problems into smaller manageable processes to jump start your overall governance strategy.

While discussing organizational governance processes in the last entry, I realized that endpoint management, UDDI and service bus is a topic probably best described as a technical governance process and thus really more about tools than people-processes. In contrast, I want to discuss a very important process that focuses more on the people and communications that should be established before even the first service is conceived. Change Management. Those 2 words alone might be enough to make even the most disciplined IT managers scurry back to the comfort of their Linux prompts. Why is it that organizations implementing SOA often minimize the importance of people processes such as change management in favor technical tasks? Undoubtedly for many reasons but maybe because they are easier than organizing a group of stakeholders in a weekly meeting and can be performed in the dead of night without anyone else's input.

So why is change management so important anyway for SOA and isn't that why IT departments create all of these environments to test, stage, retest and deploy to production? As with many things including coding, the majority of time should not be spent managing the things that go right but instead managing the exception cases or errors when things go wrong. Below is a list of questions you should ask yourself when considering the role of change management.

-Do you have a well documented back out procedure when things go wrong with a production deployment?

-Is there an approval chain of command in place to resolve and decide when a back out would be required?

-Are all of the stakeholders aware of the change schedule and understand the full impact of a failed production upgrade or deployment?

-Has the risk to external organizations to which you may have legal and financial obligations been taken into account?

-Do you regularly evaluate all of the risk(s) associated with changes to production systems?

-Have you taken into account the system loads across different time zones of your users?

-Have end users received adequate notification indicating the schedule and benefits of major changes?

-Do external auditors review the procedures and provide feedback on your processes?

This is not an exhaustive list by any means but simply meant to make you think about what people processes you have in place to address unexpected technology issues. For many, a list like this may in fact be the exact reason that you have not engaged in proper change management because it's just seems to be so hard to gain consensus. Maybe it is best to step back and take a higher view of things. I found the following definition of change management on Wikipedia that I think encapsulates the goals of change management very well.

"Change management is the process during which the changes of a system are implemented in a controlled manner by following a pre-defined framework/model with, to some extent, reasonable modifications".

Now granted this is a pretty high level view but wow, it sure says a lot in that 1 sentence. I especially like the highlighted words such as "controlled manner", "pre-defined framework" and "reasonable modifications". An organization could do a lot worse than posting reminders or mission statements in plane view for all stakeholders to see and read it regularly. I'm sure many have implemented fancy operational dashboards on large screens that show real time monitoring of critical systems for all to see. Why not pepper in some organization goals or definitions such as this that foster communications and other people oriented best practices at the same time?

Ultimately, these sorts of critical governance people-processes should be ingrained into the DNA of everyone in your organization much as code reviews (you do code reviews, right?) and extreme programming practices might be. With workers distributed across different geographies, regular change management meetings may pose a bit more of a logistical problem but really, that is just an excuse given the volume of repositories, collaboration and notification tools at your disposal. While they seem unexciting to many, these sorts of governance processes are going to be major contributors to your future SOA successes. I can only imagine some of the horror stories that readers could share on this topic. Please, don't be shy about posting them here and by all means make them anonymous if you need to, but I suspect these stories would be a good validation of why these people-process best practices are so important.