Customization Lifecycle Management: Principles and Recommendations Explained
By Richard Bingham-Oracle on Jun 10, 2013
IntroductionSo why is this important? Well it has been reported that heavy customization can add x10 the total ownership costs to any Enterprise Application. This is not just as an initial upfront development expense, but by adding customization it greatly increases the complexity of the ongoing system maintenance. So whilst the desire for tailored application products will forever remain strong, and the urgency is pushed on to developers to deliver them, there needs to be diligence from the outset to ensure the resulting solutions do not end up causing more harm than the benefit they bring.
Customization Lifecycle Management
Of course customization is just normal software development and configuration, so why should it require anything special in terms of lifecycle factors? Well in theory it doesn’t, however because small discrete customization work is often done outside of a formal application development and implementation project it loses many of the great standards, methodologies, and best practices normally associated.
For example, an outline of the very basic development lifecycle illustrated below should still remain effective for all customization work, and without proper attention to the early parts of the process customizations readily incur problems in the latter stages. Obviously the amount of upfront administrative work should reflect the size and significance of the development effort, however even just a one page design specification can bring huge benefits later in the lifecycle, especially compared with nothing documented at all.
By including some of the simple recommendations given below we can ensure customizations are not only effective but are also robust enough to tolerate general system changes. Here are a few of these often disruptive events that destabilize custom code:
- Application product patches And upgrades
- Technology component upgrades
- Applications product and technology reconfigurations
- Alterations to existing customizations
- New customizations are added
- Customizations are retired as replaced by new product features
The following represents four simple recommended capabilities that should be used to support effective customization management. Whilst most are common sense, the additional cost and time involved in following them through frequently mean they are skipped, leading to costly problems later on. We recommend you attempt to put the upfront expense down to delivering quality.
a) Documentation Repository
Often outdated customizations remain in a system because it is not now clear what they do and why they were originally seen as important. As business needs change so should the supporting Enterprise Applications. Access to design documents is useful as an ongoing resource as they explain the purpose, provide detail on the usage boundaries, and offer details on the technical implementation. These documents also help explain any dependencies and relationships to other parts of the system, also very useful information to have at hand.
In addition to the functional and technical designs, the repository may also contain supporting implementation information, such as test scripts and deployment instructions and notes. Again these may be useful resources to access to understand more about the customization.
In addition to designs, any extra documentation that was created for end users, such as training notes or help texts are useful to store with the customization record. Also with considering is access to the customization source code via the repository, as even if debugging requires more skills than available it should be annotated with comments that explain its use and the execution flow.
A single-source-of-truth repository such as this doesn’t have to be especially complex or extensive, just as long as it is easy-to-access and kept updated. A simple wiki or shared network folder would suffice in many cases. Of course it could be part of an overall enterprise IT Change Management System, and indeed most IT Service Management (ITSM) methodologies strongly advocate a change and configuration management implementation. Where these exist their management components should be leveraged for use with Enterprise Applications.
b) Built-In Diagnostics
The saying goes ‘you cannot manage what you cannot measure’ and by definition customizations are essential parts of the Enterprise Application functionality, therefore their performance should be part of the related monitoring.
Again the design should be such that metrics are exposed for such monitoring. Many times Oracle Support has seen additional custom programs that were not designed to run under a now evolved load profile causing generic process bottlenecks and failures that are often hard to pinpoint.
Once again the metrics do not have to be especially complex, and often simply ensuring the custom code uses the technologies in the recommended way is enough to ensure performance data is captured. An example might be to ensure all Java methods and classes are uniquely packaged and named, so that should JVM profilers (like JRockit Mission Control) be used in troubleshooting it is easy to spot custom code execution.
Another important aspect to monitoring is logging. It is essential that all custom code generates uniquely identifiable log output, so that testing and debugging is as easy as possible. The logging process used must also comply with the host application standards, ideally using the exact same process such as AppsLogger. Whilst again this might seem like an overhead that slows development time (compared to using standard output), the benefits are often reaped many times over later in the customization lifecycle, and indeed some sloppy logging implementations can cause issues themselves, such as poor performance.
In addition exception management is important, so that when the custom code is unable to handle the problem the appropriate diagnostic information is available for more in-depth troubleshooting. Again this should be done in the same way as the base application and its technologies to ensure consistency and quality, and for Fusion Applications this is through the Diagnostic Framework.
Finally, sometimes log output and incident records are not enough to troubleshoot problems, and indeed some capability for end users to resolve problems themselves can save huge amounts of time later on. As such in-application diagnostic features are recommended additions, especially where the custom feature has many dependencies on specific setup and transactional data that could be the source of a problem. For Fusion Applications the existing Diagnostic Test Framework provides an out-of-the-box set of hundreds of reports that provide investigative information to support the key features and business processes. This, along with the other diagnostic features, is available from the Help menu, under the Troubleshooting link.
Whether internally or externally developed customizations, make sure every one has a Service Level Agreement (SLA) document associated with them, so if they need support or change then some is on the hook. Even if there is no (costly) maintenance agreement, and your organization retains the ownership, getting the IT department to accept the responsibility and agree to devote resources important to kick start any subsequent problem resolution. Usually when someone knows they are responsible for part of the application its overall management improves substantially. The document repository mentioned above should have details on the basic SLA, maintenance, and ownership of the customization.
So whilst Oracle Applications Support will do their best to help resolve problems in the standard products and features, without the engagement of Advanced Customer Services they do not officially support custom applications.
d) Negative Testing
Whilst everyone will test the customization to ensure it works as intended, a day or two spent testing its boundaries and limitations can again pay dividends, rather than wait to find out what these are in production. Also the graceful failure (or not) in these situations can be reviewed and any improvements to the diagnostics made, to ensure troubleshooting is as effective as possible. A robust and fault-tolerant product feature is much less likely to cost money later on.
In addition to negative testing where you are looking to create a failure, performance testing is also an important activity that must not be overlooked. Simulating production load is relatively easy with modern testing packages like Oracle Application Testing Suite, with script-based programs that emulate concurrent users and load using real application data. These tools also facilitate the automation of testing runs, which is also strongly recommended as it ensures quality through consistency and substantial time-savings from the standard QA process.
A final word of testing is that it also needs to be scheduled and planned as part of accepting system changes, such as the type mentioned at the start of this article. This ensures that the business processes that include customizations remain fully functional after significant events like patches, upgrades, and reconfigurations. So whilst most of the Fusion Applications customization techniques provide upgrade-safe support through the use of Metadata Services (MDS), there still may be times when edge technology or especially invasive alterations are used, therefore testing always makes sense. This is a good example where automated testing adds assurance with little incremental cost.
So whilst the majority of the posts on this forum are concerned with the nuts-and-bolts of developing customizations and extensions on the Fusion Applications platform, its remains prudent to not forget that with great power comes great responsibility. In fact even in cloud scenarios where ongoing IT management is a delegated task it remains important to ensure customization is done in a consistent and high-quality manner. Just because something is easy to do, it doesn’t mean it should be done without proper care and consideration.