Friday Feb 08, 2013

Configuration Migration Assistant Part 8 - Differences from ConfigLab

For customers on Oracle Utilities Customer Care And Billing as well as Oracle Enterprise Taxation and Policy Management who use ConfigLab will ask what the Configuration Migration Assistant offers over ConfigLab. Here is the summary:

  • Simpler Terminology - One of the issues with users of ConfigLab was the confusing terminology used. You had supporting and supported environments and then Compare Source and Synchronize processes. In Configuration Migration Assistant we removed all that terminology and use the easier understandable Source and Target. There is no equivalent of Compare Source and Synchronize as it just export and import.
  • Minimal Setup Required - One of the other issues with ConfigLab is that you explicitly define the data migration relationships between environments and that required technical setup, called Environment registration. This process created db links between environments and needed to be maintained by the DBA across upgrades or patches. For the Configuration Migration Assistant there is no such technical setup. The only setup is to define the export and import directories on the Master Configuration record in each environment. This is retained across patches and upgrades automatically.
  • Reusable Objects - In ConfigLab, the relationships and criteria where stored in an object called DB Process and each relationship and criteria was in a DB Instruction. The issue is that structure is not reusable as you had to duplicate it to make changes. In Configuration Migration Assistant different objects are used to store relationships (Migration Plans) and criteria (Migration Requests). This allows for reuse and minimizes the need for duplications.
  • Supports data types used in product - Since the introduction of ConfigLab, the Oracle Utilities Application Framework introduced support for CLOB and XML data types but ConfigLab had issues detecting changes within these data types. In Configuration Migration Assistant you can not only detect changes in nodes within CLOB and XML data types you can now specify relationships using XPATH to model relationships within these data types.
  • Export files can be checked into version management software - One of the requirements from our partners was the ability to take the export files generated by Configuration Migration Assistant and check them into a code repository with any java code and/or Bundles. ConfigLab did not feature this facility.
  • Both Changed and Unchanged data is displayed - One of the key feedback from customers using ConfigLab was that just as important as telling them what has changed, it was important to tell them what has NOT been changed as a result of the import and compare process. This is a feature in Configuration Migration Assistant. This feature was only implied in ConfigLab.
  • Export files can be reused across environments - In line with other migration tools, it was important to be able to export once but use many times. For example, the export can represent a point in time configuration that can be migrated across environments regardless of the target state. ConfigLab could not share migrations as it was always considered live migrations.
  • Export files can be used to restore an environment - One of the features of the Configuration Migration Assistant is the ability to import data that was previously exported from the same environment. This acts as a restore like facility. ConfigLab did not feature this facility (well at least not after multiple steps and complex configuration).
  • Objects with system generated keys not supported - ConfigLab was available to migrate any data from environment to environment including master and transaction data. Feedback from our customers suggested it was not optimal for master/transaction data even with lower volumes. This is partly issues around system generated keys in the target environment. Configuration Migration Assistant is restricted to objects without system generated keys to avoid this issue. This means it cannot be used for master or transaction data. Oracle uses another product called Test Data Management Pack which is part of the Oracle Enterprise Management family of products for this purpose. We will be featuring a discussion of that product in a future posting and whitepaper.
  • Delete transactions not supported -  One of the issues we would get feedback on from ConfigLab customers was delete transactions. If the record did not exist in the Supported environment but in the Supporting environment then a delete statement would be generated. The issue is that you cannot use data that was used for any transaction due to data integrity rules. This would cause ConfigLab to generate errors when attempting to apply changes causing concerns from customers. It is in fact, expected behavior as ConfigLab was protecting data integrity and the errors indicated that. This still concerned customers so to avoid this issue, we do not support delete transactions in Configuration Migration Assistant. These should be manually managed.
  • Same monitor process used for import/export and apply changes - One of the issues with ConfigLab was the sheer number of batch jobs that were available. We simplified this by providing a single configurable monitor process to manage the objects over the lifecycle of the objects. This reduces maintenance overhead and maximizes flexibility. 

In the long run, the Configuration Migration Assistant will replace ConfigLab.

For more information about this aspect of the Configuration Migration Assistant and other aspects refer to the Configuration Migration Assistant Overview (Doc Id: 1506830.1) whitepaper available from My Oracle Support.

Thursday Feb 07, 2013

Configuration Migration Assistant Part 7 - Migration Import

After data is exported it can be imported into target environments. The original source environment can also be a target environment if you wish to use export to restore the data (Note: Deletes are not supported in Configuration Migration Assistant).

Note: Before importing data ensure that the Master Configuration Record for the Migration Assistant Configuration has been setup.

Migration Assistant Configuration

To execute an import, access the target environment and perform the following tasks:

  • Register the Migration Data Set Imports to import exported data into the environment.
  • Execute F1-MGDPR on the target environment to import the data to create a Migration Data Set outlining the changes and the data that is unchanged.
  • Optionally, Approve or Reject individual changes
  • After the approval process has been completed for the import, the Migration Data Set must be marked as Ready To Apply.
  • Execute F1-MGDPR on the target environment to apply the approved changes.

The process is illustrated as follows:

Migration Import process

The first step in the import process is to login to the target environment and create a Migration Data Set Import record using the Administration Menu --> M --> Migration Data Set Import menu option. Fill in the following information:

  • Import Directory - The import directory taken from the Master Configuration Record for the Migration Assistant Configuration.
  • File Name - Name of the export file in the Import Directory (do not include file suffix in name).
  • File Suffix - The File Suffix taken from the Master Configuration Record for the Migration Assistant Configuration.
  • Default Status For Add - The status value for any detected change that results in an Add. The valid values are:
    • Approved - Add is pre-approved.
    • Rejected - Add is pre-rejected
    • Needs Approval -  Add requires a manual approval or rejected
  • Default Status for Change - The status value for any detected change that results in an Update. The valid values are:
    • Approved - Change is pre-approved.
    • Rejected - Change is pre-rejected
    • Needs Approval -  Change requires a manual approval or rejected
For example: 

Migration Data Set Import

Once the Migration Data Set Import has been saved, as with the Migration Data Set Export it registers the intent to import, it does not perform the import at this time. To import the data, execute the F1-MGDPR batch process to execute the import and compare process. Note: Remember the L2 cache must be switched off for the threadpool used for F1-MGDPR. This will generate a Migration Data Set which will contain all the changes and also the data that has not been changed. To view the status of the Migration Data Set, open the Migration Data Set Import search dialog. For example:

Migration Data Set Import Search

Clicking on the Migration Data Set will list all the changes and unchanged data with the relevant defaults used on the Migration Data Set Import record. For example:

Migration Data Set

This will list the Migration Objects in the Migration Data Set. Note: If you want to filter out the Unchanged data use the filters on the zone and use Save Preferences to save your preferences. When selecting a change the SQL for the change is displayed. For example:

Example Change

At this point the change can be Approved or Rejected. Note: The VERSION field is included in each change to ensure that if the record changes between the approval process and the apply changes process then this change will be rejected and the import must be re-performed. This protects integrity of the objects compared.

Once all the changes have been approved the Migration Data Set Import should be marked Ready to Apply. For example:

Ready To Apply

To apply the changes execute F1-MGDPR batch process which will apply Approved changes for any Migration Data Set's that are Ready To Apply. Note: Remember the L2 cache must be switched off for the threadpool used for F1-MGDPR. To check the status of the Migration Data Set Import check the status on the Migration Data Set Import Query screen. If ALL the changes have been applied then the status will be Applied and if any change failed due to a business rule will be Applied With Errors. For example:

Migration Data Set Import Search

The state transition for an Migration Data Set Import is as follows:

For the Migration Data Set Import:

  • Initially Migration Data Set Import is set to Pending state. Before the F1-MGDPR process is executed the import may be manually Cancelled.
  • At the start of the process the Migration Data Set is created and the Migration Data Set Import is marked Ready to Compare.
  • During the compare process the Migration Data Set Import is marked as Comparing.
  • If the compare process encounters an error the Migration Data Set Import is marked in Error and import stopped.
  • If there are NO additions or changes in the export in the target environment, then the Migration Data Set Import is marked as Unchanged, otherwise is marked as Awaiting Approval.
  • After the manual approval process the Migration Data Set Import is marked as Ready to Apply.  Before the F1-MGDPR process is executed the import may be manually set to Cancel Apply.
  • During the Apply Changes process, using the F1-MGDPR batch process, the Migration Data Set Import is marked as Applying.
  • After the Apply Changes process, if all changes are applied successfully then the Migration Data Set Import set to Applied. If any errors occurred it is marked to Applied With Errors.

Migration Data Set Import state transition

For more information about this aspect of the Configuration Migration Assistant and other aspects refer to the Configuration Migration Assistant Overview (Doc Id: 1506830.1) whitepaper available from My Oracle Support.

Wednesday Feb 06, 2013

Configuration Migration Assistant Part 6 - Migration Export

Now that the configuration steps have been completed the next step is to use the Configuration Migration Assistant to migrate data across environments. The first step is to export the data from the source environment.

The steps to perform to export data using the Configuration Migration Assistant are as follows:

  • Create a Migration Data Set Export record to register the intent to export.
  • Execute the Migration Data Set Monitor batch process to perform the export.
  • Optionally, check the exported file into a code repository for cross reference with any customized code.

 For example:

Export Process Overview

The first step in the process is to create a Migration Data Set Export record. To do this use the Administration --> M --> Migration Data Set Export menu item and provide the following information:

  • Migration Request - Name of the Migration Request to Migrate
  • Export Directory - The directory taken from the Master Configuration Record for this environment. You cannot override the directory.
  • File Name - Name of the export file. The name of the file must conform to operating system constraints. We advise you do not embed blanks in the file name. Also do not include the filename suffix.
  • Export Description - Short description to attach within the export file.
  • Source Environment Reference - Defaults to the URL of the source environment but can be changed to reference the source of the information.

For example:

Migration Data Set Export

You may of noticed that Migration Request is a free format field. The easiest way to populate that field automatically is initiating the Export from the Migration Request maintenance screen. This allows you to verify the criteria prior to registering the export request. For example:

Export function from Migration Request

After saving this Migration Data Set Export, the request is registered within the product. Remember this does not perform the actual export it just registers the intent. The F1-MGDPR batch process exports the file. As the Migration Data Set Export has been registered then the request can be cancelled anytime prior to the execution of the F1-MGDPR batch process using the Migration Data Set Export maintenance dialog. For example:

Migration Data Set Export display

Now, you can register any number of Migration Data Set Exports as necessary for migrations.

To export the data the Migration Data Set Monitor batch process (F1-MGDPR) must be executed. This can be executed using any method (online submission, command line, batch scheduler etc) available to the environment by an authorized user.

The generic F1-MGDPR job is used to execute exports, process imports as well as apply changes and requires only one parameter DIST-THD-POOL. This is the threadpool to use for the export. As the product is processing data that is potentially cached, the job must run with the L2 cache turned off. This can be done using the following command:

Linux/UNIX

threadpoolworker.sh -l2 OFF -p <threadpoolname>=<number of threads> 

Windows

threadpoolworker  -l2 OFF -p <threadpoolname>=<number of threads> 

where

<threadpoolname> - Threadpool name to use in DIST-THD-POOL.

<number of threads> - Maximum limit for number of batch threads

Note: starttpw.sh can be also used on Linux/UNIX.

Note: Running this process using a threadpool where L2 cache is enabled may cause unexpected results.

During the execution of the export batch process the Migration Data Set Export will transition to a number of states:

  • The Migration Data Set Export starts in a Pending Status. This indicates that the export should be initiated the next time the F1-MGDPR process is executed.
  • At anytime before the F1-MGDPR is executed the Migration Data Set Export can be Cancelled manually using the Migration Data Set Export maintenance function.
  • During the execution the data to be migrated is identified and Migration Data Set and Migration Objects are created. The Migration Data Set groups all the records for a particular export request. The Migration Data Set Export is transitioned to Set Up Data status to indicate this activity is being performed.
  • The Migration Objects in the Migration Data Set are reviewed for duplications and relationships to be combined into transactions. This allows them to be exported in the correct sequence. The Migration Data Set Export is transitioned to Combine Transactions to indicate this activity is being performed.
  • Once the objects are checked and combined they are ready to export, with the state transitioned to Ready To Export to indicate this activity is being performed.
  • If there is some sort of processing error during any of the above transitions the Migration Data Set Export is transitioned to Error state and the batch process terminated for that Migration Data Set Export.
  • Once all the objects in the Migration Data Set Export are written to the file indicated the Migration Data Set Export is set to an Exported state. The date and time are recorded for auditing purposes.
For example the state transition model for Migration Data Set Export objects: 

Export State Transition

The state transitions can be tracked on the Log tab for the Migration Data Set Export maintenance function. For example:

Log entries for state transition

After executing the F1-MGDPR process the Migration Data Set Export will be updated with the file name and the date/time exported. For example:

Updated Migration Data Set Export

Now the data has been exported to a file. The file is in an internal format and SHOULD NOT be altered manually except through this tool. Any attempt to edit the file may result in unexpected results and even data corruption. The file can be optionally checked into your site code repository if you wish to keep it with any additional customizations such as java code etc.

The file is now available for import which we will cover in the next blog entry.

For more information about this aspect of the Configuration Migration Assistant and other aspects refer to the Configuration Migration Assistant Overview (Doc Id: 1506830.1) whitepaper available from My Oracle Support.

Monday Feb 04, 2013

Configuration Migration Assistant Part 5 - Migration Requests

Once you have Migration Plans created the next step is assembling them into Migration Requests. The Migration Request is a collection of unrelated Migration Plans and associated filter criteria to decide the scope of the migration.

To create a Migration Request navigate to the Administration Menu and select the M --> Migration Request menu option and fill in the following:

  • Migration Request - Name the Migration Request (it should be prefixed with CM to seperate it from the Migration Requests delivered with the product).
  • Description - A short description to describe the Migration Request
  • Detailed Descrption - A detailed description of the Migration Request
  • Migration Plans - A list of Migration Plans to include in this Migration Request and the Selection Criteria to use to subset the requests. This means the following:
    • Migration Plan - The Migration Plan to include in the Migration Request. The Migration Plans in the Request are NOT related. Any relationships are documented in the Migration Plans.
    • Selection Type - The criteria to select the subset of records in the Primary object in the Migration Plan. The Configuration Migration Assistant supports SQL based, XPath based or Algorithm based criteria.
    • Key Selection - The selection criteria to use in the format as specified in the Selection Type. This is SQL WHERE clause, XPATH statement or algorithm to use for selection. You can yse the help icon to find examples.

   For example:

Example Migration Request

The above example uses the (1=1) SQL clause to indicate that ALL records are migrated.

Note: The product supplies the majority of the Migration Requests you would use in the implementation. This step is only necessary if you wanted to copy a base Migration Request and alter it or add custom Business Objects to Migration Plans/Requests.

This concludes the configuration of the Configuration Migration Assistant. The next blog entries on this subject will discuss the execution components of the feature.

For more information about this aspect of the Configuration Migration Assistant and other aspects refer to the Configuration Migration Assistant Overview (Doc Id: 1506830.1) whitepaper available from My Oracle Support.

About

Anthony Shorten
Hi, I am Anthony Shorten, I am the Principal Product Manager for the Oracle Utilities Application Framework. I have been working for over 20+ years in the IT Business and am the author of many a technical whitepaper, manual and training material. I am one of the product managers working on strategy and designs for the next generation of the technology used for the Utilities and Tax markets. This blog is provided to announce new features, document tips and techniques and also outline features of the Oracle Utilities Application Framework based products. These products include Oracle Utilities Customer Care and Billing, Oracle Utilities Meter Data Management, Oracle Utilities Mobile Workforce Management and Oracle Enterprise Taxation and Policy Management. I am the product manager for the Management Pack for these products.

Search

Archives
« February 2013 »
SunMonTueWedThuFriSat
     
1
2
3
5
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
  
       
Today