Author: Chuan Shi
The ETL logic in BI Apps uses parameters in packages, interfaces, load plans, and knowledge modules (KM) to control the ETL behaviors. Parameters can be configured by customers, pre-seeded in the ETL code, or maintained internally:
ETL parameters are handled by using ODI variables. The purpose of this blog is to explain ODI variable refreshing to ensure that correct values of variables are used in Oracle BI Apps ETL.
ODI Variable Classification
To ensure correct values to be used in ETL, variables are refreshed at run time. A variable can be refreshed in either a package or a load plan. Variables can be classified into four categories based on whether and where they are refreshed in ODI (and if a variable needs to be refreshed, then there must be a refreshing logic specified for it, which will be discussed later).
The four categories are:
Definition: Generally, internal parameters used by the load plan to control the ETL execution, and therefore they shall not be refreshed
Definition: Variables that are not feasible to be refreshed in load plans. (The refreshing logic of such a variable depends on ETL run facts. It sources from DW tables and uses the QUALIFY user define function (UDF). ODI fails to interpret QUALIFY at load plan level in some cases.)
Examples: IS_INCREMENTAL, LAST_EXTARCT_DATE
Definition: Variables whose values are hardcoded by overwriting in load plans. The variables will take the hardcoded values in ETL.
Examples: DOMAIN_CODE, LOOKUP_TYPE
Definition: Variables whose values are configured in BIACM by customers. (In other words, the values of these variables come from BIACM. The refreshing logic of such a variable uses a UDF to extract its value from BIACM.)
Examples: UPDATE_ALL_HISTORY, TYPE2_FLG
For a complete list of ODI variables, please refer to this post.
Refreshing Variables in a Package
Refreshing variables in a package is straightforward. One needs to create Refresh Variable steps for the variables to be refreshed. The screenshot below shows examples of refreshing IS_INCREMENTAL.
Hard-coding Variables in a Load Plan
Some variables, such as DOMAIN_CODE and LOOKUP_TYPE are hardcoded in load plan components. To do that, go to the load plan component to which the variables are to be hardcoded, select the Overwrite checkbox and provide the hardcode values for the variables. The screenshot below shows examples of hardcoding DOMAIN_CODE and LOOKUP_TYPE.
Refreshing Variables from BIACM
BIACM is a central UI where customers can define the values of data load parameters (i.e., ODI variables), among with many other features offered. ODI variables, which are refreshed in load plans, have their values extracted from BIACM. We also refer to such variables as BIACM variables.
BIACM Variables are classified into truly global variables, PLV specific variables, and fact/dimension group level variables.
From a variable value overriding perspective:
These variables are defined in ODI. To ensure that variable refreshing works correctly, there are some rules on the definitions of ODI variables:
(Whenever the Keep History option or the Default Value of a variable is changed, the scenarios that use this variable need to be regenerated.)
Once ODI variables are defined, a refreshing logic is needed to refresh them from BIACM. In this regard,
- variable name;
- product line;
- $(param_code) is the name of the variable (e.g., TYPE2_FLG)
- $(DATASOURCE_NUM_ID) is used to specify the product line.
For PLV/group level variables, we pass #DATASOURCE_NUM_ID as $(DATASOURCE_NUM_ID);
For truly global variable, we pass #WH_DATASOURCE_NUM_ID as a pseudo-PLV ID.
- Do not pass fact/dimension group directly into the syntax. They are determined by where the variable is refreshed.
BIACM variables are refreshed in load plans. To refresh a variable in a load plan, the following three steps are required (they have been done OOTB):
Step 1: Specify the correct logical schema and refreshing logic in the refreshing tab of the variable definition.
The logical schema has to be CM_BIAPPS11G. The refreshing logic should be getcmparam() with appropriate inputs, e.g.,
Step 2: Update the variable definition in the variables tab of the load plan.
Go to the load plan component where you want to refresh the variables. In the Variables tab, right click on the variables and select ‘Refresh Variable Definition’ so that the variable definition in the LP is synchronized with its real definition. Once this is done, verify that the logical schema is showing CM_BIAPPS11G, and the select statement is showing the embedded SQL in the getcmparam() function.
Step 3: Check the refreshing checkbox at the appropriate LP step.
For truly global variables, Step 3 becomes:
The logic behind getcmparam() guarantees that appropriate value of the variable is returned from BIACM given the name of the variable, the DATASOURCE_NUM_ID passed in, and the LPC step where it is refreshed.
Values stored in BIACM are strings. Therefore all ODI variables refreshed from BIACM will come in as strings. Each of the consuming codes (where the variables are used) should make sure it converts the data type accordingly. For example, dates are returned as a string in format yyyy-mm-dd hh:mi:ss. TO_DATE_VAR UDF is used to convert the returned string to DATE format. Number values are returned as strings as well.
Checklist when Things Go Wrong
What can go wrong?
Overriding Rule (1)
· However, if a variable is refreshed and/or overwritten in a child step, the value refreshed from this step will override the value refreshed from the parent step. Other child steps of the same level will NOT be affected. They will still inherit the value refreshed in the parent step.
Overriding Rule (2)
(unlikely to happen but it exists)
If a variable is refreshed both in a package and in a load plan, then the value refreshed from the package will override the value refreshed from the load plan.
When the value of a variable returned from BIACM is not in line with expectation:
- Check its value(s) in BIACM UI. For PLV variables, check its value in each product line; for group level variables, check its group specific values as well as global default value.
- Check if the variable is refreshed in a root step of a load plan (refresh checkbox checked). In the meanwhile, check if the root step is named after a fact/dim group.
- Check if the variable is incorrectly refreshed or hardcoded in a child step belonging to the root step (avoid overriding rule 1).
- Check the ODI default value of this variable. If BIACM returns (null), i.e., nothing, for this variable, its ODI default value will be used. Also, if we check the overwrite box (but not the refresh checkbox) of a variable in a load plan step, but forget to provide the value, then the ODI default value will be used.
- Note: overriding rule (2) is unlikely to happen to BIACM variables.
When variable has no value: