Thursday Mar 27, 2014

Interested in presenting and sharing your insights around Data Integration at OpenWorld?

You have been successful in making your organization run smoother. Faster. More cost-effectively. You have come up with the perfect solution to increase your staff retention, speed up your lead to sales pipeline, or minimize your supply management costs. We want to hear your story. Submit your proposal today and share your success at OpenWorld 2014.

Send us a proposal covering your Data Integration success. If it is selected, you will share your idea, experiences, and stories with Oracle customers, developers, and partners from around the world. You will also get a complimentary full pass to the conference.

Learn more about submitting your proposal.

Conference attendees want to hear it straight from you.

Don't wait—proposals must be submitted by April 15!

Tuesday Mar 25, 2014

Useful Tips on Oracle GoldenGate 12c Installation, Instantiation, and Setup

Written by Nick Wagner, Director of Product Management, Oracle Database High Availability

This is the first of 3 blog posts that I’m writing that will cover some of the questions I get from customers doing a first time implementation of Oracle GoldenGate. I’m not trying to reinvent the wheel, so to speak, so I’m going to reference a number of different My Oracle Support notes that go into more depth about a particular subject or process.

This first post will be about getting the environment set up and installation of Oracle GoldenGate. Also, for simplicity, this will assume a like to like replication environment as well as one-way replication. Future blogs will cover bi-directional replication and environments where the source and target objects have different structures.

Installation

Installation should be done on direct attached storage unless you are installing Oracle GoldenGate in a cluster or for failover, in that case the use of a dedicated NAS or SAN device is fine. Oracle GoldenGate is a quickly evolving product and we have a standard release and patching process. For example, in Oracle GoldenGate 11.2 and 12.1, it’s typically a 3 week patching cycle. So, always use the most recent version of Oracle GoldenGate so you have the most recent stable release. If you are using Oracle GoldenGate 11.2.1, you can just download the most recent release from My Oracle Support. If you are using Oracle GoldenGate 12.1.2, then you should download the installation from OTN or edelivery.oracle.com, and then download and upgrade to the most recent version from My Oracle Support and use Opatch to patch it to the latest release.

Instantiation

There are numerous articles on Oracle GoldenGate instantiation, the two most common are mentioned below.  Essentially, if you can do an Oracle hot backup that’s the easiest way to establish the target database. If that’s not possible, look at Oracle export/import or data pump, and if that can’t be used, you can use GoldenGate's own instantiation. In a situation where you are going to be doing transformation in the Delivery process (Replicat), then GoldenGate's own instantation, while being the slowest, is probably the best choice, as any transformation done in the Delivery can be applied during the initial load as well. It saves you from having to load the data, then transform it.

For like to like, the first is Oracle GoldenGate Best Practices: Instantiation from an Oracle Source Database [Article ID 1276058.1] this document is very comprehensive and covers Oracle to Oracle as well as heterogeneous options. And the other document: Oracle GoldenGate: Initial Load Techniques and References (Doc ID 1311707.1) can be used as well and has a few examples and would be ideal for targets where transformation is going to be used.

Initial Setup

The initial setup of the production database can be quite involved. However, the most important aspect is ensuring that the source database has the correct supplemental logging enabled. In the case where all (or a majority) of the tables are being replicated, it’s easier to configure supplemental logging at the schema level instead of the table level. Using the GGSCI command ADD SCHEMATRANDATA ensures that as the application changes, and new tables are added or keys changed on a table the underlying supplementally logged columns still adhere to those required by Oracle GoldenGate.

If there are tables without primary keys I would recommend that you review this article Supplemental logging – How to Handle Tables Without Primary Keys or Unique Indexes With Oracle GoldenGate [Article ID 1271578.1] that covers the way Oracle GoldenGate handles these types of tables and why it’s important to handle them correctly. If tables without keys aren’t managed properly, you could get anything from poor performance all the way to data corruption. The default way, may not always be the best way of handling these objects. Especially if there is a unique column that just doesn’t have a unique constraint on it.

Another simple way to reduce maintenance on Oracle GoldenGate is wildcarding. For example, if you have  Capture processes (also called Extract) running as a pump and pulling data from a Trail File, and all the data in the Trail File should be sent to the target, then use TABLE *.*; and that will instruct Capture to send all the data that it reads. That way, if a table is added to the Capture reading from the redo logs, you won’t need to make any changes to the Capture running as a pump is necessary. This same concept can even be used in the Delivery, where ASSUMETARGETDEFS is valid. Just use MAP *.*, TARGET *.*; and any table added to the Capture reading from the transaction logs will be sent all the way down to the target.

And that brings the first blog post to a close. I’ll be happy to answer any questions…

Wednesday Mar 12, 2014

ODI 12c - Data Input Experts

Back in the olde days of OWB I blogged about a few utilities (see here) that were useful for collecting user input data in custom flows, users build such flows to implement accelerators to take the mundane tasks out of common activities. In ODI you can also use groovy SwingBuilder, this let's you build useful dialogs very easily. I posted some examples such as the one below for model creation in ODI and a launchpad example;

The utilities for OWB I mentioned in the blog are just basic java classes that were invoked from OWB via tcl/jacl. These utilities are written in java and can still be used from ODI via groovy. Still as useful, still as functional. Let's see how we call them now!

The required JARs need to be put on the groovy classpath, which is under the ODI IDE's Tools->Preferences option, and then under ODI->System->Groovy and set the groovy classpath to include jexpert.jar, tcljava.jar and jacl.jar. For example I have the following referencing the JARs from my 11gR2 database which has the OWB code;

  • D:\app\dallan\product\11.2.0\dbhome_1\owb\lib\int\jexpert.jar;D:\app\dallan\product\11.2.0\dbhome_1\owb\lib\int\tcljava.jar;D:\app\dallan\product\11.2.0\dbhome_1\owb\lib\int\jacl.jar

I can then launch the shuttle dialog for example as follows;

  1. import oracle.owb.jexpert.ShuttleObjects
  2. arrayOfString = [ "PRODUCT_ID", "PRODUCT_NAME", "PRODUCT_COLOR", "PRODUCT_DESC", "PRODUCT_LONG_DESC", "CATEGORY_ID", "CATEGORY_NAME", "CATEGORY_DESCRIPTION", "SUBCATEGORY_ID", "SUBCATEGORY_NAME", "SUBCATEGORY_DESCRIPTION" ]
  3. sels = ShuttleObjects.getselection("Select dimension levels", "Select columns to identify levels:", "Columns:", "Levels", (String[]) arrayOfString.toArray())

  4. println sels

I can use the returned variable sels and do whatever ODI stuff I need, you can see the code above executed from within ODI and the dialog appearing with the information;

Likewise the data entry dialog works as is, when that dialog is executed from groovy, just like in OWB we can get the information displayed, the user can enter data, we can collect it and action it in our groovy using the ODI SDK;

The blog on the 12c mapping SDK here has a good SDK reference table that gives you pointers for all parts of the product into the SDK areas. This is definitely a handy one to bookmark, I often use it myself. Learn some scripting it'll help save you are your teams a lot of time.

Friday Mar 07, 2014

Using the Unpivot Component in ODI 12c

As we’ve seen last week the latest ODI 12c patches added several new Mapping Components such as Pivot or Unpivot. In this blog post we will walk you through an example of how to use the new Unpivot Component. 

You can use the following SQL statements to recreate this example in your environment. It will create the source (UNPIVOT_TEST) and target (TRG_UNPIVOT_TEST) tables used in this article in your database then you can reverse engineer them in ODI.

CREATE TABLE unpivot_test (
year NUMBER,
q1_sales NUMBER,
q2_sales NUMBER,
q3_sales NUMBER,
q4_sales NUMBER
);
insert into unpivot_test values (2012, 10.5, 11.4, 9.5, 8.7);
insert into unpivot_test values (2013, 9.5, 10.5, 10.3, 7.6);
commit;
CREATE TABLE trg_unpivot_test (
year NUMBER,
quarter VARCHAR2(255),
sales NUMBER
);

Our goal is to unpivot the data on the Quarter columns when going from UNPIVOT_TEST into TRG_UNPIVOT_TEST as shown below


Follow these steps to add and configure an Unpivot Component in a Mapping:

  1. First add the Source table UNPIVOT_TEST into your Mapping, to do so drag and drop the datastore from the Models into the Mapping
  2. Next add an Unpivot component into the Mapping. This is done by clicking on the Unpivot Component in the Component palette and then clicking on the Mapping diagram. A new UNPIVOT component will appear in the Mapping:

  3. Drag and drop the YEAR column from UNPIVOT_TEST into the UNPIVOT component. There is no need to add the remaining source attributes yet, we will be using them in the Unpivot Transforms section later.

  4. Then click on the UNPIVOT component and select the Attributes panel. We will add 2 new attributes representing the row and value locator attributes: QUARTER (VARCHAR) and SALES (NUMERIC).

  5. Click on the General panel and select QUARTER in the Row Locator drop-down list. We will be performing the unpivot operation using this column.

  6. Next add the transformations to unpivot the data. Click on the Unpivot Transforms panel and create a transformation rule for each of the 4 quarters and match the appropriate QUARTER and SALES values as shown below:

  7. Finally add the Target datastore TRG_UNPIVOT_TEST and link the UNPIVOT component to it. Leave the default settings in the Attributes Matching panel and click OK

  8. In this example you can use the default Physical settings for your Mapping. Integration Type is set to Control Append by default and the IKM Oracle Insert is used
  9. Finally click on Run to execute the Mapping, 8 inserts are performed and you should see the following data in your target table:

  10. If you review the generated code you will notice that ODI leverages the UNPIVOT function on Oracle to perform such operation. The Unpivot component supports Oracle as well as any database

You can recreate the following example using the ODI 12c Getting Started VirtualBox image which is available on OTN: http://www.oracle.com/technetwork/middleware/data-integrator/odi-demo-2032565.html

About

Learn the latest trends, use cases, product updates, and customer success examples for Oracle's data integration products-- including Oracle Data Integrator, Oracle GoldenGate and Oracle Enterprise Data Quality

Search

Archives
« March 2014 »
SunMonTueWedThuFriSat
      
1
2
3
4
5
6
8
9
10
11
13
14
15
16
17
18
19
20
21
22
23
24
26
28
29
30
31
     
Today