Thursday Jan 10, 2013

ODI - MongoDB and Java Table Functions Anyone?

Let's see how we could integrate MongoDB using ODI, first take a step back. Derby/JavaDB introduced table functions a few years ago. Table functions are really useful, they are in the Oracle database and as much fun in JavaDB! ODI is a great platform for integration and JavaDB and table functions provide a really nice way to integrate arbitrary Java APIs into your designs. What I have done here is;

  • built a very simple java table function to project a table to represent the documents in a MongoDB collection. The collection is passed as a parameter to the KM and the column names are the keys for the MongoDB document. The data comes from the values.
  • built a very simple LKM from a Java table function to SQL

All of this will use the JavaDB in-memory, so no admin, simple to use. Rick Hillegas wrote a nice article with some handy classes that I have used. The mongo_table class I have written uses the EnumeratorTableFunction class included in Rick's examples. The MongoDB DBCursor class is a Java Iterator, which makes it really nice to pass to the  EnumeratorTableFunction class, and let it do all of the work.

The LKM I constructed declares the table function to JavaDB/Derby, for example below, the function is declared based on the source datastore name (MONGO_EMPS) and columns (my source datastore has EMPNO,ENAME,SAL, note the table function will actually project types defined in ODI's datastore), the function has the MongoDB database name and collection name as parameters.

  1. create function MONGO_EMPS( dbName varchar( 330), collectionName varchar( 30))
  2. returns table
  3. (
  4.   EMPNO VARCHAR(20),
  5.   ENAME VARCHAR(30),
  6.   SAL NUMERIC(10),
  7. )
  8. language java
  9. parameter style DERBY_JDBC_RESULT_SET
  10. no sql
  11. external name 'mongo_table.readCollection'

Then the actual code to use the function as a source is executed from a source task (the target is SQL as I mentioned earlier for the LKM). Below you can see my execution using the test MongoDB and the myStuff collection;

  1. select
  4. MON.SAL   C6_SAL
  5. from table(MONGO_EMPS('test', 'myStuff' )) MON
  6. where (1=1)
  7. And (MON.SAL > 4000)

Note I can also perform some filtering as an example, here it is being done in JavaDB and in my case its in-memory. No setup, no persistence just on the fly Java. Ideally I would push the filter down to MongoDB rather than reading and filtering in the driver - more on that later.

I had defined my documents in MongoDB using the following basic commands in the mongo shell;

  1. use test
  2. a1 = { EMPNO: "1", ENAME : "Fred", SAL : 10000 }
  3. a2 = { EMPNO: "2", ENAME : "John", SAL : 2000 }
  4. db.myStuff.insert( a1 )
  5. db.myStuff.insert( a2 )

In ODI, I can simply then use the datastore representing the MongoDB collection of documents in an interface and map it to my target;

The physical design uses the LKM JavaTableFunction to SQL KM and sets the MongoDB databases, collection and the Java table function name.

That's it. Pretty straightforward and we are reading and consuming MongoDB documents. So what about complex document types like this? These are transported, more to come. The other point here is that this is a generic LKM that you can plug in other arbitrary table functions - so you can consume from any API, its very simple. For the LKM I created, I just defined 2 parameters (because my table function only had two), but really we need a better way to handle this and ensure they are ignored if not defined etc. That's all polishing tho, anyway fun stuff you can see all with a small piece of code leveraging JavaDB!

Maximum Availability with Oracle GoldenGate Webcast: On Demand Link & FAQ

Last month we had a live webcast on a hot topic: Maximum Availability with Oracle GoldenGate. The webcast presentation focused on how Oracle GoldenGate can help achieve maximum availability for Oracle Database and other major databases. This webcast was very useful in answering a frequently asked question around GoldenGate's role for Oracle Database Maximum Availability Architecture (MAA) and when to use Oracle GoldenGate vs Active Data Guard. Product experts, Joe Meeks from Oracle Database High Availability product management team and Nick Wagner from Oracle GoldenGate product management team, addressed this question for us. Joe and Nick presented GoldenGate’s key solutions within MAA, and when to use which product along with Integrated Capture and Automated Conflict Management features of GoldenGate 11gR2. If you missed this event I highly recommend watching it on demand via the following link.

Watch On Demand:Maximum Availability with Oracle GoldenGate.

We also recorded a short podcast on this topic. Oracle GoldenGate 11gR2 for Maximum Availability

In the live webcast we had a very interactive audience who submitted over 100 questions during the event. Joe and Nick answered the most commonly asked questions about GoldenGate’s use cases and technology. Here are a few of the questions that we did not have time to answer during the event, along with the answers from the PM team.

  • Can I use the target database as warehouse for select only. If I were to allow users to this target, are there steps to ensure both source and target databases remain in sync?

 Yes you can use the target for read only or select only operations. Yes, we can do bi-directional active-active replication if you want to allow users to make changes to the target system. We can replicate those back to the source.

  • Can Integrated Capture be run on Oracle Database Standard Edition?

Yes, but all of the features that are only supported by Integrated Capture are only available with Oracle Database Enterprise Edition.

  •  Is it possible to capture database transactions from a source database using GoldenGate and publish them as messages into   a JMS queue?

Yes, GoldenGate can capture from and deliver to JMS queue. Please see our page for Java Adapter for more info. 

  • Can GoldenGate be used in environments where a number of tables do not have primary keys?

Yes. In tables without primary keys, we will use all non-LOB columns in the where clause for updates and deletes and then use ROWNUM = 1.

  • Is there a webinar that shows how to set up Oracle GoldenGate?

 Oracle GoldenGate Youtube channels has many useful videos and labs including “Oracle GoldenGate Fundamentals and How Oracle GoldenGate Works” and “Oracle GoldenGate Deep Dive Hands-on Labs” in 3 part series. You can browse and watch them here. Please check out GoldenGate training sessions with Oracle University as well.

I highly recommend subscribing to GoldenGate’s Youtube channel along with joining the GoldenGate Facebook and Twitter communities to get regular updates on the product.


Learn the latest trends, use cases, product updates, and customer success examples for Oracle's data integration products-- including Oracle Data Integrator, Oracle GoldenGate and Oracle Enterprise Data Quality


« January 2013 »