Recent Posts by Melliyal Annamalai

In Part 1 we saw how easily we can move data between a Big Data Cloud Service and Database Cloud Service. In this post we look at how data copied from Oracle Database Cloud Service to Big Data Cloud Service can be used in Machine Learning applications that use Spark MLLib. Using steps from this earlier blog post on using Spark and data copied by Copy to Hadoop, we see that a Data Frame can be created to point to the Hive external table created in Part 1.   scala> val...

  The Oracle Big Data Cloud Service (BDCS) license includes tools to move data between Apache Hadoop and Oracle Database, such as Oracle Loader for Hadoop and Copy to Hadoop.   Here is how you can use these tools with the easy-to-use CLI (OHSH).   These tools are all installed and configured in BDCS. There are just two additional configuration steps you need to perform: In the database cloud service enable the access rule to open the listener port 1521.   Access Cloudera Manager...

Thank for you attending the OracleCODE session TUT 4244 at OOW (Moving Data between Apache Hadoop and Oracle Database for Customer 360-degree Analytics).   Here is presentation Here is a recording of the session.

SQL Developer is much loved in the developer community.   A developer starting on the task of moving data between Apache Hadoop and Oracle Database can save lots of time by using a tool they know and like. Download SQL Developer 4.2 and try it out if you are moving data between Apache Hadoop and Oracle Database.   Remember that you always start with the target (destination) of the load.  To load data from Hadoop into Oracle Database: Using Oracle Loader for Hadoop: Start...

It is very easy to copy Oracle Database tables to Parquet format in HDFS. When Using Copy to Hadoop with OHSH  If using Copy to Hadoop with OHSH, with one additional step you can convert the Oracle Data Pump files into Parquet. ohsh> %hive_moviedemo create movie_sessions_tab_parquet stored as parquet as select * from movie_sessions_tab; hive_moviedemo is a Hive resource (we created that in the blog post on using Copy to Hadoop with OHSH).   Within OHSH you are using Hive to...

After following the tips below your Oracle Loader for Hadoop jobs should fly!   Summary   Check your network bandwidth Use OCIOutputFormat if possible (OCIOutputFormat requires the target table to be partitioned, if target table is not partitioned, see whether you can make it a partitioned table) Check parallelism (this is the number of reduce tasks when loading into a partitioned table) Use SDP (in particular when using InfiniBand) Confirm that balancer is enabled, a well-balanced...

Oracle Loader for Hadoop (OLH) can load Parquet data.  It does so by reading Parquet data via a Hive table. So if you have Parquet data that you want to load into Oracle Database using Oracle Loader for Hadoop, the first step is to create a Hive table over the Parquet data.  For example: create external table oracletest.my_hivetab_on_parquet(f1 decimal(38,18), i2 int, v3 varchar(50), d4 timestamp, t5 timestamp, v6 varchar(200), i7 int)       stored as parquet       location'/user/...

Copy to Hadoop copies data from an Oracle Database table to HDFS, as Oracle Data Pump files.  These files can be accessed by Hive tables using a SerDe that is part of Copy to Hadoop. So Hive queries can be run against this data. What if you would like to include this data in a Spark ML (machine learning) application? A Spark data frame can access Oracle Data Pump files via Hive. Below is an example. Configure the Spark Installation to use Hive To read Oracle Data Pump files into...

It is easy to use Oracle SQL Connector for HDFS with a client-side Oracle Wallet.  Oracle Wallet is a secure container that stores authentication and signing credentials. This is a secure way to automate connections to the database and simplify large-scale deployments that rely on password credentials to connect to the database. Refer to Oracle Wallet documentation here. You can create the OSCH external table from either thedatabase node, or from a Hadoop node, edge node, or...