Recent Posts by Melliyal Annamalai

It is very easy to copy Oracle Database tables to Parquet format in HDFS. When Using Copy to Hadoop with OHSH  If using Copy to Hadoop with OHSH, with one additional step you can convert the Oracle Data Pump files into Parquet. ohsh> %hive_moviedemo create movie_sessions_tab_parquet stored as parquet as select * from movie_sessions_tab; hive_moviedemo is a Hive resource (we created that in the blog post on using Copy to Hadoop with OHSH).   Within OHSH you are using Hive to...

After following the tips below your Oracle Loader for Hadoop jobs should fly!   Summary   Check your network bandwidth Use OCIOutputFormat if possible (OCIOutputFormat requires the target table to be partitioned, if target table is not partitioned, see whether you can make it a partitioned table) Check parallelism (this is the number of reduce tasks when loading into a partitioned table) Use SDP (in particular when using InfiniBand) Confirm that balancer is enabled, a well-balanced...

Oracle Loader for Hadoop (OLH) can load Parquet data.  It does so by reading Parquet data via a Hive table. So if you have Parquet data that you want to load into Oracle Database using Oracle Loader for Hadoop, the first step is to create a Hive table over the Parquet data.  For example: create external table oracletest.my_hivetab_on_parquet(f1 decimal(38,18), i2 int, v3 varchar(50), d4 timestamp, t5 timestamp, v6 varchar(200), i7 int)       stored as parquet       location'/user/...

Copy to Hadoop copies data from an Oracle Database table to HDFS, as Oracle Data Pump files.  These files can be accessed by Hive tables using a SerDe that is part of Copy to Hadoop. So Hive queries can be run against this data. What if you would like to include this data in a Spark ML (machine learning) application? A Spark data frame can access Oracle Data Pump files via Hive. Below is an example. Configure the Spark Installation to use Hive To read Oracle Data Pump files into...

It is easy to use Oracle SQL Connector for HDFS with a client-side Oracle Wallet.  Oracle Wallet is a secure container that stores authentication and signing credentials. This is a secure way to automate connections to the database and simplify large-scale deployments that rely on password credentials to connect to the database. Refer to Oracle Wallet documentation here. You can create the OSCH external table from either thedatabase node, or from a Hadoop node, edge node, or...

Oracle Shell for Hadoop Loaders (OHSH) is an intuitive command line tool to move data between Hadoop and Oracle Database. OHSH uses the concept of resources to identify data sources and destinations.   Hive resource: For executing Hive commands and specifying Hive as a source or destination Hadoop resource: For executing HDFS commands to navigate HDFS and use HDFS as a source or destination SQL resource: For executing SQL commands in a database schema JDBC resource: To make JDBC...

David Bayard, Oracle Big Data Pursuit Team April 2016 Multimedia Analytics with Oracle Big Data Spatial and Graph Facial Detection, Optical Character Recognition, and QR/Barcode Detection In this blog, we will explore how to use and extend the new Multimedia Analytics (MMA) framework that ships with the Oracle Big Data Spatial and Graph (BDSG) product. We will show how we can leverage the BDSG Multimedia Analytics framework to help us do things like facial recognition, optical...

Oracle Database customers install and use the database on a variety of platforms.   A frequent question is whether Oracle SQL Connector for HDFS (part of Oracle Big Data Connectors) works with Oracle Database on a specific platform. Oracle SQL Connector for HDFS can be used with Oracle Databases on Solaris, Oracle SuperCluster, and Oracle Databases on Linux/UNIX flavors like HP-UX, IBM-AIX, Amazon AWS, and so on.   Installation instructions are documented in thesesupport...

Guest post by Javier De La Torre Medina and Iwona Rajca In this post we describe how to setup and use Oracle SQL Connector for Hadoop (OSCH) for data query between Cloudera’s Distribution Including Apache Hadoop hosted on AWS and Oracle Exadata. OSCH leverages Oracle’s External tables technology and enables queries on Hadoop and Hive from the database. AWS is a good starting point to build your own test Hadoop environment. Many enterprises who are experimenting with Hadoop in...