Recent Posts by Bob Hanckel

Overview The emergence and popularity of “data-in-motion” technologies has grown with the maturity of countless technologies that generate continuous streams of data to cloud applications that monitor and perform analytics 24/7.  The data is generated by various entities such as cloud monitoring software, smart wristwatches, cell phones, financial transaction messages, and wireless medical devices, and have various forms (e.g. SWIFT messages, IoT messages, system logs etc.) For...

The following links discuss OHSH (Oracle Shell for Hadoop Loaders) , which is a CLI used for loading Oracle tables from content living in HDFS or Hive tables, and for loading Hive tables from content living in Oracle tables.   How to Load Oracle and Hive Tables using OHSH (Part 1 - Introduction) How to Load Oracle and Hive Tables using OHSH (Part 2 - Configuration and CLI Usage) How to Load Oracle and Hive Tables with OHSH (Part 3 - Loading Oracle Tables) How to Load Oracle...

Introduction This is the first of a series of blog posts that will discuss how to load data living in the Hadoop Ecosphere into Oracle tables. The goal is to give insights, discuss pros and cons, and best practices for achieving optimal load times and flexibility from an experienced developer’s point of view. Oracle and Hadoop are complementary technologies where the whole is greater than the sum of the parts. They both have parallel architectures, which, if used intelligently...

In this tutorial we deal with columns and tables.   Specifically how to map Oracle columns to specific data fields in HDFS files or to Hive columns. Previously in Part 3, we discussed the methods for loading Oracle tables for relatively simple cases.  The simplifying assumptions included the following: When loading delimited text into Oracle tables, the physical order of the delimited text fields reflects the declared order of the columns in the Oracle table When loading Hive...

In this post we will discuss the "etl" method for loading tables.  It was briefly mentioned in earlier posts, but because is a hybrid of both Hadoop map/reduce processing and Oracle external tables it made sense to defer its discussion until the other simpler methods of loading Oracle tables (i.e. "exttab", "directcopy", and "jdbc") were explained. The "etl" method runs an OLH job on the Hadoop cluster to produce data pump files living in HDFS.  This is followed by an OSCH job...

This is a first of a series of blog posts that revisits an earlier set of tutorials about loading Oracle tables from Hadoop. The intention of this tutorial is to introduce a tool formally known as OHSH (Oracle Shell for Hadoop Loaders), nick named "ohshell".  This is a CLI that supports commands to load Oracle tables from data living in Hadoop, and to create, replace, or incrementally load data from Oracle tables into Hive.  OHSH is a Hadoop application which layers on three...

  In Part 3 we described how to use OHSH to load Oracle tables with content living in either HDFS files or in Hive tables.  In this tutorial we focus on how to use OHSH to do the opposite: create, replace, and incrementally load Hive tables with content living in Oracle tables.  To use OHSH to do this you need to download and configure Copy To Hadoop feature of Big Data SQL (CP2HADOOP) as described in Part 1 and Part 2 of this series of tutorials. When launching OHSH, the...

In this tutorial we are going to talk in depth about loading Oracle tables with content from HDFS files or from Hive tables.  The focus will be on content that is stored as delimited text, but we will talk about loading other forms of content (e.g. parquet). The Oracle target table and sources of data The Oracle table we will be loading lives in a schema called "MOVIEDEMO" and is called MOVIE_RATINGS with various attributes relating customers to movies they saw and other...

In this "eat your vegetables before having dessert" tutorial we will talk about how to configure OHSH.  We will also discuss groups of OHSH commands that you will need to know about before you start using the tool to perform loading tasks. Configuration and SQL*Plus and Hadoop/Hive Dependencies Configuration of OHSH will differ depending upon where you want to run it.  The short story is that OHSH expects to run in a Linux environment with Bash shell available, and have...